Add multimodal model support (STT/TTS/image) + fix UTF-8 server crash#24
Merged
Conversation
Extend arbiterAI beyond chat/embeddings to speech-to-text, text-to-speech, and image generation, across cloud and local engines, and fix a server-wide crash on invalid UTF-8 in model output. Data model & routing - New request/response types: AudioTranscription, Speech, ImageGeneration. - BaseProvider gains non-pure transcribe/synthesizeSpeech/generateImage (default NotImplemented); ArbiterAI dispatch mirrors getEmbeddings. - `modes::` constants; ModelInfo.mode is the router key. Cloud path - OpenAI provider implements transcription (multipart), speech (binary out), and image generation (url/b64_json). - Server endpoints: POST /v1/audio/transcriptions, /v1/audio/speech, /v1/images/generations, with requireModelMode() enforcing mode. - Modality-aware pricing (per-image / per-audio-second / per-character). Local engines (default-OFF CMake flags; vcpkg overlay ports) - whisper.cpp (STT), stable-diffusion.cpp (image, PNG+base64 via zlib), vibevoice.cpp (TTS). Each overlay port strips its bundled ggml and links the llama port's ggml so exactly one ggml exists. - Orpheus TTS via chatllm.cpp: because chatllm bundles a forked ggml and its C binding exposes no audio, it is isolated behind a self-contained engine .so (orpheus-engine port) that exports arbiter_orpheus_tts; the Orpheus provider dlopens it (RTLD_LOCAL) so the forked ggml never clashes with the host llama ggml. - Mock provider implements all three modalities for tests. Feature requests (docs/feature_request.md) - Auto-download: BaseProvider::resolveDownloadableModelFile() fetches local weights from variants[].download.url to the StorageManager models dir (multi-file bundles supported); whisper/sd/vibevoice/orpheus delegate to it. - Modality speed telemetry: InferenceStats carries modality/images/steps/ audio-seconds/realtime-factor/steps-per-second/cost, computed at dispatch and exposed per-entry on /api/stats/history and aggregated on /api/stats. - Schema: whisper_options / sd_options blocks (additionalProperties kept false); parsed into ModelInfo and applied as provider defaults. Bug fix: UTF-8 crash (server-wide SIGABRT) - safeDump() serializes every model-text response body with error_handler_t::replace + try/catch (streaming deltas, non-streaming chat, transcription/image) so no model output can abort the process. - Root cause: Utf8StreamBuffer in the llama provider's generation loops holds an incomplete multi-byte tail until the next token completes it, so streaming deltas never emit a split character. Tests: provider unit tests (mock/whisper/sd/vibevoice/orpheus), modality dispatch + pricing, telemetry aggregation. Full suite green (local). Design + deployment notes in docs/tasks/multimodal_model_support.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014DA3HMkstTTLtakWda7em4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extend arbiterAI beyond chat/embeddings to speech-to-text, text-to-speech, and image generation, across cloud and local engines, and fix a server-wide crash on invalid UTF-8 in model output.
Data model & routing
modes::constants; ModelInfo.mode is the router key.Cloud path
Local engines (default-OFF CMake flags; vcpkg overlay ports)
Feature requests (docs/feature_request.md)
Bug fix: UTF-8 crash (server-wide SIGABRT)
Tests: provider unit tests (mock/whisper/sd/vibevoice/orpheus), modality dispatch + pricing, telemetry aggregation. Full suite green (local). Design + deployment notes in docs/tasks/multimodal_model_support.md.
Claude-Session: https://claude.ai/code/session_014DA3HMkstTTLtakWda7em4