Skip to content

Add multimodal model support (STT/TTS/image) + fix UTF-8 server crash#24

Merged
caseymcc merged 1 commit into
mainfrom
multimodal-model-support
Jul 19, 2026
Merged

Add multimodal model support (STT/TTS/image) + fix UTF-8 server crash#24
caseymcc merged 1 commit into
mainfrom
multimodal-model-support

Conversation

@caseymcc

Copy link
Copy Markdown
Owner

Extend arbiterAI beyond chat/embeddings to speech-to-text, text-to-speech, and image generation, across cloud and local engines, and fix a server-wide crash on invalid UTF-8 in model output.

Data model & routing

  • New request/response types: AudioTranscription, Speech, ImageGeneration.
  • BaseProvider gains non-pure transcribe/synthesizeSpeech/generateImage (default NotImplemented); ArbiterAI dispatch mirrors getEmbeddings.
  • modes:: constants; ModelInfo.mode is the router key.

Cloud path

  • OpenAI provider implements transcription (multipart), speech (binary out), and image generation (url/b64_json).
  • Server endpoints: POST /v1/audio/transcriptions, /v1/audio/speech, /v1/images/generations, with requireModelMode() enforcing mode.
  • Modality-aware pricing (per-image / per-audio-second / per-character).

Local engines (default-OFF CMake flags; vcpkg overlay ports)

  • whisper.cpp (STT), stable-diffusion.cpp (image, PNG+base64 via zlib), vibevoice.cpp (TTS). Each overlay port strips its bundled ggml and links the llama port's ggml so exactly one ggml exists.
  • Orpheus TTS via chatllm.cpp: because chatllm bundles a forked ggml and its C binding exposes no audio, it is isolated behind a self-contained engine .so (orpheus-engine port) that exports arbiter_orpheus_tts; the Orpheus provider dlopens it (RTLD_LOCAL) so the forked ggml never clashes with the host llama ggml.
  • Mock provider implements all three modalities for tests.

Feature requests (docs/feature_request.md)

  • Auto-download: BaseProvider::resolveDownloadableModelFile() fetches local weights from variants[].download.url to the StorageManager models dir (multi-file bundles supported); whisper/sd/vibevoice/orpheus delegate to it.
  • Modality speed telemetry: InferenceStats carries modality/images/steps/ audio-seconds/realtime-factor/steps-per-second/cost, computed at dispatch and exposed per-entry on /api/stats/history and aggregated on /api/stats.
  • Schema: whisper_options / sd_options blocks (additionalProperties kept false); parsed into ModelInfo and applied as provider defaults.

Bug fix: UTF-8 crash (server-wide SIGABRT)

  • safeDump() serializes every model-text response body with error_handler_t::replace + try/catch (streaming deltas, non-streaming chat, transcription/image) so no model output can abort the process.
  • Root cause: Utf8StreamBuffer in the llama provider's generation loops holds an incomplete multi-byte tail until the next token completes it, so streaming deltas never emit a split character.

Tests: provider unit tests (mock/whisper/sd/vibevoice/orpheus), modality dispatch + pricing, telemetry aggregation. Full suite green (local). Design + deployment notes in docs/tasks/multimodal_model_support.md.

Claude-Session: https://claude.ai/code/session_014DA3HMkstTTLtakWda7em4

Extend arbiterAI beyond chat/embeddings to speech-to-text, text-to-speech,
and image generation, across cloud and local engines, and fix a server-wide
crash on invalid UTF-8 in model output.

Data model & routing
- New request/response types: AudioTranscription, Speech, ImageGeneration.
- BaseProvider gains non-pure transcribe/synthesizeSpeech/generateImage
  (default NotImplemented); ArbiterAI dispatch mirrors getEmbeddings.
- `modes::` constants; ModelInfo.mode is the router key.

Cloud path
- OpenAI provider implements transcription (multipart), speech (binary out),
  and image generation (url/b64_json).
- Server endpoints: POST /v1/audio/transcriptions, /v1/audio/speech,
  /v1/images/generations, with requireModelMode() enforcing mode.
- Modality-aware pricing (per-image / per-audio-second / per-character).

Local engines (default-OFF CMake flags; vcpkg overlay ports)
- whisper.cpp (STT), stable-diffusion.cpp (image, PNG+base64 via zlib),
  vibevoice.cpp (TTS). Each overlay port strips its bundled ggml and links
  the llama port's ggml so exactly one ggml exists.
- Orpheus TTS via chatllm.cpp: because chatllm bundles a forked ggml and its
  C binding exposes no audio, it is isolated behind a self-contained engine
  .so (orpheus-engine port) that exports arbiter_orpheus_tts; the Orpheus
  provider dlopens it (RTLD_LOCAL) so the forked ggml never clashes with the
  host llama ggml.
- Mock provider implements all three modalities for tests.

Feature requests (docs/feature_request.md)
- Auto-download: BaseProvider::resolveDownloadableModelFile() fetches local
  weights from variants[].download.url to the StorageManager models dir
  (multi-file bundles supported); whisper/sd/vibevoice/orpheus delegate to it.
- Modality speed telemetry: InferenceStats carries modality/images/steps/
  audio-seconds/realtime-factor/steps-per-second/cost, computed at dispatch
  and exposed per-entry on /api/stats/history and aggregated on /api/stats.
- Schema: whisper_options / sd_options blocks (additionalProperties kept
  false); parsed into ModelInfo and applied as provider defaults.

Bug fix: UTF-8 crash (server-wide SIGABRT)
- safeDump() serializes every model-text response body with
  error_handler_t::replace + try/catch (streaming deltas, non-streaming chat,
  transcription/image) so no model output can abort the process.
- Root cause: Utf8StreamBuffer in the llama provider's generation loops holds
  an incomplete multi-byte tail until the next token completes it, so
  streaming deltas never emit a split character.

Tests: provider unit tests (mock/whisper/sd/vibevoice/orpheus), modality
dispatch + pricing, telemetry aggregation. Full suite green (local).
Design + deployment notes in docs/tasks/multimodal_model_support.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014DA3HMkstTTLtakWda7em4
@caseymcc
caseymcc merged commit 436a23b into main Jul 19, 2026
1 check passed
@caseymcc
caseymcc deleted the multimodal-model-support branch July 19, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant