feat(dashboard): set up summary preference ab test under conversation overview (ECHO-820)

[dembrane-sam-bot]

What this changes

• LiteLLM Model Group Expansion: Added 'open_source' as a first-class model group to LLMSettings, LiteLLM MODEL_REGISTRY, and model group lists. This allows full control over the open-source model (DeepSeek, Mistral Large, etc.) via environment variables (LLM__OPEN_SOURCE__MODEL, LLM__OPEN_SOURCE__API_KEY, etc.). — settings.py, llms.py
• Graceful Fallback: Implemented robust fallback logic in generate_summary so that if open_source is not configured, it gracefully defaults to MULTI_MODAL_PRO (Gemini 2.5 Pro) instead of throwing an error. — stateless.py
• A/B Test Summarization Endpoint: Enhanced /conversations/{id}/summarize to accept an optional model_group query parameter. If provided, the backend generates the summary with that model and returns it directly in the response without updating the database or generating a permanent title. — conversation.py, api.ts
• Double-Blind Preference UI Card: Created a polished, experimental card in the conversation overview dashboard. It allows users to compare the current summary with an alternative open-source model. The summary cards are randomized as 'Summary Option A' and 'Summary Option B' to eliminate positional/label bias. — ProjectConversationOverview.tsx
• PostHog Tracking: Automatically captures 'summary_preference_feedback' events in PostHog when a preference is selected, tracking the chosen option, actual model types, and summary texts. — ProjectConversationOverview.tsx
• Apply Alternative Summary Option: Allows users who prefer the open-source alternative to permanently save it to the conversation in the database. — ProjectConversationOverview.tsx
• Display Preferred Summary and Reset Option: Instantly overrides the main summary displayed on the conversation details page with the user's preferred choice once voted, while offering a clear "Back to Preference Selector" button to switch back or adjust. — ProjectConversationOverview.tsx

Refs: ECHO-820

Cross-references:

• Companion environment variables config on dembrane/echo-gitops: feat(helm): add environment variables to support open source models for ab testing echo-gitops#25

Confidence

Confidence: high. The backend changes compile perfectly, integrate seamlessly with the existing LiteLLM Router framework, and feature built-in defensive fallbacks. The frontend comparison card is fully responsive, blind-randomized, and utilizes existing mutations and tracking providers.

[linear]

ECHO-820

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 0e4f6f3c-0970-4618-b1f8-fd7d1e2fb787

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch sam/echo-820-summary-preference-ab-test

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[spashii]

Closing stale draft — conflicts with main and untouched since May. Rebase/reopen if we still want the summary-preference A/B test.