feat: support multimodal (image) tool results end-to-end#61
Open
shadaj wants to merge 1 commit into
Open
Conversation
Deploying infinity with
|
| Latest commit: |
96ce40e
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://9fdac6f6.infinity-dc7.pages.dev |
| Branch Preview URL: | https://sandbox-a179c018-4d8a-40c0-9.infinity-dc7.pages.dev |
shadaj
added a commit
that referenced
this pull request
Jul 9, 2026
…sults
Builds on the multimodal tool-result support: tool results can now also
*display* images, rendered inline in the web UI, with e2e tests exercising
the full RAP → daemon → client pipeline in both frontends.
## RAP protocol (`rap-protocol`)
* New `DisplaySegment::Image(ImageContent { data, mediaType })` display
segment type (base64), documented in the RAP spec (`tool-result.md`)
including the recommendation to follow it with a `"text"` fallback.
## Sandbox (`sandbox-core`)
* `read_file` on an image now emits `display_as: [image, text-summary]`
(image first for capable clients, text fallback for terminals). The
earlier 5 MiB cap remains removed.
## Clients
* **Web (`infinity-ui`)**: `DisplaySegment` TS union gains `image`;
`MessageItem` renders the first supported segment including images as an
inline bordered `<img>` (`data:` URL, max-height 300px,
`data-testid="tool-result-image"`).
* **TUI (`infinity-agent-cli`)**: the terminal renderer now scans for the
first *supported* segment (text/diff) instead of blindly taking the first,
so image segments gracefully fall back to the text summary. Same for the
ACP server's segment mapping.
## e2e tests
Both suites share a minimal in-process stub RAP tool server (axum, new dev
dep in both crates) exposing a `read_image` tool that answers with a fixed
96×64 indigo PNG as multimodal content plus image/text display segments,
wired to sessions via a cwd-local `.infinity/rap.json`.
* **TUI** (`e2e_daemon_tui.rs`): `rap_image_tool_result_reaches_model_and_tui`
asserts the base64 image content reaches the (image-capable) mock model
unchanged and the terminal renders the text fallback, with an insta screen
snapshot. Runs on real time (not `start_paused`) since the RAP round-trip
uses loopback TCP and paused-clock auto-advance would fire timeouts during
I/O waits. Harness `_cwd` renamed to `cwd` so tests can drop config in it.
* **Web** (`web_e2e.rs`, `e2e-web` feature): `image_tool_result_renders_inline`
asserts the follow-up model request carries the base64 image, the
transcript renders the inline `<img>` with the exact `data:` URL, and adds
the `chat-image-result.png` screenshot golden (**needs human visual
review**; pixel-verified to contain the indigo test image).
Also fixes the two e2e harnesses' `ModelEntry` initializers for the new
`supports_image_input` field (declared `true` so image content flows to the
mock model).
Co-authored-by: Infinity 🤖 <infinity@hydro.run>
PR: #61
shadaj
force-pushed
the
sandbox-a179c018-4d8a-40c0-9a3d-9bd0926bdeff
branch
from
July 9, 2026 20:55
ffeac53 to
72faf91
Compare
shadaj
added a commit
that referenced
this pull request
Jul 9, 2026
…support
Models declare image input support, RAP tools can return images in tool
results (both as model-facing content and as display segments), the sandbox
`read_file` returns image files as image attachments, the web UI renders
them inline, and e2e tests cover the full pipeline in both frontends.
## Provider protocol (`infinity-agent-core`)
* `ModelEntry` gains `supports_image_input: bool` (`#[serde(default)]`, so
the remote provider socket protocol stays backward compatible).
* `ModelProvider` gains `supports_image_input(model_id) -> bool` with a
default implementation that consults `list_models`.
* `HistoryManager::get_history` now takes `supports_image_input: bool`:
when `false`, image tool-result content is replaced in place (during the
existing message translation, no extra allocation pass) with the
`IMAGE_OMITTED_PLACEHOLDER` text. `run_completion` resolves the capability
once per completion, so images kept in history become visible after
switching to an image-capable model.
* Bedrock provider: all Claude models declare `supports_image_input: true`.
## RAP protocol (`rap-protocol`)
* New `RapToolResultContent` enum (`text` / `image { data, mediaType }`,
base64) and an optional `content: Vec<RapToolResultContent>` field on
`RapToolResult`; `text` remains the required plain-text fallback.
* New `DisplaySegment::Image(ImageContent { data, mediaType })` display
segment type for human-facing UIs.
* Both documented in the RAP spec (`tool-result.md`); provider capability
documented in `model-providers.md`.
## Daemon (`infinity-daemon`)
* RAP callbacks build the rig tool result from structured `content` when
present (images become `ToolResultContent::Image`), falling back to `text`.
## Sandbox (`sandbox-core` / `sandbox-local`)
* `read_file` detects image files by extension (png/jpg/jpeg/gif/webp, no
size limit) and returns a describing text plus base64 image content, with
`display_as: [image, text-summary]`; text files behave exactly as before.
`with_sandbox` made generic over the action payload to carry the content.
* Tool manifest description updated to mention image reads.
## Clients
* **Web (`infinity-ui`)**: `DisplaySegment` TS union gains `image`;
`MessageItem` renders images as an inline bordered `<img>` (`data:` URL,
max-height 300px, `data-testid="tool-result-image"`).
* **TUI / ACP (`infinity-agent-cli`)**: renderers scan for the first
*supported* segment instead of taking the first, so image segments fall
back to the text summary.
## New crate: `rap-test-servers` (unpublished)
Shared test-support crate (dev-dependency of `infinity-agent-cli` and
`infinity-daemon`): `start_stub_image_server()` serves a `read_image` RAP
tool answering with a fixed 96×64 indigo PNG (multimodal content + display
segments) through the standard callback flow; `write_rap_config(cwd, port)`
points sessions at it via a cwd-local `.infinity/rap.json`.
## Tests
* agent-core: mock-model tests that image tool results pass through to
image-capable models and are replaced with the placeholder otherwise;
remote provider round-trips `supports_image_input` over the socket.
* daemon: unit tests for RAP→rig content conversion (fallbacks, media types).
* sandbox-local: integration tests for PNG round-trip content and display
segments, extension → media type mapping, and text reads unchanged; the
text content item is asserted to match the plain-text fallback exactly.
* TUI e2e (`rap_image_tool_result_reaches_model_and_tui`): both the text
and base64 image tool-result content reach the mock model unchanged; the
terminal renders the text fallback (insta snapshot). Runs on real time —
paused-clock auto-advance would fire timeouts during the loopback-TCP RAP
round-trip.
* Web e2e (`image_tool_result_renders_inline`, `e2e-web` feature): follow-up
model request carries the base64 image; transcript renders the inline
`<img>` with the exact `data:` URL; new `chat-image-result.png` golden
(**needs human visual review**; pixel-verified to contain the test image).
Co-authored-by: Infinity 🤖 <infinity@hydro.run>
PR: #61
shadaj
force-pushed
the
sandbox-a179c018-4d8a-40c0-9a3d-9bd0926bdeff
branch
from
July 9, 2026 21:11
72faf91 to
13ee5bc
Compare
shadaj
added a commit
that referenced
this pull request
Jul 9, 2026
…support
Models declare image input support, RAP tools can return images in tool
results (both as model-facing content and as display segments), the sandbox
`read_file` returns image files as image attachments, the web UI renders
them inline, and e2e tests cover the full pipeline in both frontends.
## Provider protocol (`infinity-agent-core`)
* `ModelEntry` gains `supports_image_input: bool` (`#[serde(default)]`, so
the remote provider socket protocol stays backward compatible).
* `ModelProvider` gains `supports_image_input(model_id) -> bool` with a
default implementation that consults `list_models`.
* `HistoryManager::get_history` now takes `supports_image_input: bool`:
when `false`, image tool-result content is replaced in place (during the
existing message translation, no extra allocation pass) with the
`IMAGE_OMITTED_PLACEHOLDER` text. `run_completion` resolves the capability
once per completion, so images kept in history become visible after
switching to an image-capable model.
* Bedrock provider: all Claude models declare `supports_image_input: true`.
## RAP protocol (`rap-protocol`)
* New `RapToolResultContent` enum (`text` / `image { data, mediaType }`,
base64) and an optional `content: Vec<RapToolResultContent>` field on
`RapToolResult`; `text` remains the required plain-text fallback.
* New `DisplaySegment::Image(ImageContent { data, mediaType })` display
segment type for human-facing UIs.
* Both documented in the RAP spec (`tool-result.md`); provider capability
documented in `model-providers.md`.
## Daemon (`infinity-daemon`)
* RAP callbacks build the rig tool result from structured `content` when
present (images become `ToolResultContent::Image`), falling back to `text`.
## Sandbox (`sandbox-core` / `sandbox-local`)
* `read_file` detects image files by extension (png/jpg/jpeg/gif/webp, no
size limit) and returns a describing text plus base64 image content, with
`display_as: [image, text-summary]`; text files behave exactly as before.
`with_sandbox` made generic over the action payload to carry the content.
* Tool manifest description updated to mention image reads.
## Clients
* **Web (`infinity-ui`)**: `DisplaySegment` TS union gains `image`;
`MessageItem` renders images as an inline bordered `<img>` (`data:` URL,
max-height 300px, `data-testid="tool-result-image"`).
* **TUI / ACP (`infinity-agent-cli`)**: renderers scan for the first
*supported* segment instead of taking the first, so image segments fall
back to the text summary.
## New crate: `rap-test-servers` (unpublished)
Shared test-support crate (dev-dependency of `infinity-agent-cli` and
`infinity-daemon`): `start_stub_image_server()` serves a `read_image` RAP
tool answering with a fixed 96×64 indigo PNG (multimodal content + display
segments) through the standard callback flow; `write_rap_config(cwd, port)`
points sessions at it via a cwd-local `.infinity/rap.json`.
## Tests
* agent-core: mock-model tests that image tool results pass through to
image-capable models and are replaced with the placeholder otherwise;
remote provider round-trips `supports_image_input` over the socket.
* daemon: unit tests for RAP→rig content conversion (fallbacks, media types).
* sandbox-local: integration tests for PNG round-trip content and display
segments, extension → media type mapping, and text reads unchanged; the
text content item is asserted to match the plain-text fallback exactly.
* TUI e2e (`rap_image_tool_result_reaches_model_and_tui`): both the text
and base64 image tool-result content reach the mock model unchanged; the
terminal renders the text fallback (insta snapshot). Runs on real time —
paused-clock auto-advance would fire timeouts during the loopback-TCP RAP
round-trip.
* Web e2e (`image_tool_result_renders_inline`, `e2e-web` feature): follow-up
model request carries the base64 image; transcript renders the inline
`<img>` with the exact `data:` URL; new `chat-image-result.png` golden
(**needs human visual review**; pixel-verified to contain the test image).
Co-authored-by: Infinity 🤖 <infinity@hydro.run>
PR: #61
shadaj
force-pushed
the
sandbox-a179c018-4d8a-40c0-9a3d-9bd0926bdeff
branch
from
July 9, 2026 23:36
13ee5bc to
342160f
Compare
shadaj
marked this pull request as ready for review
July 9, 2026 23:41
shadaj
force-pushed
the
sandbox-a179c018-4d8a-40c0-9a3d-9bd0926bdeff
branch
4 times, most recently
from
July 16, 2026 22:37
c0bb9ff to
5627ee1
Compare
…support
Models declare image input support, RAP tools can return images in tool
results (both as model-facing content and as display segments), the sandbox
`read_file` returns image files as image attachments, the web UI renders
them inline, and e2e tests cover the full pipeline in both frontends.
## Provider protocol (`infinity-agent-core`)
* `ModelEntry` gains `supports_image_input: bool` (`#[serde(default)]`, so
the remote provider socket protocol stays backward compatible).
* `ModelProvider` gains `supports_image_input(model_id) -> bool` with a
default implementation that consults `list_models`.
* `HistoryManager::get_history` now takes `supports_image_input: bool`:
when `false`, image tool-result content is replaced in place (during the
existing message translation, no extra allocation pass) with the
`IMAGE_OMITTED_PLACEHOLDER` text. `run_completion` resolves the capability
once per completion, so images kept in history become visible after
switching to an image-capable model.
* Bedrock provider: all Claude models declare `supports_image_input: true`.
## RAP protocol (`rap-protocol`)
* New `RapToolResultContent` enum (`text` / `image { data, mediaType }`,
base64) and an optional `content: Vec<RapToolResultContent>` field on
`RapToolResult`; `text` remains the required plain-text fallback.
* New `DisplaySegment::Image(ImageContent { data, mediaType })` display
segment type for human-facing UIs.
* Both documented in the RAP spec (`tool-result.md`); provider capability
documented in `model-providers.md`.
## Daemon (`infinity-daemon`)
* RAP callbacks build the rig tool result from structured `content` when
present (images become `ToolResultContent::Image`), falling back to `text`.
## Sandbox (`sandbox-core` / `sandbox-local`)
* `read_file` detects image files by extension (png/jpg/jpeg/gif/webp, no
size limit) and returns a describing text plus base64 image content, with
`display_as: [image, text-summary]`; text files behave exactly as before.
`with_sandbox` made generic over the action payload to carry the content.
* Tool manifest description updated to mention image reads.
## Clients
* **Web (`infinity-ui`)**: `DisplaySegment` TS union gains `image`;
`MessageItem` renders images as an inline bordered `<img>` (`data:` URL,
max-height 300px, `data-testid="tool-result-image"`).
* **TUI / ACP (`infinity-agent-cli`)**: renderers scan for the first
*supported* segment instead of taking the first, so image segments fall
back to the text summary.
## New crate: `rap-test-servers` (unpublished)
Shared test-support crate (dev-dependency of `infinity-agent-cli` and
`infinity-daemon`): `start_stub_image_server()` serves a `read_image` RAP
tool answering with a fixed 96×64 indigo PNG (multimodal content + display
segments) through the standard callback flow; `write_rap_config(cwd, port)`
points sessions at it via a cwd-local `.infinity/rap.json`.
## Tests
* agent-core: mock-model tests that image tool results pass through to
image-capable models and are replaced with the placeholder otherwise;
remote provider round-trips `supports_image_input` over the socket.
* daemon: unit tests for RAP→rig content conversion (fallbacks, media types).
* sandbox-local: integration tests for PNG round-trip content and display
segments, extension → media type mapping, and text reads unchanged; the
text content item is asserted to match the plain-text fallback exactly.
* TUI e2e (`rap_image_tool_result_reaches_model_and_tui`): both the text
and base64 image tool-result content reach the mock model unchanged; the
terminal renders the text fallback (insta snapshot). Runs on real time —
paused-clock auto-advance would fire timeouts during the loopback-TCP RAP
round-trip.
* Web e2e (`image_tool_result_renders_inline`, `e2e-web` feature): follow-up
model request carries the base64 image; transcript renders the inline
`<img>` with the exact `data:` URL; new `chat-image-result.png` golden
(**needs human visual review**; pixel-verified to contain the test image).
Co-authored-by: Infinity 🤖 <infinity@hydro.run>
PR: #61
shadaj
force-pushed
the
sandbox-a179c018-4d8a-40c0-9a3d-9bd0926bdeff
branch
from
July 16, 2026 22:55
5627ee1 to
96ce40e
Compare
Member
Author
|
Confirmed that this works with a local build! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Models can now declare image input support, RAP tools can return images in
tool results, and the sandbox
read_filereturns image files as attachedimage content.
Provider protocol (
infinity-agent-core)ModelEntrygainssupports_image_input: bool(#[serde(default)], sothe remote provider socket protocol stays backward compatible).
ModelProvidergainssupports_image_input(model_id) -> boolwith adefault implementation that consults
list_models.HistoryManager::get_historynow takes asupports_image_input: boolparameter: when
false, image tool-result content is replaced in place(during the existing message translation, no extra allocation pass) with
the
IMAGE_OMITTED_PLACEHOLDERtext.run_completionresolves thecapability once per completion and passes it through, so switching to an
image-capable model retroactively exposes images kept in history.
supports_image_input: true.RAP protocol (
rap-protocol)RapToolResultContentenum (text/image { data, mediaType },base64) and an optional
content: Vec<RapToolResultContent>field onRapToolResult.textremains the required plain-text fallback.docs/docs/rap/spec/basic/tool-result.md(and the capability flag in
docs/docs/infinity-runtime/model-providers.md).Daemon (
infinity-daemon)convert_callbackbuilds the rig tool result from structuredcontentwhen present (images become
ToolResultContent::Imagewith base64 dataand MIME-derived media type), falling back to
textotherwise.Sandbox (
sandbox-core/sandbox-local)read_filedetects image files by extension (png/jpg/jpeg/gif/webp, nosize limit), returning a describing text plus base64 image content; text
files behave exactly as before.
with_sandboxwas made generic over theaction payload to carry the extra content channel.
Tests
to image-capable models and replaced with the placeholder otherwise;
remote provider round-trips
supports_image_inputover the socket.media type mapping, and text reads returning no structured content.