diff --git a/docs/design/agent-workflows/documentation/adapters/pi.md b/docs/design/agent-workflows/documentation/adapters/pi.md index c7d5cf0fb8..b53b0fe9fd 100644 --- a/docs/design/agent-workflows/documentation/adapters/pi.md +++ b/docs/design/agent-workflows/documentation/adapters/pi.md @@ -143,6 +143,23 @@ answers. For output, Pi streams pure text deltas over ACP (`agent_message_chunk`). The runner appends them in order to build the final answer. +### OpenAI-compatible custom endpoints + +A named `custom` connection (a base URL, a key, and a model id, resolved to provider `openai` +and deployment `custom`) reaches Pi through a `models.json` the runner writes into the per-run +Pi agent dir. The builder (`services/runner/src/engines/sandbox_agent/pi-model-config.ts`) is a +pure function that returns a plan keyed by the connection slug, in the shape +`{"providers": {"": {baseUrl, api: "openai-completions", apiKey: "$OPENAI_API_KEY", +models: [{id}]}}}`. The `apiKey` is an environment reference, so no secret lands on disk; the +real key rides the sandbox env. Locally the runner writes the file `0600` and atomically into an +isolated per-run dir that carries no operator `auth.json`, so a plan run never authenticates with +the operator's login. On Daytona it uploads the file to `DAYTONA_PI_DIR` before session creation. + +For a custom run the runner requests the fully qualified `/` id rather than the bare +model. This bypasses the suffix match above and its risk of silently routing to `api.openai.com`. +Builder, write, and upload failures are terminal; the run never falls through to a default +provider. + ## Daytona notes Two things differ on Daytona. The sandbox-agent `-full` image ships the `pi-acp` adapter but not the diff --git a/docs/design/agent-workflows/interfaces/cross-service/service-to-vault-and-tool-providers.md b/docs/design/agent-workflows/interfaces/cross-service/service-to-vault-and-tool-providers.md index 97038e5c04..4b86e59349 100644 --- a/docs/design/agent-workflows/interfaces/cross-service/service-to-vault-and-tool-providers.md +++ b/docs/design/agent-workflows/interfaces/cross-service/service-to-vault-and-tool-providers.md @@ -26,11 +26,15 @@ The service resolves several types, each with a clear job: one connection deterministically, and fails loud on ambiguity. The run never receives the whole vault. -**Capability gating.** A per-harness table decides which providers, deployments, and -connection modes a harness can reach. Pi reaches eight direct providers; Claude reaches -Anthropic across direct, custom, bedrock, and vertex. The service checks provider and mode -before resolution and deployment after, and rejects unreachable choices server-side. The -same table ships on `/inspect` so the form can filter ahead of time. +**Capability gating.** A resolved-pair table (`harness_allows_pair`) decides which resolved +`(provider, deployment)` pairs a harness can reach. Pi reaches its direct providers plus the +OpenAI-compatible `custom` deployment (a provider-less `custom` connection resolves to provider +`openai`, deployment `custom`); Claude reaches Anthropic across direct, custom, bedrock, and +vertex. The service checks provider and mode before resolution and deployment after, and rejects +unreachable choices server-side. The flat `deployments` lists (`["direct", "custom"]` for Pi) feed +the form and `/inspect`; the pair table is authoritative for a run. A `custom` connection with an +unusable base URL never falls back to a default endpoint. It raises `EndpointResolutionError`, +which returns HTTP 422, as do `UnsupportedProviderError` and `UnsupportedDeploymentError`. ## Owned by diff --git a/docs/design/agent-workflows/interfaces/in-service/model-connection-resolution.md b/docs/design/agent-workflows/interfaces/in-service/model-connection-resolution.md index 7891ffd9f1..cfde22125c 100644 --- a/docs/design/agent-workflows/interfaces/in-service/model-connection-resolution.md +++ b/docs/design/agent-workflows/interfaces/in-service/model-connection-resolution.md @@ -36,10 +36,13 @@ The run receives only that candidate's `env` and `endpoint`. It never receives t vault. **Capability gating** runs in two halves around resolution. Provider and connection mode are -checked before; deployment is checked after (the resolved connection is what carries it). -Each harness publishes what it can reach: Pi reaches eight direct providers; Claude reaches -Anthropic across direct, custom, bedrock, and vertex. Unnamed default connections degrade -tolerantly to an empty `env` rather than failing the run. +checked before; deployment is checked after (the resolved connection is what carries it). A +resolved-pair table (`harness_allows_pair`) is authoritative: Pi reaches its direct providers +plus the OpenAI-compatible `custom` deployment (a provider-less `custom` connection resolves to +provider `openai`, deployment `custom`); Claude reaches Anthropic across direct, custom, bedrock, +and vertex. Unnamed default connections degrade tolerantly to an empty `env` rather than failing +the run. A `custom` connection with an unusable base URL is the exception: it never resolves to a +default endpoint. It raises `EndpointResolutionError`, which returns HTTP 422. ## Owned by diff --git a/docs/design/agent-workflows/projects/pi-openai-compatible-models/README.md b/docs/design/agent-workflows/projects/pi-openai-compatible-models/README.md new file mode 100644 index 0000000000..ec84874955 --- /dev/null +++ b/docs/design/agent-workflows/projects/pi-openai-compatible-models/README.md @@ -0,0 +1,43 @@ +# Pi OpenAI-compatible models + +This project makes a vault `custom_provider` connection usable by the Pi harnesses when the +endpoint implements OpenAI Chat Completions. It keeps the current vault and `/run` schemas, gives +the existing `custom` provider kind the product label **OpenAI-compatible**, and translates the +resolved connection into Pi's native `models.json` format inside the runner. + +The implementation is intentionally narrow. It establishes one clean service-to-runner path for a +resolved model connection without introducing a general provider protocol framework. A later +Anthropic Messages implementation can add a second translation beside the OpenAI-compatible one. + +## Documents + +- [context.md](context.md) explains the problem, goals, non-goals, and boundaries. +- [research.md](research.md) records the current UI, service, wire, runner, and Pi behavior. +- [design.md](design.md) defines the proposed behavior and the decisions behind it. +- [plan.md](plan.md) breaks implementation into reviewable phases. +- [qa.md](qa.md) defines unit, contract, integration, and live acceptance coverage. +- [status.md](status.md) is the source of truth for progress and open decisions. + +## Decision summary + +1. Keep `custom_provider.data.kind = "custom"`; change only its user-facing label to + **OpenAI-compatible**. +2. Keep the current `ModelRef`, `ResolvedConnection`, `Endpoint`, `secrets`, and `/run` wire + fields. No persistent or wire schema migration is required. +3. Interpret a provider-less named `custom` connection as provider family `openai`. Preserve an + explicit provider family for existing connections, including Anthropic gateway configurations. +4. Validate named connections after vault resolution against the combination of provider family + and deployment. Do not add a wildcard provider capability. +5. For Pi plus `provider=openai` plus `deployment=custom`, generate an isolated `models.json` with + `api: "openai-completions"` and an environment reference to `OPENAI_API_KEY`. +6. Keep Claude's existing connection translation unchanged. Add regression tests around it. +7. Fail the run if a selected custom endpoint cannot be validated or materialized. Never fall back + to the default OpenAI endpoint or a different model. + +## Relationship to earlier work + +The older [model-config project](../model-config/) established that Pi needs `models.json`. Its +paths and several premises predate the current `services/runner` architecture, strict model +selection, least-privilege connection resolution, and warm Daytona sessions. This workspace is the +current implementation plan for the remaining OpenAI-compatible slice. + diff --git a/docs/design/agent-workflows/projects/pi-openai-compatible-models/context.md b/docs/design/agent-workflows/projects/pi-openai-compatible-models/context.md new file mode 100644 index 0000000000..6e4c43079c --- /dev/null +++ b/docs/design/agent-workflows/projects/pi-openai-compatible-models/context.md @@ -0,0 +1,72 @@ +# Context + +## Current user experience + +A user can create a custom provider in the model registry with a name, base URL, API key, and one +or more model IDs. The vault stores this as a `custom_provider`. The old LLM playground can use the +record, but an agent using `pi_core` or `pi_agenta` cannot. + +The agent UI already carries most of the intended connection identity. A vault model option +contains the connection slug, and selecting it writes a structured model reference. The agent +service resolves the named vault record into a model, endpoint, deployment, credential mode, and +least-privilege environment. Those values already cross the `/run` boundary. + +The feature stops at the runner. Pi does not use a generic `OPENAI_BASE_URL` variable for ordinary +OpenAI-compatible models. Pi discovers custom providers and arbitrary model IDs through +`PI_CODING_AGENT_DIR/models.json`. The runner never creates that file. + +## Goals + +- Make an OpenAI-compatible vault connection work automatically with `pi_core` and `pi_agenta`. +- Use the existing vault secret, model reference, resolved connection, and `/run` wire fields. +- Default the existing `custom` provider kind to OpenAI Chat Completions. +- Preserve least-privilege credential delivery and current secret isolation. +- Work identically on the local and Daytona runner paths. +- Keep strict model selection: the requested custom model either runs or fails clearly. +- Keep the implementation easy to extend to Anthropic Messages without designing that extension + now. +- Avoid changing Claude or other harness behavior as part of this feature. + +## Non-goals + +- No vault schema migration or new secret kind. +- No new `/run` field for provider protocol, API type, or Pi configuration. +- No generic protocol registry. +- No Anthropic Messages support in Pi in this phase. +- No OpenAI Responses support in this phase. +- No Bedrock, Vertex, Azure, or Sagemaker support in Pi. +- No automatic discovery of models from an endpoint. +- No support for unauthenticated local endpoints in the first slice. Users may use a non-secret + placeholder API key for servers that ignore authentication. +- No attempt to make a Daytona sandbox's `localhost` refer to the runner host. + +## Constraints + +### Security + +- Raw API keys remain in `ResolvedConnection.env` and the wire `secrets` field. +- `models.json` contains only an environment reference such as `$OPENAI_API_KEY`, never the key. +- Managed Pi runs must not write model configuration into an operator's shared subscription + directory. +- A blocked or malformed endpoint must fail the named connection. Silently removing the endpoint + would risk sending the request to a provider default. +- Private, loopback, and insecure HTTP endpoints remain disabled by default. Trusted single-tenant + deployments can opt in with the existing egress setting. + +### Compatibility + +- Existing standard provider keys continue to use Pi's built-in model catalog and env mapping. +- Existing Claude custom-gateway handling continues to use `ANTHROPIC_BASE_URL`. +- Existing explicit provider families on custom records remain authoritative. Only a missing + provider family on `kind=custom` defaults to `openai`. +- Unknown harnesses and unsupported provider/deployment combinations remain closed. + +### Ownership + +- The UI owns product labels and selection intent. +- The service owns vault resolution, semantic normalization, endpoint validation, and harness + capability validation. +- The runner owns conversion from the neutral resolved connection into harness-native runtime + configuration. +- Pi owns the format and behavior of `models.json`. + diff --git a/docs/design/agent-workflows/projects/pi-openai-compatible-models/design.md b/docs/design/agent-workflows/projects/pi-openai-compatible-models/design.md new file mode 100644 index 0000000000..79dd13bac9 --- /dev/null +++ b/docs/design/agent-workflows/projects/pi-openai-compatible-models/design.md @@ -0,0 +1,254 @@ +# Design + +## Overview + +The existing neutral connection boundary is sufficient. The service will normalize and validate a +named custom connection. The runner will translate that resolved connection into Pi's native +configuration before Pi starts. + +```text +UI selection + -> ModelRef(model, provider, connection.slug) + -> service resolves one vault connection + -> ResolvedConnection + secrets + -> existing /run request + -> Pi adapter builds models.json + -> local write or Daytona upload + -> ACP session creation + -> strict model selection +``` + +## Decision 1: preserve the schemas + +Keep the persisted value `custom` and the existing wire fields. Change its UI label to +"OpenAI-compatible". + +The first implementation has one protocol default: + +```text +provider family: openai +API dialect: openai-completions +``` + +This is a service and runner interpretation, not a new user-authored field. It is appropriate while +the product offers only one custom API dialect. A protocol field becomes justified when the UI +offers a real second choice, such as Anthropic Messages or OpenAI Responses. + +Rejected alternatives: + +- **Store `openai-compatible` as a new enum value.** This creates a vault and generated-client + migration for a label-level distinction that the current `custom` value can represent. +- **Add `api` to `/run` now.** The runner can derive the only supported value. The field would be + future scaffolding rather than current user intent. +- **Set `OPENAI_BASE_URL`.** Pi does not use it to register arbitrary custom models. + +## Decision 2: normalize provider intent in the service + +For a named `custom_provider` candidate: + +1. Preserve an explicit `ModelRef.provider` when it names a known family. This retains existing + Anthropic gateway behavior. +2. If the model reference has no provider and `data.kind == "custom"`, resolve the provider family + as `openai`. +3. Map the custom connection's API key to the resolved family's canonical env variable. The v1 Pi + path therefore receives `OPENAI_API_KEY`. +4. Keep `deployment = "custom"`, `endpoint.base_url`, the exact model ID, and the connection slug. + +The connection slug remains connection identity. It must not replace the semantic provider family +inside `ResolvedConnection.provider`. + +This gives the runner both concepts without a schema change: + +- `request.provider`: the API family (`openai`); +- `request.connection.slug`: the unique runtime provider ID (`my-ollama`). + +## Decision 3: validate the resolved combination + +Do not make `harness_allows_provider` permissive and do not add `"*"` to Pi. + +For an explicit named Agenta connection, the pre-resolve check should validate the connection mode +but defer provider acceptance until the vault record is resolved. The post-resolve check then +validates the complete pair: + +```text +harness + resolved provider family + resolved deployment +``` + +Initial allowed pairs: + +| Harness | Provider | Deployment | Result | +| --- | --- | --- | --- | +| pi_core / pi_agenta | openai | direct | Existing behavior | +| pi_core / pi_agenta | openai | custom | New behavior | +| pi_core / pi_agenta | arbitrary | custom | Reject | +| claude | anthropic | direct/custom | Existing behavior | +| claude | openai | custom | Reject | +| unknown harness | any | any | Reject | + +The published capability table can add `custom` to Pi's deployments so the UI can surface the +connection. Server-side pair validation remains authoritative because independent provider and +deployment lists cannot express cross-product restrictions by themselves. + +This is a small extension of the existing capability mechanism, not a new policy framework. + +## Decision 4: fail endpoint resolution loudly + +A named custom connection requires a usable base URL. The service must fail with a typed, +actionable connection error when the URL is absent, malformed, blocked, or unresolvable under the +current egress policy. + +It must not return the same connection with `endpoint=None`. Continuing would let the harness use a +default endpoint, violating the user's explicit routing choice. + +The error should identify the connection slug and explain whether trusted self-hosted deployments +can use `AGENTA_INSECURE_EGRESS_ALLOWED`. It must not include the API key. + +## Decision 5: isolate Pi translation in a small adapter + +Add a focused runner module, for example `sandbox_agent/pi-model-config.ts`, with two layers. + +### Pure planning layer + +The pure builder accepts the existing request and returns either no Pi override or a validated +internal plan: + +```ts +interface PiModelConfigPlan { + providerId: string // connection.slug + providerFamily: "openai" + api: "openai-completions" + baseUrl: string + apiKeyEnv: "OPENAI_API_KEY" + models: Array<{id: string}> +} +``` + +The builder applies only when all are true: + +- harness is Pi; +- deployment is `custom`; +- provider is `openai`; +- connection mode is `agenta` with a non-empty slug; +- endpoint has a base URL; +- credential mode is `env`; +- `OPENAI_API_KEY` is present in the resolved secret set; +- a model ID is present. + +An incomplete applicable request is an error, not a no-op. A non-applicable request returns no +plan and follows current behavior. + +The builder emits no secret values. Its `apiKeyEnv` becomes `$OPENAI_API_KEY` in JSON. + +### Materialization layer + +The materializer serializes one exact provider and its selected model: + +```json +{ + "providers": { + "my-ollama": { + "baseUrl": "https://example.test/v1", + "api": "openai-completions", + "apiKey": "$OPENAI_API_KEY", + "models": [{"id": "qwen2.5-coder:7b"}] + } + } +} +``` + +Rules: + +- Write the exact plan. Do not merge arbitrary existing providers into a managed run. +- Use mode `0600` locally and an atomic temporary-file-plus-rename pattern where supported. +- Upload the exact JSON to `DAYTONA_PI_DIR/models.json` before `createSession` on Daytona. +- Treat write or upload failure as terminal. +- Never log the document or secret values. A log may identify the connection slug, API dialect, + model ID, and sanitized endpoint host. + +This module is Pi-specific because `models.json` is a Pi mechanism. The input remains neutral. A +generic cross-harness adapter interface would add indirection before there is a second consumer. +If Anthropic Messages later needs the same builder, extend the pure plan's `api` union and provider +mapping. If another harness needs a different mechanism, add a sibling adapter and only then +extract a shared interface. + +## Decision 6: prepare managed Pi assets before session creation + +### Local + +A managed custom connection always requires an isolated Pi agent directory, even when there are no +skills or prompt overrides. Extend `prepareLocalPiAssets` so a Pi model-config plan is another +reason to create the throwaway directory. + +Write `models.json` into that directory, point `PI_CODING_AGENT_DIR` at it, and delete it during the +existing teardown. Do not write managed model configuration into the operator's shared +subscription directory. + +The implementation should reuse the existing extension/settings preparation, but it must not add +new credential copies. `models.json` references the run-scoped environment key. + +### Daytona + +Daytona already receives the resolved secrets as sandbox env. After the sandbox exists and before +the ACP session starts, upload `models.json` into `DAYTONA_PI_DIR` with the other Pi assets. + +Overwrite any previous file. Reusable sandboxes must not retain a provider from an earlier +configuration. + +## Decision 7: preserve strict selection and warm-session safety + +After Pi reads `models.json`, pi-acp should advertise `/`. The existing +strict `applyModel` suffix matching can map the wire's exact model ID to that option. No custom +fallback is needed. + +If Pi does not advertise it, the run fails with `ModelNotSettableError`. It must never continue on +the built-in OpenAI endpoint or a harness default. + +No new keepalive fingerprint is needed: + +- configuration fingerprint already covers model, provider, connection, deployment, endpoint, and + credential mode; +- credential epoch already covers secret rotation. + +Add tests that pin these assumptions so later fingerprint refactors cannot silently reuse a +mismatched session. + +## Decision 8: make the UI semantic but keep it small + +Change `PROVIDER_LABELS.custom` from "Custom Provider" to "OpenAI-compatible". Keep the underlying +value `custom`. + +For a model selected from a `custom` connection: + +- preserve a provider family encoded by the model or already stored explicitly; +- otherwise choose `openai`, not the previously selected unrelated provider; +- always preserve the selected connection slug. + +Custom model visibility should require both a supported deployment and a compatible provider +family. This prevents the UI from presenting OpenAI-compatible connections to a harness that only +supports Anthropic. + +Do not add an API protocol selector in this phase. + +## Future protocol extension + +Anthropic Messages should be added only after its product behavior is specified. It has different +model metadata, authentication, headers, compatibility details, and harness support. + +The intended extension point is the pure runner plan: + +```ts +type PiProviderApi = "openai-completions" | "anthropic-messages" +``` + +At that point: + +- an explicit Anthropic provider family can map to `anthropic-messages`; +- its key can remain `ANTHROPIC_API_KEY`; +- the same local/Daytona materializers can write the resulting plan; +- capability pair validation can allow Pi plus Anthropic plus custom; +- the UI can expose a real protocol choice or provider-specific form. + +If the second protocol needs user-authored compatibility, headers, or API selection, add those as +provider configuration under the vault connection at that time. Do not encode them in metadata or +add Pi-specific wire fields. + diff --git a/docs/design/agent-workflows/projects/pi-openai-compatible-models/open-issues.md b/docs/design/agent-workflows/projects/pi-openai-compatible-models/open-issues.md new file mode 100644 index 0000000000..38cfc8634d --- /dev/null +++ b/docs/design/agent-workflows/projects/pi-openai-compatible-models/open-issues.md @@ -0,0 +1,163 @@ +# Open issues + +Deferred TODOs and open questions for this project. Each entry carries enough context and +provenance to act on cold. See the `defer-todo` skill for the format. + +## Open issues + +### Stat-guard the Daytona stale models.json cleanup for no-plan Pi runs + +**Status:** open +**Added:** 2026-07-15 +**Commit:** 277e48d9f0 (branch `gitbutler/workspace`) +**Project:** [Pi OpenAI-compatible models](README.md) +**Source:** docs sync after the feature was implemented and end-to-end verified. + +**The problem.** A Pi run that carries no model-config plan removes any stale `models.json` from +`DAYTONA_PI_DIR` before session creation. That cleanup is best-effort right now: it swallows the +delete error and continues. On a reused Daytona sandbox a leftover `models.json` from an earlier +custom run could then register a provider the current run did not ask for. A plan run overwrites +the file, so it is safe. The gap is only the no-plan case on a warm sandbox. + +**Why it is deferred.** The end-to-end matrix passed, and a stale file only bites a reused sandbox +whose previous run was a custom endpoint. It is a hardening step, not a correctness fix for the +shipped happy path, so it was kept out of the first PR to keep the diff focused. + +**What to decide or do.** Change the no-plan cleanup from best-effort to stat-guarded and terminal. +Stat the target path. If a `models.json` exists and the delete fails, fail the run rather than +continue on a possibly stale provider. Add a runner test that a no-plan run on a sandbox seeded +with a stale `models.json` either removes it or fails. + +### Require apiBaseUrl at save time for a kind=custom connection + +**Status:** open +**Added:** 2026-07-15 +**Commit:** 277e48d9f0 (branch `gitbutler/workspace`) +**Project:** [Pi OpenAI-compatible models](README.md) +**Source:** docs sync after the feature was implemented and end-to-end verified. + +**The problem.** A `kind=custom` connection with no base URL is useless: the service rejects it at +run time with `EndpointResolutionError` (HTTP 422). The connection form +(`web/packages/agenta-entities/src/secret/core/providerFields.ts`) does not require `apiBaseUrl` +at save time, so a user can create a connection that can never run and only learns it when a run +fails. + +**Why it is deferred.** The run-time guard already fails loud with an actionable error, so no run +routes to the wrong place. Moving the check to creation is a usability improvement, not a +correctness fix, and it touches the form validation rather than the run path this project owns. + +**What to decide or do.** Make `apiBaseUrl` a required field for `kind=custom` in +`providerFields.ts` so an incomplete connection fails at save. Keep the run-time guard as the +backstop. Add a form test that a custom connection with no base URL cannot be saved. + +### Add an engine-level test that an incomplete applicable request yields ok:false + +**Status:** open +**Added:** 2026-07-15 +**Commit:** 277e48d9f0 (branch `gitbutler/workspace`) +**Project:** [Pi OpenAI-compatible models](README.md) +**Source:** docs sync after the feature was implemented and end-to-end verified. + +**The problem.** The pure builder in `pi-model-config.ts` returns a typed error when a request is +applicable (Pi plus provider `openai` plus deployment `custom`) but incomplete, for example a +missing base URL, model, or env credential. The builder has unit coverage for this. There is no +engine-level test that a run driven by such a request ends with `{ok: false}` rather than falling +through to a default provider. The terminal behavior is only proven at the builder layer. + +**Why it is deferred.** The builder unit tests and the live 422 paths give confidence the run +fails, and the engine wiring treats a builder error as terminal by construction. A dedicated +engine test pins that contract so a later refactor of the engine cannot silently reroute an +incomplete request. + +**What to decide or do.** Add a runner engine test that feeds an applicable-but-incomplete request +through the sandbox-agent engine and asserts the result is `{ok: false}` with the typed error, and +that no session is created against a default endpoint. + +### Run the updated Playwright model-hub suite live + +**Status:** open +**Added:** 2026-07-15 +**Commit:** 277e48d9f0 (branch `gitbutler/workspace`) +**Project:** [Pi OpenAI-compatible models](README.md) +**Source:** docs sync after the feature was implemented and end-to-end verified. + +**The problem.** The UI relabels the `custom` provider kind to "OpenAI-compatible endpoint". The +Playwright strings in `web/oss/tests/playwright/1-settings/model-hub.ts` (and the acceptance copy) +are updated to match, but no live Playwright run has confirmed the suite passes against the new +label. The strings are edited on trust. + +**Why it is deferred.** The entity-ui unit suite (208 passed) covers the label and picker logic, +and the manual UI click-through passed on the dev stack. A full Playwright run needs a live +browser environment that was not part of this session. + +**What to decide or do.** Run the Playwright model-hub suite against a deployment carrying the new +label. Confirm it passes and fix any string drift the unit tests did not catch. + +### Thread model_ref for the Claude template if its named-connection wire identity is wanted + +**Status:** open +**Added:** 2026-07-15 +**Commit:** 277e48d9f0 (branch `gitbutler/workspace`) +**Project:** [Pi OpenAI-compatible models](README.md) +**Source:** docs sync after the feature was implemented and end-to-end verified. + +**The problem.** The fix in `adapters/harnesses.py` threads the structured `model_ref` so a named +connection's `{mode, slug}` reaches the `/run` wire. It threads it for the Pi and Agenta harnesses +only. The Claude harness is left unchanged on purpose, because Claude does not consume the slug and +this project scoped Claude out. A named connection selected for a Claude template therefore does not +carry its `{mode, slug}` identity on the wire. + +**Why it is deferred.** Claude reaches Anthropic through its own deployment path and does not read a +Pi-style `models.json`, so it has no use for the slug today. Threading it would change the Claude +wire shape and its golden fixtures for no current consumer. + +**What to decide or do.** If a Claude named-connection wire identity is ever wanted (for example to +key a Claude-side provider config or for parity), thread `model_ref` for the Claude harness in +`adapters/harnesses.py` the same way, and update the Claude wire golden fixtures with it. + +### Add the Anthropic Messages dialect + +**Status:** open +**Added:** 2026-07-15 +**Commit:** 277e48d9f0 (branch `gitbutler/workspace`) +**Project:** [Pi OpenAI-compatible models](README.md) +**Source:** design.md "Future protocol extension"; carried into this log during the docs sync. + +**The problem.** The runner speaks only `openai-completions` in v1. The pure builder in +`pi-model-config.ts` exposes `api` as a discriminator so a second dialect can be added later, but +Anthropic Messages is not implemented. Anthropic Messages has different model metadata, +authentication, headers, and request and response semantics, so it cannot be inferred from a +provider label. + +**Why it is deferred.** The first release covers one custom API dialect, which is the actual +requirement. Anthropic Messages needs its own product behavior specified before it is built, so it +was kept out of scope. + +**What to decide or do.** When Anthropic Messages is specified, extend the pure plan's `api` union +to `"openai-completions" | "anthropic-messages"`, map an explicit Anthropic provider family to it, +keep its key as `ANTHROPIC_API_KEY`, let capability pair validation allow Pi plus Anthropic plus +`custom`, and expose the protocol choice in the UI. See [design.md](design.md) "Future protocol +extension" for the full extension point. + +### Give the provider-kind UI label a design-doc home + +**Status:** open +**Added:** 2026-07-16 +**Commit:** 5d29b1938c (branch `gitbutler/workspace`) +**Project:** [Pi OpenAI-compatible models](README.md) +**Source:** keep-docs-in-sync sweep after the feature was implemented and end-to-end verified. + +**The problem.** The UI relabels the `custom` provider kind from "Custom provider" to +"OpenAI-compatible endpoint" in `web/`. The keep-docs-in-sync sweep found no page in +`docs/design/agent-workflows/documentation/` or `interfaces/` that describes the settings +provider-kind label, so there is nothing to update. The runner and capability changes have homes +(the Pi adapter page, the model-connection-resolution and vault inventory pages); the label string +does not. + +**Why it is deferred.** The label is a UI string, not a wire or config contract, and inventing a +new design-doc page for one string is out of scope for the sweep. The label is covered by the +frontend PR body and the entity-ui unit suite. + +**What to decide or do.** Decide whether the vault provider-kind labels belong in the design docs +at all. If they do, add a short reference table (kind value, UI label) to an existing page rather +than a new page. If not, close this as intentionally undocumented. diff --git a/docs/design/agent-workflows/projects/pi-openai-compatible-models/plan.md b/docs/design/agent-workflows/projects/pi-openai-compatible-models/plan.md new file mode 100644 index 0000000000..4eda3e9ff9 --- /dev/null +++ b/docs/design/agent-workflows/projects/pi-openai-compatible-models/plan.md @@ -0,0 +1,160 @@ +# Implementation plan + +## Phase 0: pin current behavior + +Before changing behavior, add or update tests that describe the current boundaries: + +- Pi rejects `deployment=custom`. +- A generic custom key with no provider family does not reach a canonical env variable. +- The custom endpoint already reaches the neutral `/run` payload. +- Claude maps an Anthropic custom gateway to `ANTHROPIC_BASE_URL`. +- `configFingerprint` changes when connection, model, or endpoint changes. +- credential epoch changes when the custom key rotates. + +These tests distinguish intended changes from regressions. + +## Phase 1: service normalization and capability validation + +Primary files: + +- `sdks/python/agenta/sdk/agents/platform/connections.py` +- `sdks/python/agenta/sdk/agents/handler.py` +- `sdks/python/agenta/sdk/agents/capabilities.py` +- SDK agent connection and composition tests + +Work: + +1. Normalize a provider-less `kind=custom` named connection to provider family `openai`. +2. Preserve an explicit known provider family for existing connections. +3. Route the normalized custom key through `OPENAI_API_KEY`. +4. Require a valid base URL for an explicit named custom connection. Replace silent URL dropping + with a typed resolution error. +5. For named Agenta connections, defer provider acceptance until after resolve. +6. Add an authoritative resolved-pair validation helper. Allow Pi plus OpenAI plus custom; retain + current direct and Claude combinations; reject arbitrary combinations. +7. Publish `custom` in Pi's deployment capabilities after the server can consume it. + +Exit criteria: + +- A named OpenAI-compatible connection resolves to provider `openai`, deployment `custom`, exact + model, endpoint, and only `OPENAI_API_KEY`. +- Invalid endpoints fail before the runner call. +- Direct providers, self-managed connections, Claude, and unknown harnesses retain their current + behavior. + +## Phase 2: pure Pi model-config builder + +Primary files: + +- new `services/runner/src/engines/sandbox_agent/pi-model-config.ts` +- new focused unit test file + +Work: + +1. Define the internal `PiModelConfigPlan`. +2. Build it from the existing `AgentRunRequest` and resolved secrets. +3. Serialize the exact `models.json` shape with `api: "openai-completions"`. +4. Reject incomplete applicable requests with typed errors. +5. Prove that raw API keys never appear in the plan or serialized file. +6. Prove that standard Pi, subscription, and Claude requests return no plan. + +Exit criteria: + +- The pure builder has exhaustive unit coverage and no filesystem or sandbox dependencies. +- No wire field or protocol type changes. + +## Phase 3: local Pi materialization + +Primary files: + +- `services/runner/src/engines/sandbox_agent/pi-assets.ts` +- `services/runner/src/engines/sandbox_agent.ts` +- Pi asset and orchestration tests + +Work: + +1. Make a model-config plan a reason to create an isolated managed Pi agent directory. +2. Write `models.json` before creating the ACP session. +3. Use restrictive permissions and atomic replacement. +4. Keep the environment key in `secrets`; write only `$OPENAI_API_KEY` to disk. +5. Make materialization failure terminal. +6. Reuse existing teardown for the throwaway directory. + +Exit criteria: + +- Local Pi advertises and selects `/`. +- The operator's shared Pi login directory is unchanged. +- A failed write cannot fall through to a default provider. + +## Phase 4: Daytona parity + +Primary files: + +- `services/runner/src/engines/sandbox_agent/daytona.ts` +- `services/runner/src/engines/sandbox_agent.ts` +- Daytona unit and orchestration tests + +Work: + +1. Upload the exact config to `DAYTONA_PI_DIR/models.json` before `createSession`. +2. Overwrite a stale file on every cold environment acquisition. +3. Keep using `daytonaEnvVars` for the resolved API key. +4. Make upload failure terminal. +5. Verify config and credential fingerprint changes evict or cold-start rather than reuse a + mismatched live session. + +Exit criteria: + +- The local and Daytona paths produce equivalent Pi-visible configuration. +- No personal subscription credential is uploaded. +- Reused sandboxes do not retain a prior custom provider configuration. + +## Phase 5: UI terminology and selection behavior + +Primary files: + +- `web/packages/agenta-entities/src/secret/core/types.ts` +- `web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/connectionUtils.ts` +- relevant entity UI tests + +Work: + +1. Rename the `custom` label to "OpenAI-compatible" without changing its stored value. +2. Default provider-less custom model selections to family `openai`. +3. Preserve explicit provider families and connection slugs. +4. Filter custom model options using both deployment and provider compatibility. +5. Keep the form limited to Chat Completions. Add concise helper text if needed. + +Exit criteria: + +- Existing vault records still load and edit. +- A new OpenAI-compatible connection is automatically selected after creation. +- Pi displays it after the capability update. +- Claude and other harnesses do not gain incompatible choices. + +## Phase 6: integration and live QA + +1. Run the Python SDK agent suite. +2. Run runner unit tests and typecheck. +3. Run affected web package unit tests and `pnpm lint-fix` before any commit. +4. Exercise a fake OpenAI-compatible HTTP server through the service-to-runner path. +5. Verify local Pi against one real compatible endpoint. +6. Verify Daytona against an endpoint reachable from Daytona. +7. Verify a blocked loopback endpoint fails clearly by default. +8. Verify a trusted self-hosted local endpoint with the existing insecure-egress opt-in. +9. Re-run Claude direct and custom-gateway smoke cases. + +## Suggested review slices + +The safest review order is: + +1. Service normalization, validation, and capabilities. +2. Pure runner builder plus local materialization. +3. Daytona materialization. +4. UI label and picker behavior. +5. Live QA evidence. + +The slices can share one design but should remain separately reviewable. Do not enable Pi's custom +deployment capability until the runner slice that consumes it is available in the same release +line. + diff --git a/docs/design/agent-workflows/projects/pi-openai-compatible-models/qa.md b/docs/design/agent-workflows/projects/pi-openai-compatible-models/qa.md new file mode 100644 index 0000000000..12aa91d702 --- /dev/null +++ b/docs/design/agent-workflows/projects/pi-openai-compatible-models/qa.md @@ -0,0 +1,135 @@ +# QA plan + +## What was exercised (2026-07-15) + +The plan below is the intended coverage. The result of running it: + +- SDK agents suite: 669 passed. Covers the resolution, capability, and wire cases below. +- Runner suite: 1190 passed, plus the wire golden fixtures. Covers the pure builder, local + assets, Daytona assets, and session-reuse cases below. +- Frontend entity-ui suite: 208 passed. Covers the frontend cases below. +- Live acceptance ran on the EE dev stack on 2026-07-14 and 2026-07-15. The happy path passed + local and on Daytona: the custom endpoint reply was proven, the `models.json` contents were + verified, and `auth.json` isolation was verified. The 422 fail-loud paths passed. Standard Pi + and Claude regression runs passed. The UI click-through passed. + +The Playwright `model-hub.ts` strings are updated but a live Playwright run is still outstanding +(see [open-issues.md](open-issues.md)). + +A replay regression test is being added from the recorded happy-path `/run` pair (the capture +lives in the session scratchpad). It pins the SDK and runner behavior against the recorded runner +response so the custom-endpoint path replays without a live LLM. See the `agent-replay-test` skill +for the capture-and-redact flow. + +## Service and SDK unit coverage + +### Resolution + +- `kind=custom`, named connection, unknown bare model, API key, public HTTPS URL: + resolves provider `openai`, deployment `custom`, env only `OPENAI_API_KEY`, exact model, endpoint, + and credential mode `env`. +- Explicit provider `openai` produces the same result. +- Explicit provider `anthropic` remains Anthropic and does not receive `OPENAI_API_KEY`. +- Two named custom connections with the same model select by connection slug. +- Missing named connection fails loudly. +- Missing base URL on a named OpenAI-compatible connection fails loudly. +- Blocked HTTP, loopback, link-local, private, and metadata URLs fail loudly. +- API keys never appear in error strings, reprs, or non-secret wire fields. + +### Capabilities + +- Pi accepts resolved `(openai, custom)`. +- Pi rejects `(anthropic, custom)` in this phase. +- Pi continues accepting current direct providers and subscription modes. +- Claude continues accepting its current Anthropic direct/custom path. +- Claude rejects `(openai, custom)`. +- Unknown harnesses remain closed. +- Direct callers cannot bypass post-resolve pair validation. + +### Wire + +- Existing golden fixtures remain byte-identical unless their test input explicitly includes a + resolved custom connection. +- A custom connection continues to emit the existing `provider`, `connection`, `deployment`, + `endpoint`, `credentialMode`, and `secrets` fields only. +- No `api`, `protocol`, `modelsJson`, or Pi-specific wire field appears. + +## Runner unit coverage + +### Pure builder + +- Applicable Pi request produces one provider keyed by connection slug. +- Output uses `openai-completions` and `$OPENAI_API_KEY`. +- Arbitrary model IDs and characters valid in model IDs round-trip unchanged. +- Raw secret value does not appear in the plan or serialized JSON. +- Missing slug, endpoint, model, env credential, or wrong credential mode fails. +- Standard direct Pi, self-managed Pi, Claude, and unknown harness requests do not produce a Pi + custom config. + +### Local assets + +- Managed custom run creates a throwaway Pi directory even without skills or prompt overrides. +- `models.json` has mode `0600` and valid JSON. +- Existing unrelated shared files are not modified. +- Materialization precedes ACP session creation. +- Write failure aborts the run. +- Teardown deletes the throwaway directory. + +### Daytona assets + +- Upload targets `${DAYTONA_PI_DIR}/models.json`. +- Upload precedes ACP session creation. +- Existing file is replaced rather than merged. +- Upload failure aborts the run. +- The sandbox env contains only the resolved provider key plus existing required Pi variables. +- No runner subscription login is uploaded. + +### Session reuse + +- Changing connection slug, base URL, model, provider, or deployment changes config fingerprint. +- Rotating the API key changes credential epoch. +- Either mismatch prevents reuse of a parked incompatible environment. +- An unchanged request can still use the current keepalive path. + +## Frontend unit coverage + +- `PROVIDER_LABELS.custom` renders "OpenAI-compatible". +- Stored and submitted kind remains `custom`. +- Selecting a bare model from a custom connection writes provider `openai` and the exact slug. +- Selecting an explicitly family-qualified model preserves that family. +- Switching from an unrelated prior provider does not leak the prior provider into the new custom + selection. +- Pi shows compatible custom connections when its custom deployment is enabled. +- Claude does not show an OpenAI-compatible bare-model connection. +- Existing explicit Anthropic gateway behavior remains covered. + +## Integration coverage + +Use a deterministic fake OpenAI Chat Completions endpoint that records the request host, path, +model, and authorization presence without recording the secret. + +Assert: + +- the service resolves exactly one named vault connection; +- the runner calls the configured host, not `api.openai.com`; +- the request model equals the configured custom model; +- a second connection with the same model ID routes to its own host; +- an invalid model fails instead of using a harness default; +- rotating the key causes a cold configuration refresh. + +## Live acceptance matrix + +| Runner | Endpoint | Expected | +| --- | --- | --- | +| Local | Public HTTPS OpenAI-compatible | Pass | +| Daytona | Public HTTPS OpenAI-compatible | Pass | +| Local trusted self-host | HTTP/private with opt-in | Pass when reachable | +| Local default security | HTTP/private | Fail before runner | +| Daytona | Runner host `localhost` | Fail with reachability guidance | +| Claude | Existing Anthropic direct | Unchanged pass | +| Claude | Existing Anthropic gateway | Unchanged pass | + +For successful Pi cases, verify the provider/model reported by Pi or the runner is the custom +connection provider ID and exact selected model. Do not accept only a plausible text response as +proof because the harness could have fallen back to another model. + diff --git a/docs/design/agent-workflows/projects/pi-openai-compatible-models/research.md b/docs/design/agent-workflows/projects/pi-openai-compatible-models/research.md new file mode 100644 index 0000000000..194f51223c --- /dev/null +++ b/docs/design/agent-workflows/projects/pi-openai-compatible-models/research.md @@ -0,0 +1,173 @@ +# Research + +## Existing UI flow + +`web/packages/agenta-entity-ui/src/secretProvider/CustomProviderForm.tsx` offers standard provider +kinds plus `azure`, `bedrock`, `vertex_ai`, and `custom`. The shared label for `custom` is currently +"Custom Provider" in `web/packages/agenta-entities/src/secret/core/types.ts`. + +`web/packages/agenta-entities/src/secret/core/transforms.ts` writes the existing vault shape: + +```json +{ + "kind": "custom_provider", + "data": { + "kind": "custom", + "provider": { + "url": "https://example.test/v1", + "extras": {"api_key": "..."} + }, + "models": [{"slug": "model-id"}] + } +} +``` + +No schema addition is needed to retain the endpoint, key, connection name, or model IDs. + +The agent model picker combines the harness catalog with custom-provider records in +`connectionUtils.ts::vaultModelGroups`. Each custom model option carries the connection slug. +`useModelHarness.tsx` writes that slug into `llm.connection.slug`. The picker currently treats +`custom` as a deployment kind and hides it from Pi because Pi publishes only `direct`. + +One UI edge matters. For a bare custom model ID, the picker cannot infer a provider family from the +model string. It currently falls back to the previous provider. For the renamed +OpenAI-compatible option, it should resolve that missing family to `openai`; an explicit family +must continue to win. + +## Existing service and SDK flow + +`sdks/python/agenta/sdk/agents/handler.py` performs two independent checks: + +1. Provider and connection mode before vault resolution. +2. Deployment after vault resolution. + +This is correct for default/direct connections, but a named custom connection can use a portable +connection slug as its provider identifier. Rejecting that untrusted string before resolving the +trusted vault record prevents valid custom connections. + +`sdks/python/agenta/sdk/agents/platform/connections.py` already: + +- fetches the existing `GET /secrets/` list; +- selects a named custom connection by slug; +- extracts its URL, key, models, deployment, and extras; +- returns a `ResolvedConnection` with `endpoint.base_url` and `env`. + +For `data.kind=custom`, the resolver has no known provider env mapping. If the model reference also +lacks a known provider, the custom API key does not enter `ResolvedConnection.env`. The minimal +normalization is: + +- preserve an explicit provider family; +- otherwise map `kind=custom` to provider family `openai`; +- resolve its API key as `OPENAI_API_KEY`. + +The endpoint guard currently logs and drops a blocked custom URL. That is unsafe for a named +connection because the rest of the connection can continue without its routing configuration. The +named resolve should instead raise an actionable `ConnectionResolutionError`. + +## Existing wire contract + +The wire already carries: + +```text +model +provider +connection.mode +connection.slug +deployment +endpoint.baseUrl +credentialMode +secrets +``` + +`ResolvedConnection.to_wire()` excludes `env`; the secret environment travels in `secrets`. This +is the correct semantic split: + +- model and provider are selection data; +- connection slug is portable intent; +- deployment and endpoint are resolved routing/configuration; +- credential mode is authentication policy; +- secrets are credentials. + +Adding a Pi-specific `modelsJson`, `apiType`, or `openaiBaseUrl` field would leak harness mechanism +into a neutral boundary. It is unnecessary for the first protocol. + +## Existing runner flow + +`services/runner/src/engines/sandbox_agent.ts` builds a run plan, clears inherited provider +credentials for managed runs, applies exactly the resolved secrets, prepares harness assets, starts +the sandbox, creates the ACP session, and applies the requested model strictly. + +Claude has a harness-specific adapter function that maps the neutral endpoint to +`ANTHROPIC_BASE_URL`. Pi has no equivalent endpoint translation. + +`services/runner/src/engines/sandbox_agent/pi-assets.ts` already owns: + +- local Pi agent-directory preparation; +- local extension, prompt, and skill files; +- Daytona Pi asset upload; +- the distinction between shared subscription directories and isolated managed-run directories. + +The new model config belongs beside those Pi assets. It must be correctness-critical, unlike the +best-effort extension upload. If config creation fails, the run must stop before `createSession`. + +Daytona receives provider secrets explicitly through `daytonaEnvVars`. `KNOWN_PROVIDER_ENV_VARS` +in `daemon.ts` is the local clear/inherit inventory, not the mechanism that permits an environment +variable to enter Daytona. + +## Pi 0.80.6 behavior + +The installed Pi documentation at +`services/runner/node_modules/@earendil-works/pi-coding-agent/docs/models.md` defines +`models.json` as the supported path for Ollama, LM Studio, vLLM, proxies, and other custom models. +The minimal provider entry contains: + +```json +{ + "providers": { + "provider-id": { + "baseUrl": "https://example.test/v1", + "api": "openai-completions", + "apiKey": "$OPENAI_API_KEY", + "models": [{"id": "model-id"}] + } + } +} +``` + +Pi does not expose a general `OPENAI_BASE_URL` override for this path. An arbitrary model ID must +also be registered before pi-acp can advertise it as a settable model. + +The connection slug is the best Pi provider ID: + +- it is stable and portable across the model reference and vault lookup; +- it prevents one custom endpoint from overriding Pi's built-in `openai` provider; +- it disambiguates identical model IDs on two custom connections; +- it is already constrained by the UI's slug validation. + +## Warm session behavior + +The runner's `configFingerprint` already includes model, provider, connection, deployment, +endpoint, and credential mode. A change to any custom model configuration causes a cold +environment instead of continuing a mismatched session. + +The credential epoch hashes resolved secret values in memory. Rotating the key for the same slug +also causes a cold environment. No new warm-session key or persisted secret hash is required. + +The implementation must still overwrite the sandbox's `models.json` with the exact current plan +before creating a fresh Pi session. It must not merge with stale custom providers from a prior use +of a reusable sandbox. + +## Local endpoint behavior + +The existing endpoint guard rejects HTTP and private, loopback, link-local, and reserved addresses +by default. Trusted self-hosted installations can opt in with `AGENTA_INSECURE_EGRESS_ALLOWED`. + +Even with that opt-in, endpoint reachability depends on execution topology: + +- Local runner: the endpoint must be reachable from the runner container/process. +- Daytona: the endpoint must be reachable from the Daytona sandbox. `localhost` refers to the + sandbox itself. + +This feature should report routing failures clearly, but it should not invent tunnels or weaken +network policy. + diff --git a/docs/design/agent-workflows/projects/pi-openai-compatible-models/status.md b/docs/design/agent-workflows/projects/pi-openai-compatible-models/status.md new file mode 100644 index 0000000000..3855cf19b9 --- /dev/null +++ b/docs/design/agent-workflows/projects/pi-openai-compatible-models/status.md @@ -0,0 +1,97 @@ +# Status + +Last updated: 2026-07-15 + +## State + +**Implemented, end-to-end verified, and committed on two lanes. PRs against `main` pending.** + +Everything in this plan is built and passes its unit suites and a live end-to-end matrix on the +EE dev stack (2026-07-14 and 2026-07-15). The design held: no vault schema change and no `/run` +schema change. The service formalizes the existing `custom` kind as an OpenAI-compatible Chat +Completions endpoint, the runner writes Pi's `models.json`, and the UI relabels the kind. Two +lanes carry the change: `feat/pi-openai-compatible-models` (the SDK and runner backend) and +`feat/pi-openai-compatible-ui` (the frontend). Both PRs target `main`; the frontend PR merges +after or with the backend one, because the custom entries stay hidden until the backend +capability ships. See the PR bodies in the session scratchpad. + +## What shipped, by area + +### Service and SDK (`sdks/python/agenta/sdk/agents/`) + +- A provider-less named connection with `kind=custom` resolves to provider `openai`, deployment + `custom`, and its key in `OPENAI_API_KEY`. An explicit provider family is preserved, so existing + Anthropic gateway connections are unchanged. +- `harness_allows_pair` implements the resolved-pair table. Pi plus `openai` plus `custom` is newly + allowed. Arbitrary provider plus `custom` on Pi is still rejected. +- Named Agenta connections defer provider acceptance until after the vault record resolves. The + pre-resolve check validates only the connection mode. +- `pi_core` and `pi_agenta` publish deployments `["direct", "custom"]`. +- An unusable base URL raises `EndpointResolutionError`, a new subclass that returns HTTP 422. The + connection never resolves with `endpoint=None`, so the harness cannot fall back to a default + endpoint. +- The connection `{mode, slug}` rides the `/run` wire through `model_ref` threading in + `adapters/harnesses.py`, for the Pi and Agenta harnesses. The slug no longer leaks in as a + provider family; it stays connection identity. + +### Runner (`services/runner/src/engines/sandbox_agent/`) + +- A new `pi-model-config.ts` holds a pure builder that returns a `PiModelConfigPlan`. It applies + only for Pi plus provider `openai` plus deployment `custom` plus connection mode `agenta`. An + incomplete applicable request returns a typed error, not a silent no-op. +- The runner writes `models.json` in the shape + `{"providers": {"": {baseUrl, api: "openai-completions", apiKey: "$OPENAI_API_KEY", + models: [{id}]}}}`. Locally it writes the file `0600` and atomically into the isolated per-run Pi + directory. A plan run never seeds the operator's `auth.json`. On Daytona it uploads the file to + `DAYTONA_PI_DIR` before session creation. A no-plan run removes any stale file best-effort. +- Builder, write, and upload failures are terminal. The run never falls through to a default + provider. +- When a plan exists, the runner requests the fully qualified model id `/`. This fixes + a suffix collision that could silently route a request to `api.openai.com`. + +### UI (`web/`) + +- The `custom` provider-kind label reads "OpenAI-compatible endpoint". The stored value is + unchanged. +- Selecting a custom connection defaults the provider family to `openai` and preserves the + connection slug. +- Custom vault models appear only where the harness reaches them (`openai`, `custom`). +- The Playwright `model-hub.ts` strings are updated. They still need a live Playwright run (see + the open-issues log). + +## Deviations from the plan + +These are the places where the built code differs from the plan text, with the reason for each. + +- **`model_ref` threading in `adapters/harnesses.py`.** The plan assumed the connection `{mode, + slug}` already reached the wire. It did not for a named custom connection. Live end-to-end + testing surfaced the gap, and the fix threads the structured `model_ref` for Pi and Agenta. + Claude is left unchanged on purpose. +- **`EndpointResolutionError` and the 422 status.** Review added a dedicated typed error for an + unusable base URL, and set `EndpointResolutionError`, `UnsupportedProviderError`, and + `UnsupportedDeploymentError` to return HTTP 422. Without the explicit `status_code` these fell + through to 500 in the invoke remap. +- **The fully qualified `/` request.** Review found that the bare model id could + suffix-match the wrong provider and route to `api.openai.com`. The runner now requests the + qualified id whenever a plan exists. +- **Fail-loud endpoint applies to all custom deployments.** The endpoint resolution guard is not + Pi-only. Any `custom` deployment with an unusable base URL fails loudly, including a Claude + Anthropic gateway. + +## Test evidence + +- SDK agents suite: 669 passed. +- Runner suite: 1190 passed, plus the wire golden fixtures. +- Frontend entity-ui suite: 208 passed. +- Live end-to-end (EE dev stack, 2026-07-14 and 2026-07-15): the happy path passes local and on + Daytona, with the custom endpoint reply proven, the `models.json` contents verified, and + `auth.json` isolation verified. The 422 fail-loud paths pass. Standard Pi and Claude regression + runs pass. The UI click-through passes. + +## Deferred follow-ups + +Tracked with full context in [open-issues.md](open-issues.md). In brief: harden the Daytona stale +`models.json` cleanup, require `apiBaseUrl` at save time for `kind=custom`, add an engine-level +test for an incomplete applicable request, run the updated Playwright suite live, thread +`model_ref` for the Claude template if its named-connection wire identity is wanted, and add the +Anthropic Messages dialect. diff --git a/sdks/python/agenta/sdk/agents/__init__.py b/sdks/python/agenta/sdk/agents/__init__.py index ce28c8c8f3..9ab8ef6e2f 100644 --- a/sdks/python/agenta/sdk/agents/__init__.py +++ b/sdks/python/agenta/sdk/agents/__init__.py @@ -42,6 +42,7 @@ ConnectionResolutionError, ConnectionResolver, Endpoint, + EndpointResolutionError, EnvConnectionResolver, MissingProviderError, ModelRef, @@ -241,6 +242,7 @@ "StaticConnectionResolver", "AgentConnectionError", "ConnectionResolutionError", + "EndpointResolutionError", "ConnectionNotFoundError", "MissingProviderError", "AmbiguousConnectionError", diff --git a/sdks/python/agenta/sdk/agents/adapters/harnesses.py b/sdks/python/agenta/sdk/agents/adapters/harnesses.py index af959e9230..e04c60be75 100644 --- a/sdks/python/agenta/sdk/agents/adapters/harnesses.py +++ b/sdks/python/agenta/sdk/agents/adapters/harnesses.py @@ -67,6 +67,10 @@ def _to_harness_config(self, config: SessionConfig) -> PiAgentTemplate: return PiAgentTemplate( agents_md=config.agent.instructions, model=config.agent.model, + # Thread the structured ref so the author's connection {mode, slug} reaches the /run + # wire (via wire_model_ref). Without it a named custom (OpenAI-compatible) connection + # loses its slug and the runner cannot build its models.json plan. + model_ref=config.agent.model_ref, resolved_connection=config.resolved_connection, builtin_names=list(config.builtin_names), tool_specs=list(config.tool_specs), @@ -130,6 +134,9 @@ def _to_harness_config(self, config: SessionConfig) -> AgentaAgentTemplate: return AgentaAgentTemplate( agents_md=compose_instructions(config.agent.instructions), model=config.agent.model, + # See PiHarness: thread the structured ref so a named custom connection's {mode, slug} + # reaches the /run wire and the runner can build its models.json plan. + model_ref=config.agent.model_ref, resolved_connection=config.resolved_connection, builtin_names=force_tools(list(config.builtin_names)), tool_specs=list(config.tool_specs), diff --git a/sdks/python/agenta/sdk/agents/capabilities.py b/sdks/python/agenta/sdk/agents/capabilities.py index 1fef2d5a93..cc994abe99 100644 --- a/sdks/python/agenta/sdk/agents/capabilities.py +++ b/sdks/python/agenta/sdk/agents/capabilities.py @@ -17,9 +17,14 @@ subscription), which Pi reaches through its own OAuth login rather than a vault key — usable under ``self_managed`` (and the ``agenta`` default's ``runtime_provided`` fallback). Pi also reaches ~24 more providers that have no Agenta vault kind; those are out of scope unless a - ``custom_provider`` secret is made for them, so they are not enumerated here. Pi's cloud + ``custom_provider`` secret is made for them, so they are not enumerated here. Pi consumes the + ``direct`` deployment for all of them, plus the ``custom`` (OpenAI-compatible) deployment for + the ``openai`` family only — the runner's ``models.json`` builder speaks openai-completions. + The published ``custom`` capability lets the UI surface the connection; server-side pair + validation (:func:`harness_allows_pair`) is authoritative because the flat ``providers`` and + ``deployments`` lists cannot express the openai-only cross-product on their own. Pi's cloud deployments (azure/bedrock/vertex) are *declared* but Pi *consumption* of them stages with the - model-config sibling, so v1 fails loud: ``deployments`` is ``["direct"]`` for the live reach. + model-config sibling, so v1 fails loud for those. - **Claude** reaches anthropic only, direct, via a custom gateway, or through Anthropic on Bedrock/Vertex. The runner passes the selected model id through to Claude Code and lets the configured backend fail loudly if it rejects it. @@ -179,9 +184,11 @@ class HarnessConnectionCapabilities(BaseModel): """The connection-relevant capabilities of one harness (the ``/inspect`` ``meta`` shape). - ``providers``: the provider families the harness can reach (a literal list; never ``"*"``). - - ``deployments``: the deployment surfaces it can *consume* in v1 (``direct`` for both - harnesses today; Claude additionally consumes custom gateway, Bedrock, and Vertex - deployments. + - ``deployments``: the deployment surfaces it can *consume* in v1 (``direct`` and ``custom`` + for Pi, where ``custom`` is the OpenAI-compatible surface; Claude additionally consumes the + Anthropic custom gateway, Bedrock, and Vertex deployments). The list is per-axis only; + cross-product pairing (which provider a ``custom`` deployment accepts) lives in + :func:`harness_allows_pair`. - ``connection_modes``: which :class:`Connection` ``mode`` values it supports (``["agenta", "self_managed"]``). - ``model_selection``: how a model is named for the harness (``"provider/id"`` exact for Pi, @@ -208,16 +215,20 @@ class HarnessConnectionCapabilities(BaseModel): HARNESS_CONNECTION_CAPABILITIES: Dict[str, HarnessConnectionCapabilities] = { "pi_core": HarnessConnectionCapabilities( + # ``custom`` is published so the UI surfaces OpenAI-compatible connections; the + # openai-only pairing is enforced server-side by ``harness_allows_pair``, not by this + # flat list (which cannot express the cross-product on its own). providers=list(PI_VAULT_PROVIDERS) + list(PI_SUBSCRIPTION_PROVIDERS), - deployments=["direct"], + deployments=["direct", "custom"], connection_modes=list(_ALL_MODES), model_selection="provider/id", models=_pi_models(), model_catalog=_model_catalog("pi_core"), ), "pi_agenta": HarnessConnectionCapabilities( + # See ``pi_core``: ``custom`` is UI-surface only; ``harness_allows_pair`` is authoritative. providers=list(PI_VAULT_PROVIDERS) + list(PI_SUBSCRIPTION_PROVIDERS), - deployments=["direct"], + deployments=["direct", "custom"], connection_modes=list(_ALL_MODES), model_selection="provider/id", models=_pi_models(), @@ -301,10 +312,55 @@ def harness_allows_deployment(harness: str, deployment: str) -> bool: A harness with no entry is unknown, so it gets no capability (closed). The cloud surfaces are allowed only when the harness lists them as consumable. ``pi_core``/``pi_agenta`` list - only ``direct``; Claude also lists ``custom``/``bedrock``/``vertex_ai``. + ``direct`` and ``custom`` (the OpenAI-compatible surface); Claude also lists + ``bedrock``/``vertex_ai``. """ entry = HARNESS_CONNECTION_CAPABILITIES.get(harness) if entry is None: return False normalized = "vertex_ai" if deployment == "vertex" else deployment return normalized in entry.deployments + + +# The provider family each harness's ``custom`` deployment surface accepts. This is Decision 3's +# cross-product restriction: Pi's custom surface speaks openai-completions (the ``openai`` family +# only), and Claude's custom gateway is Anthropic only. The flat ``providers``/``deployments`` +# lists cannot express this pairing, so :func:`harness_allows_pair` consults this map. A harness +# absent here accepts no ``custom`` deployment. +HARNESS_CUSTOM_DEPLOYMENT_PROVIDERS: Dict[str, str] = { + "pi_core": "openai", + "pi_agenta": "openai", + "claude": "anthropic", +} + + +def harness_allows_pair(harness: str, provider: str, deployment: str) -> bool: + """Whether ``harness`` can consume the full (provider family, deployment) pair. + + The authoritative resolved-pair check (design Decision 3). ``harness_allows_provider`` and + ``harness_allows_deployment`` gate each axis independently; this gates their cross product, + which those flat lists cannot express on their own. A ``direct`` (or cloud) deployment is + allowed for any provider the harness already reaches, so the pair reduces to the two + independent checks there. A ``custom`` deployment is narrower: Pi consumes it only with the + ``openai`` family and Claude only with ``anthropic`` (per + :data:`HARNESS_CUSTOM_DEPLOYMENT_PROVIDERS`). An unknown harness is closed. + + The allowed triples: + + - ``pi_core``/``pi_agenta`` + ``openai`` + ``direct`` or ``custom`` -> allowed; + - ``pi_core``/``pi_agenta`` + any other family + ``custom`` -> rejected; + - ``claude`` + ``anthropic`` + ``direct``/``custom``/``bedrock``/``vertex_ai`` -> allowed; + - ``claude`` + ``openai`` + anything -> rejected (Claude reaches anthropic only); + - unknown harness -> rejected. + """ + if HARNESS_CONNECTION_CAPABILITIES.get(harness) is None: + return False + if not harness_allows_provider(harness, provider): + return False + if not harness_allows_deployment(harness, deployment): + return False + normalized = "vertex_ai" if deployment == "vertex" else deployment + if normalized == "custom": + allowed = HARNESS_CUSTOM_DEPLOYMENT_PROVIDERS.get(harness) + return allowed is not None and provider.lower() == allowed.lower() + return True diff --git a/sdks/python/agenta/sdk/agents/connections/__init__.py b/sdks/python/agenta/sdk/agents/connections/__init__.py index 7db5d9716b..7254684804 100644 --- a/sdks/python/agenta/sdk/agents/connections/__init__.py +++ b/sdks/python/agenta/sdk/agents/connections/__init__.py @@ -11,6 +11,7 @@ AmbiguousConnectionError, ConnectionNotFoundError, ConnectionResolutionError, + EndpointResolutionError, MissingProviderError, ProviderMismatchError, UnsupportedConnectionModeError, @@ -45,6 +46,7 @@ # Errors "AgentConnectionError", "ConnectionResolutionError", + "EndpointResolutionError", "ConnectionNotFoundError", "MissingProviderError", "AmbiguousConnectionError", diff --git a/sdks/python/agenta/sdk/agents/connections/errors.py b/sdks/python/agenta/sdk/agents/connections/errors.py index ac40c6b2de..eaac35bb4a 100644 --- a/sdks/python/agenta/sdk/agents/connections/errors.py +++ b/sdks/python/agenta/sdk/agents/connections/errors.py @@ -25,6 +25,18 @@ class ConnectionResolutionError(AgentConnectionError): """Raised when a connection cannot be resolved into a credential plan.""" +class EndpointResolutionError(ConnectionResolutionError): + """Raised when a chosen custom connection cannot resolve to a usable base URL. + + A named custom (OpenAI-compatible or gateway) connection routes through an explicit + endpoint. A missing or egress-blocked base URL is a config problem, not a server fault, so + it fails loud with a client-error status rather than degrading to a provider default. + """ + + # The invoke remap reads `status_code` off the exception; without it this fell through to 500. + status_code = 422 + + class ConnectionNotFoundError(ConnectionResolutionError): """Raised when a named connection (``mode == agenta`` + ``slug``) does not exist.""" @@ -91,6 +103,9 @@ def __init__(self, *, expected: str, actual: str) -> None: class UnsupportedProviderError(ConnectionResolutionError): """Raised when the requested provider cannot be reached by the selected harness.""" + # The invoke remap reads `status_code` off the exception; without it this fell through to 500. + status_code = 422 + def __init__(self, *, provider: str, harness: Optional[str] = None) -> None: suffix = f" by harness '{harness}'" if harness else "" super().__init__(f"provider '{provider}' is not supported{suffix}") diff --git a/sdks/python/agenta/sdk/agents/handler.py b/sdks/python/agenta/sdk/agents/handler.py index 5080aba8ff..40117ebfc1 100644 --- a/sdks/python/agenta/sdk/agents/handler.py +++ b/sdks/python/agenta/sdk/agents/handler.py @@ -18,6 +18,7 @@ from agenta.sdk.agents.capabilities import ( harness_allows_deployment, harness_allows_mode, + harness_allows_pair, harness_allows_provider, ) from agenta.sdk.agents.connections import ( @@ -114,10 +115,19 @@ def _check_harness_pre_resolve(model_ref: ModelRef, harness: Optional[str]) -> N """ if not harness: return + connection = model_ref.connection + is_named = connection.mode == "agenta" and bool( + connection.slug and connection.slug.strip() + ) provider = model_ref.provider - if provider and not harness_allows_provider(harness, provider): + # A named Agenta connection defers provider acceptance to the post-resolve pair check: the + # vault record is authoritative for the family (a provider-less OpenAI-compatible custom + # normalizes to ``openai``, and the model may even carry the connection slug as a placeholder + # provider that no harness lists). Rejecting that untrusted pre-resolve string would block a + # valid custom connection, so a named connection validates only its mode here. + if provider and not is_named and not harness_allows_provider(harness, provider): raise UnsupportedProviderError(provider=provider, harness=harness) - mode = model_ref.connection.mode + mode = connection.mode if not harness_allows_mode(harness, mode): raise UnsupportedConnectionModeError(mode=mode, harness=harness) @@ -125,17 +135,23 @@ def _check_harness_pre_resolve(model_ref: ModelRef, harness: Optional[str]) -> N def _check_harness_post_resolve( resolved: ResolvedConnection, harness: Optional[str] ) -> None: - """The POST-resolve half of the capability check: reject an unconsumable deployment. - - A slug-less ``agenta`` connection only reveals its deployment once the vault selects the - secret, so the deployment reject runs after the resolve returns. + """The POST-resolve half of the capability check: reject an unconsumable resolved pair. + + The resolved provider family and deployment surface are only known once the vault selects + the secret (a slug-less ``agenta`` connection reveals its deployment there; a named custom + connection normalizes its provider there), so the authoritative pair check runs after the + resolve returns. ``harness_allows_pair`` gates the cross-product (design Decision 3): Pi + consumes ``custom`` only with ``openai``, Claude only with ``anthropic``. Decompose the + reject into the most specific error. """ if not harness: return - if not harness_allows_deployment(harness, resolved.deployment): - raise UnsupportedDeploymentError( - deployment=resolved.deployment, harness=harness - ) + if not harness_allows_pair(harness, resolved.provider, resolved.deployment): + if not harness_allows_deployment(harness, resolved.deployment): + raise UnsupportedDeploymentError( + deployment=resolved.deployment, harness=harness + ) + raise UnsupportedProviderError(provider=resolved.provider, harness=harness) async def _default_resolve_session_connection( diff --git a/sdks/python/agenta/sdk/agents/platform/connections.py b/sdks/python/agenta/sdk/agents/platform/connections.py index 09c131010b..5cd046e0f0 100644 --- a/sdks/python/agenta/sdk/agents/platform/connections.py +++ b/sdks/python/agenta/sdk/agents/platform/connections.py @@ -28,6 +28,7 @@ AmbiguousConnectionError, ConnectionNotFoundError, ConnectionResolutionError, + EndpointResolutionError, Endpoint, MissingProviderError, ModelRef, @@ -230,6 +231,9 @@ class _ConnectionCandidate: api_key: Optional[str] = None env: Dict[str, str] = field(default_factory=dict) endpoint: Optional[Endpoint] = None + # True when a base URL was supplied but the egress policy rejected it (so the endpoint was + # dropped). Distinguishes "blocked" from "absent" for the fail-loud endpoint error below. + endpoint_blocked: bool = False model_slugs: Set[str] = field(default_factory=set) model_keys: Set[str] = field(default_factory=set) @@ -256,7 +260,44 @@ def selected_model_id(self, model: ModelRef) -> str: return model.model def resolved_provider(self, model: ModelRef) -> str: - return model.provider or self.provider or self.slug + if model.provider: + return model.provider + if self.provider: + return self.provider + # A provider-less named custom (OpenAI-compatible) connection normalizes to the + # ``openai`` provider family: its key then rides ``OPENAI_API_KEY`` (via ``resolved_env``) + # and the harness pair check reads ``openai``. An explicit provider (e.g. an Anthropic + # gateway) always wins above; the connection slug stays runtime identity + # (``request.connection.slug``) and must never become the semantic provider family. + if self.deployment == "custom": + return "openai" + return self.slug + + def requires_endpoint(self) -> bool: + """Whether this candidate must resolve to a usable base URL. + + A named custom (OpenAI-compatible or gateway) connection routes through an explicit + endpoint. Continuing with no base URL would let the harness fall back to a provider + default, silently violating the user's routing choice, so resolution fails loud instead + (design Decision 4). Known-family custom records resolve to ``direct`` and keep the + provider's own default endpoint, so they are exempt. + """ + return self.deployment == "custom" + + def endpoint_resolution_error(self) -> EndpointResolutionError: + """The typed, key-free failure for a custom connection missing a usable base URL.""" + if self.endpoint_blocked: + detail = ( + "its endpoint URL is blocked by the outbound egress policy (non-HTTPS or a " + "private, loopback, link-local, or reserved address); trusted self-hosted " + "deployments can allow it with AGENTA_INSECURE_EGRESS_ALLOWED" + ) + else: + detail = "it has no base URL configured" + # Names the connection slug for the operator; never includes the API key. + return EndpointResolutionError( + f"custom connection '{self.slug}' cannot be resolved: {detail}" + ) def resolved_env(self, provider: str) -> Dict[str, str]: env = dict(self.env) @@ -313,12 +354,17 @@ def _custom_provider_candidate( env = _normalized_extra_env(extras) region = env.get("AWS_REGION") or env.get("AWS_DEFAULT_REGION") raw_url = _stripped(settings.get("url")) + endpoint_blocked = False if raw_url: try: assert_endpoint_url_allowed(raw_url) except ValueError: - log.warning("agent: custom_provider url blocked by SSRF guard, dropping") + # Drop the blocked URL here (the candidate may not be the chosen one). A named + # custom connection that IS chosen fails loud in `_resolve_from_secrets` instead of + # continuing endpoint-less (design Decision 4); `endpoint_blocked` shapes that error. + log.warning("agent: custom_provider url blocked by egress policy, dropping") raw_url = None + endpoint_blocked = True endpoint = Endpoint( base_url=raw_url, api_version=_stripped(settings.get("version")), @@ -343,6 +389,7 @@ def _custom_provider_candidate( api_key=api_key, env=env, endpoint=endpoint, + endpoint_blocked=endpoint_blocked, model_slugs=_model_slugs(data), # Stored model keys remain namespaced by the vault provider kind. Runtime # deployment normalization must not change how a committed model selector matches. @@ -468,6 +515,14 @@ def _resolve_from_secrets( else _choose_default(candidates, model, harness) ) provider = chosen.resolved_provider(model) + # A chosen custom connection must carry a usable base URL. Failing here (rather than + # returning endpoint=None) keeps the harness from falling back to a provider default and + # silently ignoring the user's routing choice (design Decision 4). The error names the slug + # and never carries the API key. + if chosen.requires_endpoint() and not ( + chosen.endpoint and chosen.endpoint.base_url + ): + raise chosen.endpoint_resolution_error() env = chosen.resolved_env(provider) return ResolvedConnection( provider=provider, diff --git a/sdks/python/oss/tests/pytest/integration/agents/recordings/e2-pi-custom-openai-compatible.json b/sdks/python/oss/tests/pytest/integration/agents/recordings/e2-pi-custom-openai-compatible.json new file mode 100644 index 0000000000..8c8e857f29 --- /dev/null +++ b/sdks/python/oss/tests/pytest/integration/agents/recordings/e2-pi-custom-openai-compatible.json @@ -0,0 +1,60 @@ +{ + "cell": "e2-pi-custom-openai-compatible", + "provenance": { + "captured_from": "live EE dev stack POST /services/agent/v0/invoke (Pi, local sandbox)", + "endpoint": "https://openrouter.ai/api/v1 (OpenAI-compatible; free model)", + "captured_on": "2026-07-14", + "note": "The custom OpenAI-compatible connection green cell. A real vault custom_provider connection (data.kind=custom) pointed at OpenRouter was resolved by the service and run through Pi; the assistant returned the forced token. REDACTIONS: vault_secrets[].data.provider.key -> 'sk-test' placeholder (never a real key); the vault record id and lifecycle timestamps are dropped (unused by resolution); result.sessionId -> 'sess-replay'.", + "reply": "REPLAY-COMPAT-OK", + "stop_reason": "end_turn" + }, + "request": { + "harness": "pi_core", + "sandbox": "local", + "agents_md": "Reply with exactly the requested token and nothing else.", + "model": "openai/gpt-oss-20b:free", + "connection": {"mode": "agenta", "slug": "replay-compat"}, + "messages": [ + {"role": "user", "content": "Reply with exactly: REPLAY-COMPAT-OK"} + ], + "vault_secrets": [ + { + "kind": "custom_provider", + "data": { + "kind": "custom", + "provider": { + "url": "https://openrouter.ai/api/v1", + "key": "sk-test" + }, + "models": [ + {"slug": "openai/gpt-oss-20b:free"} + ], + "provider_slug": "replay-compat", + "model_keys": [ + "replay-compat/custom/openai/gpt-oss-20b:free" + ] + }, + "header": { + "name": "replay-compat", + "description": "replay regression custom OpenAI-compatible connection" + } + } + ] + }, + "result": { + "ok": true, + "output": "REPLAY-COMPAT-OK", + "messages": [ + {"role": "assistant", "content": "REPLAY-COMPAT-OK"} + ], + "events": [ + {"type": "message", "text": "REPLAY-COMPAT-OK"}, + {"type": "done", "stopReason": "end_turn"} + ], + "usage": {"input": 1, "output": 1, "total": 2, "cost": 0.0}, + "stopReason": "end_turn", + "capabilities": {"textMessages": true, "mcpTools": false}, + "sessionId": "sess-replay", + "model": "openai/gpt-oss-20b:free" + } +} diff --git a/sdks/python/oss/tests/pytest/integration/agents/test_custom_connection_replay.py b/sdks/python/oss/tests/pytest/integration/agents/test_custom_connection_replay.py new file mode 100644 index 0000000000..a7e659fb61 --- /dev/null +++ b/sdks/python/oss/tests/pytest/integration/agents/test_custom_connection_replay.py @@ -0,0 +1,198 @@ +"""Replay the custom OpenAI-compatible connection cell through the real SDK + service path. + +The agent-workflows QA matrix cell ``e2-pi-custom-openai-compatible`` is green: a real vault +``custom_provider`` connection (``data.kind == custom``) pointed at an OpenAI-compatible gateway +(OpenRouter) was resolved by the service and run through Pi, and the assistant returned the +forced token. This pins that cell so it stays green with no live LLM and no live vault. + +Two halves of the same recorded run are asserted, both against the SAME real code path the +service uses: + +1. The REQUEST-shaping half. The service resolves the author's ``{mode: agenta, slug}`` + connection against the vault (here the recorded, redacted ``custom_provider`` secret, fed + through the real ``_StaticSecretsResolver`` / ``_default_resolve_session_connection`` — the + offline stand-in for the live ``GET /secrets/`` fetch), threads the resulting + ``ResolvedConnection`` onto the ``SessionConfig``, and ``request_to_wire`` spreads it onto the + ``/run`` payload. The test drives that through the real subprocess transport and asserts the + wire carries ``deployment=custom``, ``provider=openai``, the author's ``connection``, the + endpoint ``baseUrl``, the exact resolved ``model``, and the provider key present in + ``secrets`` by NAME (``OPENAI_API_KEY``), value redacted. + +2. The RESULT-parsing half. The recorded runner response replays back through + ``result_from_wire`` / the transport, proving the SDK folds a real recorded custom-connection + run into the expected ``AgentResult`` shape. + +Provenance and redactions live in the fixture's own ``provenance`` block. See the +``agent-replay-test`` skill for the capture/redact/replay conventions. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import pytest + +from agenta.sdk.agents import ( + AgentTemplate, + EndpointResolutionError, + Environment, + Message, + ModelRef, + PiHarness, + RuntimeAuthContext, + SessionConfig, +) +from agenta.sdk.agents.handler import ( + _agent_model_ref, + _default_resolve_session_connection, +) +from agenta.sdk.agents.platform.connections import _StaticSecretsResolver + +from ._fake_runner_backend import FakeRunnerBackend + +pytestmark = [pytest.mark.integration, pytest.mark.cost_free] + +REC = Path(__file__).parent / "recordings" + + +def _load(name: str) -> dict: + """Load a recorded ``/run`` cell verbatim (never hand-copied into the test).""" + return json.loads((REC / name).read_text(encoding="utf-8")) + + +def _template_from_recording(rec: dict) -> AgentTemplate: + """Build today's :class:`AgentTemplate` from the recorded request, exactly as the service + handler does (``AgentTemplate.from_params`` over ``parameters.agent``). The nested + ``llm.connection`` is what makes ``model_ref`` structured, so the author's ``{mode, slug}`` + survives to the wire.""" + req = rec["request"] + params = { + "agent": { + "instructions": {"agents_md": req["agents_md"]}, + "llm": {"model": req["model"], "connection": req["connection"]}, + "harness": {"kind": req["harness"]}, + "sandbox": {"kind": req["sandbox"]}, + } + } + return AgentTemplate.from_params(params) + + +async def _resolve_connection(template: AgentTemplate, rec: dict): + """Resolve the connection over the recorded (redacted) vault secret, through the same + ``_default_resolve_session_connection`` the handler calls -- only the live ``GET /secrets/`` + fetch is swapped for a static list, which is the whole point of an offline replay.""" + model_ref = _agent_model_ref(template) + assert model_ref is not None + ctx = RuntimeAuthContext(harness=template.harness, backend=template.sandbox) + static = _StaticSecretsResolver(rec["request"]["vault_secrets"]) + return await _default_resolve_session_connection( + model_ref, ctx, resolve_connection=static.resolve + ) + + +def _replay_backend( + tmp_path: Path, result: dict, capture_path: Path +) -> FakeRunnerBackend: + """A runner that records the ``/run`` request it received (for the request-shaping asserts) + and echoes the recorded runner ``result`` verbatim (for the result-parsing asserts). The + recorded result rides a sidecar JSON file so control characters in a real reply stay valid.""" + result_path = tmp_path / "recorded_result.json" + result_path.write_text(json.dumps(result), encoding="utf-8") + runner = tmp_path / "custom_connection_replay_runner.py" + runner.write_text( + "import sys, json\n" + "req = json.load(sys.stdin)\n" + f"open({json.dumps(str(capture_path))}, 'w', encoding='utf-8')" + ".write(json.dumps(req))\n" + f"sys.stdout.write(open({json.dumps(str(result_path))}, encoding='utf-8').read())\n", + encoding="utf-8", + ) + return FakeRunnerBackend(command=[sys.executable, str(runner)], cwd=str(tmp_path)) + + +async def test_custom_openai_compatible_connection_replays(tmp_path): + rec = _load("e2-pi-custom-openai-compatible.json") + template = _template_from_recording(rec) + + # The service's connection resolution over the recorded, redacted vault secret. Guard the + # resolved plan first so a stale fixture is caught before the wire assertions ride on it. + resolved = await _resolve_connection(template, rec) + assert resolved.provider == "openai" + assert resolved.deployment == "custom" + assert resolved.model == "openai/gpt-oss-20b:free" + assert resolved.endpoint is not None + assert resolved.endpoint.base_url == "https://openrouter.ai/api/v1" + assert resolved.credential_mode == "env" + assert set(resolved.env) == {"OPENAI_API_KEY"} + + session_config = SessionConfig( + agent=template, + secrets=resolved.env, # Slice 1 ships the credential through `secrets` on the wire + resolved_connection=resolved, + ) + messages = [ + Message(role=m["role"], content=m["content"]) + for m in rec["request"]["messages"] + ] + capture_path = tmp_path / "run_request.json" + harness = PiHarness( + Environment(_replay_backend(tmp_path, rec["result"], capture_path)) + ) + + result = await harness.prompt(session_config, messages) + + # 1) REQUEST-shaping half: the /run wire carries the resolved custom-connection descriptor. + sent = json.loads(capture_path.read_text(encoding="utf-8")) + assert sent["deployment"] == "custom" + assert sent["provider"] == "openai" + assert sent["connection"] == {"mode": "agenta", "slug": "replay-compat"} + assert sent["endpoint"] == {"baseUrl": "https://openrouter.ai/api/v1"} + assert sent["model"] == "openai/gpt-oss-20b:free" + assert sent["credentialMode"] == "env" + # The provider key rides `secrets` by NAME; the value is the redacted placeholder, and no + # real key ever reaches the wire (the fixture carries only `sk-test`). + assert "OPENAI_API_KEY" in sent["secrets"] + assert sent["secrets"]["OPENAI_API_KEY"] == "sk-test" + + # 2) RESULT-parsing half: the recorded runner response folds back cleanly, no live LLM. + assert result.output == rec["result"]["output"] == "REPLAY-COMPAT-OK" + assert result.stop_reason == "end_turn" + assert result.model == "openai/gpt-oss-20b:free" + assert result.capabilities is not None and result.capabilities.mcp_tools is False + assert result.session_id == "sess-replay" + + +async def test_url_less_custom_connection_fails_loud_with_422(): + """A named custom (OpenAI-compatible) connection with NO base URL cannot route the request. + + The companion to the recorded green cell: the same resolver, fed a custom connection whose + vault record has no ``url``, must fail loud with a 422 ``EndpointResolutionError`` (not degrade + to a provider default) -- the design's Decision 4. The error names the slug and never carries + the key. This pins the ``no base URL`` branch and the ``status_code`` the ``/invoke`` remap + reads (the existing http tests cover only egress-BLOCKED urls, and none assert the 422).""" + secret = { + "kind": "custom_provider", + "header": {"name": "no-url"}, + "data": { + "kind": "custom", + "provider_slug": "no-url", + "provider": {"key": "sk-test"}, # placeholder; never a real key + "models": [{"slug": "some-model"}], + "model_keys": ["no-url/custom/some-model"], + }, + } + model = ModelRef( + model="some-model", connection={"mode": "agenta", "slug": "no-url"} + ) + + with pytest.raises(EndpointResolutionError) as exc: + await _StaticSecretsResolver([secret]).resolve( + model=model, + context=RuntimeAuthContext(harness="pi_core", backend="local"), + ) + + assert exc.value.status_code == 422 + assert "no-url" in str(exc.value) + assert "sk-test" not in str(exc.value) diff --git a/sdks/python/oss/tests/pytest/unit/agents/connections/test_capabilities.py b/sdks/python/oss/tests/pytest/unit/agents/connections/test_capabilities.py index aa8130ae8f..6ff41db882 100644 --- a/sdks/python/oss/tests/pytest/unit/agents/connections/test_capabilities.py +++ b/sdks/python/oss/tests/pytest/unit/agents/connections/test_capabilities.py @@ -16,6 +16,7 @@ PI_VAULT_PROVIDERS, harness_allows_deployment, harness_allows_mode, + harness_allows_pair, harness_allows_provider, harness_capabilities_document, ) @@ -70,13 +71,42 @@ def test_two_modes_supported_on_all_known_harnesses(): assert harness_allows_mode("pi_core", "bogus") is False -def test_pi_only_consumes_direct_deployment_in_v1(): +def test_pi_consumes_direct_and_custom_deployment_in_v1(): + # Pi now publishes `custom` (the OpenAI-compatible surface) alongside `direct` so the UI can + # surface those connections. The cloud surfaces remain unconsumed in v1. The openai-only + # pairing on `custom` is enforced by `harness_allows_pair`, not this per-axis list. for harness in ("pi_core", "pi_agenta"): assert harness_allows_deployment(harness, "direct") is True - for deployment in ("custom", "bedrock", "vertex_ai", "azure"): + assert harness_allows_deployment(harness, "custom") is True + for deployment in ("bedrock", "vertex_ai", "azure"): assert harness_allows_deployment(harness, deployment) is False +def test_resolved_pair_validation_matches_decision_3_table(): + # Every row of design Decision 3's allowed-pairs table. + for harness in ("pi_core", "pi_agenta"): + # Pi + openai + direct/custom -> allowed. + assert harness_allows_pair(harness, "openai", "direct") is True + assert harness_allows_pair(harness, "openai", "custom") is True + # Pi + an arbitrary family + custom -> rejected (custom is openai-only for Pi), even for + # a family Pi otherwise reaches directly (anthropic). + assert harness_allows_pair(harness, "anthropic", "custom") is False + assert harness_allows_pair(harness, "anything-custom", "custom") is False + # A family Pi cannot reach at all is rejected on any deployment. + assert harness_allows_pair(harness, "anything-custom", "direct") is False + + # Claude + anthropic + direct/custom -> allowed; the cloud surfaces it consumes too. + for deployment in ("direct", "custom", "bedrock", "vertex_ai", "vertex"): + assert harness_allows_pair("claude", "anthropic", deployment) is True + # Claude + openai + custom -> rejected (Claude reaches anthropic only). + assert harness_allows_pair("claude", "openai", "custom") is False + assert harness_allows_pair("claude", "openai", "direct") is False + + # Unknown harness -> rejected for any pair (closed, not permissive). + assert harness_allows_pair("some-future-harness", "openai", "direct") is False + assert harness_allows_pair("some-future-harness", "openai", "custom") is False + + def test_claude_consumes_custom_gateway_bedrock_and_vertex(): for deployment in ("direct", "custom", "bedrock", "vertex_ai", "vertex"): assert harness_allows_deployment("claude", deployment) is True @@ -92,7 +122,7 @@ def test_capabilities_document_shape(): PI_SUBSCRIPTION_PROVIDERS ) assert doc["pi_core"]["connection_modes"] == ["agenta", "self_managed"] - assert doc["pi_core"]["deployments"] == ["direct"] + assert doc["pi_core"]["deployments"] == ["direct", "custom"] assert doc["claude"]["deployments"] == [ "direct", "custom", diff --git a/sdks/python/oss/tests/pytest/unit/agents/platform/test_connections_http.py b/sdks/python/oss/tests/pytest/unit/agents/platform/test_connections_http.py index bb574075ca..49cbba1709 100644 --- a/sdks/python/oss/tests/pytest/unit/agents/platform/test_connections_http.py +++ b/sdks/python/oss/tests/pytest/unit/agents/platform/test_connections_http.py @@ -364,7 +364,12 @@ async def test_custom_gateway_api_key_from_extras_and_endpoint(fake_http, connec assert resolved.endpoint.base_url == "https://93.184.216.34/v1" -async def test_custom_provider_private_url_is_dropped_not_pinned(fake_http, connection): +async def test_custom_provider_private_url_fails_loud_not_dropped( + fake_http, connection +): + # Decision 4: a chosen named custom connection whose endpoint is blocked must fail loud, not + # silently continue endpoint-less onto a provider default. The error names the slug, points + # at AGENTA_INSECURE_EGRESS_ALLOWED, and never carries the API key. fake_http( connections, payload=[ @@ -377,14 +382,18 @@ async def test_custom_provider_private_url_is_dropped_not_pinned(fake_http, conn ) ], ) - resolved = await VaultConnectionResolver(connection).resolve( - model=_model("internal-gw", provider="anthropic", model="gpt-5.5"), - context=RuntimeAuthContext(harness="claude"), - ) - assert resolved.endpoint is None + with pytest.raises(ConnectionResolutionError) as exc: + await VaultConnectionResolver(connection).resolve( + model=_model("internal-gw", provider="anthropic", model="gpt-5.5"), + context=RuntimeAuthContext(harness="claude"), + ) + message = str(exc.value) + assert "internal-gw" in message + assert "AGENTA_INSECURE_EGRESS_ALLOWED" in message + assert "sk-gw" not in message -async def test_custom_provider_loopback_url_is_dropped_not_pinned( +async def test_custom_provider_loopback_url_fails_loud_not_dropped( fake_http, connection ): fake_http( @@ -399,11 +408,12 @@ async def test_custom_provider_loopback_url_is_dropped_not_pinned( ) ], ) - resolved = await VaultConnectionResolver(connection).resolve( - model=_model("loopback-gw", provider="anthropic", model="gpt-5.5"), - context=RuntimeAuthContext(harness="claude"), - ) - assert resolved.endpoint is None + with pytest.raises(ConnectionResolutionError) as exc: + await VaultConnectionResolver(connection).resolve( + model=_model("loopback-gw", provider="anthropic", model="gpt-5.5"), + context=RuntimeAuthContext(harness="claude"), + ) + assert "loopback-gw" in str(exc.value) async def test_custom_provider_ssrf_guard_defaults_secure(fake_http, connection): @@ -425,12 +435,75 @@ async def test_custom_provider_ssrf_guard_defaults_secure(fake_http, connection) ) ], ) + # Blocked by default (secure) — and a chosen custom connection fails loud rather than + # continuing endpoint-less. + with pytest.raises(ConnectionResolutionError) as exc: + await VaultConnectionResolver(connection).resolve( + model=_model("private-gw", provider="anthropic", model="gpt-5.5"), + context=RuntimeAuthContext(harness="claude"), + ) + assert "AGENTA_INSECURE_EGRESS_ALLOWED" in str(exc.value) + + +async def test_openai_compatible_custom_normalizes_to_openai(fake_http, connection): + # Decision 2: a provider-less named custom connection resolves to the openai provider family, + # keeps deployment=custom and the exact model id + endpoint, and routes its key through + # OPENAI_API_KEY only (the family's canonical env var). The connection slug stays identity, + # never the provider family. + endpoint = "https://93.184.216.34/v1" + model_id = "qwen2.5-coder:7b" + fake_http( + connections, + payload=[ + _custom_provider( + "my-ollama", + "custom", + key="sk-oai-compatible", + url=endpoint, + models=[model_id], + ) + ], + ) resolved = await VaultConnectionResolver(connection).resolve( - model=_model("private-gw", provider="anthropic", model="gpt-5.5"), - context=RuntimeAuthContext(harness="claude"), + model=ModelRef( + model=model_id, connection={"mode": "agenta", "slug": "my-ollama"} + ), + context=_context(), + ) + assert resolved.provider == "openai" + assert resolved.deployment == "custom" + assert resolved.model == model_id + assert resolved.endpoint.base_url == endpoint + assert resolved.credential_mode == "env" + assert resolved.env == {"OPENAI_API_KEY": "sk-oai-compatible"} + + +async def test_openai_compatible_custom_missing_url_fails_loud(fake_http, connection): + # Decision 4: an explicit named custom connection with no base URL fails loud (naming the + # slug), rather than resolving with endpoint=None and letting the harness pick a default. + fake_http( + connections, + payload=[ + _custom_provider( + "my-ollama", + "custom", + key="sk-oai-compatible", + url=None, + models=["qwen2.5-coder:7b"], + ) + ], ) - # Blocked with no env var required — secure by default. - assert resolved.endpoint is None + with pytest.raises(ConnectionResolutionError) as exc: + await VaultConnectionResolver(connection).resolve( + model=ModelRef( + model="qwen2.5-coder:7b", + connection={"mode": "agenta", "slug": "my-ollama"}, + ), + context=_context(), + ) + message = str(exc.value) + assert "my-ollama" in message + assert "sk-oai-compatible" not in message async def test_full_custom_model_key_selects_and_strips_to_backend_model( diff --git a/sdks/python/oss/tests/pytest/unit/agents/test_agent_composition_seam.py b/sdks/python/oss/tests/pytest/unit/agents/test_agent_composition_seam.py index ec3b92b0a0..57181dfaf6 100644 --- a/sdks/python/oss/tests/pytest/unit/agents/test_agent_composition_seam.py +++ b/sdks/python/oss/tests/pytest/unit/agents/test_agent_composition_seam.py @@ -225,6 +225,82 @@ async def _resolve(*, model, context): ) +async def test_named_connection_defers_provider_reject_to_post_resolve(): + """A named Agenta connection must NOT be rejected pre-resolve on an unknown-family provider + placeholder (the UI can store the connection slug as the provider). The vault normalizes it, + and Pi + openai + custom is an allowed pair, so the run proceeds.""" + backend = _FakeBackend(output="ok") + + async def _resolve(*, model, context): + # The vault record normalized the provider-less custom to openai + custom deployment. + return ResolvedConnection( + provider="openai", + model="qwen2.5-coder:7b", + deployment="custom", + credential_mode="env", + env={"OPENAI_API_KEY": "sk-oai"}, + endpoint={"base_url": "https://93.184.216.34/v1"}, + ) + + comp = AgentComposition( + select_backend=lambda template: backend, + resolve_connection=_resolve, + ) + handler = make_agent_handler(comp) + + result = await handler( + request=_request(), + messages=[{"role": "user", "content": "hi"}], + # provider is the connection slug (no harness lists it); pre-resolve must defer it. + parameters=_params( + "pi_core", + model={ + "provider": "my-ollama", + "model": "qwen2.5-coder:7b", + "connection": {"mode": "agenta", "slug": "my-ollama"}, + }, + ), + ) + assert result == {"messages": [{"role": "assistant", "content": "ok"}]} + + +async def test_pi_custom_with_non_openai_family_rejected_post_resolve(): + """Pi + a non-openai family + custom is not an allowed pair (Pi's custom surface is + openai-only), so it must fail loud post-resolve even though anthropic is otherwise reachable + by Pi directly.""" + backend = _FakeBackend() + + async def _resolve(*, model, context): + return ResolvedConnection( + provider="anthropic", + model="some-model", + deployment="custom", + credential_mode="env", + env={"ANTHROPIC_API_KEY": "sk-ant"}, + endpoint={"base_url": "https://93.184.216.34/v1"}, + ) + + comp = AgentComposition( + select_backend=lambda template: backend, + resolve_connection=_resolve, + ) + handler = make_agent_handler(comp) + + with pytest.raises(UnsupportedProviderError): + await handler( + request=_request(), + messages=[{"role": "user", "content": "hi"}], + parameters=_params( + "pi_core", + model={ + "provider": "my-gateway", + "model": "some-model", + "connection": {"mode": "agenta", "slug": "my-gateway"}, + }, + ), + ) + + async def test_default_composition_degrades_default_connection_failure(): """An unconfigured default-mode connection degrades to runtime_provided, no raise -- even with NO composition override (the SDK default now has the degradation policy diff --git a/sdks/python/oss/tests/pytest/unit/agents/test_harness_adapters.py b/sdks/python/oss/tests/pytest/unit/agents/test_harness_adapters.py index bafad4557c..4a29d4ad2a 100644 --- a/sdks/python/oss/tests/pytest/unit/agents/test_harness_adapters.py +++ b/sdks/python/oss/tests/pytest/unit/agents/test_harness_adapters.py @@ -67,6 +67,54 @@ def test_pi_keeps_builtins_and_native_tools(make_env): assert result.model == "m" +def test_pi_threads_model_ref_so_connection_reaches_wire(make_env): + """Regression: a named custom connection's ``{mode, slug}`` must reach the ``/run`` wire. + + ``_to_harness_config`` builds the wire-producing harness template. If it drops + ``model_ref`` (passing only the plain ``model`` string), ``wire_model_ref`` returns ``{}`` + and the runner never sees the connection slug, so it cannot build its ``models.json`` plan + for an OpenAI-compatible custom connection (the run then falls back to a default provider). + """ + harness = PiHarness(make_env(supported=[HarnessType.PI])) + agent = AgentTemplate( + instructions="hi", + model={ + "model": "gpt-4o-mini", + "connection": {"mode": "agenta", "slug": "my-compat"}, + }, + ) + + result = harness._to_harness_config(_session_config(agent=agent)) + + assert result.model_ref is not None + assert result.model_ref.connection.slug == "my-compat" + # The connection intent reaches the /run wire so the runner can register the provider. + assert result.wire_model_ref().get("connection") == { + "mode": "agenta", + "slug": "my-compat", + } + + +def test_agenta_threads_model_ref_so_connection_reaches_wire(make_env): + """Same guarantee as Pi for the ``pi_agenta`` harness (it also runs Pi).""" + harness = AgentaHarness(make_env(supported=[HarnessType.AGENTA])) + agent = AgentTemplate( + instructions="hi", + model={ + "model": "gpt-4o-mini", + "connection": {"mode": "agenta", "slug": "my-compat"}, + }, + ) + + result = harness._to_harness_config(_session_config(agent=agent)) + + assert result.model_ref is not None + assert result.wire_model_ref().get("connection") == { + "mode": "agenta", + "slug": "my-compat", + } + + def test_pi_reads_its_harness_extras_slice(make_env): harness = PiHarness(make_env(supported=[HarnessType.PI])) agent = AgentTemplate( diff --git a/sdks/python/oss/tests/pytest/unit/agents/test_invoke_failure_status.py b/sdks/python/oss/tests/pytest/unit/agents/test_invoke_failure_status.py index 0ad76dcf9e..7c42229e69 100644 --- a/sdks/python/oss/tests/pytest/unit/agents/test_invoke_failure_status.py +++ b/sdks/python/oss/tests/pytest/unit/agents/test_invoke_failure_status.py @@ -9,7 +9,11 @@ import json -from agenta.sdk.agents.connections import UnsupportedDeploymentError +from agenta.sdk.agents.connections import ( + EndpointResolutionError, + UnsupportedDeploymentError, + UnsupportedProviderError, +) from agenta.sdk.decorators.routing import handle_invoke_failure @@ -26,6 +30,23 @@ async def test_unsupported_deployment_remaps_to_422(): assert "bedrock" in json.dumps(body) +async def test_unsupported_provider_remaps_to_422(): + status, body = await _status_and_body( + UnsupportedProviderError(provider="cohere", harness="claude") + ) + assert status == 422 + assert "cohere" in json.dumps(body) + + +async def test_endpoint_resolution_error_remaps_to_422(): + # A chosen custom connection with no usable base URL is a config problem, not a server fault. + status, body = await _status_and_body( + EndpointResolutionError("custom connection 'my-ollama' cannot be resolved") + ) + assert status == 422 + assert "my-ollama" in json.dumps(body) + + async def test_unremarkable_exception_still_remaps_to_500(): # The fallback must stay 500 — 422 is opt-in via `status_code`, not the new default. status, _ = await _status_and_body(RuntimeError("boom")) diff --git a/services/oss/tests/pytest/unit/agent/test_invoke_handler.py b/services/oss/tests/pytest/unit/agent/test_invoke_handler.py index 3fea258515..00d1becaa8 100644 --- a/services/oss/tests/pytest/unit/agent/test_invoke_handler.py +++ b/services/oss/tests/pytest/unit/agent/test_invoke_handler.py @@ -62,12 +62,15 @@ async def _no_mcp(mcp_servers, **_kw): return [] async def _no_connection(*, model, context): - # a no-credential plan so existing response-body/lifecycle/cross-harness tests run clean - return ResolvedConnection( - provider="openai", - model="m", - credential_mode="runtime_provided", - env={}, + # No connection is configured for the default model, so the resolve fails and the handler + # degrades to a no-credential ``runtime_provided`` plan (harness login / self-managed). + # This is the realistic "no connection" simulation: it exercises the degraded path that + # every harness tolerates, so the response-body / lifecycle / cross-harness tests run + # clean regardless of harness. A stubbed *successful* resolve would instead pin a single + # provider and be rejected by the post-resolve capability gate on a mismatched harness + # (e.g. ``openai`` on ``claude``), which is not what these tests are exercising. + raise ConnectionResolutionError( + "no connection configured for the default model" ) monkeypatch.setattr(app, "resolve_tools", _tools) diff --git a/services/runner/src/engines/sandbox_agent.ts b/services/runner/src/engines/sandbox_agent.ts index e7d63d2363..ef959193a5 100644 --- a/services/runner/src/engines/sandbox_agent.ts +++ b/services/runner/src/engines/sandbox_agent.ts @@ -92,12 +92,17 @@ import { configurePiSessionWorkspace, configurePiSkillSnapshot, PI_PERMISSION_EXTENSION_UNAVAILABLE_MESSAGE, + PI_MODEL_CONFIG_WRITE_FAILED_MESSAGE, prepareLocalPiAssets, resolvePiSkillSnapshot, uploadSystemPromptToSandbox, writeSystemPromptLocal, writeOtlpAuthFile, } from "./sandbox_agent/pi-assets.ts"; +import { + buildPiModelConfigPlan, + type PiModelConfigPlan, +} from "./sandbox_agent/pi-model-config.ts"; import { uploadToolMcpAssets, type ToolMcpAssets, @@ -298,8 +303,7 @@ function shouldSuppressPausedToolCallUpdate( pause: PendingApprovalPauseController, ): boolean { const frame = update as - | { sessionUpdate?: unknown; toolCallId?: unknown } - | undefined; + { sessionUpdate?: unknown; toolCallId?: unknown } | undefined; const kind = frame?.sessionUpdate; if (kind !== "tool_call" && kind !== "tool_call_update") return false; const toolCallId = @@ -646,8 +650,7 @@ export interface SessionEnvironment { } export type AcquireEnvironmentResult = - | { ok: true; env: SessionEnvironment } - | { ok: false; error: string }; + { ok: true; env: SessionEnvironment } | { ok: false; error: string }; /** * Sign the session's durable mount up front so keep-alive can build a pool key (the mount's @@ -857,10 +860,44 @@ export async function acquireEnvironment( ); } } + // Translate a managed OpenAI-compatible custom connection into Pi's native models.json plan + // (design Decision 5). Non-applicable requests yield no plan (current behavior); an applicable + // but incomplete request throws — captured here and re-thrown inside the try below so the + // engine's own catch turns it into `{ ok: false, error }` and a visible error frame (fail loud, + // never a silent fall-back to a default provider). Only the env var NAME enters the plan. + let piModelConfig: PiModelConfigPlan | undefined; + let piModelConfigError: Error | undefined; + if (plan.isPi) { + try { + piModelConfig = buildPiModelConfigPlan(request, plan.secrets); + } catch (err) { + piModelConfigError = err as Error; + } + } + if (piModelConfig) { + logger( + `pi model-config plan provider=${piModelConfig.providerId} api=${piModelConfig.api} ` + + `model=${piModelConfig.models.map((m) => m.id).join(",")}`, + ); + } + // undefined is fine: the local provider runs its own resolution and errors clearly. const binaryPath = (deps.resolveDaemonBinary ?? resolveDaemonBinary)(); - const localPiAssets = prepareLocalPiAssets({ plan, env, log: logger }); + const localPiAssets = prepareLocalPiAssets({ + plan, + env, + piModelConfig, + log: logger, + }); let runAgentDir = localPiAssets.dir; + // Fail closed (Decision 6): a local managed custom run whose models.json could not be written + // must stop rather than run on a default provider. Recorded here (the write ran above) and + // thrown inside the try below, like the permission-extension gate. + const localModelConfigUnwritable = + plan.isPi && + !plan.isDaytona && + !!piModelConfig && + !localPiAssets.modelConfigWritten; // Fail closed (Decision 2): when the policy could gate a Pi built-in tool but the permission // extension did not install, the run must stop rather than run those tools unprotected. Recorded // here (the install ran above) and thrown inside the try below so the engine's own catch turns it @@ -1237,11 +1274,21 @@ export async function acquireEnvironment( }; try { + // Fail loud before any sandbox/mount infra spins up: an applicable-but-incomplete + // OpenAI-compatible custom request is a hard error, never a silent fall-back (Decision 5). + if (piModelConfigError) { + throw piModelConfigError; + } // Fail closed before any sandbox/mount infra spins up: a local Pi run whose policy could gate a // built-in tool cannot proceed without the permission extension installed (Decision 2). if (localBuiltinGatingUnenforceable) { throw new Error(PI_PERMISSION_EXTENSION_UNAVAILABLE_MESSAGE); } + // Fail closed: a local managed custom run whose models.json could not be materialized cannot + // fall through to a default provider (Decision 6). + if (localModelConfigUnwritable) { + throw new Error(PI_MODEL_CONFIG_WRITE_FAILED_MESSAGE); + } // Persist events in-process so a follow-up turn can resume by session id. const persist = deps.createPersist?.() ?? new InMemorySessionPersistDriver(); @@ -1362,6 +1409,7 @@ export async function acquireEnvironment( )({ sandbox: environment.sandbox, plan: { ...plan, skillDirs: [] }, + piModelConfig, log: logger, }); // Fail closed (Decision 2): same guarantee as the local path. A genuine upload failure on the @@ -1698,9 +1746,22 @@ export async function acquireEnvironment( // Resolve the model first: when the harness rejects the requested id and keeps its own // default, `model` is undefined and the chat span is labelled "chat". + // + // For a managed OpenAI-compatible custom run, request the FULLY QUALIFIED + // `/` that pi-acp advertises for this provider, not the bare wire + // model id (design Decision 7). `applyModel`/`pickModel` fall back to suffix matching, which + // returns the FIRST advertised id whose suffix matches — so a built-in `openai/` that + // Pi still advertises (the vault key rides in as `OPENAI_API_KEY`, keeping Pi's built-in + // openai provider live) would be selected ahead of the custom `/` when both share + // the model id. That would silently route to api.openai.com instead of the user's endpoint. + // The qualified id is an EXACT match, so it always wins over any bare-suffix collision. + const wantedModel = + piModelConfig && piModelConfig.models.length > 0 + ? `${piModelConfig.providerId}/${piModelConfig.models[0].id}` + : request.model; environment.model = await (deps.applyModel ?? applyModel)( environment.session, - request.model, + wantedModel, logger, { strict: strictModel }, ); diff --git a/services/runner/src/engines/sandbox_agent/daytona.ts b/services/runner/src/engines/sandbox_agent/daytona.ts index 52cea644ab..1746236914 100644 --- a/services/runner/src/engines/sandbox_agent/daytona.ts +++ b/services/runner/src/engines/sandbox_agent/daytona.ts @@ -6,6 +6,11 @@ import { uploadSkillsToSandbox, uploadSystemPromptToSandbox, } from "./pi-assets.ts"; +import { + PI_MODELS_JSON_FILENAME, + serializePiModelsJson, + type PiModelConfigPlan, +} from "./pi-model-config.ts"; import { type RunPlan } from "./run-plan.ts"; type Log = (message: string) => void; @@ -139,6 +144,49 @@ export async function ensurePiInSandbox( } } +/** + * Upload the exact Pi `models.json` into a Daytona sandbox's Pi agent dir, overwriting any stale + * file left by an earlier configuration on a reused sandbox. THROWS on failure so the caller makes + * materialization terminal — a managed custom run must never fall through to a default provider + * (design Decision 6). The document carries only the `$OPENAI_API_KEY` reference; the key value + * itself rides `daytonaEnvVars` into the sandbox env. + */ +export async function uploadPiModelsConfigToSandbox( + sandbox: any, + agentDir: string, + plan: PiModelConfigPlan, + log: Log = () => {}, +): Promise { + await sandbox.mkdirFs({ path: agentDir }); + await sandbox.writeFsFile( + { path: `${agentDir}/${PI_MODELS_JSON_FILENAME}` }, + serializePiModelsJson(plan), + ); + log( + `pi models.json uploaded provider=${plan.providerId} api=${plan.api} ` + + `model=${plan.models.map((m) => m.id).join(",")}`, + ); +} + +/** + * Remove any stale Pi `models.json` from a reused sandbox when the current run has NO model-config + * plan, so a reused/parked sandbox never retains a custom provider from an earlier configuration. + * Best-effort: an absent file (or a provider without a delete op) is fine. + */ +export async function removePiModelsConfigFromSandbox( + sandbox: any, + agentDir: string, + log: Log = () => {}, +): Promise { + try { + await sandbox.deleteFsEntry?.({ + path: `${agentDir}/${PI_MODELS_JSON_FILENAME}`, + }); + } catch (err) { + log(`pi models.json cleanup skipped: ${(err as Error).message}`); + } +} + export interface PrepareDaytonaPiAssetsInput { sandbox: any; plan: Pick< @@ -149,6 +197,12 @@ export interface PrepareDaytonaPiAssetsInput { | "systemPrompt" | "appendSystemPrompt" >; + /** + * A managed OpenAI-compatible custom run's Pi provider config. When set, its `models.json` is + * uploaded before the ACP session starts; when absent, any stale `models.json` on a reused + * sandbox is removed so no earlier provider survives. + */ + piModelConfig?: PiModelConfigPlan; log?: Log; } @@ -161,6 +215,7 @@ export interface PrepareDaytonaPiAssetsInput { export async function prepareDaytonaPiAssets({ sandbox, plan, + piModelConfig, log = () => {}, }: PrepareDaytonaPiAssetsInput): Promise { if (!plan.isPi) return true; @@ -174,6 +229,19 @@ export async function prepareDaytonaPiAssets({ DAYTONA_PI_DIR, log, ); + // Managed OpenAI-compatible custom provider: upload the exact models.json (overwriting stale) + // before the session starts. No plan: remove any stale file so a reused sandbox keeps no earlier + // provider. Upload failure THROWS here and is terminal in the engine's acquire try. + if (piModelConfig) { + await uploadPiModelsConfigToSandbox( + sandbox, + DAYTONA_PI_DIR, + piModelConfig, + log, + ); + } else { + await removePiModelsConfigFromSandbox(sandbox, DAYTONA_PI_DIR, log); + } if (plan.skillDirs.length > 0) { await uploadSkillsToSandbox(sandbox, DAYTONA_PI_DIR, plan.skillDirs, log); } diff --git a/services/runner/src/engines/sandbox_agent/pi-assets.ts b/services/runner/src/engines/sandbox_agent/pi-assets.ts index a8ee344530..548317356c 100644 --- a/services/runner/src/engines/sandbox_agent/pi-assets.ts +++ b/services/runner/src/engines/sandbox_agent/pi-assets.ts @@ -19,6 +19,11 @@ import type { AgentRunRequest, ResolvedToolSpec } from "../../protocol.ts"; import { advertisedToolSpecs } from "../../tools/public-spec.ts"; import type { MaterializedSkill } from "../skills.ts"; import { PKG_ROOT } from "./daemon.ts"; +import { + PI_MODELS_JSON_FILENAME, + serializePiModelsJson, + type PiModelConfigPlan, +} from "./pi-model-config.ts"; import type { RunPlan } from "./run-plan.ts"; type Log = (message: string) => void; @@ -270,6 +275,43 @@ export const PI_PERMISSION_EXTENSION_UNAVAILABLE_MESSAGE = "Ask your deployment operator to make the runner's Pi agent directory writable, or rebuild and " + "republish the runner image."; +/** + * Thrown (via the engine's named-message pattern) when a managed OpenAI-compatible custom run has + * a model-config plan but its `models.json` could not be materialized. Fail closed: the custom + * provider would not be registered, so the run must stop rather than fall back to a default + * provider (design Decision 6). Single line so `conciseError` surfaces it verbatim. + */ +export const PI_MODEL_CONFIG_WRITE_FAILED_MESSAGE = + "The agent could not write its model configuration (models.json), so the OpenAI-compatible " + + "custom provider could not be registered. The run was stopped rather than fall back to a " + + "default provider. Ask your deployment operator to make the runner's Pi agent directory writable."; + +/** + * Write the Pi `models.json` into a local (throwaway) agent dir with mode `0600` via an atomic + * temp-file-plus-rename. THROWS on failure so the caller can make materialization terminal — a + * managed custom run must never fall through to a default provider (design Decision 6). The file + * carries only the `$OPENAI_API_KEY` reference, never the key value. + */ +export function writePiModelsConfigLocal( + agentDir: string, + plan: PiModelConfigPlan, +): void { + const document = serializePiModelsJson(plan); + const target = join(agentDir, PI_MODELS_JSON_FILENAME); + const staging = join( + agentDir, + `.${PI_MODELS_JSON_FILENAME}.${randomUUID()}.tmp`, + ); + mkdirSync(agentDir, { recursive: true }); + writeFileSync(staging, document, { encoding: "utf-8", mode: 0o600 }); + try { + renameSync(staging, target); + } catch (err) { + rmSync(staging, { force: true }); + throw err; + } +} + /** * Env the Agenta Pi extension reads. Tool env contains only public metadata plus the * relay directory; private specs/auth stay in the runner. @@ -471,13 +513,24 @@ export interface PreparedLocalAgentDir { * into it. The temp dir lives under the system temp path, which the runtime user owns, so the * install never depends on the operator-configured `PI_CODING_AGENT_DIR` being writable. Reports * whether the extension install succeeded. + * + * `seedCredentials` (default true) controls whether the operator's `auth.json` is copied in. A + * managed OpenAI-compatible custom run sets it FALSE: that run authenticates purely from the + * vault-resolved `$OPENAI_API_KEY` plus its `models.json`, so the operator's personal Pi login + * must not leak into it (a copied `auth.json` could let Pi fall back to the operator's own + * provider). The non-credential `settings.json` is always carried. */ export function prepareLocalAgentDir( sourceAgentDir: string, log: Log = () => {}, + opts: { seedCredentials?: boolean } = {}, ): PreparedLocalAgentDir { + const seedCredentials = opts.seedCredentials ?? true; const dir = mkdtempSync(join(tmpdir(), "agenta-pi-agentdir-")); - for (const name of ["auth.json", "settings.json"]) { + const seedFiles = seedCredentials + ? ["auth.json", "settings.json"] + : ["settings.json"]; + for (const name of seedFiles) { const src = join(sourceAgentDir, name); try { if (existsSync(src)) copyFileSync(src, join(dir, name)); @@ -502,6 +555,12 @@ export interface PrepareLocalPiAssetsInput { | "sourcePiAgentDir" >; env: Record; + /** + * A managed OpenAI-compatible custom run's Pi provider config. When set, the isolated per-run + * agent dir receives a `models.json` for this provider and does NOT get the operator's personal + * `auth.json`; the run authenticates from the vault key referenced by `$OPENAI_API_KEY`. + */ + piModelConfig?: PiModelConfigPlan; log?: Log; } @@ -517,6 +576,12 @@ export interface PrepareLocalPiAssetsResult { * fails the run closed when the policy could gate a Pi built-in tool (`builtinGatingActive`). */ extensionInstalled: boolean; + /** + * False only when a model-config plan was present but its `models.json` could not be written; + * true when there was nothing to write or the write succeeded. The caller fails the run closed + * when this is false (materialization is terminal — design Decision 6). + */ + modelConfigWritten: boolean; } /** @@ -543,14 +608,20 @@ export interface PrepareLocalPiAssetsResult { export function prepareLocalPiAssets({ plan, env, + piModelConfig, log = () => {}, }: PrepareLocalPiAssetsInput): PrepareLocalPiAssetsResult { // Not a local Pi run: nothing to install here, so enforcement is not this path's concern. if (!plan.isPi || plan.isDaytona) - return { dir: undefined, extensionInstalled: true }; + return { + dir: undefined, + extensionInstalled: true, + modelConfigWritten: true, + }; // buildRunPlan already rejected a local runtime_provided run with no configured - // PI_CODING_AGENT_DIR, so `sourcePiAgentDir` here IS the operator's mount. + // PI_CODING_AGENT_DIR, so `sourcePiAgentDir` here IS the operator's mount. A model-config plan + // never reaches here (it requires credentialMode "env"), so there is no models.json to write. if (plan.credentialMode === "runtime_provided") { const agentDir = plan.sourcePiAgentDir; const extensionInstalled = installPiExtensionLocal(agentDir, log); @@ -564,14 +635,17 @@ export function prepareLocalPiAssets({ } env.PI_CODING_AGENT_DIR = agentDir; // Deliberately NOT returned as a throwaway: this is the operator's login, not a temp dir. - return { dir: undefined, extensionInstalled }; + return { dir: undefined, extensionInstalled, modelConfigWritten: true }; } // Managed / none: always route through a throwaway per-run dir the runtime user owns, so the - // extension install never depends on the configured PI_CODING_AGENT_DIR being writable. + // extension install never depends on the configured PI_CODING_AGENT_DIR being writable. A + // managed OpenAI-compatible custom run (a model-config plan is present) additionally gets a + // models.json and does NOT receive the operator's personal auth.json. const { dir: runAgentDir, extensionInstalled } = prepareLocalAgentDir( plan.sourcePiAgentDir, log, + { seedCredentials: !piModelConfig }, ); if (plan.hasSystemPrompt) { writeSystemPromptLocal( @@ -581,8 +655,22 @@ export function prepareLocalPiAssets({ log, ); } + let modelConfigWritten = true; + if (piModelConfig) { + try { + writePiModelsConfigLocal(runAgentDir, piModelConfig); + log( + `pi models.json written provider=${piModelConfig.providerId} ` + + `api=${piModelConfig.api} model=${piModelConfig.models.map((m) => m.id).join(",")}`, + ); + } catch (err) { + // Terminal: the caller stops the run rather than fall back to a default provider. + modelConfigWritten = false; + log(`pi models.json write failed: ${(err as Error).message}`); + } + } env.PI_CODING_AGENT_DIR = runAgentDir; - return { dir: runAgentDir, extensionInstalled }; + return { dir: runAgentDir, extensionInstalled, modelConfigWritten }; } /** Upload materialized skill dirs into a Daytona sandbox's Pi `skills/` user scope. */ diff --git a/services/runner/src/engines/sandbox_agent/pi-model-config.ts b/services/runner/src/engines/sandbox_agent/pi-model-config.ts new file mode 100644 index 0000000000..61f42b58b8 --- /dev/null +++ b/services/runner/src/engines/sandbox_agent/pi-model-config.ts @@ -0,0 +1,147 @@ +/** + * Pure Pi model-config builder (design Decision 5, planning layer). + * + * Translates a neutral `/run` request for a managed OpenAI-compatible custom connection into Pi's + * native `models.json` plan, WITHOUT any filesystem or sandbox dependency. The materialization + * (local write / Daytona upload) lives in `pi-assets.ts` and `daytona.ts`; this module only + * decides whether a plan applies and produces the exact document shape. + * + * `models.json` is a Pi mechanism (see the bundled Pi docs `docs/models.md`): it registers a + * custom provider so pi-acp advertises `/` as a settable model. The input + * here stays neutral — no Pi-specific wire field is introduced. + */ +import type { AgentRunRequest } from "../../protocol.ts"; + +/** The API dialect this builder emits. The only value v1 supports (design Decision 1). */ +export type PiProviderApi = "openai-completions"; + +/** + * The canonical env var a managed OpenAI-compatible key resolves into (design Decision 2). The + * document references it as `$OPENAI_API_KEY`; the raw value never enters the plan or the file. + */ +export const OPENAI_API_KEY_ENV = "OPENAI_API_KEY"; + +/** The file Pi reads a custom provider registry from, inside its agent dir (PI_CODING_AGENT_DIR). */ +export const PI_MODELS_JSON_FILENAME = "models.json"; + +/** + * A validated internal plan for one custom provider and its selected model. Holds only the env var + * NAME (`apiKeyEnv`), never the key value — the raw key stays in the resolved `secrets` set. + */ +export interface PiModelConfigPlan { + /** Pi provider id — the connection slug (stable, portable, disambiguates two custom endpoints). */ + providerId: string; + /** The resolved provider family. Only "openai" in v1. */ + providerFamily: "openai"; + /** The API dialect Pi speaks to the endpoint. */ + api: PiProviderApi; + /** The custom endpoint base URL (e.g. https://host/v1). */ + baseUrl: string; + /** The env var Pi reads the key from; the file carries only `$OPENAI_API_KEY`. */ + apiKeyEnv: typeof OPENAI_API_KEY_ENV; + /** The exact selected model(s). v1 registers exactly one. */ + models: Array<{ id: string }>; +} + +/** + * Thrown when a request is APPLICABLE (a managed OpenAI-compatible custom Pi run) but INCOMPLETE: + * a required piece (slug, base URL, env credential mode, key, or model) is missing. Fail loud — + * a run must never silently fall back to a default provider (design Decision 5). Single line so + * `conciseError` surfaces it verbatim; it never carries the key value. + */ +export class PiModelConfigError extends Error { + constructor(message: string) { + super(message); + this.name = "PiModelConfigError"; + } +} + +/** Pi identity check, mirroring `buildRunPlan` (an empty harness defaults to `pi_core`). */ +function isPiHarness(harness: string | undefined): boolean { + const resolved = harness || "pi_core"; + return resolved === "pi_core" || resolved === "pi_agenta"; +} + +/** + * Build the Pi model-config plan from the neutral run request and the resolved secrets, or return + * `undefined` when the request is not a managed OpenAI-compatible custom Pi run (current behavior). + * + * Applicability (which KIND of run this is) — ALL must hold, else no plan: + * - the harness is Pi; + * - the provider family is "openai"; + * - the deployment is "custom"; + * - the connection is a named Agenta connection (`mode === "agenta"`). + * + * Completeness (the applicable run has everything it needs) — once applicable, ALL must hold or + * the request is INCOMPLETE and throws `PiModelConfigError`: + * - a non-empty connection slug; + * - an endpoint base URL; + * - credential mode "env"; + * - `OPENAI_API_KEY` present in the resolved secrets; + * - a model id. + * + * The plan holds only the env var NAME; the raw key never enters it. + */ +export function buildPiModelConfigPlan( + request: AgentRunRequest, + secrets: Record, +): PiModelConfigPlan | undefined { + const applicable = + isPiHarness(request.harness) && + request.provider === "openai" && + request.deployment === "custom" && + request.connection?.mode === "agenta"; + if (!applicable) return undefined; + + const slug = request.connection?.slug?.trim(); + const baseUrl = request.endpoint?.baseUrl?.trim(); + const model = request.model?.trim(); + const hasKey = !!secrets[OPENAI_API_KEY_ENV]?.trim(); + + const missing: string[] = []; + if (!slug) missing.push("a connection slug"); + if (!baseUrl) missing.push("an endpoint base URL"); + if (request.credentialMode !== "env") + missing.push( + `credential mode "env" (got "${request.credentialMode ?? "none"}")`, + ); + if (!hasKey) missing.push(`${OPENAI_API_KEY_ENV} in the resolved secrets`); + if (!model) missing.push("a model id"); + + if (missing.length > 0) { + throw new PiModelConfigError( + `OpenAI-compatible custom connection ${slug ? `'${slug}' ` : ""}is incomplete: ` + + `missing ${missing.join(", ")}. The run was stopped rather than fall back to a ` + + `default provider.`, + ); + } + + return { + providerId: slug as string, + providerFamily: "openai", + api: "openai-completions", + baseUrl: baseUrl as string, + apiKeyEnv: OPENAI_API_KEY_ENV, + models: [{ id: model as string }], + }; +} + +/** + * Serialize the plan to the exact Pi `models.json` document. The provider is keyed by slug; the + * key is referenced as `$OPENAI_API_KEY` (never the raw value). Only this plan's one provider and + * model are written — arbitrary existing providers are never merged into a managed run. Pretty + * printed with a trailing newline. + */ +export function serializePiModelsJson(plan: PiModelConfigPlan): string { + const document = { + providers: { + [plan.providerId]: { + baseUrl: plan.baseUrl, + api: plan.api, + apiKey: `$${plan.apiKeyEnv}`, + models: plan.models.map((model) => ({ id: model.id })), + }, + }, + }; + return `${JSON.stringify(document, null, 2)}\n`; +} diff --git a/services/runner/tests/unit/sandbox-agent-daytona.test.ts b/services/runner/tests/unit/sandbox-agent-daytona.test.ts index d40dbae805..fa39c257cb 100644 --- a/services/runner/tests/unit/sandbox-agent-daytona.test.ts +++ b/services/runner/tests/unit/sandbox-agent-daytona.test.ts @@ -15,7 +15,19 @@ import { createCookieFetch, daytonaEnvVars, ensurePiInSandbox, + removePiModelsConfigFromSandbox, + uploadPiModelsConfigToSandbox, } from "../../src/engines/sandbox_agent/daytona.ts"; +import type { PiModelConfigPlan } from "../../src/engines/sandbox_agent/pi-model-config.ts"; + +const MODEL_CONFIG_PLAN: PiModelConfigPlan = { + providerId: "my-ollama", + providerFamily: "openai", + api: "openai-completions", + baseUrl: "https://example.test/v1", + apiKeyEnv: "OPENAI_API_KEY", + models: [{ id: "qwen2.5-coder:7b" }], +}; const envKeys = ["PI_CODING_AGENT_DIR"]; const previousEnv = new Map(); @@ -147,6 +159,85 @@ describe("ensurePiInSandbox (probe and pinned-install repair)", () => { }); }); +describe("uploadPiModelsConfigToSandbox", () => { + it("writes the exact models.json into the Pi agent dir (key never inlined)", async () => { + const writes: Array<{ path: string; body: string }> = []; + const sandbox = { + mkdirFs: async () => {}, + writeFsFile: async ({ path }: { path: string }, body: string) => { + writes.push({ path, body }); + }, + }; + + await uploadPiModelsConfigToSandbox( + sandbox, + DAYTONA_PI_DIR, + MODEL_CONFIG_PLAN, + ); + + assert.equal(writes.length, 1); + assert.equal(writes[0].path, `${DAYTONA_PI_DIR}/models.json`); + assert.deepEqual(JSON.parse(writes[0].body), { + providers: { + "my-ollama": { + baseUrl: "https://example.test/v1", + api: "openai-completions", + apiKey: "$OPENAI_API_KEY", + models: [{ id: "qwen2.5-coder:7b" }], + }, + }, + }); + assert.equal(writes[0].body.includes("$OPENAI_API_KEY"), true); + }); + + it("throws when the upload fails (materialization is terminal)", async () => { + const sandbox = { + mkdirFs: async () => {}, + writeFsFile: async () => { + throw new Error("sandbox fs write failed"); + }, + }; + + await assert.rejects( + () => + uploadPiModelsConfigToSandbox( + sandbox, + DAYTONA_PI_DIR, + MODEL_CONFIG_PLAN, + ), + /sandbox fs write failed/, + ); + }); +}); + +describe("removePiModelsConfigFromSandbox", () => { + it("deletes a stale models.json so a reused sandbox keeps no earlier provider", async () => { + const deletes: string[] = []; + const sandbox = { + deleteFsEntry: async ({ path }: { path: string }) => { + deletes.push(path); + }, + }; + + await removePiModelsConfigFromSandbox(sandbox, DAYTONA_PI_DIR); + + assert.deepEqual(deletes, [`${DAYTONA_PI_DIR}/models.json`]); + }); + + it("swallows a missing-file / unsupported-delete error (best effort)", async () => { + const sandbox = { + deleteFsEntry: async () => { + throw new Error("not found"); + }, + }; + // Must not throw. + await removePiModelsConfigFromSandbox(sandbox, DAYTONA_PI_DIR); + + // A provider without a delete op is also tolerated. + await removePiModelsConfigFromSandbox({}, DAYTONA_PI_DIR); + }); +}); + describe("createCookieFetch", () => { it("persists Daytona preview cookies per host", async () => { const seenCookies: Array = []; diff --git a/services/runner/tests/unit/sandbox-agent-model.test.ts b/services/runner/tests/unit/sandbox-agent-model.test.ts index 9870c6f6bb..2779d6c656 100644 --- a/services/runner/tests/unit/sandbox-agent-model.test.ts +++ b/services/runner/tests/unit/sandbox-agent-model.test.ts @@ -16,12 +16,24 @@ import { describe("pickModel", () => { it("matches exact ids first", () => { - assert.equal(pickModel(["openai-codex/gpt-5.5", "anthropic/sonnet"], "anthropic/sonnet"), "anthropic/sonnet"); + assert.equal( + pickModel( + ["openai-codex/gpt-5.5", "anthropic/sonnet"], + "anthropic/sonnet", + ), + "anthropic/sonnet", + ); }); it("matches by provider suffix", () => { - assert.equal(pickModel(["openai-codex/gpt-5.5"], "gpt-5.5"), "openai-codex/gpt-5.5"); - assert.equal(pickModel(["openai-codex/gpt-5.5"], "other/gpt-5.5"), "openai-codex/gpt-5.5"); + assert.equal( + pickModel(["openai-codex/gpt-5.5"], "gpt-5.5"), + "openai-codex/gpt-5.5", + ); + assert.equal( + pickModel(["openai-codex/gpt-5.5"], "other/gpt-5.5"), + "openai-codex/gpt-5.5", + ); }); it("returns undefined when no model matches", () => { @@ -42,14 +54,56 @@ describe("pickModel", () => { it("does not fall back from a hinted request to a bare id (never shrinks context)", () => { // Only "sonnet" is offered (no "[1m]" sibling): a caller that explicitly asked for the // long-context variant must not be silently downgraded to the short-context one. - assert.equal(pickModel(["default", "sonnet", "opus"], "sonnet[1m]"), undefined); + assert.equal( + pickModel(["default", "sonnet", "opus"], "sonnet[1m]"), + undefined, + ); + }); + + it("maps a custom connection's bare model id to Pi's advertised /", () => { + // After models.json, pi-acp advertises the custom provider as `/`. + // The wire carries the bare model id; the existing suffix match resolves it (design Decision 7). + assert.equal( + pickModel(["my-ollama/qwen2.5-coder:7b"], "qwen2.5-coder:7b"), + "my-ollama/qwen2.5-coder:7b", + ); + // A model id that the custom provider does not advertise does not resolve, so the run fails + // loud (ModelNotSettableError) instead of continuing on a default. + assert.equal( + pickModel(["my-ollama/qwen2.5-coder:7b"], "llama3:70b"), + undefined, + ); + }); + + it("bare-suffix matching is order-dependent and can pick a built-in over the custom provider", () => { + // Collision (design Decision 7 hazard): the vault key rides into Pi as OPENAI_API_KEY, so Pi + // keeps advertising its built-in `openai/` (pointing at api.openai.com) ALONGSIDE the + // custom `my-conn/`. When the custom model id equals a built-in one (e.g. "gpt-4o"), + // bare-suffix matching returns the FIRST advertised id with that suffix — the built-in — which + // would silently route to the wrong provider/endpoint. This proves why the runner must request + // the fully qualified id (below) instead of the bare wire id for a managed custom run. + assert.equal( + pickModel(["openai/gpt-4o", "my-conn/gpt-4o"], "gpt-4o"), + "openai/gpt-4o", + ); + // The fully qualified `/` is an EXACT match, so it wins regardless of order and + // regardless of a colliding built-in — this is the id the runner now passes when a + // PiModelConfigPlan exists. + assert.equal( + pickModel(["openai/gpt-4o", "my-conn/gpt-4o"], "my-conn/gpt-4o"), + "my-conn/gpt-4o", + ); }); }); describe("allowedFromError", () => { it("parses allowed values from harness errors", () => { assert.deepEqual( - allowedFromError(new Error("Unsupported value. Allowed values: openai-codex/gpt-5.5, anthropic/sonnet")), + allowedFromError( + new Error( + "Unsupported value. Allowed values: openai-codex/gpt-5.5, anthropic/sonnet", + ), + ), ["openai-codex/gpt-5.5", "anthropic/sonnet"], ); }); @@ -70,7 +124,10 @@ describe("allowedModels", () => { }, ], }; - assert.deepEqual(await allowedModels(session), ["openai-codex/gpt-5.5", "anthropic/sonnet"]); + assert.deepEqual(await allowedModels(session), [ + "openai-codex/gpt-5.5", + "anthropic/sonnet", + ]); }); }); @@ -79,7 +136,10 @@ describe("applyModel", () => { const calls: string[] = []; const session = { setModel: async (id: string) => void calls.push(id) }; - assert.equal(await applyModel(session, "anthropic/sonnet"), "anthropic/sonnet"); + assert.equal( + await applyModel(session, "anthropic/sonnet"), + "anthropic/sonnet", + ); assert.deepEqual(calls, ["anthropic/sonnet"]); }); @@ -99,7 +159,9 @@ describe("applyModel", () => { setModel: async (id: string) => { calls.push(id); if (id === "gpt-5.5") { - throw new Error("Unsupported value. Allowed values: openai-codex/gpt-5.5"); + throw new Error( + "Unsupported value. Allowed values: openai-codex/gpt-5.5", + ); } }, }; @@ -139,7 +201,9 @@ describe("applyModel", () => { }; assert.equal( - await applyModel(session, "gpt-5.5", (m) => logs.push(m), { strict: false }), + await applyModel(session, "gpt-5.5", (m) => logs.push(m), { + strict: false, + }), undefined, ); assert.match(logs[0], /using harness default/); diff --git a/services/runner/tests/unit/sandbox-agent-orchestration.test.ts b/services/runner/tests/unit/sandbox-agent-orchestration.test.ts index db7ad02bb1..b7cce0d9ab 100644 --- a/services/runner/tests/unit/sandbox-agent-orchestration.test.ts +++ b/services/runner/tests/unit/sandbox-agent-orchestration.test.ts @@ -422,6 +422,37 @@ describe("runSandboxAgent orchestration", () => { assert.ok(calls.mkdirFsPaths.includes(expected)); }); + it("requests the fully qualified / for a managed OpenAI-compatible custom run", async () => { + // Design Decision 7 hazard: the vault key rides in as OPENAI_API_KEY, so Pi keeps advertising + // its built-in `openai/` alongside the custom `my-conn/`. Passing the bare wire + // id "gpt-4o" would let suffix matching pick the built-in (api.openai.com) over the user's + // endpoint. The runner must pass the exact `/` so it always wins. + const { calls, deps } = fakeHarness(); + deps.prepareDaytonaPiAssets = (async () => true) as any; + + const result = await runSandboxAgent( + { + harness: "pi_core", + sandbox: "daytona", + messages: [{ role: "user", content: "hello" }], + model: "gpt-4o", + provider: "openai", + deployment: "custom", + connection: { mode: "agenta", slug: "my-conn" }, + endpoint: { baseUrl: "https://proxy.test/v1" }, + credentialMode: "env", + secrets: { OPENAI_API_KEY: "sk-vault-xyz" }, + }, + undefined, + undefined, + deps, + ); + + assert.equal(result.ok, true); + assert.equal(calls.applyModelArgs.length, 1); + assert.equal(calls.applyModelArgs[0].model, "my-conn/gpt-4o"); + }); + it("passes the same Pi skill snapshot to local Pi and workspace preparation", async () => { const cwd = join( tmpdir(), diff --git a/services/runner/tests/unit/sandbox-agent-pi-assets.test.ts b/services/runner/tests/unit/sandbox-agent-pi-assets.test.ts index dddcabc200..746be24929 100644 --- a/services/runner/tests/unit/sandbox-agent-pi-assets.test.ts +++ b/services/runner/tests/unit/sandbox-agent-pi-assets.test.ts @@ -9,6 +9,7 @@ import { existsSync, mkdirSync, mkdtempSync, + readdirSync, readFileSync, rmSync, statSync, @@ -31,8 +32,19 @@ import { resolvePiSkillSnapshot, uploadDirToSandbox, writeOtlpAuthFile, + writePiModelsConfigLocal, writeSystemPromptLocal, } from "../../src/engines/sandbox_agent/pi-assets.ts"; +import type { PiModelConfigPlan } from "../../src/engines/sandbox_agent/pi-model-config.ts"; + +const MODEL_CONFIG_PLAN: PiModelConfigPlan = { + providerId: "my-ollama", + providerFamily: "openai", + api: "openai-completions", + baseUrl: "https://example.test/v1", + apiKeyEnv: "OPENAI_API_KEY", + models: [{ id: "qwen2.5-coder:7b" }], +}; describe("Pi session workspace", () => { it("uses one stable transcript directory inside the conversation cwd", () => { @@ -388,6 +400,60 @@ describe("prepareLocalAgentDir", () => { else process.env.SANDBOX_AGENT_EXTENSION_BUNDLE = previous; } }); + + it("with seedCredentials=false, skips the operator's auth.json but keeps settings.json", () => { + const source = tempDir("agenta-pi-source-nocreds-"); + writeFileSync(join(source, "auth.json"), '{"token":"personal"}', "utf-8"); + writeFileSync(join(source, "settings.json"), '{"model":"gpt"}', "utf-8"); + + const { dir: runDir } = prepareLocalAgentDir(source, undefined, { + seedCredentials: false, + }); + dirs.push(runDir); + + // The operator's personal login never leaks into a managed custom run's isolated dir... + assert.equal(existsSync(join(runDir, "auth.json")), false); + // ...but non-credential settings are still carried. + assert.equal( + readFileSync(join(runDir, "settings.json"), "utf-8"), + '{"model":"gpt"}', + ); + }); +}); + +describe("writePiModelsConfigLocal", () => { + it("writes an exact 0600 models.json via an atomic temp-file + rename", () => { + const dir = tempDir("agenta-pi-models-config-"); + + writePiModelsConfigLocal(dir, MODEL_CONFIG_PLAN); + + const path = join(dir, "models.json"); + assert.equal(statSync(path).mode & 0o777, 0o600); + assert.deepEqual(JSON.parse(readFileSync(path, "utf-8")), { + providers: { + "my-ollama": { + baseUrl: "https://example.test/v1", + api: "openai-completions", + apiKey: "$OPENAI_API_KEY", + models: [{ id: "qwen2.5-coder:7b" }], + }, + }, + }); + // No staging file lingers. + assert.equal( + readdirSync(dir).some((n) => n.startsWith(".models.json.")), + false, + ); + }); + + it("throws when the target cannot be written (materialization is terminal)", () => { + const dir = tempDir("agenta-pi-models-config-fail-"); + // A non-empty directory occupying models.json makes the rename fail. + mkdirSync(join(dir, "models.json")); + writeFileSync(join(dir, "models.json", "keep.txt"), "x", "utf-8"); + + assert.throws(() => writePiModelsConfigLocal(dir, MODEL_CONFIG_PLAN)); + }); }); describe("prepareLocalPiAssets (managed/none routes through a throwaway dir)", () => { @@ -439,6 +505,66 @@ describe("prepareLocalPiAssets (managed/none routes through a throwaway dir)", ( else process.env.SANDBOX_AGENT_EXTENSION_BUNDLE = previous; } }); + + it("reports modelConfigWritten=true for a plain run with no model-config plan", () => { + const { dir: runDir, modelConfigWritten } = prepareLocalPiAssets({ + plan: plainPiPlan, + env: {}, + }); + if (runDir) dirs.push(runDir); + assert.equal(modelConfigWritten, true); + assert.equal(existsSync(join(runDir as string, "models.json")), false); + }); + + it("still copies the operator's auth.json for a managed run WITHOUT a model-config plan (unchanged)", () => { + const source = tempDir("agenta-pi-managed-noplan-source-"); + writeFileSync(join(source, "auth.json"), '{"token":"managed"}', "utf-8"); + + const { dir: runDir } = prepareLocalPiAssets({ + plan: { ...plainPiPlan, sourcePiAgentDir: source }, + env: {}, + }); + assert.ok(runDir); + dirs.push(runDir as string); + assert.equal( + readFileSync(join(runDir as string, "auth.json"), "utf-8"), + '{"token":"managed"}', + ); + }); + + it("for a model-config plan: writes models.json, omits auth.json, and points PI_CODING_AGENT_DIR at the dir", () => { + const source = tempDir("agenta-pi-managed-plan-source-"); + writeFileSync(join(source, "auth.json"), '{"token":"personal"}', "utf-8"); + writeFileSync(join(source, "settings.json"), '{"model":"x"}', "utf-8"); + const env: Record = {}; + + const { dir: runDir, modelConfigWritten } = prepareLocalPiAssets({ + plan: { ...plainPiPlan, sourcePiAgentDir: source }, + env, + piModelConfig: MODEL_CONFIG_PLAN, + }); + + assert.ok(runDir); + dirs.push(runDir as string); + assert.equal(modelConfigWritten, true); + assert.equal(env.PI_CODING_AGENT_DIR, runDir); + // The managed custom run authenticates from $OPENAI_API_KEY, never the operator's login. + assert.equal(existsSync(join(runDir as string, "auth.json")), false); + // Non-credential settings are still carried. + assert.equal( + readFileSync(join(runDir as string, "settings.json"), "utf-8"), + '{"model":"x"}', + ); + // The exact models.json is present and references only the env var. + const modelsText = readFileSync( + join(runDir as string, "models.json"), + "utf-8", + ); + assert.equal(modelsText.includes("$OPENAI_API_KEY"), true); + assert.deepEqual(JSON.parse(modelsText).providers["my-ollama"].models, [ + { id: "qwen2.5-coder:7b" }, + ]); + }); }); describe("Pi skill snapshots", () => { diff --git a/services/runner/tests/unit/sandbox-agent-pi-model-config.test.ts b/services/runner/tests/unit/sandbox-agent-pi-model-config.test.ts new file mode 100644 index 0000000000..dbbb61ff4e --- /dev/null +++ b/services/runner/tests/unit/sandbox-agent-pi-model-config.test.ts @@ -0,0 +1,230 @@ +/** + * Unit tests for the pure Pi model-config builder (design Decision 5, planning layer). + * + * Exhaustive over the applicability + completeness gate, and a no-secret-leak proof. Pure module: + * no filesystem or sandbox dependency. + * + * Run: pnpm test (or: pnpm exec vitest run tests/unit/sandbox-agent-pi-model-config.test.ts) + */ +import { describe, it } from "vitest"; +import assert from "node:assert/strict"; + +import type { AgentRunRequest } from "../../src/protocol.ts"; +import { + buildPiModelConfigPlan, + PiModelConfigError, + serializePiModelsJson, +} from "../../src/engines/sandbox_agent/pi-model-config.ts"; + +const RAW_KEY = "sk-super-secret-value-do-not-leak"; + +/** A complete, applicable managed OpenAI-compatible custom Pi request. */ +function completeRequest(over: Partial = {}): AgentRunRequest { + return { + harness: "pi_core", + provider: "openai", + deployment: "custom", + connection: { mode: "agenta", slug: "my-ollama" }, + endpoint: { baseUrl: "https://example.test/v1" }, + credentialMode: "env", + model: "qwen2.5-coder:7b", + ...over, + }; +} + +const completeSecrets = { OPENAI_API_KEY: RAW_KEY }; + +describe("buildPiModelConfigPlan (applicable + complete)", () => { + it("builds a plan for a managed OpenAI-compatible custom Pi run", () => { + const plan = buildPiModelConfigPlan(completeRequest(), completeSecrets); + assert.deepEqual(plan, { + providerId: "my-ollama", + providerFamily: "openai", + api: "openai-completions", + baseUrl: "https://example.test/v1", + apiKeyEnv: "OPENAI_API_KEY", + models: [{ id: "qwen2.5-coder:7b" }], + }); + }); + + it("applies to pi_agenta as well as pi_core, and to an empty (defaulted) harness", () => { + assert.ok( + buildPiModelConfigPlan( + completeRequest({ harness: "pi_agenta" }), + completeSecrets, + ), + ); + assert.ok( + buildPiModelConfigPlan( + completeRequest({ harness: undefined }), + completeSecrets, + ), + ); + }); + + it("holds only the env var NAME — never the raw key value", () => { + const plan = buildPiModelConfigPlan(completeRequest(), completeSecrets); + assert.ok(plan); + assert.equal(plan.apiKeyEnv, "OPENAI_API_KEY"); + assert.equal(JSON.stringify(plan).includes(RAW_KEY), false); + }); +}); + +describe("buildPiModelConfigPlan (non-applicable -> no plan, current behavior)", () => { + it("returns no plan for a Claude request", () => { + assert.equal( + buildPiModelConfigPlan( + completeRequest({ harness: "claude" }), + completeSecrets, + ), + undefined, + ); + }); + + it("returns no plan for a standard Pi (direct) request", () => { + assert.equal( + buildPiModelConfigPlan( + completeRequest({ deployment: "direct", connection: undefined }), + completeSecrets, + ), + undefined, + ); + }); + + it("returns no plan for a subscription (runtime_provided) request", () => { + // A subscription run is direct + self-managed, not a named custom connection. + assert.equal( + buildPiModelConfigPlan( + completeRequest({ + deployment: "direct", + credentialMode: "runtime_provided", + connection: { mode: "default" }, + }), + {}, + ), + undefined, + ); + }); + + it("returns no plan when the provider family is not openai", () => { + assert.equal( + buildPiModelConfigPlan( + completeRequest({ provider: "anthropic" }), + completeSecrets, + ), + undefined, + ); + }); + + it("returns no plan when the connection is not a named agenta connection", () => { + assert.equal( + buildPiModelConfigPlan( + completeRequest({ connection: { mode: "default", slug: "x" } }), + completeSecrets, + ), + undefined, + ); + assert.equal( + buildPiModelConfigPlan( + completeRequest({ connection: undefined }), + completeSecrets, + ), + undefined, + ); + }); +}); + +describe("buildPiModelConfigPlan (applicable but incomplete -> typed error)", () => { + const cases: Array<{ + name: string; + over: Partial; + secrets?: Record; + hint: RegExp; + }> = [ + { + name: "empty connection slug", + over: { connection: { mode: "agenta", slug: " " } }, + hint: /connection slug/, + }, + { + name: "missing endpoint base URL", + over: { endpoint: { baseUrl: " " } }, + hint: /base URL/, + }, + { + name: "credential mode is not env", + over: { credentialMode: "none" }, + hint: /credential mode "env"/, + }, + { + name: "OPENAI_API_KEY absent from the secret set", + over: {}, + secrets: {}, + hint: /OPENAI_API_KEY/, + }, + { + name: "OPENAI_API_KEY present but blank", + over: {}, + secrets: { OPENAI_API_KEY: " " }, + hint: /OPENAI_API_KEY/, + }, + { + name: "no model id", + over: { model: " " }, + hint: /model id/, + }, + ]; + + for (const { name, over, secrets, hint } of cases) { + it(`throws a typed error when ${name} (never a silent no-op)`, () => { + assert.throws( + () => + buildPiModelConfigPlan( + completeRequest(over), + secrets ?? completeSecrets, + ), + (err: unknown) => { + assert.ok(err instanceof PiModelConfigError); + assert.match(err.message, hint); + // Fail-loud, never a silent fall-back. + assert.match(err.message, /stopped/); + return true; + }, + ); + }); + } + + it("never leaks the raw key in the incomplete-request error message", () => { + try { + buildPiModelConfigPlan(completeRequest({ model: "" }), completeSecrets); + assert.fail("expected an incomplete-request error"); + } catch (err) { + assert.ok(err instanceof PiModelConfigError); + assert.equal(err.message.includes(RAW_KEY), false); + } + }); +}); + +describe("serializePiModelsJson (exact shape, no key leak)", () => { + it("serializes the exact Pi models.json document keyed by slug", () => { + const plan = buildPiModelConfigPlan(completeRequest(), completeSecrets); + assert.ok(plan); + const text = serializePiModelsJson(plan); + + assert.deepEqual(JSON.parse(text), { + providers: { + "my-ollama": { + baseUrl: "https://example.test/v1", + api: "openai-completions", + apiKey: "$OPENAI_API_KEY", + models: [{ id: "qwen2.5-coder:7b" }], + }, + }, + }); + // The file references the env var, never the raw key. + assert.equal(text.includes("$OPENAI_API_KEY"), true); + assert.equal(text.includes(RAW_KEY), false); + // Trailing newline for a well-formed file. + assert.equal(text.endsWith("\n"), true); + }); +}); diff --git a/services/runner/tests/unit/session-pool.test.ts b/services/runner/tests/unit/session-pool.test.ts index c7e8d4ce58..1d8cf921c6 100644 --- a/services/runner/tests/unit/session-pool.test.ts +++ b/services/runner/tests/unit/session-pool.test.ts @@ -248,6 +248,42 @@ describe("configFingerprint", () => { configFingerprint({ ...base, tools: ["read"] }), ); }); + + // Custom OpenAI-compatible warm-session safety (design Decision 7): a change to the connection, + // model, or endpoint must cold-start rather than reuse a mismatched live session. No new + // fingerprint field is needed — these already ride configFingerprint. + it("changes when the connection changes (custom provider identity)", () => { + const withConn = { + ...base, + provider: "openai", + deployment: "custom", + connection: { mode: "agenta", slug: "ollama-a" }, + endpoint: { baseUrl: "https://a.test/v1" }, + }; + assert.notEqual( + configFingerprint(withConn), + configFingerprint({ + ...withConn, + connection: { mode: "agenta", slug: "ollama-b" }, + }), + ); + }); + + it("changes when the endpoint base URL changes", () => { + const withEndpoint = { + ...base, + deployment: "custom", + connection: { mode: "agenta", slug: "ollama-a" }, + endpoint: { baseUrl: "https://a.test/v1" }, + }; + assert.notEqual( + configFingerprint(withEndpoint), + configFingerprint({ + ...withEndpoint, + endpoint: { baseUrl: "https://b.test/v1" }, + }), + ); + }); }); describe("historyFingerprint (pruned-array contract)", () => { @@ -412,6 +448,20 @@ describe("credential epoch", () => { ); }); + it("a rotated OPENAI_API_KEY invalidates the epoch (custom provider key rotation)", () => { + // Rotating the custom OpenAI-compatible connection's key for the same slug must cold-start + // with the fresh key rather than reuse a warm session baked with the old one (design + // Decision 7 — the credential epoch already covers this; no new key is needed). + const parked = computeCredentialEpoch({ + secrets: { OPENAI_API_KEY: "sk-old" }, + }); + const rotated = computeCredentialEpoch({ + secrets: { OPENAI_API_KEY: "sk-new" }, + }); + assert.notEqual(parked.secretsHash, rotated.secretsHash); + assert.equal(credentialEpochValid(parked, rotated, Date.now()), false); + }); + it("a re-minted tool-callback bearer does NOT change the hash (per-turn material)", () => { // The backend re-mints the callback bearer on its auth-cache cadence (~60s); the turn's // relay always uses the incoming bearer, so a warm continue must not evict over it. @@ -587,10 +637,7 @@ describe("SessionPool", () => { ); // Only the awaiting_approval entry matches, and only by its own session id. assert.equal(pool.awaitingApproval("s-idle"), undefined); - assert.equal( - pool.awaitingApproval("s-gated")?.environment, - parked.env, - ); + assert.equal(pool.awaitingApproval("s-gated")?.environment, parked.env); assert.equal(pool.awaitingApproval("s-unknown"), undefined); }); @@ -654,11 +701,9 @@ describe("SessionPool", () => { stoppingEnv.state.reasons.push(reason); }, }; - const pool = new SessionPool( - { poolMax: 1 }, - () => {}, - { strictCapacity: true }, - ); + const pool = new SessionPool({ poolMax: 1 }, () => {}, { + strictCapacity: true, + }); await pool.park(parkInput("a", stoppingEnv).input, 10_000); const replacement = parkInput("b"); @@ -673,17 +718,19 @@ describe("SessionPool", () => { releaseTeardown?.(); assert.equal(await parked, true); - assert.equal(teardownCompleted, true, "teardown completes before park resolves"); + assert.equal( + teardownCompleted, + true, + "teardown completes before park resolves", + ); assert.equal(pool.get("a"), undefined); assert.equal(pool.get("b")?.state, "idle"); }); it("strict capacity returns false at cap when no idle entry exists", async () => { - const pool = new SessionPool( - { poolMax: 1 }, - () => {}, - { strictCapacity: true }, - ); + const pool = new SessionPool({ poolMax: 1 }, () => {}, { + strictCapacity: true, + }); const busy = parkInput("busy"); await pool.park(busy.input, 10_000); pool.checkoutIdle("busy"); @@ -696,16 +743,10 @@ describe("SessionPool", () => { }); it("strict approval checkout stays seated while it is busy", async () => { - const pool = new SessionPool( - { poolMax: 1 }, - () => {}, - { strictCapacity: true }, - ); - await pool.park( - parkInput("approval").input, - 10_000, - "awaiting_approval", - ); + const pool = new SessionPool({ poolMax: 1 }, () => {}, { + strictCapacity: true, + }); + await pool.park(parkInput("approval").input, 10_000, "awaiting_approval"); const live = pool.checkoutApproval("approval"); @@ -725,11 +766,9 @@ describe("SessionPool", () => { releaseTeardown = resolve; }), }; - const pool = new SessionPool( - { poolMax: 1 }, - () => {}, - { strictCapacity: true }, - ); + const pool = new SessionPool({ poolMax: 1 }, () => {}, { + strictCapacity: true, + }); await pool.park(parkInput("a", environment).input, 10_000); const stopping = pool.get("a")!; const replacement = pool.park(parkInput("b").input, 10_000); @@ -739,14 +778,22 @@ describe("SessionPool", () => { assert.equal(pool.checkoutIdle("a"), undefined); assert.equal(pool.checkoutApproval("a"), undefined); assert.equal( - await pool.repark(stopping, { - configFingerprint: "new", - historyFingerprint: "new", - credentialEpoch: epoch, - }, 10_000), + await pool.repark( + stopping, + { + configFingerprint: "new", + historyFingerprint: "new", + credentialEpoch: epoch, + }, + 10_000, + ), false, ); - assert.equal(pool.get("a"), stopping, "repark does not clobber the seated stop"); + assert.equal( + pool.get("a"), + stopping, + "repark does not clobber the seated stop", + ); releaseTeardown?.(); assert.equal(await replacement, true);