docs(design): plan warm and resumable Daytona sessions (F-020)

[mmabrouk]

What a user sees today

When you chat with an agent that runs on Daytona, every message waits about twenty seconds before the agent starts to answer. The answer itself is correct, because durable continuity replays the conversation into a fresh sandbox, but every turn pays the full setup time. This is QA finding F-020.

What we measured (new in this revision)

A credit-controlled measurement against the Daytona API (2026-07-11, two repetitions, all sandboxes deleted afterwards) changed the plan's shape:

• Creating a sandbox takes 1.2 to 1.7 seconds, and starting a stopped one takes under a second. So the twenty-second turn is not sandbox creation; it is our own per-turn pipeline (daemon startup, asset upload, mounts, harness startup, session reload). Only a sandbox that stays running skips that pipeline.
• Restoring an archived sandbox (33 to 66 seconds) is slower than creating a fresh one, and the disk archive frees costs under a tenth of a cent per hour. Archive is dropped from the design.
• A parked running sandbox costs about $0.0028 per minute. A 60-second keep-warm window costs about a third of a cent per turn.

Full numbers and prices are in research.md.

What this PR proposes

One progressive sequence of five slices, each shippable on its own, ending with warm running sandboxes:

1. Correctness base, everything behind a default-off flag. The two Daytona provider functions the runner already calls (pause, reconnect), two teardown leak fixes in the vendored package, guarded sandbox-pointer writes, and a compatibility fingerprint.
2. Park-to-stopped. Stop the sandbox at a clean turn end instead of deleting it; restart the same one next turn. Parked cost is disk storage only (about $0.0009/hour).
3. Provider-aware pool refactor. The existing local keepalive pool (session-pool.ts) gets per-provider configuration and typed teardown reasons instead of the local-only gate. Local behavior is preserved; no second pooling mechanism.
4. Park-to-running. Keep the sandbox running for a short window after each turn (default 60 seconds, hard cap of 4 concurrently running sandboxes, enforced by an admission gate before creation, not by the pool).
5. Live verification, then flip the flags. Park-to-stopped and park-to-running graduate independently.

Pending approvals follow the F-018 gate plan's resume model (daytona-gate-delivery/): below park-to-running a pending approval always takes the cold path, because a stopped sandbox kills the waiting process.

What the design reviews changed

Two adversarial review rounds ran. Round 1 found the teardown leaks and races now in the must-fix list. Round 2 reviewed the five-slice reshape and found two design properties that were false as drafted, both fixed: the pool alone cannot enforce the running cap (fresh misses create sandboxes before the pool sees them; the cap now lives in an admission gate taken before creation), and a single eviction disposition cannot implement the teardown matrix (teardown reasons are now typed). status.md records both rounds in full.

Decisions that still need you

• Where the compare-and-set guard on the stored sandbox id lives (open-questions.md item 1).
• The shutdown split: delete an in-flight turn, stop an idle one (item 2).
• Queue briefly or reject when the running cap is saturated (item 3).
• Two proposed defaults to confirm: the 15-minute stop timer, and the 60-second window with cap 4 (items 5 and 6).

How to read it

Start with README.md (reading order), then context.md for the story. The inline comments on this PR mark the exact decision points.

https://claude.ai/code/session_018MaXPNpvzN22kngHno3VMj

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	Jul 11, 2026 2:07pm

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3e539d84-b0c2-4f91-b255-1c03e3246c95

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely summarizes the design docs for warm and resumable Daytona sessions.
Description check	✅ Passed	The description is detailed and directly matches the design-only warm Daytona sessions plan in the PR.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch docs/warm-daytona-sessions-plan

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mmabrouk]

Feedback needed on three decisions before implementation starts: (1) the crash-orphan compute budget and the DAYTONA_AUTOSTOP value (5 min can stop a sandbox under a live long tool call; the old default was 15) — needs the billing owner; (2) where the sandbox-pointer fence lives (session_states generation column vs the existing turn counter vs the Redis owner claim) — see open-questions.md #3; (3) whether Tier 2 should ship at all before Tier 1's real second-turn latency is measured on E3. Everything else in plan.md is ready to implement once these are settled.

[coderabbitai]

Actionable comments posted: 5

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 62da98bc-f563-4773-a4f5-c26e6426585e

📥 Commits

Reviewing files that changed from the base of the PR and between e6f7962 and 07accab.

📒 Files selected for processing (6)

• docs/design/agent-workflows/projects/warm-daytona-sessions/README.md
• docs/design/agent-workflows/projects/warm-daytona-sessions/context.md
• docs/design/agent-workflows/projects/warm-daytona-sessions/open-questions.md
• docs/design/agent-workflows/projects/warm-daytona-sessions/plan.md
• docs/design/agent-workflows/projects/warm-daytona-sessions/research.md
• docs/design/agent-workflows/projects/warm-daytona-sessions/status.md

[coderabbitai]

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Make the auto-stop decision a Tier 1 enablement blocker.

With a 5-minute auto-stop and a 300-second maximum silent interval, Daytona can stop a still-live tool call or approval wait. Before enabling Tier 1, either set auto-stop above the maximum silent interval with margin or add a reliable keepalive mechanism and test this boundary explicitly.

[coderabbitai]

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Do not clear the stored ID for every reconnect failure.

Only confirmed terminal cases such as not-found, deleted, or unrecoverable error states should clear the pointer. Transient API failures and transitional-state timeouts should retain it for retry; otherwise one temporary failure forces a cold create and can overwrite a still-recoverable sandbox.

[coderabbitai]

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Bound and verify stop-before-replacement.

Awaiting stop() is insufficient if the provider call hangs or returns before the sandbox reaches stopped. Add a deadline, poll/reconcile the actual state, and admit a replacement only after the old sandbox is stopped or deleted; otherwise MAX_RUNNING cannot reliably cap billed sandboxes.

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Show the relevant section with line numbers.
sed -n '170,220p' docs/design/agent-workflows/projects/warm-daytona-sessions/research.md

printf '\n---\n'

# Search for measurement / latency claims in the document.
rg -n "measure|measured|latenc|resume cost|about 1s|session/load|mount" docs/design/agent-workflows/projects/warm-daytona-sessions/research.md

Repository: Agenta-AI/agenta

Length of output: 5295

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

sed -n '130,160p' docs/design/agent-workflows/projects/warm-daytona-sessions/research.md

Repository: Agenta-AI/agenta

Length of output: 2006

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Look for any explicit "Phase 3" measurement requirement or latency estimate caveats nearby.
rg -n "Phase 3|measure|measured|estimate|about 1s|resume|restart path|latency" docs/design/agent-workflows/projects/warm-daytona-sessions/research.md

Repository: Agenta-AI/agenta

Length of output: 1374

---

Qualify the “about 1s” resume estimate

The Stopped (warm) row presents a concrete resume latency, but the surrounding text doesn’t show an end-to-end measurement for stop/start + remount + session/load. Mark it as an estimate or remove the number until that path is measured.

[coderabbitai]

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Separate Tier 1 blockers from Tier 2 prerequisites.

The status currently lists the Tier 2 TTL and running cap as a shipping gate, while plan.md explicitly defers Tier 2 and recommends enabling Tier 1 first. Split this into “Tier 1 enablement blockers” and “Tier 2 prerequisites” so approval of the first rollout is unambiguous.

[mmabrouk]

Navigation aids for the rewrite. These inline notes mark where to start, what each key section answers, and the one or two decisions that need a human. The design and every technical claim are unchanged from the previous version; only the structure, vocabulary, and ordering moved.

[mmabrouk]

Start reading here. This lists the five files in order and says what each one answers, so you can stop at the first that gives you what you need.

[mmabrouk]

The story starts here: what a user actually experiences today (every Daytona turn waits about twenty seconds), before any mechanism or jargon.

[mmabrouk]

This section answers the 'so what is actually wrong' question in one paragraph: the warm-reuse code is already written, and only two missing Daytona functions plus a few gaps block it.

[mmabrouk]

This is the code-level evidence for that key finding. It names the exact three consequences of the two missing provider functions.

[mmabrouk]

The proposal in one screen: park-to-stopped (cheaper, ships first) and park-to-running (pricier, deferred). Read just this if you only want the shape.

[mmabrouk]

This section answers why the recommendation is 'flag off, then turn on after a live test' and not 'on by default'. These are the correctness gaps the design review found.

[mmabrouk]

This is the decision to weigh in on: ship park-to-stopped behind a default-off flag now, and defer park-to-running until a billing owner sets its cost limits.

[mmabrouk]

This is the decision that most needs you or a billing owner. With the sandbox no longer auto-deleted on stop, a crashed runner can bill compute for up to five minutes. Acceptable, or should the stop timer drop?

[mmabrouk]

Reading guide for this revision

Reading order: README.md (orientation and the one-paragraph recommendation) then context.md (the story) then plan.md (the decision; slices at the bottom) then open-questions.md (what needs you) then status.md (decision log and both review rounds). research.md is the evidence file; dip into it when a claim in plan.md needs backing.

What changed since the version you last saw:

• Real numbers replaced estimates. A measurement against the Daytona API (two sandboxes, created and deleted): create 1.2 to 1.7 s, start from stopped 0.7 to 0.8 s, restore from archive 33 to 66 s. Prices: running about $0.17/hour, stopped about $0.0009/hour for our shape. Tables in research.md.
• Archive is gone. Restoring from archive is slower than creating fresh, so the demotion ladder is stop, then delete.
• Park-to-running moved from "deferred" into the main line (your direction), as slices 3 and 4: the local keepalive pool refactored provider-aware, then a Daytona live window (default 60 s, hard cap 4) with a capacity admission gate that enforces the cap before sandbox creation.
• The approval story is bound to the F-018 gate plan (daytona-gate-delivery/): below park-to-running, a pending approval always re-asks the model and answers from the stored decision, exactly like the local cold case; holding a gate open live needs park-to-running plus that plan's file-transport parked-gate variant.
• A second codex review ran on the reshape. It confirmed the slice ordering and found two real gaps, both folded in: cap enforcement moved out of the pool into an admission gate, and teardown got typed reasons. status.md has the full list.

Open for you (inline comments below mark the exact lines): the pointer-write guard location, the shutdown split, saturation behavior of the capacity gate (queue or reject), and two defaults to confirm (15-minute stop timer; 60-second window with cap 4).

[mmabrouk]

Decision: stop timer 15 minutes (was 5). Cost of the longer timer is about 4 cents per crash orphan versus 1.4; the gain is never stopping a sandbox under a live silent turn. Confirm 15, or pick another value.

[mmabrouk]

Decision: the park-to-running defaults. Window 60 seconds (about a third of a cent per parked turn), cap 4 (worst case about $0.67/hour). Both env-tunable later, so this only sets the shipping default.

[mmabrouk]

Decision: when the cap is saturated by active turns, queue briefly or reject with a clear error? Queue is friendlier, reject is simpler and easier to observe. open-questions.md item 3.

[mmabrouk]

Note the disposition here: pool eviction stops the sandbox (drops it to park-to-stopped), never deletes it, and is awaited so the cap holds. Object here if you want evictions to delete instead.

[mmabrouk]

The approval rule bound to the F-018 gate plan: below park-to-running, a pending approval always ends the turn and re-asks the model next turn (the stored decision answers it). Byte-exact gate holding waits for park-to-running plus the gate plan's file-transport parked-gate variant. Confirm this sequencing is acceptable.

[mmabrouk]

Reviewer decision 1: where the compare-and-set guard on the stored sandbox id lives (new generation column, existing turn counter, or the Redis owner claim from #5197). One mechanism should probably also cover #5197's unguarded continuity write.

[mmabrouk]

Reviewer decision 2: shutdown split. Delete when a turn is in flight (partial transcript), stop when idle; /kill stays a hard delete. Confirm.

[mmabrouk]

Slight question: where does this Project 3D come from? I had a previous version where it came with a mount, but I wanted to refactor this. I just wanted to know whether it's still the case.

[mmabrouk]

It is still both sources, in order. The preferred source is the project id the services layer stamps on the run context; the mount's owning project is only the fallback when that is missing. I clarified the sentence in research.md so the source is stated where the key is introduced.

[mmabrouk]

Update: the full where-does-the-time-go measurement is now in the PR

The earlier revision had only the sandbox-lifecycle numbers (create ~1.5 s, start from stopped ~0.8 s), which told you the sandbox is cheap but not where the fifteen seconds actually goes. This push adds the full stage split from three real E3 chat runs plus micro-measurements inside the runner container, and recosts both park levels from it.

What changed:

• research.md has a new section, "Where the time goes in a real turn": a stage table for the cold Daytona Pi turn (~15 s wall, ~12.3 s our pipeline), the local Pi and local Claude contrast numbers, and the log lines the runner is missing to repeat this without hand instrumentation.
• The two dominant stages are fixable outside this plan: a redundant ~5.2 s Pi install (the snapshot already bakes the CLI; the skip is live on the dev sidecar, E3 smoke dropped to 12.2 s) and ~2 s of pi-acp version probes inside the harness spawn (removal planned in PR [parked] docs(design): plan removal of cold-turn startup costs (pi-acp probes, eager mount) #5221). Together ~7.2 s.
• The park-level savings are now stated from measurement everywhere the plan reasons about them: park-to-stopped saves about 1 second (a restarted sandbox still pays the ~5.15 s harness respawn plus mounts); park-to-running takes the turn to an estimated 2 to 3 seconds (model plus small overheads).
• Slice 5 now starts by adding the missing duration log lines and re-measures against this baseline.
• All files updated from "about twenty seconds" to the measured numbers.

Inline comments on this commit mark the stage table and the two savings statements so you can react line by line.

[mmabrouk]

The measured stage split for a cold Daytona Pi turn (three real E3 runs, 2026-07-11). The two big rows are the redundant Pi install (skip already live on the dev sidecar) and the harness spawn (~2 s of it is pi-acp probes, removal in PR #5221).

[mmabrouk]

The headline: ~12.3 s of the ~15 s turn is our pipeline, and ~7.2 s of that falls to the two config/patch fixes independent of this plan. The lifecycle work in this plan competes with the ~5 s remainder.

[mmabrouk]

The recosted savings, from measurement: park-to-stopped saves about 1 second (a stopped restart still pays the ~5.15 s harness respawn plus mounts); park-to-running takes the turn to an estimated 2 to 3 seconds. This is the strongest evidence for your call to not defer park-to-running.

[mmabrouk]

Park-to-stopped's win, restated from measurement: about 1 second out of a ~10 s pipeline once the install skip is live. Its value is as the eviction target and correctness base, not as a latency fix. React here if that changes how much of slices 1-2 you want to fund.

[mmabrouk]

Park-to-running's target number: a resumed turn at roughly 2 to 3 seconds against today's ~15 s, estimated from the stage split and confirmed in Slice 5.

[mmabrouk]

I honestly don't understand this point: this fire-and-forget, last-writer-wins, and go. I really don't have the context to make a decision here. When does this happen? What are the trade-offs of building this fix, etc., etc.? I remember from the F-018 PR comment that this is supposed to be quite complex and requires re-architecting, so I really want to understand its requirements before jumping to an implementation.

[mmabrouk]

Rewrote this item so it stands alone (plan.md must-fix item 3). Short version: the runner records which sandbox belongs to a conversation in a database row. It bites when two turns race on one conversation (a second message before the first turn finishes, or cancel and immediately re-send): the older write can land last and point the row at a dying sandbox. The cost is one doomed reconnect next turn plus an orphaned sandbox that bills cents until the timers reap it. The fix is small: wait for the write and add a compare-and-set so an older turn cannot overwrite a newer one. It is plumbing in one function plus one API check, not the re-architecture the F-018 gate needed. The only open choice is which counter to compare against; open-questions item 1 now lists the three candidates with trade-offs.

[mmabrouk]

From the research above, we will never ever use AutoArchive. It doesn't make sense to have the AutoArchive, so we should remove this whole logic.

[mmabrouk]

Done. The plan now removes the auto-archive logic entirely: the create call drops the autoArchiveInterval field and the DAYTONA_AUTOARCHIVE override with it. Daytona's own archive default is 7 days, far past our 30-minute delete, so archive can never fire.

[mmabrouk]

I think it makes sense. We just need to make sure to have these editable as environment variables at the end of the day. Even if we don't do the plumbing right now end to end, it needs to be easy later to expose these and modify these as you wish.

[mmabrouk]

Recorded as confirmed: 15 minutes. It is already an env override (DAYTONA_AUTOSTOP, whole minutes) and stays one; the delete timer (DAYTONA_AUTODELETE) too. The plan text now says all timer values stay env-overridable.

[mmabrouk]

What is the smell overhead? Why do we have small overhead if it's already running?

[mmabrouk]

Named them in the text now. Even with the sandbox up, a turn still pays the hop through the services layer (measured 100 to 220 ms locally), the pool checkout with its validation (config and history fingerprints, credential expiry), and forwarding the prompt over the open session. That plus the model's own generation time is the 2 to 3 seconds.

[mmabrouk]

Update: review comments addressed

All fifteen of your comments are answered in their threads, and the documents changed accordingly. The substance:

• The warm cap follows your model. It bounds idle warm sandboxes only (default 20, its own env var); active turns are never blocked, and when every slot is taken a finishing turn parks to stopped, which is the stop-and-restart path you described. Queue-or-reject is gone.
• No feature flags. The off switch is the keep-alive time-to-live: 0 disables, default is two minutes per your call. Defaults stay at today's behavior until the live verification passes; the final slice changes the defaults.
• Auto-archive is removed entirely, not configured around.
• Stop timer 15 minutes confirmed, env-overridable, and the text now answers your reset question: every message resets it, so an active conversation never hits it.
• The two open decisions are now self-contained in open-questions.md (the pointer-write guard with its three candidate mechanisms, and the shutdown split), each with the scenario and trade-offs needed to decide from the file alone.
• Leak safety is a named plan item: the cap, the create-time self-reaping timers, the no-recurring-pings rule, and a zero-live-sandboxes verification check.

Also applied your two writing corrections to the passages touched in this pass: provenance and status narration moved out of the document bodies (they live in these comments), and changes are just made in place with an inline comment at the spot.

[mmabrouk]

5197 has been already merged. and yes it makes sense let\s go with the simple solution that does not require any schema migration

[mmabrouk]

makes sense

[mmabrouk]

ok