docs(design): sandbox network story: tunnels today, NetBird overlay plan#5219
docs(design): sandbox network story: tunnels today, NetBird overlay plan#5219mmabrouk wants to merge 2 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| no channel back to the runner. Every feature that needs the reverse direction, or needs the | ||
| sandbox to reach a runner-side service, has grown its own workaround: | ||
|
|
||
| 1. **Mounts ride a public ngrok tunnel.** Durable sessions mount the S3 store as the |
There was a problem hiding this comment.
so if i understand correctly we open a reverse proxy to our local private docker network for seeweedfs iwht a public URL (unauthenticated). is that correct?
There was a problem hiding this comment.
🤖 The AI agent says: Yes at the network layer, but not at the storage layer. ngrok publishes the SeaweedFS S3 and STS endpoint. Its authtoken authenticates the tunnel agent, not callers of the public URL. Agenta still requires S3 authentication and mints short-lived credentials restricted to one mount prefix. The revised recommendation now separates public reachability from anonymous access and calls out the added network attack surface.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
docs/design/agent-workflows/projects/netbird-sandbox-network/research.md (1)
338-347: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winCall out the multipart-size check before standardizing quick tunnels.
docs/design/agent-workflows/projects/netbird-sandbox-network/research.md: add multipart PUT/GET acceptance tests for the S3 path.docs/design/agent-workflows/projects/netbird-sandbox-network/recommendation.md: make the default-path rollout explicitly contingent on that run.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e01cd152-6395-40a5-8657-0e402a6c8b54
📒 Files selected for processing (4)
docs/design/agent-workflows/projects/netbird-sandbox-network/README.mddocs/design/agent-workflows/projects/netbird-sandbox-network/plan.mddocs/design/agent-workflows/projects/netbird-sandbox-network/recommendation.mddocs/design/agent-workflows/projects/netbird-sandbox-network/research.md
| 2. **MVP behind a flag, about 1 week.** Daytona only, default off. The runner mints a | ||
| per-run ephemeral setup key and injects it into the sandbox env; the sandbox joins on | ||
| boot; the geesefs mount targets the store at the runner's overlay address. NetBird cloud | ||
| as the control plane first; the self-hosted control plane (three services: management, | ||
| signal, relay) follows for self-hosters. | ||
| 3. **ACL hardening, 1 to 2 weeks.** Default-deny policy generated per run: a sandbox reaches | ||
| only the runner ports its run needs, never another sandbox. Key and peer revocation on | ||
| teardown. Fail-loud when the overlay does not come up (the F-017 lesson: never a silent | ||
| fallback that hangs). Observability and self-hosting docs. |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
The MVP must include the minimum ACL boundary. Enabling untrusted sandbox peers before deny-by-default, runner-only, port-scoped policies are in place contradicts the security model and creates an unsafe rollout window.
docs/design/agent-workflows/projects/netbird-sandbox-network/plan.md#L96-L104: move baseline ACL enforcement into Phase 1.docs/design/agent-workflows/projects/netbird-sandbox-network/research.md#L173-L198: define the mandatory baseline policy and teardown behavior.docs/design/agent-workflows/projects/netbird-sandbox-network/recommendation.md#L59-L64: require those controls before the flagged MVP can be enabled.
📍 Affects 3 files
docs/design/agent-workflows/projects/netbird-sandbox-network/plan.md#L96-L104(this comment)docs/design/agent-workflows/projects/netbird-sandbox-network/research.md#L173-L198docs/design/agent-workflows/projects/netbird-sandbox-network/recommendation.md#L59-L64
There was a problem hiding this comment.
🤖 The AI agent says: Agreed. There is no later ACL-hardening phase now. If NetBird is reopened, the first sandbox cannot enroll until the all-to-all default policy is removed and a one-way, paired-peer, TCP and port-scoped policy exists. The same first MVP includes explicit teardown for peers, policies, groups, listeners, and keys.
There was a problem hiding this comment.
@mmabrouk, that resolves the security concern cleanly — folding ACL hardening into the MVP phase itself (rather than deferring it) removes the unsafe rollout window where untrusted sandbox peers could exist without a deny-by-default policy. The explicit teardown requirement for peers, policies, groups, listeners, and keys also closes the resource-leak/orphaned-access gap that a phased approach would have left open temporarily. Thanks for confirming — I'll treat this as the direction for plan.md, research.md, and recommendation.md once they're updated to reflect the merged phase.
✏️ Learnings added
Learnt from: mmabrouk
Repo: Agenta-AI/agenta PR: 5219
File: docs/design/agent-workflows/projects/netbird-sandbox-network/plan.md:0-0
Timestamp: 2026-07-11T15:48:02.814Z
Learning: In the `docs/design/agent-workflows/projects/netbird-sandbox-network/` design docs (plan.md, research.md, recommendation.md) describing the NetBird WireGuard overlay for Daytona sandbox networking, there is no separate later "ACL hardening" phase. The MVP (first phase) itself must include: removal of NetBird's all-to-all default policy, a one-way paired-peer TCP and port-scoped policy before the first sandbox can enroll, and explicit teardown of peers, policies, groups, listeners, and keys. Untrusted sandbox peers must never be enrolled before these deny-by-default, runner-only, port-scoped controls exist.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
| Option C), and carry the residual ask-gates over the same polled file relay (its | ||
| Option B narrowed). | ||
| 4. **MCP delivery is swapped out on Daytona.** Claude only accepts tools over MCP, so the | ||
| runner synthesizes an MCP server. It binds the runner's `127.0.0.1`, which inside a |
There was a problem hiding this comment.
I want to understand more: what's the state of this implementation of MCP, both in local and in Daytona, for Pi and for Cloudflare? For an HTTP MCP, I guess we don't have that even implemented for either case, correct me if I'm wrong. For here, what you're talking about is probably the gateway tools MCP. You're saying that this thing we do create in the runner works on localhost because they're in the same container, but in the case of Daytona, that doesn't work because it's in a different place. Is that accurate ?
There was a problem hiding this comment.
🤖 The AI agent says: Your distinction is right, interpreting “Cloudflare” here as Claude. User remote HTTP MCP is already implemented for Claude locally and on Daytona when the HTTPS URL passes the SSRF guard and sandbox egress reaches it. Pi rejects user MCP, and stdio MCP is disabled for every harness. The loopback issue applies to Agenta gateway/custom-tool MCP: local Claude reaches the runner's 127.0.0.1 server, while Daytona Claude cannot, so that combination is rejected. Pi uses its extension and file relay instead. The new matrix documents all four cases.
| ([quick tunnels docs](https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/do-more-with-tunnels/trycloudflare/)). | ||
| The tunnel is therefore always up on every dev box and self-hosted deployment, which removes | ||
| exactly the F-017 failure mode. The trade-aways are stated plainly in | ||
| [research.md](research.md): no SLA, testing-only intent, 200 in-flight requests, one edge |
There was a problem hiding this comment.
Yeah, I think the 200 in-flight requests makes it not very usable, even in dev. How does it compare to ngrok and the number of in-flight requests?
There was a problem hiding this comment.
🤖 The AI agent says: The 200 figure is simultaneous in-flight requests, not requests per minute. ngrok does not document the same concurrency cap; its free limits are instead 4,000 requests per minute, 20,000 per month, and 1 GB outbound transfer per month. GeeseFS can issue parallel uploads, reads, and metadata calls, so 200 may fit one light dev mount but is not enough evidence for a default. The revised plan makes concurrent mounts and multipart traffic a rollout gate.
|
|
||
| The concrete change, one to two days (**UNVERIFIED**): | ||
|
|
||
| - `mount.ts`: `discoverTunnelEndpoint` gains a cloudflared branch that queries |
There was a problem hiding this comment.
I'm surprised that mount.ts does not need to have any knowledge of whether we're using Cloudflare or using ngrok. It seems to me that the mount would just get a URL, and where this URL and hostname come from is not something that it's responsible for. Can you explain how it's implemented right now and why we have even a discovered channel endpoint? Why don't we just use the URL of wherever the seaweedfs is?
There was a problem hiding this comment.
🤖 The AI agent says: Your model is correct. geesefs only needs an effective endpoint URL. Today the API returns the configured S3 endpoint; public endpoints go straight to geesefs, while a private address such as seaweedfs:8333 triggers ngrok discovery because ngrok assigns the public hostname dynamically. The revised plan moves that provider-specific step behind resolveSandboxStoreEndpoint, with direct endpoint, explicit public override, and dynamic sidecar sources. The mount primitive stays provider-neutral.
| vars). ngrok stays available for anyone who already holds a token. | ||
| - No sandbox-side change: geesefs takes whatever endpoint URL discovery returns. | ||
|
|
||
| Production (Agenta Cloud) is unaffected: its store is genuinely public and uses no tunnel. |
There was a problem hiding this comment.
I don't want to understand, actually, depending on the differences in production and non-production. What's the difference between using seaweedfs in production or not? Are they both authenticated, etc., etc., etc., or are there also differences in the authentication? How does this authentication work? Basically, can I use them with S3 or also without in production? If I use them without, do they stay kind of usable, and can they be public?
There was a problem hiding this comment.
🤖 The AI agent says: The relevant distinction is endpoint topology, not production versus non-production. Bundled SeaweedFS uses S3 plus web-identity STS; external S3-compatible storage uses its STS path. In both cases the API mints short-lived credentials scoped to one mount prefix and geesefs signs S3 requests. A public endpoint is routable but not anonymously readable under Agenta's configuration. Current durable mounts require an S3-compatible endpoint; they do not use SeaweedFS's other protocols. A private endpoint needs a tunnel, overlay, or another route from Daytona.
|
|
||
| ## Longer term: the NetBird overlay | ||
|
|
||
| The overlay puts the runner and each sandbox on one private WireGuard mesh so each can dial |
There was a problem hiding this comment.
If I understand correctly, the solution that NetBird brings is for two reasons:
- The mounts. If you are using mounts and seaweedfs is not public, then you need a reverse proxy.
- Now assume you don't have that problem. What do you gain? I think what you gain is that now the sandbox can talk to the runner, but then we said there is a proxy or something already in Daytona. I don't understand why the sandbox cannot talk to the runner already right now. How do they actually talk with an ACP, etc.? Is the issue that right now they talk through this kind of SSE, so the runner opens the connection, and then everything goes through that? That is not working fine, as we hope it is.
I want to understand: when is it that the sandbox needs to reach the runner, and why is it not happening right now? Do we need that bird? What are the actions that the sandbox would say? "Actually, I need to call the runner for this." I guess permissions, yes, but what else? If it's in permissions, does the ACP, in the fact that you have an open connection, if I understand, not already solve this problem? Maybe also two calls, but then again, probably with two calls if you all have an open connection, and then you say, "I want a tool call." You wait, and then the runner does what is needed to call the tool for you, and then it gets back to it? Again, I am not sure I understand that.
There was a problem hiding this comment.
🤖 The AI agent says: Your reading of ACP is right. The runner initiates the Daytona connection, but the protocol is already bidirectional: runner messages use POST, daemon notifications and reverse requests share one SSE GET, and the runner answers requests with another POST. PR #5218 found that stale pi-acp never created session/request_permission, so NetBird is not needed for permissions. Pi custom tools use a runner-polled file relay; user HTTP MCP is a sandbox-to-user-server connection; telemetry is exported by the runner. Only private S3 mounts currently require direct sandbox-to-Agenta infrastructure access. A future Daytona Claude gateway endpoint could become a second case, but it would still need an explicit listener, authentication, and ACLs. The revised recommendation therefore rejects NetBird as the general answer.
|
|
||
| ## Recommendation | ||
|
|
||
| Do not build NetBird as a general sandbox-to-runner transport. |
There was a problem hiding this comment.
🤖 The AI agent says: This is the central change from the first draft. PR #5218 proved that permission delivery was an adapter-version defect before the network transport. The rest of this document now classifies each flow independently.
|
|
||
| ## Corrected model | ||
|
|
||
| Network direction and application direction are different. |
There was a problem hiding this comment.
🤖 The AI agent says: This distinction answers the main question in discussion_r3564426767. The runner initiates the HTTP connections, but daemon requests still travel back over SSE and are answered with POST.
|
|
||
| ## Decision | ||
|
|
||
| Do not implement a general NetBird overlay now. First fix the mount endpoint boundary and |
There was a problem hiding this comment.
🤖 The AI agent says: The recommendation is now a decision gate, not a deferred commitment. The near-term work is the smaller mount endpoint and readiness boundary.
| requests. GeeseFS 0.43.0 defaults can issue parallel flushes and uploads, so one developer | ||
| may fit while several busy mounts may not. | ||
|
|
||
| Before adding a Cloudflare sidecar: |
There was a problem hiding this comment.
🤖 The AI agent says: This acceptance list incorporates CodeRabbit's multipart concern and the owner's 200-concurrency question. A Quick Tunnel rollout is contingent on real geesefs data, including large multipart parts and several concurrent mounts.
| - advertise the overlay URL only to the paired Daytona Claude session; | ||
| - keep the feature rejected until the listener, authentication, and policy are ready. | ||
|
|
||
| ### Security baseline before the first sandbox joins |
There was a problem hiding this comment.
🤖 The AI agent says: This answers the ACL and setup-key review comments. Default-deny, paired-peer and port-scoped policy, safe enrollment, and teardown are all prerequisites for the first possible MVP.
| Conclusion: runner-initiated TCP does not mean runner-only application messages. NetBird is | ||
| not required for Pi or Claude permission delivery. | ||
|
|
||
| ## 2. Capability and transport matrix |
There was a problem hiding this comment.
🤖 The AI agent says: This matrix answers the MCP implementation question directly. User HTTP MCP, runner-loopback gateway MCP, Pi relay tools, and permissions are four different paths.
| - If NetBird is reopened, service exposure, per-run authentication, default-deny ACLs, and | ||
| one-off setup-key handling belong in the first MVP. | ||
|
|
||
| ## Review threads addressed |
There was a problem hiding this comment.
🤖 The AI agent says: This section is the review map for this revision. Every original owner and CodeRabbit thread has a corresponding design change and a signed reply.
|
🤖 The AI agent says: Revision complete at commit The main outcome is a changed recommendation: do not build NetBird as the general sandbox transport. ACP already carries reverse application requests; PR #5218 showed that Pi permissions failed before the transport. Private S3 mounts are the one current direct sandbox-to-Agenta infrastructure need. This revision:
The PR is labeled |
|
🤖 The AI agent says: Closing this design PR with a no-action conclusion for the MVP. The review established that the MVP does not need a new sandbox network layer:
NetBird and a Cloudflare tunnel replacement are therefore not MVP work. The only possible follow-up is smaller readiness and fail-loud hardening for private durable mounts, if that becomes an MVP requirement. No implementation will follow from this PR. |
1 participant
Context
This design started from a real remote-sandbox problem: geesefs inside Daytona cannot reach a private S3 endpoint such as
seaweedfs:8333. The first draft generalized that into a missing sandbox-to-runner connection and proposed NetBird as the common answer for mounts, permissions, tools, and MCP.The revised design separates those flows. ACP already carries daemon-to-runner requests over SSE, and PR #5218 showed that Pi permissions failed because the old adapter never emitted the request. Private S3 connectivity is the only current direct sandbox-to-Agenta infrastructure need.
Decision
Do not build NetBird as a general sandbox transport. Keep ngrok while we:
NetBird remains a conditional private-store option. If reopened, its first MVP must include an explicit SeaweedFS route or proxy, authenticated remote MCP if needed, default-deny port-scoped ACLs, and one-off setup-key handling outside sandbox environment variables.
What changed
context.mdandstatus.mdso the design workspace can be continued without session context.How to review
README.md.recommendation.mdfor the decision and alternatives.plan.md.research.mdfor the implementation evidence, authentication model, limits, and NetBird security details.status.mdfor the mapping from every review thread to the revised design.Feedback requested
Please confirm whether the narrowed decision is right: no general NetBird project now, provider-neutral mount endpoint resolution first, and a Quick Tunnel experiment before choosing the self-hosted connectivity default.