Phase 0 + Phase 1: repo health, CA persistence, and a trustworthy HTTP debugger#7
Open
epli2 wants to merge 10 commits into
Open
Phase 0 + Phase 1: repo health, CA persistence, and a trustworthy HTTP debugger#7epli2 wants to merge 10 commits into
epli2 wants to merge 10 commits into
Conversation
Analyze current implementation and open PRs (#2, #4), redefine phantom's concept as a local-first API development toolbox (observe / perturb / record-replay / spec-gen), and lay out a 5-phase roadmap (P0-P4) with per-task dependencies, implementation steps, acceptance criteria, and an LLM-agent working protocol. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EgPcomgY5nj8RuU8V1G82Y
…3, P0-6) - Persist the proxy CA under <data-dir>/ca (0600 key, .gitignore guard), reusing it across runs so clients can durably trust one certificate. Corrupt/missing key regenerates the CA; unusable dirs fall back to an ephemeral CA. Unit tests cover first-use, reload stability, and corruption. - Introduce clap subcommands with full backward compatibility: 'phantom -- CMD' == 'phantom run -- CMD'. New 'phantom cert path|print|export' prints/exports the CA with OS trust-store instructions. - Spawned processes now get HTTPS_PROXY (non-Node; Node's preload handles HTTPS itself and axios would double-proxy), a combined CA bundle (system/user roots + phantom CA) via SSL_CERT_FILE, CURL_CA_BUNDLE, REQUESTS_CA_BUNDLE, NODE_EXTRA_CA_CERTS and DENO_CERT, and cleared NO_PROXY/ALL_PROXY/npm_config_* so no traffic bypasses or leaks past the phantom proxy. - New integration test: proxied curl verifies the MITM'd certificate with NO client-side --insecure flag; CA path/bytes stable across runs. Implements ROADMAP.md P0-3 and P0-6 (gaps G1, G2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EgPcomgY5nj8RuU8V1G82Y
…ode in CI (P0-4, P0-5) - README: full rewrite around the toolbox concept (30-second demo, feature matrix, how-it-works, CA trust, JSONL schema pointer, roadmap link). - docs/how-to-use.ja.md: subcommand CLI reference, CA/HTTPS section, cleared-proxy-vars note, roadmap section now points at ROADMAP.md. - AGENTS.md: CLI structure (subcommands), CA persistence + child trust environment tables, new integration test docs. - plan.md: frozen-notice pointing to ROADMAP.md. - CI: pin Node 22 via actions/setup-node so proxy integration tests never silently skip on runner image drift. Implements ROADMAP.md P0-4 and P0-5 (gap G9). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EgPcomgY5nj8RuU8V1G82Y
…(P1-1, P1-2)
- crates/phantom-capture/src/proxy.rs: transparently decode gzip/deflate/br/
zstd request and response bodies for the recorded trace, applying the size
limit before AND after decoding (zip-bomb guard). The wire bytes forwarded
to the real client/server are always the complete, unmodified original —
collect_body now separates 'wire_bytes' (untouched, full) from 'recorded'
(the size-capped copy used for the trace), fixing a latent bug where large
(>1MB) request/response bodies were silently truncated on the wire, not
just in the recorded copy.
- HttpTrace gains request/response_content_encoding, *_body_truncated, and
*_body_binary fields (serde(default) for backward-compatible deserialization
of traces persisted before this change).
- New --max-body <SIZE> flag (accepts "512kb"/"1mb"/"2gb", "0" = unlimited,
default "1mb") replaces the hardcoded MAX_BODY_SIZE constant at the CLI
layer; ProxyCaptureBackend::with_max_body_size wires it through.
- JSONL output gains request/response_body_encoding ("utf-8"|"base64" —
binary bodies are now base64-encoded instead of lossily mangled) and the
new HttpTrace fields, plus a schema_version field (bumped to 2 here since
these are the first v2 additions; full docs land in the next commit).
- TUI detail pane shows '[binary body, <size>, <content-type>]' and
'[body truncated at <size> — rerun with --max-body 0]' placeholders instead
of a bare byte count.
- Unit tests: gzip/brotli/zstd round-trip, corrupt-gzip doesn't panic, post-decode
truncation, binary detection (NUL bytes, invalid UTF-8 ratio, empty body).
- Integration test (tests/proxy_gzip_integration.rs): proxied curl receives the
gzip body byte-for-byte unmodified while the JSONL trace shows the decoded
plaintext.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EgPcomgY5nj8RuU8V1G82Y
…(P1-3) Replace the duplicated JSONL schema tables in AGENTS.md and --help with summaries pointing at docs/jsonl-schema.md, which documents the full schema_version 2 field set (added in the previous commit) and the additive-only compatibility policy within a version. Extend the gzip integration test to assert every required field is present on a real JSONL record. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EgPcomgY5nj8RuU8V1G82Y
- New crates/phantom-core/src/redact.rs: RedactionConfig + redact_trace(), applied once right after an HttpTrace is built (both the proxy and ldpreload backends), before it reaches the trace channel — so storage, TUI, and JSONL all see the same redacted copy. - --redact enables the default lists (headers: authorization, proxy-authorization, cookie, set-cookie, x-api-key; JSON body fields: password, token, access_token, refresh_token, client_secret, api_key). --redact-header/--redact-body-field add further names on top of whatever --redact already selected (repeatable, composable). Off by default. - Body redaction only touches parseable JSON, matching keys recursively (objects and arrays) case-insensitively; non-JSON bodies are left as-is. Matched values become the literal string "[REDACTED]" — no length or content of the original is retained. - Unit tests in phantom-core (empty config no-op, case-insensitive header match, nested/array JSON redaction, non-JSON and unconfigured-field passthrough) plus an end-to-end integration test proving --redact masks both a request header and a response JSON field through the real proxy, and that omitting the flag leaves values untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EgPcomgY5nj8RuU8V1G82Y
…(P1-6)
- tests/apps/python-app/client.py + tests/proxy_python_integration.rs: Python
3 stdlib (urllib.request) HTTP+HTTPS capture, verified working out of the
box via phantom's auto-injected HTTP_PROXY/HTTPS_PROXY + SSL_CERT_FILE.
- tests/apps/go-app/client.go + tests/proxy_go_integration.rs: Go net/http
HTTP capture, verified working — scoped to HTTP only after discovering two
real limitations while building this (documented, not silently worked
around):
1. Go's net/http.ProxyFromEnvironment unconditionally refuses to proxy
requests to "localhost" or any loopback IP, regardless of
HTTP_PROXY/HTTPS_PROXY/NO_PROXY — confirmed directly against go1.24.
Not a phantom bug; the test backend binds to a dynamically-discovered
non-loopback address so the request actually reaches the proxy.
2. phantom's MITM leaf certificates only ever carry a DNS-name SAN, never
an IP SAN, even for IP-literal CONNECT targets — so any client with
strict RFC 6125 IP-literal verification (Go's crypto/tls confirmed;
likely others) rejects the cert. This is a phantom/hudsucker gap
affecting IP-literal HTTPS targets for every language, not Go-specific;
tracked as future work rather than fixed here.
- docs/compatibility.md: new runtime support matrix (Node/curl/Python
verified; Go verified with the HTTP-only caveat; Java per PR #4; several
runtimes honestly marked untested) plus write-ups of both limitations
above with the exact repro. Linked from README and AGENTS.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EgPcomgY5nj8RuU8V1G82Y
…export (P1-4) - Add a small filter query language (status:/method:/host:/path: plus plain-text substring) to the TUI, replacing plain URL substring matching. - Split the detail pane into scrollable Request/Response/Headers/Timing tabs ([ / ] to switch, j/k to scroll) with a scroll-position indicator. - Add a `?` help overlay listing all keybindings and filter syntax. - Add `c` (copy selected trace as a curl command via arboard, falling back to stderr when no clipboard is available) and `w` (write selected trace to phantom-trace-<span_id>.json) export keys. - Document the new keybindings and filter syntax in docs/how-to-use.ja.md. This completes Phase 1 (P1-1 through P1-6) of ROADMAP.md.
- tests/integration/lib.sh: run_phantom_capture matched JSONL lines with
a hardcoded `{"timestamp_ms"...}` anchor. Adding schema_version as the
new first field (P1-3) shifted timestamp_ms out of that position, so
the grep matched nothing and every LD_PRELOAD integration test failed
with "no output". Match on `{"schema_version"` instead.
- tests/proxy_curl_https_integration.rs: curl's own response body shares
phantom's inherited stdout with no synchronization between the two
processes, so it can interleave with a JSONL line and corrupt it
(intermittent on macOS runners). Redirect curl's output to /dev/null,
matching the fix already applied in redaction_integration.rs.
…n test macOS CI surfaced a pre-existing failure once the earlier interleaving bug stopped masking it (cargo test aborts after the first failing binary): test_proxy_captures_python_stdlib_client's HTTPS leg fails on the runner's Homebrew Python 3.14 with "Missing Authority Key Identifier". Root cause: hudsucker::certificate_authority::RcgenAuthority builds every MITM leaf certificate from CertificateParams::default(), which never sets use_authority_key_identifier_extension. Confirmed unchanged on hudsucker's latest upstream main branch, so this can't be fixed by a version bump. Most TLS stacks tolerate a leaf cert without an AKI extension; some stricter OpenSSL/LibreSSL builds (observed here) don't. This is the same class of issue as the already-documented Go IP-SAN gap: a real, environment-dependent limitation in phantom's cert generation, not a regression. Documented as limitation #3 in docs/compatibility.md. client.py now detects exactly this SSL error and reports it distinctly instead of crashing; the Rust test tolerates that specific, known outcome (HTTP-only) while still fully asserting both legs wherever the local TLS stack is lenient enough to accept the cert (e.g. Linux CI).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements Phase 0 and Phase 1 of
ROADMAP.md.Phase 0 — repository health (v0.1.x)
<data-dir>/ca/) and auto-inject trust env vars (SSL_CERT_FILE,CURL_CA_BUNDLE,REQUESTS_CA_BUNDLE,NODE_EXTRA_CA_CERTS,DENO_CERT) into spawned children.phantom run,phantom cert path|print|export) while keeping the legacy no-subcommand form working forever.README.md, strengthen CI (macOS runner, separated integration tests, pinned Node version).Phase 1 — "信頼できる HTTP デバッガ" (a trustworthy HTTP debugger) (v0.2)
schema_versionfield, additive-only evolution policy) documented as the source of truth indocs/jsonl-schema.md.--redact,--redact-header,--redact-body-field) applied once at trace-construction time so storage, TUI, and JSONL all see the same redacted copy.net/http) with real integration tests; documented two genuine upstream limitations found along the way (Go's loopback-proxy exclusion, missing IP SAN in generated MITM certs) indocs/compatibility.md.status:,method:,host:,path:), a scrollable tabbed detail pane (Request/Response/Headers/Timing), a?help overlay, andc/wkeys to export the selected trace as acurlcommand or a JSON file.Test plan
cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-features -- -D warningscargo test --workspace --all-targets --all-features(unit tests + Node/curl/Python/Go/gzip/redaction integration tests)Generated by Claude Code