feat(tls): verify against the OS trust store by default (closes #2004)

[fangkangmi]

Summary

By default, apm now verifies HTTPS against the operating-system trust store (via truststore) — the same source git and curl use — instead of the bundled certifi set. This makes apm work out-of-the-box behind a corporate CA or a TLS-inspecting proxy (closes #2004).

Coverage is the Python-based paths: apm install (in-process) and the Python llm child runtime spawned by apm run (a self-contained .pth bootstrap + truststore are installed into its venv at setup time). The frozen binary honours the system store as well, with certifi as a genuine fallback. An explicit REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE still wins, and APM_DISABLE_TRUSTSTORE=1 restores the previous certifi-only behaviour. Node (Copilot) and Rust (Codex) child runtimes are not yet covered — tracked in #2034.

Type of change

• Bug fix
• New feature
• Documentation
• Maintenance / refactor

Testing

• Tested locally
• All existing tests pass
• Added tests for new functionality (if applicable)

Added unit tests (tests/unit/core/test_tls_trust.py, every branch of configure_tls_trust()), an integration test that verifies against a freshly-minted private CA over a loopback HTTPS server (tests/integration/test_tls_custom_ca.py), and a regression test that drives the real frozen SSL runtime hook (tests/integration/test_tls_frozen_hook.py). The child-delivery chain is covered end-to-end: wheel/sdist content guards (tests/integration/test_tls_wheel_content.py), foreign-venv .pth injection (tests/integration/test_tls_child_runtime.py), and the round-2/round-3 verification suites. 69 TLS tests green.

Also validated the end-to-end effect by mimicking the enterprise condition (corporate CA present in the OS trust store but absent from certifi):

Scenario	Result
Before (no truststore), CA in OS store only	TLS failure — reproduces the reported bug
After (truststore injected), CA in OS store	Success (HTTP 200) — fixed via OS store
Control: injected, CA not in OS store	TLS failure — verification remains enforced

Not yet exercised on a real PyInstaller build; a make build smoke against a custom CA on Linux is recommended before release (the code degrades safely to certifi regardless).

Hardening

Four adversarial red-team + remediation cycles (enterprise-TLS/CA, packaging/distribution, proxy/runtime, python-integration lenses) fixed 2 CRITICAL (child propagation; frozen SSL_CERT_FILE handling), 1 HIGH (the .pth was dropped from the wheel), and several MEDIUM/LOW items (best-effort pinned truststore install, bundled-certifi child drop, atomic bootstrap write, certifi component-boundary match, honest macOS-py3.9 docs caveat). The round-4 confirmation pass returned SHIP_NOW on all four lenses.

NOTICE

truststore (MIT, © 2022 Seth Michael Larson) is added to scripts/notice-metadata.yaml and NOTICE is regenerated per the Microsoft CELA manual-NOTICE process.

Spec conformance (OpenAPM v0.1)

• N/A -- this PR does not introduce a new OpenAPM package-format requirement. It touches runtime/install critical-path files for HTTPS transport trust only, which is orthogonal to the normative manifest/lockfile/resolver/policy/registry/runtime-contract surface. Mode-B waiver below (also carried as a commit trailer).

apm-spec-waiver: TLS transport trust is orthogonal to the OpenAPM package-format normative surface; no new manifest/lockfile/resolver/policy/registry/runtime-contract requirement is introduced by this HTTPS trust change.

[fangkangmi]

@microsoft-github-policy-service agree

[Copilot]

Pull request overview

Adds best-effort OS trust-store TLS verification to align APM’s requests HTTPS behavior with git/curl in enterprise (corporate CA / TLS-inspecting proxy) environments, while preserving explicit CA-bundle overrides and providing an opt-out.

Changes:

• Introduces apm_cli.core.tls_trust.configure_tls_trust() (truststore injection with safe fallback to certifi) and runs it at CLI startup.
• Adds truststore as a runtime dependency and ensures it is included in the PyInstaller build.
• Adds unit + integration coverage (custom CA loopback server + frozen runtime hook regression) and updates SSL troubleshooting docs + changelog.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
uv.lock	Locks truststore into the resolved dependency set.
pyproject.toml	Declares truststore>=0.9.1 as a runtime dependency.
build/apm.spec	Adds truststore to PyInstaller hidden imports for frozen builds.
src/apm_cli/core/tls_trust.py	Implements best-effort OS trust-store injection with override/opt-out behavior.
src/apm_cli/cli.py	Calls configure_tls_trust() early at process startup before HTTPS usage.
tests/unit/core/test_tls_trust.py	Unit coverage for all decision branches of configure_tls_trust().
tests/integration/test_tls_custom_ca.py	End-to-end TLS verification behavior tests with a freshly minted private CA.
tests/integration/test_tls_frozen_hook.py	Regression coverage ensuring the frozen runtime hook’s SSL_CERT_FILE does not suppress injection.
docs/src/content/docs/troubleshooting/ssl-issues.md	Documents new default behavior + override/opt-out details.
CHANGELOG.md	Records the behavioral change under Unreleased.

[danielmeppiel]

@fangkangmi Final apm-review-panel convergence pass for #2005.

CEO recommendation

ship_now: the OS trust-store TLS feature is ready for maintainer review. The panel converged with zero remaining foldable items after the shepherd folds below. The performance lazy-injection idea was not adopted because the maintainer explicitly required startup-level injection before requests is imported.

Folded in this run

• (panel) Rebased feat(tls): verify against the OS trust store by default (closes #2004) #2005 onto current main; resolved CHANGELOG faithfully and kept the [FEATURE] Verify TLS using the system trust store by default (breaks in enterprise/proxy environments) #2004/feat(tls): verify against the OS trust store by default (closes #2004) #2005 TLS entry -- resolved in 1137ddd6f / 1756d1a6a.
• (panel) Updated _log_tls_failure to lead with the OS trust-store default and REQUESTS_CA_BUNDLE fallback -- resolved in bc27114af.
• (panel) Folded common-errors / ssl-issues / apm-guide TLS guidance, including the SSL_CERT_FILE caveat -- resolved in bc27114af.
• (panel) Bumped truststore floor to >=0.10.0 and refreshed uv.lock -- resolved in bc27114af.
• (panel) Added injectable env: Mapping override detection while preserving the intentional SSL_CERT_FILE / SSL_CERT_DIR exclusion -- resolved in bc27114af.
• (panel) Added import-timing subprocess coverage proving truststore injection runs before requests import -- resolved in bc27114af.
• (panel) Added canonical env-var docs, install-failures guidance, and CLI bootstrap idempotency coverage -- resolved in d652ee540.
• (panel) Linked enterprise registry-proxy readers to the TLS trust guidance -- resolved in 93a3692f.
• (panel) Isolated the private-CA loopback server from prior process-wide truststore injection so the TLS e2e suite runs with the broader integration set -- resolved in 503e698f.

Copilot signals reviewed

• Copilot posted a summary review but no inline comments were present in the pull-request review comments API on either fetch round. No LEGIT Copilot items remained to fold.

Regression-trap evidence (mutation-break gate)

• tests/unit/core/test_tls_trust.py::test_cli_bootstrap_injects_before_requests_import -- deleted the import-time _configure_process_tls_trust() call in src/apm_cli/cli.py; the test FAILED with a missing sentinel file; guard restored and the test passed.

Lint contract

python3 -m uv run --extra dev ruff check src/ tests/ and python3 -m uv run --extra dev ruff format --check src/ tests/ both passed before push. Full local lint mirror also passed: YAML I/O guard equivalent, file length guard, relative_to guard equivalent, pylint R0801, and scripts/lint-auth-signals.sh.

CI and local validation

GitHub reported no check suite for the pushed fork head (gh pr checks 2005 --watch returned no checks reported on the 'feat/truststore-os-trust' branch), so I did not claim remote CI green. Local CI-equivalent evidence on the pushed head:

• openssl version: LibreSSL 3.3.6.
• python3 -m uv run --extra dev pytest --collect-only -q tests/integration/test_tls_custom_ca.py: 5 tests collected, not skipped.
• Targeted TLS validation: 20 passed in 2.87s for tests/unit/core/test_tls_trust.py, tests/integration/test_tls_frozen_hook.py, tests/integration/test_tls_custom_ca.py, and TestLogTlsFailure.
• Scoped integration suite touching TLS / marketplace / install / registry: 641 passed, 15 skipped in 16.38s.

Mergeability status

Captured from gh pr view 2005 --json number,headRefOid,mergeable,mergeStateStatus,statusCheckRollup after the push.

PR	head SHA	CEO stance	iters	folds	defers	Copilot rounds	CI	mergeable	mergeStateStatus	notes
#2005	503e698	ship_now	2	9	0	2	local green, GitHub no checks	MERGEABLE	BLOCKED	awaiting maintainer/required checks

Convergence

2 outer iterations; 2 Copilot fetch rounds. Final panel stance: ship_now. Ready for maintainer review; no deferred items.

[danielmeppiel]

Remediation cycle: red-team blockers folded + empirically verified

A CEO-arbitrated red-team of four adversarial reviewers (enterprise-proxy, SSL/CA/truststore internals, npm/uv/pip precedent, Python ssl/httpx integration) analyzed this PR and surfaced two release blockers plus supporting concerns. This cycle folds the must-fix set and proves each fix empirically. Commits: dc2826042, 134c989aa, 8323444ce, merge 9ae788003.

Fixed and verified

• B1 -- apm run model traffic now honors OS trust (CRITICAL). The parent-process truststore.inject_into_ssl() cannot cross exec() into the runtime child processes (llm/codex/... are subprocess children), so apm install worked behind a corporate proxy while apm run still failed. Fix: a silent, never-raise sitecustomize shim delivered to children via build_child_tls_env() (prepended PYTHONPATH), so each Python child re-runs the trust bootstrap at its own startup. Verified e2e: a real child subprocess doing requests.get() against a private-CA server returns RESULT:OK when trust is delivered via the env, while a plain-env control returns RESULT:SSLERROR.

• B2 -- frozen binary now uses the OS store, certifi only as genuine fallback (CRITICAL). The PyInstaller hook set SSL_CERT_FILE=certifi before app code; truststore's Linux backend honors it, so the shipped Linux binary silently verified against certifi. Fix: the hook marks its bundled default (APM_SSL_CERT_FILE_IS_BUNDLED_DEFAULT), and configure_tls_trust() pops that bundled default before injection (letting the system store win) while leaving a genuine user SSL_CERT_FILE untouched, and restores certifi if injection fails (never zero-trust on minimal/musl containers). Verified: bundled default neutralized -> system store wins; user override honored; inject-failure restores certifi.

• H2 -- visible trust-source diagnostic. configure_tls_trust() now emits an ASCII [i] TLS: ... line naming the active source (OS store / certifi fallback / explicit bundle / disabled), so a --verbose run shows which CA source is live.

Tests authored by an independent verifier (not the implementer): tests/integration/test_tls_child_runtime.py (new), extended test_tls_frozen_hook.py and test_tls_trust.py. 28 TLS tests pass; the broad runtime/integration subset stays green; full lint contract clean (ruff + pylint R0801 10.00/10 + auth-signals); source and output ASCII-only.

Deferred as fast-follow (filed)

• Add additive corporate-CA support: APM_EXTRA_CA_BUNDLE (npm NODE_EXTRA_CA_CERTS parity) #2034 -- additive APM_EXTRA_CA_BUNDLE (npm NODE_EXTRA_CA_CERTS parity); REQUESTS_CA_BUNDLE only replaces.
• Add apm doctor tls trust-source diagnostic + test-suite ssl-isolation fixture #2035 -- apm doctor tls trust-source diagnostic + shared ssl-isolation test fixture.

Node-based runtimes are not covered by the child-shim propagation (Python llm is the default runtime); documented as a known limitation. Merge remains a human gate.

[danielmeppiel]

Round-2 red-team remediation folded (child-trust delivery rearchitected)

A second adversarial red-team pass (4 enterprise-TLS/PKI + package-manager reviewers, CEO-arbitrated) found that the round-1 child-propagation mechanism was a silent no-op in the shipped layout: the llm runtime runs in its own venv (~/.apm/runtimes/llm-venv) that cannot import apm_cli, so the old sitecustomize shim's import apm_cli... failed and was swallowed -- apm run still fell back to certifi behind a corporate proxy. The round-1 e2e test masked this by spawning apm's own interpreter.

What changed

• Delivery rearchitected to a self-contained .pth bootstrap (_apm_tls_bootstrap.py, zero apm_cli dependency) installed into the child venv at setup time, plus truststore pip-installed into the llm venv. Works in any foreign venv and under a frozen apm binary (the child is still a real venv).
• Removed the PYTHONPATH-prepend sitecustomize mechanism entirely -- this also fixes a HIGH-severity side effect where it shadowed a user's/corporate sitecustomize.py. The .pth approach (coverage.py pattern) is additive, not shadowing.
• Marker leak fixed: APM_SSL_CERT_FILE_IS_BUNDLED_DEFAULT is now cleared unconditionally in configure_tls_trust() so it can never leak into child/git subprocess environments.
• Honest scope: CHANGELOG + ssl-issues.md now claim OS-trust only for apm install and the Python llm runtime. Node (Copilot) and Rust (Codex) runtimes are called out as not yet covered (tracked in Add additive corporate-CA support: APM_EXTRA_CA_BUNDLE (npm NODE_EXTRA_CA_CERTS parity) #2034).

Verification (independent)

A separate verifier wrote foreign-venv e2e tests from scratch proving: trust injects in a venv that cannot import apm_cli (control: no .pth -> plain ssl); a user sitecustomize.py still runs alongside the bootstrap (non-shadowing); the delivery helper lands both files idempotently; marker no-leak across all 5 configure_tls_trust branches (with B2 pop/restore intact); and docs contain no unqualified Node/Rust claim. Full lint chain clean, TLS + runtime-manager suites green.

Parent-side apm install trust, the frozen-binary SSL_CERT_FILE pop/restore (B2), and the verbose trust-source line (H2) were re-confirmed holding. Re-running the red team for a round-3 confirmation pass.

[danielmeppiel]

Round-3 red-team remediation folded (packaging-channel fix + delivery hardening)

A third adversarial red-team pass (4 enterprise-TLS / packaging / proxy reviewers, CEO-arbitrated) found the round-2 rearchitecture was correct in design but undelivered on the PyPI channel, plus a set of correctness/robustness gaps. All folded and independently verified against the built wheel.

HIGH -- the child-trust .pth was dropped from the wheel

setuptools packages.find ships only .py modules; with no package-data/MANIFEST.in for *.pth, the built wheel omitted _apm_tls.pth. On pip/pipx/uv tool installs, ensure_child_tls_bootstrap then couldn't deliver the bootstrap -> llm child-trust was a silent no-op on the primary distribution channel (frozen binaries and editable dev installs masked it). Fixed two ways: the .pth is now generated in code (its content is a fixed one-liner, so delivery no longer depends on packaging at all) and [tool.setuptools.package-data] ships it as belt-and-suspenders. A wheel-content regression test guards it.

MEDIUM correctness/robustness

• Best-effort setup: pip install truststore in setup-llm.sh/.ps1 was a fatal set -e gate that could abort an otherwise-working llm setup; now best-effort (warns) and pinned truststore>=0.10.0.
• Frozen-Linux trust leak: a frozen parent whose inject failed restored SSL_CERT_FILE=certifi and leaked it to the child; truststore-on-Linux honors SSL_CERT_FILE, so the child verified against certifi, not the OS store. build_child_tls_env now drops the bundled-certifi SSL_CERT_FILE for children (a genuine user SSL_CERT_FILE is preserved).
• Atomic delivery: bootstrap files are now written atomically (temp + os.replace, module before .pth), eliminating a corrupt-file class that could cause per-invocation stderr tracebacks.

Docs honesty

Node (Copilot) / Rust (Codex) "not yet covered" caveat surfaced up-front with the NODE_EXTRA_CA_CERTS workaround; CHANGELOG scoped to "Python-based" llm; added a PIP_CERT setup-behind-proxy note and a stale-REQUESTS_CA_BUNDLE diagnostic (it replaces, not augments -- matching git/curl).

Verification (independent, against the built wheel)

A separate verifier built the wheel, pip installed it into a clean venv, then delivered the bootstrap into a second foreign venv without apm_cli and proved ssl.SSLContext.__module__ == truststore._api (control without the .pth: plain ssl) -- the exact channel the HIGH broke. Plus: certifi SSL_CERT_FILE dropped / user cert preserved; atomic-write leaves no partial file on failure; best-effort pip; docs assertions. Full lint chain clean; TLS + runtime suites green.

Deferred (tracked, non-blocking): pip --use-feature=truststore for the remaining setup-time installs (chicken-and-egg; PIP_CERT documented as the workaround today), direct-bash setup-llm.sh coupling, pypy venv glob. Re-running the red team for a round-4 confirmation pass.

[danielmeppiel]

Red-team round 4 (confirmation pass) -- SHIP_NOW, loop closed

Ran the same four-expert adversarial panel a fourth time against the round-3 head, each with a dual mandate: confirm the round-3 fixes hold against the CEO ship bar, and hunt for any new high/critical in the folded surfaces.

All four lenses returned SHIP_NOW -- no new HIGH/CRITICAL.

Lens	Verdict	Ship-bar confirmations (probe-verified)
enterprise-TLS / OpenSSL / CA	SHIP_NOW	wheel ships both files; M2 drops bundled certifi + frozen _MEIPASS drop fires; genuine user path preserved; B2/H2 marker no-leak on all 6 configure_tls_trust branches; child silent when truststore absent
packaging / distribution	SHIP_NOW	wheel + sdist + frozen all deliver the bootstrap module AND .pth; module-missing case fails safe (returns False, no dangling .pth, no startup ImportError); no double-delivery conflict; Windows LF-only .pth
proxy / runtime integration	SHIP_NOW	apm install in-proc trust holds; llm child httpx picks up the truststore patch end-to-end; proxy vars pass through untouched; no zero-trust child on any platform
python-runtime integration	SHIP_NOW	69/69 TLS tests green; .pth byte-exact (import _apm_tls_bootstrap, no BOM/CRLF); atomic module-before-.pth; idempotent re-run; non-interfering for unrelated processes in the venv

LOW residuals folded (e8993be)

None were blocking; folded because they cheaply sharpen the round-3 surfaces:

• certifi boundary -- _is_bundled_certifi matched on a raw string suffix, so a genuine user bundle at /opt/mycertifi/cacert.pem was wrongly dropped for children. Now matches on path components (certifi/cacert.pem as the last two). Regression test added.
• macOS py3.9 honesty -- truststore>=0.10.0 needs Python 3.10+; stock macOS python3 is 3.9, so the llm child silently falls back to certifi. The setup-llm warning now names the Python-version cause, and a Known-limitations caveat was added to ssl-issues.md.
• sdist regression guard -- added an sdist-content test paralleling the wheel guard.
• docstring -- corrected the stale "copies the .pth" wording (the .pth is generated inline).

Deferred (tracked, non-blocking)

• SSL_CERT_FILE=certifi.where() is widened to the OS store in children (a superset for public roots; APM_DISABLE_TRUSTSTORE / REQUESTS_CA_BUNDLE remain the honest pins) -- INFO, deliberate asymmetry.
• Node/Rust child runtimes remain out of scope -> Add additive corporate-CA support: APM_EXTRA_CA_BUNDLE (npm NODE_EXTRA_CA_CERTS parity) #2034.
• pip's own bootstrap-time cert resolution (PIP_CERT workaround documented).

Four remediation cycles complete (rounds 1-3 fixed 2 CRITICAL + 1 HIGH + several MEDIUMs; round 4 is clean). Lint 10.00/10 across all four gates. MERGEABLE. Merging remains a human gate.

[danielmeppiel]

APM Review Panel: ship_with_followups

APM defaults to the OS trust store for HTTPS verification, aligning with git and curl behind corporate proxies.

cc @fangkangmi @sergio-sisternes-epam -- a fresh advisory pass is ready for your review.

Nine panelists converged on a clear signal: this PR delivers a correctly structured trust-boundary shift that keeps verification enabled, isolates fallback behavior, and remains auth-neutral. The 71 passing focused TLS tests provide strong regression evidence across the critical surfaces.

The highest-signal gap is the missing RuntimeManager-to-bootstrap integration test. Leaf-level coverage proves the delivery helper, but no single assertion proves that setup_runtime("llm") delivers both bootstrap files into the created venv. The other focused follow-ups improve verbose observability, enterprise security documentation, setup diagnostics, and planned-feature wording.

On early CLI timing, the architecture and logging lenses agree that trust setup must remain early for import-order safety. Cache and replay of the selected trust source after verbose logging is configured preserves that safety while restoring observability.

Dissent. Combining the truststore pip install with the core llm install could save one resolver pass, but would couple optional trust delivery failure to llm setup. Keep the best-effort isolation. The unhashed child pip surface already applies to llm; track a holistic constraints solution separately.

Aligned with: secure by default: verification remains enabled and explicit bundles retain precedence; pragmatic as npm: OS-installed corporate CAs work without APM-specific setup; governed by policy: APM_DISABLE_TRUSTSTORE=1, REQUESTS_CA_BUNDLE, and CURL_CA_BUNDLE provide deterministic controls.

Growth signal. This is a strong enterprise adoption story: APM now behaves like git and curl behind corporate proxies, with explicit documentation of child-runtime limitations.

Panel summary

Persona	B	R	N	Takeaway
Python Architect	0	1	1	The TLS owner and fallback chain are clean; preserve early setup while centralizing observable outcome.
CLI Logging Expert	0	3	1	Trust-source diagnostics are lost before verbose logging, and bootstrap failure guidance needs a next step.
DevX UX Expert	0	1	2	The OS-store mental model is strong; setup diagnostics must distinguish proxy and Python-version failures.
Supply Chain Security Expert	0	2	2	Verification remains enabled, delivery is atomic, and internal markers do not leak.
OSS Growth Hacker	0	0	1	The enterprise docs funnel is honest and useful.
Auth Expert	0	0	0	Tokens, AuthResolver, credential precedence, and authorization headers are untouched.
Doc Writer	0	2	1	Docs match implementation; remove duplication and avoid naming an unimplemented variable as available.
Test Coverage Expert	0	1	1	Critical surfaces have strong integration evidence; one RuntimeManager wiring gap remains.
Performance Expert	0	1	3	Per-invocation overhead is negligible; keep optional install failure isolated.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Top 5 follow-ups

1. [Test Coverage Expert] Add an integration test proving RuntimeManager.setup_runtime("llm") delivers both TLS bootstrap files into the child venv -- this is the missing guard at the secure-by-default integration seam.
2. [CLI Logging Expert] Cache the selected trust source and re-emit it after verbose logging is configured -- setup must stay early, but --verbose currently loses the diagnostic.
3. [Supply Chain Security Expert] Add a concise transport-trust section to the enterprise security page -- enterprise reviewers need the OS-store default, child bootstrap, overrides, and opt-out documented together.
4. [DevX UX Expert] Differentiate Python-version and pip/proxy failures in the llm truststore setup warning, including PIP_CERT recovery guidance -- the target proxy user currently receives a misleading cause.
5. [Doc Writer] Remove the unimplemented APM_EXTRA_CA_BUNDLE name or present it as an explicit planned feature -- current prose can be mistaken for a supported variable.

Architecture

classDiagram
    class tls_trust
    class RuntimeManager
    class LLMRuntime
    class cli_main
    RuntimeManager ..> tls_trust : bootstrap
    LLMRuntime ..> tls_trust : child env
    cli_main ..> tls_trust : configure

Loading

flowchart TD
    A["CLI import"] --> B{"override or opt-out?"}
    B -->|yes| C["certifi or explicit bundle"]
    B -->|no| D["truststore injection"]
    E["Runtime setup"] --> F["child venv bootstrap"]

Loading

Recommendation

Fold the five focused follow-ups into this PR, then re-run the panel on the converged head. The core trust shift is sound, keeps verification enabled, and addresses a high-impact enterprise onboarding failure.

---

Full per-persona findings

Python Architect

• [recommended] Module-level TLS setup in src/apm_cli/cli.py couples behavior to import order. Preserve early injection, centralize the cached outcome in the TLS owner, and expose it for verbose replay.
• [nit] The runtime subprocess helper now defaults every child to TLS env hygiene; the cost is negligible but the coupling should remain intentional.

CLI Logging Expert

• [recommended] --verbose never sees the selected trust source because setup runs before DEBUG logging and the later guarded call is a no-op.
• [recommended] The llm bootstrap warning gives no recovery step or troubleshooting link.
• [recommended] A defensive bare except silently swallows any break in the helper's never-raise contract.
• [nit] [i] inside DEBUG messages duplicates the log level.

DevX UX Expert

• [recommended] The setup warning attributes all truststore install failures to Python version, masking the target user's pip/proxy chicken-and-egg failure.
• [nit] The hardcoded stock macOS Python 3.9 statement will age.
• [nit] A future apm doctor TLS posture check could improve discoverability, but it is a separate command surface.

Supply Chain Security Expert

• [recommended] Child-venv truststore installation lacks hash constraints, matching the existing unhashed llm install surface; address the whole child-runtime dependency channel together.
• [recommended] docs/src/content/docs/enterprise/security.md does not yet describe the new transport-trust model.
• [nit] Bundled-certifi detection uses string path splitting; current boundary tests cover the relevant cases.
• [nit] The child truststore requirement has no upper bound; fallback limits the impact of future incompatibility.

OSS Growth Hacker

• [nit] Lead the SSL troubleshooting section with the one-line OS-store fix before explaining mechanism details.

Auth Expert

No findings.

Doc Writer

• [recommended] The Node/Codex caveat is repeated almost verbatim within one section; keep the early statement and cross-reference it later.
• [recommended] APM_EXTRA_CA_BUNDLE is named as future configuration without the required planned-feature treatment.
• [nit] The default-behavior paragraph combines too many facts for quick scanning.

Test Coverage Expert

• [recommended] No integration-with-fixtures test proves the RuntimeManager.setup_runtime("llm") to child-bootstrap delivery chain. Existing leaf tests do not cover that wiring.
  Proof (missing at): tests/integration/test_tls_runtime_manager_wiring.py::test_setup_runtime_llm_installs_tls_bootstrap -- proves that llm runtime setup installs the OS-trust bootstrap. [secure-by-default,devx]
• [nit] The changed TLS warning test asserts bundle guidance but not the new system-trust-store wording.

Performance Expert

• [recommended] A separate truststore pip invocation adds a resolver pass, but combining it must not destroy best-effort semantics.
• [nit] Child env copies and bundled-certifi checks are negligible compared with subprocess launch.
• [nit] Early trust setup adds less than 5 ms cold startup and is justified by import-order safety.

_{This panel is advisory. It does not block merge. Re-apply the panel-review label after addressing feedback to re-run.}

[danielmeppiel]

APM Review Panel: ship_with_followups

The converged TLS trust change is sound; one static architecture fence and four small consistency folds remain.

cc @fangkangmi @sergio-sisternes-epam -- a fresh advisory pass is ready for your review.

Eight validated lenses converge on ship. TLS verification stays enabled, explicit CA bundles retain precedence, child delivery is guarded at integration tier, and auth behavior is unchanged. Full local tests and the canonical lint mirror pass; CI run 29182906259 is green across lint, test shards, binary smoke, CodeQL, coverage, NOTICE, spec, docs, and merge gate.

The highest-signal current item is the Python Architect finding: architecture.instructions.md requires a behavioral regression test plus a static boundary check whenever a durable decision is centralized. The TLS behavior tests exist, but the architecture lint does not yet prevent direct truststore.inject_into_ssl() calls outside the two intended owners.

The Performance Expert return failed schema validation twice, so no performance finding was admitted. Existing evidence still shows the TLS setup cost is a one-time startup operation rather than a per-request hot path.

Aligned with: secure by default: verification remains enabled with OS trust as default and certifi as fallback; governed by policy: APM_DISABLE_TRUSTSTORE=1, REQUESTS_CA_BUNDLE, and CURL_CA_BUNDLE provide explicit controls; pragmatic as npm: corporate CAs installed in the OS work without APM-specific setup.

Growth signal. The release story is direct: APM now behaves like git and curl behind corporate proxies. Lead the changelog entry with that user benefit.

Panel summary

Persona	B	R	N	Takeaway
Python Architect	0	1	1	Add the static canonical-owner fence and an env-policy parity guard.
CLI Logging Expert	0	0	1	Parent logging is sound; remove redundant child DEBUG prefixes.
DevX UX Expert	0	0	0	Happy path is invisible and failures are actionable.
Supply Chain Security Expert	0	0	1	Trust flow is safe; make offline llm probes use the same child env hygiene.
OSS Growth Hacker	0	0	1	Lead the changelog with the corporate-proxy benefit.
Auth Expert	0	0	0	Token and credential behavior is unchanged.
Doc Writer	0	0	1	Add the missing changelog section separator.
Test Coverage Expert	0	0	0	Critical promises have regression traps at their required tiers.
Performance Expert	0	0	0	Schema placeholder; no finding admitted.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Top 5 follow-ups

1. [Python Architect] Add an architecture lint that rejects direct truststore.inject_into_ssl() calls outside core/tls_trust.py and core/_child_tls/_apm_tls_bootstrap.py -- this completes the repository's dual-guardrail contract.
2. [Python Architect] Add a static parity test for parent and child TLS environment-variable policy -- the child is intentionally standalone, so literals must not drift silently.
3. [Supply Chain Security Expert] Pass build_child_tls_env() to the two offline llm --version probes -- this keeps subprocess env handling uniform if those probes evolve.
4. [CLI Logging Expert] Remove redundant [i] prefixes from child-bootstrap DEBUG messages -- the log level already carries severity.
5. [OSS Growth Hacker] Lead the changelog entry with the user benefit, then retain the mechanism and scope detail -- this makes the upgrade value scannable.

Architecture

classDiagram
    class tls_trust {
        +configure_process_tls_trust() bool
        +configure_tls_trust(env) bool
        +build_child_tls_env(env) dict
        +ensure_child_tls_bootstrap(venv) bool
    }
    class RuntimeManager
    class LLMRuntime
    class ChildBootstrap
    RuntimeManager ..> tls_trust : deliver bootstrap
    LLMRuntime ..> tls_trust : scrub child env
    tls_trust ..> ChildBootstrap : install into venv

Loading

flowchart TD
    A["CLI entry"] --> B["Configure OS trust once"]
    B --> C["Replay trust source after logging"]
    D["Runtime setup"] --> E["Install child bootstrap"]
    E --> F["Child interpreter loads OS trust"]

Loading

Recommendation

Fold the five bounded items, re-run the lint and CI evidence, then perform the terminal advisory pass. No design discussion is needed.

---

Full per-persona findings

Python Architect

• [recommended] scripts/lint-architecture-boundaries.sh does not prevent direct truststore.inject_into_ssl() calls outside the canonical parent and standalone-child owners.
• [nit] Parent and child TLS environment-variable policy needs a static parity assertion because the standalone child cannot import parent constants.

CLI Logging Expert

• [nit] _apm_tls_bootstrap.py still prefixes DEBUG messages with [i], unlike the parent logger.

DevX UX Expert

No findings.

Supply Chain Security Expert

• [nit] Two llm --version probes omit build_child_tls_env() even though network-bearing sibling subprocesses use it. The probes are offline today.

OSS Growth Hacker

• [nit] The changelog mechanism detail precedes the user benefit, reducing scanability.

Auth Expert

No findings.

Doc Writer

• [nit] The PR feat(tls): verify against the OS trust store by default (closes #2004) #2005 changelog entry needs a blank line before the next heading.

Test Coverage Expert

No findings.

Performance Expert

Schema failure after two correction attempts; no finding admitted.

_{This panel is advisory. It does not block merge. Re-apply the panel-review label after addressing feedback to re-run.}

[danielmeppiel]

APM Review Panel: ship_with_followups

The TLS runtime contract and CI are sound; two edge-case hardening items and three documentation corrections remain in scope.

cc @fangkangmi @sergio-sisternes-epam -- a fresh advisory pass is ready for your review.

Six validated lenses returned no findings. Python architecture raised one cosmetic redundancy, and the documentation lens found three current troubleshooting issues. Full local tests and lint pass; CI run 29183496602 is green across lint, test shards, binary smoke, CodeQL, coverage, NOTICE, spec, docs, and merge gate.

Auth and supply-chain-security returned malformed JSON twice, so schema placeholders were used. Their original reviews still surfaced two concrete facts for maintainer visibility: a genuine SSL_CERT_FILE ending in /certifi/cacert.pem can be mistaken for APM's bundled certifi path, and child bootstrap delivery can follow a site-packages symlink outside the managed venv. Both are bounded and will be folded before terminal.

Aligned with: secure by default: covered Python paths use OS trust without disabling verification; governed by policy: APM_DISABLE_TRUSTSTORE=1, REQUESTS_CA_BUNDLE, and CURL_CA_BUNDLE remain explicit controls; pragmatic as npm: corporate CAs installed in the OS work without APM-specific setup.

Panel summary

Persona	B	R	N	Takeaway
Python Architect	0	0	1	Dual guardrails are complete; remove one redundant Codex env argument.
CLI Logging Expert	0	0	0	TLS logging and verbose replay are clean.
DevX UX Expert	0	0	0	Happy path is silent and recovery is actionable.
Supply Chain Security Expert	0	0	0	Schema placeholder after two malformed returns.
OSS Growth Hacker	0	0	0	Benefit-first changelog copy is effective.
Auth Expert	0	0	0	Schema placeholder after two malformed returns.
Doc Writer	0	3	0	Correct verification guidance, remove an ungrounded caveat, and mark planned scope.
Test Coverage Expert	0	0	0	Critical TLS surfaces have regression traps at required tiers.
Performance Expert	0	0	0	No meaningful hot-path overhead.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Top 5 follow-ups

1. [Doc Writer] Replace the bare python -c "import requests" verification step with an actual APM debug/install path -- the bare process never runs APM's trust injection and can report a false negative.
2. [Supply Chain Security Expert] Make bundled-certifi detection exact before stripping SSL_CERT_FILE from child env -- a genuine user path ending /certifi/cacert.pem must survive.
3. [Supply Chain Security Expert] Reject a child site-packages path that resolves outside its managed venv before writing the bootstrap -- local symlink redirection should not escape the venv.
4. [Doc Writer] Remove the ungrounded Windows Schannel caveat -- it names no supported symptom or recovery.
5. [Doc Writer] Present future Node/Rust coverage (Add additive corporate-CA support: APM_EXTRA_CA_BUNDLE (npm NODE_EXTRA_CA_CERTS parity) #2034) in the documented Planned callout -- shipped scope and roadmap should be visually distinct.

Architecture

flowchart TD
    A["APM Python process"] --> B["Canonical TLS owner"]
    B --> C["OS trust or certifi fallback"]
    D["llm runtime setup"] --> E["Contained venv bootstrap"]
    E --> F["Standalone child TLS owner"]

Loading

Recommendation

Fold the five items plus the small Codex call-site cleanup, re-run full lint and CI, then perform the final advisory pass.

---

Full per-persona findings

Python Architect

• [nit] codex_runtime.py passes the same child env that _stream_subprocess_output already supplies by default.

CLI Logging Expert

No findings.

DevX UX Expert

No findings.

Supply Chain Security Expert

Schema failure after two correction attempts. Original review context surfaced the contained-venv and exact-certifi-path edge cases listed above.

OSS Growth Hacker

No findings.

Auth Expert

Schema failure after two correction attempts. Original review context overlapped the exact-certifi-path edge case listed above.

Doc Writer

• [recommended] Bare Requests verification bypasses APM's injection and can false-negative.
• [recommended] The Windows Schannel caveat is ungrounded and non-actionable.
• [recommended] Node/Rust future scope should use the Planned callout convention.

Test Coverage Expert

No findings.

Performance Expert

No findings.

_{This panel is advisory. It does not block merge. Re-apply the panel-review label after addressing feedback to re-run.}

[danielmeppiel]

APM Review Panel: ship_now

OS trust-store TLS is ready for Python APM paths, with explicit policy controls, contained child delivery, deterministic fallback, and green release evidence.

cc @fangkangmi @sergio-sisternes-epam -- a fresh advisory pass is ready for your review.

All nine panelists converged on an empty findings set. The architecture now has one parent owner plus the intentionally standalone child owner, a static injection fence, parent-child policy parity, exact bundled-cert tracking, and contained writes. Supply-chain and auth lenses found no current issue; CLI, DevX, docs, performance, and coverage lenses also returned clean.

This fourth pass confirms clean convergence after current main was merged. The final head passed the complete local test suite, the full canonical lint mirror, CI test shards, binary smoke, CodeQL, coverage, NOTICE, spec conformance, docs build, and merge gate.

Aligned with: secure by default: covered Python paths use OS trust while verification always remains enabled; governed by policy: APM_DISABLE_TRUSTSTORE=1, REQUESTS_CA_BUNDLE, and CURL_CA_BUNDLE are explicit controls; pragmatic as npm: corporate roots trusted by the OS work without APM-specific shell setup.

Panel summary

Persona	B	R	N	Takeaway
Python Architect	0	0	0	Canonical ownership and dual guardrails are complete.
CLI Logging Expert	0	0	0	Quiet and verbose TLS diagnostics are consistent and actionable.
DevX UX Expert	0	0	0	Happy path is zero-config; recovery paths are explicit.
Supply Chain Security Expert	0	0	0	Dependency, trust, path-containment, and fallback boundaries are sound.
OSS Growth Hacker	0	0	0	Benefit-first changelog and enterprise docs are complete.
Auth Expert	0	0	0	AuthResolver and token handling are untouched.
Doc Writer	0	0	0	Verification, planned scope, security guidance, and links match code.
Test Coverage Expert	0	0	0	Critical TLS promises have regression traps at required tiers.
Performance Expert	0	0	0	No meaningful hot-path or algorithmic regression.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Recommendation

Merge as-is after required human review. No current panel follow-up remains.

Folded in this run

• (panel) Replay the early trust-source choice after verbose logging is configured -- resolved in 83eb7e568.
• (panel) Prove RuntimeManager.setup_runtime("llm") delivers the child bootstrap -- resolved in 8b9d27ecf.
• (panel) Differentiate Python-version and proxy/PIP_CERT setup failures -- resolved in 8b9d27ecf.
• (panel) Make bootstrap failure output actionable and unexpected errors debug-visible -- resolved in 8b9d27ecf.
• (panel) Document the enterprise transport-trust boundary -- resolved in f4d51b0ab.
• (panel) Remove duplicated caveats and the unimplemented extra-bundle variable name -- resolved in f4d51b0ab.
• (panel) Add a static architecture fence for direct truststore injection -- resolved in 6483e2284.
• (panel) Lock parent-child TLS environment policy parity -- resolved in 6483e2284.
• (panel) Apply child TLS env hygiene to llm availability probes -- resolved in 6483e2284.
• (panel) Normalize child DEBUG diagnostics and lead the changelog with user value -- resolved in 6483e2284.
• (panel) Reject child site-packages symlink escapes -- resolved in a1ab076f9.
• (panel) Preserve user SSL_CERT_FILE unless it exactly matches the recorded bundled cert -- resolved in a1ab076f9.
• (panel) Verify trust through an actual APM path and mark planned Node/Rust scope -- resolved in a1ab076f9.
• (panel) Remove the ungrounded Schannel caveat and redundant Codex env argument -- resolved in a1ab076f9.
• (panel) Reconcile the changelog with the v0.25.0 release cut after merging main -- resolved in e08aa2145.
• (copilot) Correct stale SSL_CERT_FILE test documentation -- resolved in 1756d1a6a.
• (copilot) Replace non-ASCII dashes in TLS docs and changelog -- resolved in 1756d1a6a.
• (copilot) Reference PR feat(tls): verify against the OS trust store by default (closes #2004) #2005 as well as issue [FEATURE] Verify TLS using the system trust store by default (breaks in enterprise/proxy environments) #2004 in the changelog -- resolved in 1756d1a6a.

Copilot signals reviewed

• tests/unit/core/test_tls_trust.py:8 -- LEGIT: the module docstring contradicted the implemented SSL_CERT_FILE precedence (resolved in 1756d1a6a).
• docs/src/content/docs/troubleshooting/ssl-issues.md:57 -- LEGIT: a non-ASCII em dash violated repository encoding conventions (resolved in 1756d1a6a).
• docs/src/content/docs/troubleshooting/ssl-issues.md:61 -- LEGIT: a second non-ASCII em dash violated repository encoding conventions (resolved in 1756d1a6a).
• CHANGELOG.md:18 -- LEGIT: the entry referenced only issue [FEATURE] Verify TLS using the system trust store by default (breaks in enterprise/proxy environments) #2004 instead of the PR that carries the change (resolved in 1756d1a6a).
• CHANGELOG.md:14 -- LEGIT: a non-ASCII em dash violated repository encoding conventions (resolved in 1756d1a6a).

Deferred (out-of-scope follow-ups)

• (panel) Extend OS trust propagation to Node (Copilot) and Rust (Codex) child runtimes -- scope boundary: this PR explicitly covers Python paths; runtime-specific work is tracked in Add additive corporate-CA support: APM_EXTRA_CA_BUNDLE (npm NODE_EXTRA_CA_CERTS parity) #2034; suggested follow-up: Add additive corporate-CA support: APM_EXTRA_CA_BUNDLE (npm NODE_EXTRA_CA_CERTS parity) #2034.
• (panel) Add hash constraints for the entire child-runtime pip installation channel -- scope boundary: the existing unhashed llm installation is a broader child-runtime dependency policy, not the TLS transport-trust change.
• (panel) Add a TLS posture section to apm doctor -- scope boundary: this introduces a new diagnostic command surface beyond the trust behavior and troubleshooting documentation in this PR.

Regression-trap evidence (mutation-break gate)

• test_cached_trust_source_can_be_replayed_after_logging_configuration -- deleted cached trust-source replay; test FAILED as expected; guard restored.
• test_runtime_manager_setup_llm_installs_tls_bootstrap -- deleted RuntimeManager bootstrap handoff; test FAILED as expected; guard restored.
• test_runtime_manager_bootstrap_warning_is_actionable -- deleted PIP_CERT/Python/docs recovery guidance; test FAILED as expected; guard restored.
• test_runtime_manager_bootstrap_exception_is_visible_in_debug_log -- deleted unexpected-exception DEBUG record; test FAILED as expected; guard restored.
• test_v4_setup_llm_truststore_is_best_effort_and_pinned -- deleted proxy recovery guidance in the setup script; test FAILED as expected; guard restored.
• test_ssl_docs_keep_planned_configuration_generic -- restored APM_EXTRA_CA_BUNDLE as an advertised name; test FAILED as expected; guard restored.
• test_enterprise_security_docs_transport_trust_model -- deleted HTTPS transport trust heading; test FAILED as expected; guard restored.
• test_tls_injection_has_one_canonical_authority -- deleted architecture injection boundary; test FAILED as expected; guard restored.
• test_child_bootstrap_tls_policy_matches_parent_constants -- changed REQUESTS_CA_BUNDLE in the child policy; test FAILED as expected; guard restored.
• test_child_bootstrap_debug_messages_do_not_embed_console_symbols -- restored an inline [i] prefix; test FAILED as expected; guard restored.
• test_is_available_uses_child_tls_env -- deleted llm availability env hygiene; test FAILED as expected; guard restored.
• test_ensure_child_tls_bootstrap_rejects_site_packages_symlink_escape -- deleted managed-venv containment; test FAILED as expected; guard restored.
• test_build_child_tls_env_preserves_user_certifi_component_path -- restored suffix-only bundled-cert detection; test FAILED as expected; guard restored.
• test_ssl_docs_verify_apm_path_and_mark_planned_scope -- removed the Planned callout marker; test FAILED as expected; guard restored.

Lint contract

Full current-main CI mirror exited 0: ruff lint, ruff format, YAML I/O, 2100-line, relative_to, pylint R0801, auth boundary, and architecture boundary checks all passed.

CI

All checks green on final head in CI run 29184285439, including both test shards, binary smoke, CodeQL, coverage, NOTICE, spec conformance, docs build, and merge gate (after 1 CI recovery iteration for a conflicting-main synchronize event).

Mergeability status

PR	head SHA	CEO stance	iters	folds	defers	Copilot rounds	CI	mergeable	mergeStateStatus	notes
#2005	e08aa21	ship_now	4	18	3	2	green	MERGEABLE	BLOCKED	awaiting required review

Convergence

4 outer iterations; 2 Copilot rounds. Final panel stance: ship_now.

Ready for maintainer review.

---

Full per-persona findings

Python Architect

No findings.

CLI Logging Expert

No findings.

DevX UX Expert

No findings.

Supply Chain Security Expert

No findings.

OSS Growth Hacker

No findings.

Auth Expert

No findings.

Doc Writer

No findings.

Test Coverage Expert

No findings.

Performance Expert

No findings.

_{This panel is advisory. It does not block merge. Re-apply the panel-review label after addressing feedback to re-run.}