P4.1 Update vMCP architecture docs + doc.go

[tgrunnagle]

Description

Update the vMCP architecture documentation and package docs to reflect the
domain/transport split delivered by this refactor. After the prior phases, the
additive VMCP interface, New(cfg) -> VMCP core constructor, and
Serve(ctx, VMCP, serverCfg) -> *Server transport helper exist, server.New is
a thin stable wrapper, and decoration is the supported extension mechanism — but
the docs still describe only the old server.New path. This task brings
docs/arch/vmcp-library.md, docs/arch/10-virtual-mcp-architecture.md, and
pkg/vmcp/doc.go into line with the new public API so embedders learn the
intended (New/Serve/decorator) shape, not just the legacy wrapper.

This is docs-only (Phase 4, P4.1). No production Go behavior changes.

Context

Per architecture.md "Phase 4 — Docs + example" (P4.1, lines 434-437) and
research.md "Phase-4 docs targets confirmed" (line 302), Phase 4 updates three
doc targets: docs/arch/vmcp-library.md (add New/Serve/the decorator
extension model and refresh the stability table to reflect the additive API),
docs/arch/10-virtual-mcp-architecture.md (overview), and pkg/vmcp/doc.go
(package docs describing the domain/transport seam + decorator extension).

The substantive structural work is already merged by the time this task runs:
Phase 1 added the VMCP interface + New; Phase 2 added Serve and re-homed
the transport; Phase 3 (#5445) reduced server.New's body to
Serve(ctx, New(deriveCoreConfig(cfg)), deriveServerConfig(cfg)) while keeping
its 7-param signature and observable behavior unchanged. This task documents the
end state. docs/arch/README.md already indexes vmcp-library.md as entry #13
(docs/arch/README.md:109) — only the linked content changes here, not the
index entry.

Parent Story: ##5433
Dependencies: ##5445 (P3.2 — server.New reduced to the wrapper; the docs describe the post-Phase-3 end state)
Blocks: #5447 (P4.2 — the runnable decorator example embedder, which the decorator section of these docs points to)

Acceptance Criteria

• docs/arch/vmcp-library.md documents the New(cfg) -> VMCP core constructor, the Serve(ctx, VMCP, serverCfg) -> *Server transport helper, and the decorator extension model (decorators wrap an inner VMCP and may only subtract reachability, never widen access).
• docs/arch/vmcp-library.md documents the (*Server).Handler(ctx) outermost-wrapping pattern (RFC §"Extending the Transport Layer", case 2) as a first-class, supported transport-extension mechanism: an embedder mounts the fully-composed vMCP handler in its own mux, wraps it with its own outermost middleware, and serves sibling routes — without reordering vMCP's internal chain (C3).
• docs/arch/vmcp-library.md makes clear that server.New is retained as a stable thin wrapper over Serve(ctx, New(cfg), serverCfg) (the brood-box / thv vmcp serve path is unchanged), and that VMCP/New/Serve are additive.
• The vmcp-library.md stability table reflects the additive API — the root pkg/vmcp and pkg/vmcp/server rows acknowledge the new VMCP interface + New/Serve alongside the still-Stable server.New/Start/Stop, with stability levels assigned consistently with the existing table.
• docs/arch/10-virtual-mcp-architecture.md overview reflects the domain/transport seam (New = core/domain, Serve = transport) so the high-level picture is consistent with the library doc.
• pkg/vmcp/doc.go package docs describe the domain/transport split (VMCP + New for domain logic; Serve for transport) and the decorator extension mechanism; the stale "capabilities are stored in request context by the discovery middleware" usage note (pkg/vmcp/doc.go:129-130) is corrected/removed since the transport now calls VMCP.ListTools/CallTool directly.
• docs/arch/README.md index entry Trigger CI on PRs #13 is unchanged (only the linked vmcp-library.md content changes).
• task docs runs as a guard with no generated diff (no CLI flags added; generated docs/cli/thv_vmcp_*.md expected unchanged).
• pkg/vmcp/doc.go retains its SPDX header and remains valid (the package still builds; task build / task lint-fix clean).
• PR is docs-only (a docs-only PR may exceed the standard 400-LOC / 10-file limits per the repo's docs-only exception); each doc edit is one logical change.
• server.New signature and observable behavior unchanged (no code change in this task).
• Code reviewed and approved.

Technical Approach

Recommended Implementation

Edit the three doc targets to describe the post-refactor public API. Pull the
canonical shapes and wording from architecture.md so the docs match what was
actually built:

1. docs/arch/vmcp-library.md — Extend the "Library Embedding Pattern"
   section so it presents the new public API as the recommended path:
   • Document New(cfg *vmcp.Config) (vmcp.VMCP, error) (core/domain) and
     Serve(ctx, v vmcp.VMCP, cfg *server.ServerConfig) (*server.Server, error)
     (transport), and note that server.New(...) is preserved as a stable thin
     wrapper Serve(ctx, New(deriveCoreConfig(cfg)), deriveServerConfig(cfg))
     (architecture.md "Overview" lines 11-17, "API Contracts" lines 84-134).
   • Add a short decorator extension subsection: a decorator wraps an inner
     VMCP, holds only inner VMCP, and may only subtract reachability
     (filter List* output / refuse a call before delegating) — it cannot widen
     access (architecture.md "Core Principles" Bump golangci/golangci-lint-action from 2f856675483cb8b9378ee77ee0beb67955aca9d7 to 4696ba8babb6127d732c3c6dde519db15edab9ea #3, lines 355-357). Point forward
     to the runnable example (P4.2 Runnable decorator example embedder #5447) as "see the decorator example embedder".
   • Add a short transport-extension note documenting (*Server).Handler(ctx)
     outermost-wrapping (RFC §"Extending the Transport Layer", case 2): an embedder
     mounts the fully-composed vMCP handler in its own mux, wraps it with its own
     outermost middleware, and serves sibling routes — without reordering vMCP's
     internal chain. This is a first-class supported pattern alongside
     New/Serve/decorators (C3).
   • Refresh the stability table (lines 41-62): the root pkg/vmcp row and
     the pkg/vmcp/server row should acknowledge the additive VMCP interface +
     New/Serve while keeping server.New/Start/Stop marked Stable.
     Update the brood-box "Reference Implementation" steps (lines 26-35) only if
     needed to mention the additive New/Serve path; do not change the
     statement that brood-box requires no changes.
2. docs/arch/10-virtual-mcp-architecture.md — Update the "Overview" /
   "Architecture" prose (lines 5-19+) so the high-level description names the
   domain/transport seam (New builds the core VMCP; Serve owns HTTP + MCP
   protocol transport). Keep the change scoped to the overview framing; do not
   rewrite the full document.
3. pkg/vmcp/doc.go — Update the package doc comment (158 lines):
   • In "# Architecture" / "# Core Concepts", describe the domain/transport split
     (the VMCP interface + New are the domain/core seam; Serve is the
     transport seam) and the decorator extension mechanism.
   • Fix the stale usage-example comment that says capabilities are stored in
     request context by the discovery middleware (doc.go:129-130) — the
     transport now calls VMCP.ListTools/CallTool directly (architecture.md
     "Discovery-into-Context Replacement", lines 331-345). Keep comments
     synchronized with code (go-style "Keep Comments Synchronized With Code").
   • Preserve the SPDX header (lines 1-2) and the package vmcp clause.

Keep the prose accurate to the merged code: identity is an explicit *auth.Identity
parameter on every VMCP method (never read from context), mcp-go types do not
cross the VMCP boundary, and server.New is retained (not deprecated).

Patterns & Frameworks

• Match the existing doc voice and structure in docs/arch/ (Markdown,
  stability-table format already in vmcp-library.md).
• Source the API shapes and rationale from architecture.md "API Contracts"
  (the VMCP interface, New, Serve, ServerConfig) and "Core Principles"
  rather than re-deriving them.
• go-style "Keep Comments Synchronized With Code" — every doc statement
  describing old behavior (e.g. discovery-middleware-into-context) must be
  updated to the new direct-call model.
• .claude/rules/go-style.md SPDX header preserved on doc.go; this is a
  doc-comment-only edit (no symbols added/removed).

Code Pointers

• docs/arch/vmcp-library.md — the library embedding doc to update. Stability table at lines 41-62 (currently pkg/vmcp/server row at line 47 describes only New/Start/Stop; root pkg/vmcp row at line 43); brood-box reference steps at lines 26-35 (currently describe only server.New(...)); "Library Embedding Pattern" at lines 11-35.
• docs/arch/10-virtual-mcp-architecture.md — overview to refresh; "Overview" lines 5-15, "Architecture" prose + mermaid lines 17-40.
• pkg/vmcp/doc.go (158 lines) — package docs to rewrite for the domain/transport seam + decorator extension; stale context-storage comment at lines 129-130; SPDX header at lines 1-2; package vmcp at line 158.
• docs/arch/README.md:109 — index entry Trigger CI on PRs #13 ([vMCP Library Embedding](vmcp-library.md)); must remain unchanged (index unchanged, only linked content changes).
• architecture.md "Phase 4 — Docs + example" (P4.1, lines 434-437); "Overview" (11-17); "API Contracts" — VMCP/New/Serve/ServerConfig (84-134, 243-275); "Core Principles" Do we want the container monitor? #2, Bump golangci/golangci-lint-action from 2f856675483cb8b9378ee77ee0beb67955aca9d7 to 4696ba8babb6127d732c3c6dde519db15edab9ea #3, Implement secret injection #5 (351-360); "Discovery-into-Context Replacement" (331-345).
• research.md "Phase-4 docs targets confirmed" (line 302) — confirms the three targets, the README index is untouched, and task docs is expected to be a no-op.

Component Interfaces

No new code interfaces are defined by this task. The docs describe the
already-merged additive public API (shapes from architecture.md, for reference
when writing the prose):

// Documented in vmcp-library.md + doc.go (already implemented by prior phases):
func New(cfg *Config) (VMCP, error)                                   // core / domain
func Serve(ctx context.Context, v VMCP, cfg *ServerConfig) (*Server, error) // transport
// server.New stays a stable thin wrapper over Serve(ctx, New(...), ...):
//   func New(ctx, cfg, rt, backendClient, discoveryMgr, backendRegistry, workflowDefs) (*Server, error)

// Decorator extension model to document: wrap inner VMCP, subtract-only.
type filteringDecorator struct{ inner VMCP }
// ListTools filters inner's output; CallTool refuses-then-delegates. It cannot
// widen access because backends are reachable only through inner.

Testing Strategy

This is a docs-only change; verification is via build/lint/docs guards and
review, not new unit tests.

Guards / Verification

• task docs produces no diff (no CLI flags added; generated docs/cli/thv_vmcp_*.md unchanged) — the guard required by architecture.md "Constraints" (line 492) and research.md (line 302).
• task build / task lint-fix clean (pkg/vmcp/doc.go still compiles as a valid doc comment with its SPDX header intact).
• Markdown links in the edited docs resolve (notably the cross-links in vmcp-library.md "Related Documentation" and any new forward-reference to the decorator example).

Review checks

• Documented API shapes (New/Serve/ServerConfig, decorator subtract-only contract) match the merged code (architecture.md API Contracts).
• No stale references remain to the discovery-middleware-into-context aggregation path in doc.go.
• Stability table assigns the additive API rows consistently with the existing Stable/Experimental/Internal scheme.

Edge Cases

• docs/arch/README.md index entry Trigger CI on PRs #13 is byte-for-byte unchanged.
• No claim that server.New is deprecated or removed (it is retained as a stable wrapper).

Out of Scope

• The runnable decorator example embedder itself — that is P4.2 Runnable decorator example embedder #5447 (P4.2). This task only adds the decorator explanation and a forward reference to that example.
• Any production Go behavior change in pkg/vmcp/ (this is docs-only; the structural work landed in Phases 1-3).
• Adding/removing CLI flags or changing the thv vmcp serve surface (would change generated docs/cli/*.md; task docs must stay a no-op).
• Changing the docs/arch/README.md index or adding new arch-doc entries (only the linked vmcp-library.md content changes).
• Restructuring/rewriting docs/arch/10-virtual-mcp-architecture.md beyond the overview framing needed to reflect the seam.

References

• RFC THV-0076: vMCP Core Interface — §Documentation (Phase 4)
• architecture.md "Phase 4 — Docs + example" (P4.1, lines 434-437); "Overview" (11-17); "API Contracts" (84-134, 243-275); "Core Principles" Bump golangci/golangci-lint-action from 2f856675483cb8b9378ee77ee0beb67955aca9d7 to 4696ba8babb6127d732c3c6dde519db15edab9ea #3 (355-357); "Discovery-into-Context Replacement" (331-345); "Constraints" task docs guard (492)
• research.md "Phase-4 docs targets confirmed" (line 302)
• Doc targets: docs/arch/vmcp-library.md, docs/arch/10-virtual-mcp-architecture.md, pkg/vmcp/doc.go; index (unchanged): docs/arch/README.md:109
• Parent story: #Phase 4: Documentation + runnable decorator example #5433
• Epic: vMCP interface refactor #5419