Feat/websub hub#36
Open
andrewshell wants to merge 19 commits into
Open
Conversation
Lay out the full WebSub hub implementation plan in TODO.md as ordered TDD vertical slices spanning @rsscloud/core, @rsscloud/express, and apps/server. Settled decisions: async-202 intent verification behind an in-process best-effort VerificationScheduler seam (persisted queue + retry deferred); both thin-publish (re-fetch) and fat-ping content sourcing; honor the requested lease clamped to a configurable range; HMAC-SHA256 signatures. Headline use case: an rssCloud publisher adds <link rel="hub"> and keeps pinging via rssCloud, while WebSub subscribers to the same topic receive full content distribution — which falls out of core's existing resource-keyed fan-out. Each flow gets an e2e acceptance test as its TDD outer loop, and server integration (plugin registration, route mount, config) is spelled out per file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-0002 captures the settled WebSub hub design: async-202 intent verification behind an in-process best-effort VerificationScheduler seam (a persisted queue + retry is a future refactor behind the same seam), plus the lease (honor-requested-clamped) and HMAC-SHA256 signature decisions. CONTEXT.md gains a WebSub vocabulary cluster (Topic vs Resource, Callback vs Subscription.url, Intent verification, VerificationScheduler, Lease, Content distribution, Fat ping, X-Hub-Signature), ties the Hub and Notification entries to their WebSub roles, and adds a dialogue exchange. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add parseSubscribe in protocols/websub-dispatcher.ts: validates hub.mode
(subscribe), hub.callback (a valid absolute URL), and hub.topic
(present), returning {status:400} for anything malformed. A valid
request builds a 'websub' SubscribeRequest directly
(callbackUrl=hub.callback, resourceUrls=[hub.topic]) without
buildSubscribeRequest, which gates on rssCloud-only protocols and
assembles callbacks from port/path/domain.
Internal for now; createWebSubDispatcher and the index export land with
the express factory (S1.4). 100% coverage maintained.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add createWebSubProtocolPlugin (protocols: ['websub']). verify() always performs the WebSub intent-verification GET — never the rssCloud same-domain test-notify, so it ignores diffDomain — appending hub.mode=subscribe / hub.topic / hub.challenge to the callback (preserving any existing query) and requiring a 2xx with an exact challenge echo, else throwing. fetch and the challenge generator are injectable. deliver() is an interim stub reporting failure (it must not throw; the engine's deliverTo does not catch); real content distribution is S2.1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce VerificationScheduler — the seam behind WebSub's async-202 intent verification. The default in-process scheduler runs each verify→persist task on the microtask queue (best-effort, one attempt) and routes a rejection to onError; a future persisted queue can satisfy the same interface (ADR-0002). core.acceptSubscription(req) returns immediately and schedules the work via the scheduler. It is a new caller of the unchanged subscribe, so a successful verify persists the subscription and a refusal persists nothing — the synchronous rssCloud subscribe path is untouched. The default scheduler surfaces a thrown task through the existing error event (scope: websub-verification), coercing any non-Error throwable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add createWebSubDispatcher in core: parse the hub.* body and, on a valid
subscribe, hand the built request to core.acceptSubscription and answer
202; a malformed body is 400. Add the thin express websub({ core })
factory mirroring ping/pleaseNotify — it parses the urlencoded body and
copies the dispatcher's status onto the reply, with the hub.* logic
owned by core. Export both, plus createWebSubProtocolPlugin, from their
package indexes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Register createWebSubProtocolPlugin in the core composition root so
core.subscribe accepts the 'websub' protocol, and mount
websub({ core }) at config.webSubPath (default /websub). Add WEBSUB_PATH
and HUB_URL config (hubUrl defaults to domain/port/path; consumed once
content distribution lands). The plugin gets requestTimeoutMs for now;
hubUrl wiring follows with deliver().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the WebSub subscribe acceptance suite against the running server: a challenge-echoing callback (the existing mock's function responseBody returning req.query['hub.challenge']) is recorded after the async 202, polled via the test API; a refusing callback is never recorded within a bounded timeout; and a malformed hub.* body (missing callback/topic, or an unsupported mode) returns 400. Full suite: 138 e2e passing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement the WebSub plugin's deliver(): POST the changed feed body to each subscriber's callback, relaying the origin Content-Type verbatim (falling back to application/octet-stream when absent) and advertising the hub/self Link rels. Delivery follows 3xx redirects like the rssCloud REST notify path; any non-2xx is a failed delivery. The hub's public URL is injected as a createWebSubProtocolPlugin option and wired from config.hubUrl in apps/server. With this in place a single rssCloud ping already fans content out to WebSub subscribers through the engine's existing resource-keyed fan-out — no new publish path needed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a cross-protocol fan-out acceptance test: an rssCloud subscriber and a WebSub subscriber share one topic, and a single ordinary rssCloud /ping fires both — the rssCloud sub gets its notify, the WebSub callback gets a POST carrying the changed feed body, relayed Content-Type, and hub/self Link rels. This is the headline "free WebSub for rssCloud publishers" proof; no hub.mode=publish is involved. Extend the shared mock subscriber with content-capture: a catch-all bodyParser.text records raw, non-urlencoded POST bodies (the WebSub delivery) while leaving rssCloud notify bodies parsed as objects. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Authenticate content distribution for subscribers that supply a hub.secret. parseSubscribe carries the secret through as details.secret; the plugin then signs each delivery body with HMAC and adds X-Hub-Signature: <algo>=<hex>. No secret means no header. The HMAC algorithm is a plugin option (default sha256, names both the digest and the header method prefix), wired from a new WEBSUB_SIGNATURE_ALGO env knob in apps/server. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an authenticated-distribution suite: subscribe with a hub.secret, fire one rssCloud ping, and recompute HMAC-SHA256(secret, body) over the body the WebSub callback received to confirm it matches X-Hub-Signature. Cover the negative too — no hub.secret means no signature header. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
WebSub unsubscribe, like subscribe, must confirm the subscriber's intent before the hub acts — but core.unsubscribe has no verify hook. Add the verified path: - VerifyContext gains an optional `mode`, threaded onto the plugin's challenge GET as hub.mode (defaults to subscribe; rssCloud ignores it). - core.acceptUnsubscription schedules a challenge GET in unsubscribe mode and calls unsubscribe only once confirmed — a no-op when the sub is absent or the callback refuses to echo. - The websub dispatcher branches on hub.mode (subscribe/unsubscribe → 202, anything else → 400) via a shared hub.callback/hub.topic parser; the express factory's core Pick widens to acceptUnsubscription. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Subscribe, then drive hub.mode=unsubscribe: when the callback echoes the unsubscribe-mode challenge the subscription is removed; when it refuses, the subscription survives. Polls the store for removal since verification is async. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add WebSub lease handling. RssCloudConfig gains
webSubLease{Default,Min,Max}Secs; the dispatcher parses hub.lease_seconds
into details, and subscribeOne clamps it to [min, max] (or grants the
default when omitted), records the chosen value in details.leaseSeconds,
and maps it to whenExpires = now + chosen. The chosen lease is threaded
through VerifyContext so the plugin echoes hub.lease_seconds on the
subscribe challenge GET. removeExpired drops a lapsed lease unchanged.
Lease bounds are wired from WEBSUB_LEASE_* env in apps/server.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Subscribe with a below-minimum hub.lease_seconds and assert the chosen lease is clamped up to the bound, recorded in details, and echoed on the verification GET. Separately, force a recorded lease to lapse and confirm removeExpired drops it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Let a pure-WebSub publisher (no rssCloud ping) notify the hub that a topic changed via hub.mode=publish. The dispatcher parses the topic from hub.url (falling back to hub.topic) and calls a new core.acceptPublish, which — per WebSub §7 — acknowledges immediately (202) and re-fetches the topic out of band, reusing ping's existing fetch→payload→fanOut. A failed fetch is surfaced on the error event (scope websub-publish) rather than thrown. The dispatcher and express factory core Picks widen accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A WebSub subscriber subscribes to a topic, then a pure-WebSub publisher POSTs hub.mode=publish for it; poll for the out-of-band delivery and assert the subscriber receives the changed feed body. Also retarget the "unsupported hub.mode" rejection at a bogus mode now that publish is a supported mode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The WebSub hub is functionally complete (subscribe, content distribution, HMAC signatures, unsubscribe, leases, native publish). Remove TODO.md now that the roadmap is done — durable decisions live in docs/adr, CONTEXT.md, and git history per CLAUDE.md. Annotate CONTEXT.md's Fat ping entry as out of scope: it is non-standard (a PubSubHubbub-era extension with no WebSub wire format), so the hub only ever does thin publishes. The term is kept solely to explain the naming. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.