Deduplicate custom background tasks#806
Open
2chanhaeng wants to merge 9 commits into
Open
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughAdds a complete custom background task API to Fedify: ChangesCustom Background Task API
Sequence Diagram(s)sequenceDiagram
participant App
participant Federatable
participant Context
participant enqueueTasks
participant KvStore
participant MessageQueue
participant Worker as Task Worker (`#listenTaskMessage`)
participant TaskHandler
App->>Federatable: defineTask("sendDigest", { schema, handler })
Federatable-->>App: TaskDefinition handle
App->>Context: enqueueTask(handle, payload, { deduplicationKey: "k" })
Context->>enqueueTasks: task, [payload], options
enqueueTasks->>enqueueTasks: planDeduplication → cas plan
enqueueTasks->>enqueueTasks: encodeTaskMessage (codec.encode + envelope)
enqueueTasks->>KvStore: cas(taskDeduplication/k, token, ttl)
KvStore-->>enqueueTasks: claimed (proceed=true)
enqueueTasks->>MessageQueue: enqueue(TaskMessage)
MessageQueue-->>enqueueTasks: ok
MessageQueue-->>Worker: deliver TaskMessage
Worker->>Worker: codec.decode(schema, message.data)
Worker->>TaskHandler: handler(context, validatedPayload)
TaskHandler-->>Worker: ok
Note over App,KvStore: Duplicate enqueue within TTL
App->>Context: enqueueTask(handle, payload2, { deduplicationKey: "k" })
Context->>enqueueTasks: task, [payload2], options
enqueueTasks->>KvStore: cas(taskDeduplication/k, token2, ttl)
KvStore-->>enqueueTasks: lost (proceed=false)
enqueueTasks-->>Context: (no enqueue)
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related issues
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Member
Author
|
@codex review |
|
✅ Action performedReview finished.
|
|
To use Codex here, create an environment for this repo. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a custom background task API for Fedify 2.x.x, allowing applications to define, enqueue, and process arbitrary background jobs. It adds defineTask to Federation and FederationBuilder, and enqueueTask/enqueueTaskMany to Context. Payloads are validated using Standard Schema and serialized with devalue to support complex types and Activity Vocabulary objects. The API supports customizable retry policies, queue routing, and deduplication (either native or via a key-value fallback). Additionally, the PR updates testing utilities, documentation, and mise tasks. Feedback on the changes suggests implementing a depth limit in the recursive revival traversal of TaskCodec to mitigate potential denial-of-service (DoS) attacks or stack overflows from deeply nested payloads.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a custom background task API for Fedify, enabling the definition, enqueuing, and background processing of arbitrary application-defined jobs with robust serialization, retry policies, queue routing, and deduplication. The code review feedback highlights several critical compatibility, security, and performance improvements: replacing the ES2024 Array.fromAsync calls with standard loops to maintain compatibility with Node.js 18 and 20, implementing a depth limit in the recursive deserialization function to prevent potential Denial of Service (DoS) attacks, preserving null prototypes during object revival to avoid prototype pollution, and optimizing the enqueue pipeline by validating payloads before claiming deduplication keys.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
packages/testing/src/mock.ts (1)
515-527:⚠️ Potential issue | 🟠 Major | ⚡ Quick win
queueStartedshould not flip to true for non-outbox queue startup.Starting only
"task"currently setsqueueStarted = true, so latersendActivity()is misclassified as queued outbox delivery.Suggested fix
async startQueue( contextData: TContextData, options?: FederationStartQueueOptions, ): Promise<void> { this.contextData = contextData; - this.queueStarted = true; // If a specific queue is specified, only activate that one if (options?.queue) { this.activeQueues.add(options.queue); } else { // If no specific queue, activate all four this.activeQueues.add("inbox"); this.activeQueues.add("outbox"); this.activeQueues.add("fanout"); this.activeQueues.add("task"); } + this.queueStarted = this.activeQueues.has("outbox"); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/testing/src/mock.ts` around lines 515 - 527, The queueStarted flag is being unconditionally set to true when any queue starts, but it should only be true when the "outbox" queue is activated. This causes sendActivity() to incorrectly classify non-outbox queue activity as queued outbox delivery. Move the assignment of this.queueStarted = true to only execute when either no specific queue is specified (meaning all queues including outbox are activated) or when the specific queue being activated is "outbox". Update the conditional logic to check if options?.queue is either undefined or explicitly equals "outbox" before setting queueStarted to true.packages/fedify/src/federation/middleware.ts (1)
902-906: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick winWire task queues into the existing queue observability path.
Task queues are omitted from
#queueDepthGaugeEntries, andmessage.type === "task"bypasses the span/started/outcome/in-flight wrapper used by inbox/outbox/fanout workers. That makes task backlog and handler failures invisible to the existing queue metrics/traces even though tasks are first-class queued work.Also applies to: 1239-1240
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/fedify/src/federation/middleware.ts` around lines 902 - 906, Task queues are not included in the `#queueDepthGaugeEntries` array and the task message type handler bypasses the observability wrapper (span/started/outcome/in-flight) used by inbox/outbox/fanout workers, making task metrics and traces invisible. Add the task queue entry to `#queueDepthGaugeEntries` following the same pattern as the inbox, outbox, and fanout entries with role "task" and the corresponding task queue. Additionally, update the message handler to apply the same observability wrapper and metrics collection to task queue messages that are currently applied to other queue types, ensuring all queued work is tracked consistently.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@AGENTS.md`:
- Around line 137-139: Replace the placeholder syntax `mise run
test:<runtime:deno,node,bun>` with concrete, runnable task names. List the
actual commands explicitly as `mise run test:deno`, `mise run test:node`, and
`mise run test:bun` so users can copy and paste them directly without having to
interpret pseudo-syntax. This makes the documentation immediately actionable and
prevents confusion.
In `@packages/fedify/src/federation/context.ts`:
- Around line 460-473: Update the JSDoc comment for the `enqueueTaskMany` method
to accurately document its behavior with deduplicated batches. First, modify the
statement that currently describes falling back to parallel single enqueues to
make it conditional, clarifying that this fallback only occurs when the queue's
bulk enqueue operation is available or in certain situations. Second, add an
additional `@throws {TypeError}` entry documenting that a TypeError is thrown
when atomic `enqueueMany` is unavailable for deduplicated multi-item batches,
ensuring the documentation fully captures all error conditions the method can
encounter.
In `@packages/fedify/src/federation/middleware.test.ts`:
- Around line 10568-10573: The module-scoped taskFederationOptions object reuses
a single MemoryKvStore instance across all tests, causing potential state
pollution and test interdependencies. Move the taskFederationOptions definition
from module scope into each test function or a beforeEach hook, ensuring a fresh
MemoryKvStore instance is created for each test. Then update all test functions
that reference the module-scoped taskFederationOptions to use the local version
instead.
In `@packages/fedify/src/federation/middleware.ts`:
- Around line 944-949: The resolveTaskQueue method correctly identifies which
queue to use for task messages, but the _startQueueInternal method does not
properly handle the "task" selector when that queue is shared with standard
queues or routed as a fallback. Fix _startQueueInternal to map the "task"
selector to its resolved task queue instance by calling
resolveTaskQueue("task"), then ensure that resolved queue is started/listened to
exactly once by tracking started queues appropriately. This ensures that when
queue.task is configured as the same instance as inbox/outbox/fanout, or when
task messages are fallback-routed to outboxQueue, the actual queue receiving
those messages has an active listener.
In `@packages/fedify/src/federation/mq.ts`:
- Around line 454-462: The error check in the enqueueMany method around the
deduplicationKey validation is too strict and prevents single-item batches from
using the deduplicated enqueue fallback. Modify the condition to only throw the
TypeError when both deduplicationKey is present AND the batch contains multiple
items. For single-item batches, allow the code to proceed with the fallback to
the wrapped queue's enqueue() method, since atomicity is maintained for
individual items. Check the length of the messages array being enqueued (this
will be available through the method parameters) and only enforce the
restriction when multiple items are being batched.
In `@packages/fedify/src/federation/tasks/codec.ts`:
- Around line 7-15: The TaskCodec class and its public methods (serialize,
deserialize, and validate) lack JSDoc documentation. Add comprehensive JSDoc
comments for the TaskCodec class constructor describing its purpose and the
options parameter, the serialize method explaining that it converts data to a
JSON string and documenting any potential errors, the deserialize method
explaining that it parses a JSON string back into data and documenting any
parsing errors, and the validate method documenting its validation behavior and
error conditions. Each JSDoc should follow the established pattern in the
codebase by including descriptions of parameters, return types, and any errors
that might be thrown.
In `@packages/fedify/src/federation/tasks/tasks.test.ts`:
- Around line 302-323: The startQueue() task worker test currently only covers
the scenario with a distinct task queue, but lacks coverage for when the
effective task queue falls back to shared instances. Add additional test steps
within the same test function to cover the cases where queue.task is undefined
(falling back to the outbox queue) and where queue.task shares an instance with
inbox, outbox, or fanout. For each fallback scenario, verify that only the
appropriate fallback queue (like outbox when task is undefined) has its
listenCount incremented to 1 while the other queues remain at 0, using the same
assertion pattern as the existing test step but with the appropriate queue
configuration.
In `@packages/fedify/src/testing/tasks.ts`:
- Around line 127-134: The MockQueue.listen method does not handle pre-aborted
signals. If the signal passed via options is already aborted before the event
listener is registered, the abort event will never fire and the Promise will
never resolve, causing tests to hang. Check if options?.signal?.aborted is
already true and resolve the Promise immediately in that case. Only register the
abort event listener when the signal is not already aborted.
In `@packages/testing/src/mock.test.ts`:
- Around line 1728-1830: The tests in mock.test.ts are using the old Deno-style
test harness (`@std/assert`) instead of the required Node.js test harness for the
packages/testing package. Replace the import statements to use `import { test }
from "node:test"` and `import assert from "node:assert/strict"`, then update all
assertion calls throughout the tests: replace each `assertEquals()` call with
`assert.equal()` or `assert.strictEqual()`, replace each `assertRejects()` call
with `assert.rejects()`, and ensure the test function signatures and assertion
call syntax conform to the Node.js assert/strict API.
---
Outside diff comments:
In `@packages/fedify/src/federation/middleware.ts`:
- Around line 902-906: Task queues are not included in the
`#queueDepthGaugeEntries` array and the task message type handler bypasses the
observability wrapper (span/started/outcome/in-flight) used by
inbox/outbox/fanout workers, making task metrics and traces invisible. Add the
task queue entry to `#queueDepthGaugeEntries` following the same pattern as the
inbox, outbox, and fanout entries with role "task" and the corresponding task
queue. Additionally, update the message handler to apply the same observability
wrapper and metrics collection to task queue messages that are currently applied
to other queue types, ensuring all queued work is tracked consistently.
In `@packages/testing/src/mock.ts`:
- Around line 515-527: The queueStarted flag is being unconditionally set to
true when any queue starts, but it should only be true when the "outbox" queue
is activated. This causes sendActivity() to incorrectly classify non-outbox
queue activity as queued outbox delivery. Move the assignment of
this.queueStarted = true to only execute when either no specific queue is
specified (meaning all queues including outbox are activated) or when the
specific queue being activated is "outbox". Update the conditional logic to
check if options?.queue is either undefined or explicitly equals "outbox" before
setting queueStarted to true.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 37da6ef5-700b-44ba-83f7-0f5f4faec66b
⛔ Files ignored due to path filters (2)
deno.lockis excluded by!**/*.lockpnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (31)
AGENTS.mdCHANGES.mddocs/.vitepress/config.mtsdocs/manual/tasks.mdmise.tomlpackages/fedify/deno.jsonpackages/fedify/package.jsonpackages/fedify/src/federation/builder.tspackages/fedify/src/federation/context.tspackages/fedify/src/federation/federation.tspackages/fedify/src/federation/middleware.test.tspackages/fedify/src/federation/middleware.tspackages/fedify/src/federation/mod.tspackages/fedify/src/federation/mq.test.tspackages/fedify/src/federation/mq.tspackages/fedify/src/federation/queue.tspackages/fedify/src/federation/tasks/codec.test.tspackages/fedify/src/federation/tasks/codec.tspackages/fedify/src/federation/tasks/enqueue.test.tspackages/fedify/src/federation/tasks/enqueue.tspackages/fedify/src/federation/tasks/mod.tspackages/fedify/src/federation/tasks/task.tspackages/fedify/src/federation/tasks/tasks.test.tspackages/fedify/src/testing/context.tspackages/fedify/src/testing/mod.tspackages/fedify/src/testing/tasks.tspackages/testing/package.jsonpackages/testing/src/context.tspackages/testing/src/mock.test.tspackages/testing/src/mock.tsscripts/check_fixture_usage.ts
Context.enqueueTask() and enqueueTaskMany() now accept a
deduplicationKey requesting at-most-once enqueue for tasks that share
it (new TaskEnqueueOptions.deduplicationKey).
Resolution follows the queue and key-value store capabilities:
- A queue declaring the new MessageQueue.nativeDeduplication owns the
check; the key is forwarded through the new
MessageQueueEnqueueOptions.deduplicationKey.
- Otherwise Fedify applies a best-effort guard through the optional
KvStore.cas primitive under a new taskDeduplication key prefix,
tunable with the new FederationOptions.taskDeduplicationTtl and
taskDeduplicationFallback options.
For enqueueTaskMany(), a single key governs the whole batch. A native
queue that does not implement enqueueMany() cannot express batch-level
at-most-once with a per-message key, so such a multi-item enqueue is
rejected with a TypeError instead of silently leaking duplicates.
Configuration errors that are decidable without a payload (a native
queue lacking enqueueMany, or a closed fallback without cas) are
checked before payloads are validated and encoded, so they reject
before any user schema runs or any key is reserved.
fedify-dev#798
Assisted-by: Claude Code:claude-opus-4-8
The #enqueueTasks and #encodeTaskMessage methods made ContextImpl oversized, so move the handle validation, deduplication planning, payload encoding, and queue dispatch into a new tasks/enqueue.ts module. ContextImpl now delegates to enqueueTasks(), passing only the small slice of itself (federation, codec, origin, data) the pipeline needs. Pull the shared task-test helpers (the schema factory, stock schemas, base federation options, and the recording MockQueue) into a new testing/mq-tasks.ts module, and split the enqueue-specific cases out of tasks.test.ts into enqueue.test.ts. Teach the fixture-usage check to expand glob patterns in its allowlist so the whole testing/ directory is covered by a single entry instead of one path per file. Assisted-by: Claude Code:claude-opus-4-8
Two branches both touched the task testing utilities and diverged: one split MockQueue and the shared schemas/options out into mq-tasks.ts, while the other kept evolving them in tasks.ts. After rebasing the common edits, consolidate everything back into a single tasks.ts and drop the now-redundant mq-tasks.ts. Assisted-by: Claude Code:claude-opus-4-8
The key-value deduplication path reserved a marker before dispatching to the queue but never undid it when the dispatch failed. A transient backend failure therefore left the marker behind, so the retry was silently deduplicated against a task that had never reached the queue. The cas claim now stores a unique token instead of a bare `true`, and a failed dispatch conditionally clears it (cas succeeds only while the stored value is still our token). The conditional clear keeps a stale rollback from deleting a marker that another concurrent enqueue has already re-claimed. A rollback that itself fails is logged and swallowed so the original enqueue error still reaches the caller. The enqueueMany requirement for deduplicated multi-item batches now keys on whether deduplication is actually applied—a native queue or the cas fallback—rather than on nativeDeduplication alone. Under the "open" fallback (no native dedup, no cas) no marker is taken, so the batch fans out without deduplication instead of throwing. ParallelMessageQueue likewise rejects a deduplicated batch when the wrapped queue lacks enqueueMany, since fanning out cannot carry one key atomically. fedify-dev#798 Assisted-by: Claude Code:claude-opus-4-8
startQueue({ queue: "task" }) started no worker when the effective
task queue was the outbox fallback (no dedicated task queue) or an
instance shared with inbox/outbox/fanout: the standard-queue branches
were gated on their own selector, and the task branch skipped a queue
that equalled a standard one. Enqueued tasks then had no listener.
Rather than patch the branch conditions, _startQueueInternal now lists
every (role, queue) target—including the outbox fallback and each
per-task queue—and starts each instance at most once through a single
identity Set, which also subsumes the four per-role started flags and
the dedicated-task-queue Set into one field.
fedify-dev#806 (comment)
fedify-dev#806 (comment)
Assisted-by: Claude Code:claude-opus-4-8
MockQueue.listen() only resolved from an "abort" event listener, so a
signal already aborted before the call left the promise pending
forever and could hang queue-listener teardown in tests. Return a
resolved promise when the signal is already aborted, and register the
listener with { once: true }.
fedify-dev#806 (comment)
Assisted-by: Claude Code:claude-opus-4-8
ParallelMessageQueue.enqueueMany threw whenever the wrapped queue lacked enqueueMany and a deduplicationKey was set, regardless of how many messages the batch held. The atomicity limitation it guards against only applies to multi-item fan-out: a single-item batch can carry its deduplicationKey straight to the wrapped queue's enqueue() without splitting it across calls. Narrow the guard to multi-item batches and fall back to enqueue() for a single item, so direct ParallelMessageQueue callers are no longer rejected for a case that is perfectly safe. The task enqueue path already routes single items through enqueue() before reaching this branch, so its behavior is unchanged. fedify-dev#806 (comment) Assisted-by: Claude Code:claude-opus-4-8
The task suites in middleware.test.ts spread a module-scope taskFederationOptions object that held a single shared MemoryKvStore, so every createFederation() call reused the same key-value store. As more task cases are added, a deduplication marker written by one test could leak into another, making the suite order-dependent and flaky. Replace the shared constant with a mockOptions() factory that returns fresh options—including a fresh MemoryKvStore—on every call, and spread mockOptions() at each use site. fedify-dev#806 (comment) Assisted-by: Claude Code:claude-opus-4-8
Address review feedback on the custom background task API:
- The enqueueTaskMany JSDoc makes its fallback wording conditional
and now documents the TypeError thrown when a deduplicated
multi-item batch cannot be enqueued atomically because the queue
does not implement bulk enqueue.
- TaskCodec is marked @internal, and serialize(), deserialize(), and
the static validate() helper each gain a one-line JSDoc.
- The deduplication section of the manual lists Cloudflare Workers KV
among the key-value stores that do not yet implement cas.
fedify-dev#806
Assisted-by: Claude Code:claude-opus-4-8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #798, the second sub-issue of #206.
Important
This branch is stacked on #803 (#797, the first sub-issue), which is not yet merged. Until #803 lands, this PR's range includes #803's commits too, so the diff shown here is the core
defineTask/enqueueTaskAPI plus this sub-issue's deduplication work. Please merge #803 first; once it lands onfeat/custom-worker, rebasing this branch drops the overlap and leaves this sub-issue's deduplication changes alone.Summary
Background tasks frequently need at-most-once-per-key enqueue semantics: a digest mailer must not send twice when a request is retried, and a cleanup job should coalesce duplicate triggers. This sub-issue adds an opt-in
deduplicationKeyto the task enqueue path, mirroring the native-capability flag pattern thatnativeRetrial(#250) established—a backend that deduplicates natively owns the check; otherwise Fedify provides a best-effort key–value fallback with the race-condition tradeoff documented explicitly.It is kept separate from the core API (#797 / #803) so the first PR stays small and the deduplication semantics—including the documented best-effort limitation—get their own reviewable boundary.
Public API
MQ-layer primitives
These are message-queue-layer primitives, not task-layer concepts, so they survive a future
Workerextraction unchanged and are reusable by any enqueue path:InProcessMessageQueuedeclaresnativeDeduplication = false;ParallelMessageQueueinherits the wrapped queue's flag and rejects a batch carrying adeduplicationKeywhen the wrapped queue has noenqueueMany(it would otherwise fan out to singleenqueue()calls that cannot share one key atomically).Task-API surface
A new
taskDeduplicationkey–value prefix (default["_fedify", "taskDeduplication"]) holds fallback markers, kept separate fromactivityIdempotence.Resolution path
The enqueue pipeline was extracted out of
ContextImplinto a dedicatedtasks/enqueue.tsmodule (handle validation, deduplication planning, payload encoding, and queue dispatch live in one place). Deduplication is decided once, before any payload is encoded:nativeDeduplication: true, Fedify forwardsdeduplicationKeyinMessageQueueEnqueueOptionsand the backend owns the check. The key–value store is never touched.KvStoreexposes the optional compare-and-swap (KvStore.cas) primitive, Fedify claims the key under thetaskDeduplicationprefix withtaskDeduplicationTtl. A present marker skips the enqueue; a won claim proceeds. If the subsequent dispatch fails, the marker is rolled back (via a conditionalcasclear) so a transient failure does not suppress the retry.nativeDeduplication, and aKvStorewithoutcas), behavior followstaskDeduplicationFallback:"open"(default) proceeds without deduplication after a debug-level log;"closed"throws aTypeErrorbefore enqueuing.Among the first-party adapters, the in-memory, Deno KV, SQLite, and MySQL key–value stores implement
cas; PostgreSQL and Redis do not yet, so those deployments take the fallback branch until per-adapter follow-ups add it.Batch semantics
For
enqueueTaskMany, a singlededuplicationKeyapplies to the whole batch—it enqueues as a unit or is skipped as a unit, never partially. Per-item deduplication means callingenqueueTaskin a loop, each with its own key. When deduplication is actually applied (a native queue, or aKvStorewithcas), a multi-item batch with adeduplicationKeyon a queue withoutenqueueManyis rejected rather than risking duplicates, since fanning the key across separateenqueue()calls cannot enqueue the batch as one unit. Under the"open"fallback no marker is taken, so the batch simply fans out.Documented limitation
The key–value fallback is best-effort, not transactional: the marker write and the enqueue are separate operations. Fedify rolls the marker back on enqueue failure, but a crash before that rollback, the
"open"fallback under concurrency, a non-atomic third-partycas, or key reuse within the TTL window can still admit a duplicate or suppress a task. Cleanup is by TTL expiry, not active deletion on handler success (active cleanup introduces a success→crash-before-delete window; deferred). This is stated in the public JSDoc fordeduplicationKeyand in a warning callout in docs/manual/tasks.md. Deployments needing strict guarantees use a queue withnativeDeduplication: true.Out of scope
nativeDeduplication: true/casto the remaining first-party adapters (PostgreSQL, Redis) — tracked as per-adapter follow-ups; this sub-issue ships the core flag plus the key–value fallback.Acceptance criteria
deduplicationKeyon anativeDeduplication: truequeue is forwarded; Fedify does not write to the key–value store.deduplicationKeyon a default queue: a second enqueue inside the TTL is skipped; re-enqueue after TTL expiry succeeds.taskDeduplicationFallback: "closed"throws synchronously when no conditional write is available;"open"proceeds with a debug log.taskDeduplicationkey–value prefix does not collide withactivityIdempotence.enqueueTaskManyapplies one batch-leveldeduplicationKey.Tests
tasks/enqueue.test.tscovers the three resolution paths, TTL skip/expiry, the"open"/"closed"fallback, the prefix isolation, batch-level dedup, theenqueueMany-required rejection, and marker rollback on dispatch failure.mq.test.tscovers the newnativeDeduplicationflag and theParallelMessageQueuebatch rejection.@fedify/testing) honorsdeduplicationKeyso applications can assert dedup behavior in their own tests.@fedify/fixturetest()and pass under Deno, Node.js, and Bun.Notes for reviewers
feat/custom-workerand this is stacked on the unmerged Add custom background task API (defineTask/enqueueTask) #803; see the callout at the top.KvStore.casis a pre-existing primitive; this PR consumes it rather than introducing it.AI disclosure
Assisted-by: Claude Code:claude-opus-4-8
Assisted-by: Codex:gpt-5.5
Codex was used only in review.