Skip to content

Add durable-functions@4.0.0 — Azure Functions Durable provider on the gRPC core (+ core host helpers, E2E CI, and release pipeline) - #282

Merged
YunchuWang merged 53 commits into
mainfrom
yunchuwang/functions-grpc-support
Jul 22, 2026
Merged

Add durable-functions@4.0.0 — Azure Functions Durable provider on the gRPC core (+ core host helpers, E2E CI, and release pipeline)#282
YunchuWang merged 53 commits into
mainfrom
yunchuwang/functions-grpc-support

Conversation

@YunchuWang

@YunchuWang YunchuWang commented Jun 29, 2026

Copy link
Copy Markdown
Member

TL;DR for reviewers

This PR began as "add two Functions gRPC helpers to the core worker" (the original title), but it now delivers the full Azure Functions Durable provider on the gRPC core. The headline change is a new npm package durable-functions@4.0.0 (packages/azure-functions-durable/, ~30 source files — does not exist on main) that re-implements the classic Durable Functions programming model on top of @microsoft/durabletask-js. It is the gRPC-core successor to the classic v3 (extension‑HTTP) durable-functions package and targets the @azure/functions v4 programming model. The core worker helpers, the E2E suite enablement, and the release pipeline are the supporting cast.

Scope: 58 files, +4,648 / −523, across four areas:

# Area What Key paths
1 New provider package (the bulk) durable-functions@4.0.0 on the gRPC core — v3‑compatible programming model packages/azure-functions-durable/**
2 Core host helpers (original scope) Single‑work‑item byte processors on TaskHubGrpcWorker packages/durabletask-js/src/worker/task-hub-grpc-worker.ts, src/index.ts
3 E2E suite + CI Wire the (pre‑existing) Functions host E2E suite to run green on PRs .github/workflows/functions-e2e-tests.yaml, test/e2e-functions/**
4 Build & release Pack + ESRP‑publish the new package eng/templates/build.yml, eng/ci/release.yml

1. New package: durable-functions@4.0.0 (packages/azure-functions-durable/)

A v3‑API‑compatible Azure Functions Durable provider built on the gRPC core, so existing v3 orchestrator/activity/entity/client code type‑checks and runs against the consolidated gRPC path. Added whole in this PR (not on main).

  • Registration surface (app.ts, app-client.ts, trigger.ts, input.ts, worker.ts): app.orchestration / app.activity / app.entity, the app.client.* durable‑client starter helpers (HTTP / timer / blob / queue / service‑bus / event‑hub / event‑grid / cosmos), and trigger + input bindings.
  • Client (client.ts, 616 lines): DurableFunctionsClient / DurableOrchestrationClient — start, terminate, suspend, resume, raiseEvent, getStatus, purge, rewind, restart. Includes _mapControlPlaneError, which translates opaque gRPC control‑plane errors back to the v3 error surface (see Error fidelity below).
  • Execution contexts (orchestration-context.ts, entity-context.ts): DurableOrchestrationContext / DurableEntityContext plus wrapOrchestrator / wrapEntity adapters bridging the classic generator model to the core.
  • Status / query / types: orchestration-status.ts, orchestration-filter.ts, metadata.ts, retry-options.ts, entity-id.ts, entity-state-response.ts, purge-history-result.ts, http-management-payload.ts, get-client.ts, durable-grpc.ts, user-agent.ts.
  • Top‑level v3 compat aliases: re‑exports TaskFailedError from core for instanceof guards; type aliases (DurableClient, EntityContext<T>, OrchestrationHandler, …) so v3 code compiles unchanged. README.md + CHANGELOG.md document migration notes and intentional omissions (DurableError / AggregatedError).
  • ~15 unit spec files (~2,000 lines) covering client, client‑query, app, app‑client, orchestration‑context, entity‑context, query‑types, status, retry‑options, trigger, worker, get‑client, compat‑exports, durable‑grpc, entity‑id — 91/91 passing.

2. Core host helpers on TaskHubGrpcWorker (original PR scope)

Two methods that let a host drive one durable work item per invocation without the long‑lived gRPC worker loop — mirroring the Python consolidation in durabletask-python #155:

  • processOrchestratorRequest(request: Uint8Array): Promise<Uint8Array>
  • processEntityBatchRequest(request: Uint8Array): Promise<Uint8Array>

Each deserializes a TaskHubSidecarService protobuf, executes one work item via the worker's existing internal path (_executeOrchestratorInternal / _executeEntityInternal) using an in‑process CapturingSidecarStub (null‑stub pattern, same as Python's AzureFunctionsNullStub), and returns the serialized response. Transport encoding (e.g. base64) stays in the host, out of core. Covered by packages/durabletask-js/test/functions-grpc-support.spec.ts.

3. E2E suite + CI enablement

The Functions host E2E suite already exists on main; this PR makes it actually run — and pass — against the new provider:

  • Workflow (.github/workflows/functions-e2e-tests.yaml): runs on pull_request for the Functions surface (test/e2e-functions/**, packages/azure-functions-durable/**, the workflow file) plus workflow_dispatch; adds --skipApiVersionCheck to Azurite; fixes build ordering; and fixes a latent invalid‑YAML step name that was blocking the workflow entirely (including manual dispatch).
  • Test app (test/e2e-functions/test-app/host.json): targets the Preview extension bundle [4.29.0, 5.0.0) (gRPC path) and sets maxGrpcMessageSizeInBytes: 2147483647 to lift the host's default 4 MiB gRPC receive cap (which otherwise fails large‑output / get‑all‑instances). See Cross‑language note.
  • Specs / harness: harness.ts expected‑error strings aligned to the v4 gRPC‑core fidelity; suspend-resume.spec.ts / terminate.spec.ts updated, with 3 terminal‑instance control‑plane cases skipped pending Functions v4 client: terminal/invalid-state control-plane ops surface opaque 2 UNKNOWN (HTTP 400) instead of a meaningful status #315.
  • Docs / runner: README.md, test/e2e-functions/README.md, scripts/test-e2e-functions.sh.
  • Result: 32 passed / 0 failed / 13 skipped (the 13 = 3 new Functions v4 client: terminal/invalid-state control-plane ops surface opaque 2 UNKNOWN (HTTP 400) instead of a meaningful status #315 skips + pre‑existing environment/feature skips).

4. Build & release pipeline

  • eng/templates/build.yml: pack durable-functions and stage the azure-functions-durable artifact.
  • eng/ci/release.yml: new durable_functions ESRP release job to publish the package.

Error fidelity (_mapControlPlaneError)

The consolidated gRPC path collapses control‑plane failures into two opaque codes; the compat layer translates them back to the v3 surface:

Activity/entity failure text now surfaces the v4‑core shape (TaskFailedError: Activity task #N failed: …, EntityOperationFailedException: …) rather than v3's Activity function '<name>' failed:. The core error does not carry the function name, and reconstructing the exact v3 string would require a replay‑risky change in the shared core, so the E2E expectations were aligned to the v4‑native shape. (Name‑in‑error is a possible future core enhancement benefiting all SDKs.)

Cross‑language note

The 4 MiB gRPC cap is an extension‑defined default (.NET DurableTaskOptions.MaxGrpcMessageSizeInBytes = 4194304), sent to and honored consistently by every SDK worker. Raising it via maxGrpcMessageSizeInBytes in host.json is the same knob .NET‑isolated and Python use — no SDK‑side divergence.

Follow‑ups

Testing

  • Compat unit: 91/91 (npm run test:unit -w durable-functions).
  • Core: functions-grpc-support.spec.ts green.
  • Functions host E2E: 32 passed / 0 failed / 13 skipped (Preview bundle + Azurite).
  • npm run build -w durable-functions clean (core + compat).

YunchuWang and others added 4 commits June 29, 2026 15:30
Expose low-level protobuf codecs and single work-item execution helpers for Azure Functions Durable JS gRPC consolidation without adding Functions metadata support.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Expose stable byte-oriented worker processing methods and endpoint/taskHub client options for Azure Functions Durable JS gRPC consolidation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the work-item-executor extraction and base64 protobuf helpers with a
minimal, Python-aligned implementation. processOrchestratorRequest and
processEntityBatchRequest now reuse the existing worker execution path via an
in-process capturing stub (mirroring durabletask-python PR #155's null-stub
pattern) instead of refactoring the worker. Removed the base64 helper module,
the object-level execute* methods, the V2 EntityRequest host path, and the
related exports and tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@YunchuWang
YunchuWang marked this pull request as ready for review June 30, 2026 22:02
Copilot AI review requested due to automatic review settings June 30, 2026 22:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a minimal “single work-item” execution surface to the Durable Task JS worker so host integrations (e.g., Azure Functions) can execute one orchestrator/entity batch request per invocation without running the long-lived gRPC streaming loop. This aligns with the SDK’s role as a low-level Durable Task Scheduler (sidecar) client/worker implementation.

Changes:

  • Added byte-level helper methods on TaskHubGrpcWorker to process serialized OrchestratorRequest / EntityBatchRequest payloads and return serialized responses.
  • Introduced an in-process CapturingSidecarStub that captures completion payloads instead of sending them over gRPC.
  • Added end-to-end tests and README documentation for the new host-integration surface.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
README.md Documents the new low-level host integration APIs and clarifies the non–Durable Functions programming model scope.
packages/durabletask-js/test/functions-grpc-support.spec.ts Adds e2e-style tests covering the new byte processors for orchestrations and entities.
packages/durabletask-js/src/worker/task-hub-grpc-worker.ts Implements the new public byte-processing methods and the capturing sidecar stub.

Comment thread packages/durabletask-js/src/worker/task-hub-grpc-worker.ts
Comment thread packages/durabletask-js/src/worker/task-hub-grpc-worker.ts
Comment thread packages/durabletask-js/src/worker/task-hub-grpc-worker.ts
YunchuWang and others added 5 commits July 1, 2026 15:34
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ator

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t glue

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ease pipeline

- Add DurableOrchestrationContext/DurableEntityContext (context.df.* adapters) + wrapOrchestrator/wrapEntity
- Add RetryOptions, callHttp (throws), parentInstanceId; align newGuid/callSubOrchestrator with v3 signatures
- Add v3 client query-return types (DurableOrchestrationStatus, OrchestrationRuntimeStatus, EntityStateResponse, PurgeHistoryResult)
- Add client getStatus/getStatusAll/getStatusBy/readEntityState/purgeInstanceHistory/startNew/waitForCompletionOrCreateCheckStatusResponse
- Add deprecated DurableOrchestrationClient alias
- Wire host-provided maxGrpcMessageSizeInBytes into gRPC channel options (Python parity)
- Remove dead CapturingSidecarStub.abandonRequest field (keep required no-op method)
- Add durable-functions package to build/release pipelines
Comment thread packages/azure-functions-durable/CHANGELOG.md Outdated
Comment thread packages/azure-functions-durable/README.md
Comment thread packages/azure-functions-durable/package.json Outdated
YunchuWang and others added 7 commits July 7, 2026 16:25
Drop rootDir, outDir, include, and exclude since they are inherited from ./tsconfig.json. Keep only the baseUrl + paths override that points @microsoft/durabletask-js imports at the built dist output.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… Functions

- Introduced `EntityId` class for classic v3 entity identifiers.
- Enhanced `DurableEntityContext` to include `isNewlyConstructed` and `entityId` properties.
- Implemented `signalEntity` method in `DurableEntityContext` for signaling other entities.
- Updated `DurableOrchestrationContext` to track and set custom status.
- Added unit tests for new features and behaviors.
Collapse the 4.0.0-alpha.0 changelog to a single bullet and rewrite the README to focus on what the package supports and why it is needed, dropping the implementation-plan / phase-status / open-questions sections.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
YunchuWang and others added 4 commits July 8, 2026 15:42
…onContext type aliases

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rator

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… context.log to classic orchestration context

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…3-compatible)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… exists (v3 parity)

DurableEntityContext.getState called the initializer on every invocation, so an
expensive or side-effecting initializer ran even when the entity already had state.
Durable Functions v3 (Azure/azure-functions-durable-js Entity.getState) only uses the
initializer as a fallback when there is no existing state. Use the core state object's
hasState flag to short-circuit: return the stored state when present, otherwise fall
back to the initializer.

Adds a regression test asserting the initializer is not called when state exists.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
Copilot AI review requested due to automatic review settings July 21, 2026 20:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 57 out of 59 changed files in this pull request and generated 2 comments.

Files not reviewed (1)
  • test/e2e-functions/test-app/package-lock.json: Generated file

Comment thread packages/azure-functions-durable/src/get-client.ts Outdated
Comment thread packages/azure-functions-durable/src/client.ts
…+ evict stopped clients from getClient cache

P0 (PR #282 functions-e2e CI RED on node 22.x): terminate of a TERMINAL instance
returned 400 instead of the v3 no-op 200. Reproduced locally against the Preview
extension bundle and captured the real gRPC codes: terminate-of-terminal surfaces
FAILED_PRECONDITION(9) ("...because instance is in the <State> state."), NOT
UNKNOWN(2) (suspend/resume) nor NOT_FOUND(5) (a genuinely missing instance). No
transport code alone separates terminal / non-terminal-wrong-state / missing, so
_mapControlPlaneError now consults getStatus for the control-plane verbs on any of
UNKNOWN | NOT_FOUND | FAILED_PRECONDITION and keys the outcome off the instance's
actual runtime state: terminal -> no-op (v3 410 parity, 200); non-terminal ->
"Cannot <verb> orchestration instance in the <State> state."; missing (getStatus
finds nothing) -> "No instance with ID '<id>' found.". raiseEvent (no verb) keeps
its NOT_FOUND -> not-found mapping.

Bot #B (client.ts): mapped control-plane throws now preserve the original gRPC error
as { cause: error }.

Bot #A (get-client.ts): getClient's per-binding client cache could hand back a
stopped client whose gRPC channel is closed. The cached client's stop() is now
wrapped to evict its cache entry, so the next getClient with the same binding builds
a fresh client instead of reusing a dead channel.

Verified: build 0; compat unit 96/96 (+ terminate FAILED_PRECONDITION no-op test,
terminate NOT_FOUND missing test, getClient evict-on-stop test); functions-e2e full
suite 35 passed / 0 failed / 10 skipped (the 2 terminate-terminal tests now 200;
suspend-resume + external-event unaffected).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
Copilot AI review requested due to automatic review settings July 21, 2026 21:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 57 out of 59 changed files in this pull request and generated no new comments.

Files not reviewed (1)
  • test/e2e-functions/test-app/package-lock.json: Generated file

@YunchuWang
YunchuWang requested a review from andystaples July 21, 2026 22:19
…ckage CHANGELOG to TBD

Bill's decision, verified against v3 source (Azure/azure-functions-durable-js v3.x
DurableClient.raiseEvent, L301-311): v3 raiseEvent handled a terminal instance as
`case 410: return` — a no-op success. Our compat previously called
_mapControlPlaneError WITHOUT a verb, so raise-to-terminal skipped the getStatus
terminal->no-op path and rethrew the raw gRPC error.

raiseEvent now passes an operationVerb, and _mapControlPlaneError keys off the actual
runtime state for all four control-plane verbs (terminate/suspend/resume/raiseEvent):
- TERMINAL -> no-op (v3 410 swallow), including raiseEvent.
- MISSING (getStatus finds nothing) -> "No instance with ID '<id>' found.".
- NON-terminal wrong-state -> suspend/resume/terminate throw the friendly
  "Cannot <op> orchestration instance in the <State> state."; raiseEvent instead
  resurfaces the ORIGINAL error, faithfully mirroring v3 raiseEvent's switch whose
  `default` branch produced a generic error (v3 had no friendly wrong-state message
  for raiseEvent). Captured gRPC codes unchanged: suspend/resume UNKNOWN(2),
  terminate/raiseEvent FAILED_PRECONDITION(9) when non-running, NOT_FOUND(5) missing.

This also makes raise-to-completed swallow (no-op), consistent with the documented v3
Node behavior in external-event.spec.ts (issue #645).

CHANGELOG: per Bill, the package changelog is authored at release time, so
packages/azure-functions-durable/CHANGELOG.md is reduced to a TBD placeholder (the
detailed 4.0.0 section will be written at release). Root CHANGELOG.md untouched.

Verified: build 0; compat unit 98/98 (+2: raiseEvent FAILED_PRECONDITION->terminal
swallow, raiseEvent non-terminal wrong-state rethrows original error; existing
raiseEvent NOT_FOUND test now mocks getOrchestrationState for the missing-instance
path); eslint 0; functions-e2e full suite 35 passed / 0 failed / 10 skipped (13/13
suites — external-event, terminate, suspend-resume all green, no regression).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
Copilot AI review requested due to automatic review settings July 21, 2026 23:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 57 out of 59 changed files in this pull request and generated 1 comment.

Files not reviewed (1)
  • test/e2e-functions/test-app/package-lock.json: Generated file

Comment thread packages/azure-functions-durable/src/client.ts Outdated
… so transient lookup failures aren't masked as not-found

_mapControlPlaneError previously disambiguated terminal/wrong-state/missing
by calling getStatus inside a broad try/catch that mapped ANY failure to
"No instance with ID found." But getStatus throws in two distinct cases: a
genuinely missing instance (getOrchestrationState resolves undefined) and a
transient lookup failure (e.g. an unavailable channel). The broad catch
masked the latter as not-found, losing the real control-plane error.

Query getOrchestrationState directly: its undefined return is the true
"missing" sentinel, while a rejection now resurfaces the ORIGINAL
control-plane error instead of being converted to not-found. Terminal->no-op
(v3 410 swallow), non-terminal wrong-state, and missing->not-found outcomes
are unchanged. Adds unit tests locking the transient-lookup-failure contract
for terminate and raiseEvent.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
Copilot AI review requested due to automatic review settings July 21, 2026 23:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 57 out of 59 changed files in this pull request and generated 1 comment.

Files not reviewed (1)
  • test/e2e-functions/test-app/package-lock.json: Generated file

Comment thread packages/durabletask-js/src/worker/task-hub-grpc-worker.ts
…LOG ref, correct removed exports, document entity-lock + callHttp)

The package CHANGELOG is intentionally TBD (authored at release time), so the
migration highlights live in the README. This corrects three regressions Andy
flagged:

- Drops the "See CHANGELOG.md for the full list" link, which now pointed at the
  empty TBD changelog (broken ref).
- Removes DurableError / AggregatedError / TokenSource from the removed-exports
  bullet — they were never v3 top-level exports.
- Documents the genuinely-removed surfaces: entity locking / critical sections
  moved to core (context.entities.lockEntities / LockHandle.release /
  isInCriticalSection; v3 df.lock/isLocked + DurableLock/LockState/
  LockingRulesViolationError removed, restoration tracked in #317), and
  context.df.callHttp now throws (restoration tracked in #318).

Docs-only; no source, CHANGELOG, or version changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
Copilot AI review requested due to automatic review settings July 22, 2026 16:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 57 out of 59 changed files in this pull request and generated 1 comment.

Files not reviewed (1)
  • test/e2e-functions/test-app/package-lock.json: Generated file

Comment thread packages/azure-functions-durable/src/get-client.ts
…ify abandon-path error to point at failureStrategy

get-client.ts: getClient computed the cache key via JSON.stringify(bindingData)
BEFORE validating the input. A missing (undefined) value produced a non-string
key, and a non-serializable object (e.g. a BigInt field) threw a raw
JSON.stringify TypeError instead of the friendly "not a valid durable client
input" error. Now validate/coerce with asClientInput first, then derive the key
defensively via a deriveCacheKey helper that returns undefined when no reliable
key can be produced (non-serializable config) — in which case getClient returns
an uncached client rather than throwing. Keys are identical to before for all
valid string/object inputs, so existing caching + evict-on-stop behavior is
unchanged. Adds tests for the missing-value friendly error and the
non-serializable uncached fallback.

task-hub-grpc-worker.ts: the abandon-path error on processOrchestratorRequest
implied versioning is wholly unsupported. Only VersionFailureStrategy.Reject
(the default abandon path) cannot be honored on the single-work-item path;
VersionFailureStrategy.Fail (or no versioning) resolves in-process and returns a
response. Reword to point users at the failure strategy. Keeps the word
"abandoned" so the existing /abandoned/i assertion still matches.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
Copilot AI review requested due to automatic review settings July 22, 2026 17:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 57 out of 59 changed files in this pull request and generated 1 comment.

Files not reviewed (1)
  • test/e2e-functions/test-app/package-lock.json: Generated file

Comment thread packages/azure-functions-durable/src/worker.ts
@YunchuWang
YunchuWang merged commit df03e6b into main Jul 22, 2026
30 checks passed
@YunchuWang
YunchuWang deleted the yunchuwang/functions-grpc-support branch July 22, 2026 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants