Auto-generated sub-orchestration IDs collide across continue_as_new generations

[pinodeca]

Summary

When an orchestration both calls ctx.continue_as_new(...) and schedules sub-orchestrations without an explicit instance ID, the auto-generated child IDs collide across generations. A child spawned in generation N+1 re-derives the same ID as the child from generation N, which already exists in the provider store as Completed. The parent's await on the new child never resolves → permanent stall.

This proposes making auto-generated child ID derivation stable and unique across continue_as_new generations so that "loop-like" orchestrations that spawn children just work, without callers having to supply explicit, generation-qualified IDs.

Mechanism

Auto-generated child IDs are derived from the child's position in the current execution's history:

• A sub::pending_… placeholder is created at schedule time and replaced with sub::{event_id} during action processing (SUB_ORCH_PENDING_PREFIX, SUB_ORCH_AUTO_PREFIX in src/lib.rs).
• build_child_instance_id prefixes the parent → final ID is {parent}::sub::{event_id}.
• event_id is "a monotonically increasing position in history. The first event (OrchestrationStarted) always has event_id = 1."

Within a single execution this is perfectly replay-stable. The problem is that continue_as_new starts a new execution with a fresh history that again begins at OrchestrationStarted { event_id: 1 }. The execution_id increments (1 → 2 → 3…), but event_id restarts from 1 each generation.

So the first sub-orchestration scheduled in generation 2 gets the same event_id — hence the same {parent}::sub::{event_id} — as the first one in generation 1. That ID is already Completed in the store, so the schedule is treated as already-done and no fresh completion is delivered → the parent hangs.

In short: event_id is unique within an execution but not across continue_as_new generations, yet auto-ID derivation only uses event_id.

Impact

Any orchestration that loops via continue_as_new and also spawns sub-orchestrations in the loop body hits this. We hit it in pg_durable, where df.loop runs as a child orchestration whose body can contain parallel JOIN/RACE branches (each scheduled as a sub-orchestration). See microsoft/pg_durable#230 and #233 for the downstream symptoms.

pg_durable currently works around it (in microsoft/pg_durable#228) by using schedule_sub_orchestration_with_id with a hand-built, generation-qualified ID:

{instance}::e{execution_id}::{tag}::{node_id}

That works because it folds execution_id into the ID, but every consumer that combines continue_as_new with sub-orchestrations has to know about this sharp edge and re-implement the same trick.

Proposed fix (Option A — fold execution_id into the auto ID)

duroxide already tracks execution_id (it increments per continue_as_new and already appears in history/poison metadata, e.g. PoisonMessageType::Orchestration { instance, execution_id }). The auto derivation simply doesn't incorporate it. Change it to include the generation:

// today (conceptually):
let child = format!("sub::{event_id}");
// proposed:
let child = format!("sub::{execution_id}.{event_id}");
// → final: {parent}::sub::{execution_id}.{event_id}

• Within a generation: still derived from the replay-stable event_id (plus a constant execution_id), so replay re-derives the same ID. Determinism preserved.
• Across generations: execution_id differs, so generation 2's first child is …sub::2.N, never colliding with generation 1's …sub::1.N.

This is essentially the pg_durable workaround promoted into the runtime so callers get it for free.

Alternative (Option B — carry a monotonic child counter across generations)

Thread a "next sub-orchestration ordinal" through the continue_as_new carry-forward state so it keeps climbing (1, 2, 3, …) across all generations instead of resetting. More faithful to a globally unique child sequence, but heavier — it requires persisting and threading extra state through every continue_as_new. Option A is a smaller, stateless change and is preferred unless there's a reason to want strictly monotonic ordinals.

Backward / replay compatibility (must verify)

The change is only safe if duroxide records the resolved child ID in the SubOrchestrationScheduled history event at schedule time (the sub::pending_… placeholder is replaced before being written to history). If so:

• In-flight instances that already scheduled children under the old sub::{event_id} format read those IDs back from history during replay — derivation isn't re-run for already-recorded events. They keep working.
• Only newly scheduled children use the new format.

The one thing to audit before shipping: ensure no code path re-derives an existing child's ID from scratch instead of reading the persisted value (the dispatcher poison / new-history path — the same area touched by #31 — is where this kind of derivation gets re-evaluated). If every path reads the recorded ID for already-scheduled children, the format change is safe.

Related

• Preserve sub-orchestration parent link across ContinueAsNew #31 — preserves the parent link across continue_as_new in a child orchestration (the enabling fix for running loops as child orchestrations). This issue is the sibling concern: making the children those loops spawn collision-free across generations.
• Fix df.loop: run loops as child sub-orchestrations and fix cross-generation child ID collision pg_durable#228 — downstream PR that currently works around this with explicit generation-qualified IDs.
• JOIN/RACE inside a loop stalls on iteration ≥2 (continue_as_new resets sub-orchestration id counter → child id collision) pg_durable#230, loop inside a JOIN/RACE branch fails terminally on iteration ≥2 (continue_as_new restarts ExecuteSubtree with wrong input) pg_durable#233 — downstream stall symptoms.