fix(video): recover from a VOD producer that dies mid-session at the tail (#169)

[superuser404notfound]

Context

Round-2 reopen of #169 by rrgomes with a decisive trace: the earlier "fixed on 5.16.2" verdict was their host-side near-end workaround firing, not the engine. With the workaround removed, the tail failure is unchanged on 5.18.4, and it is a different mechanism than the 5.16.2 symptoms: the producer provably stops mid-session before writing the final segment (seg719 never produced; avioFetchedMB/packetsWritten frozen across four 30 s memprobes), and the request for seg719 re-arms a 30 s backpressure wait forever (cache miss after 30005ms (cache=46 restarted=false) x11) until AVPlayer dies with -12889.

Root cause (three defects lined up)

1. No recovery arm for a mid-session VOD readError exit. handlePumpFinished only surfaced the nothing-ever-produced case as fatal (Unknown-length HTTP MP4 enters streaming mode and returns ready with zero packets; the same URL plays in VLC #126) and assumed "the scrub/wedge arms cover recovery" for the rest. Both arms are unreachable here: the Presented frame drifts ~6 s ahead of currentTime/sourceTime after a rapid bidirectional seek burst (loopback-HLS native path) — frameAhead/producerShift/hostShift all report 0 #65 wedge detector runs inside the pump and died with it, and the provider's restart escalation (2) never fires.
2. Forward-wait liveness gap (the mechanism rrgomes identified, confirmed verbatim): in the r.1 < index <= r.1 + forwardWaitWindow branch, needsRestart = index > activeMarchFront + forwardWaitWindow answers "will this arrive?" by index distance alone. A dead producer freezes the front just below the request, so the same 30 s wait re-arms indefinitely with no escalation. The final segment is maximally exposed: there is nothing beyond it for the march to advance into.
3. producerCovers veto. Even an escalated restart would not have fired: the restart loop's producerCovers guard reads the dead producer's still-installed base, which covers the requested index.

Fix (defense in depth, VOD only)

• Event-driven revive: a mid-session VOD readError exit now gets a bounded revive (own gate, 2 attempts) mirroring the VOD start at resume position with bridged audio (mp3→EAC3) never becomes ready; recovery re-anchor kills the muxer ("Cannot write moov atom before EAC3 packets parsed") with no escalation (#93 residual) #99 muxerFailed arm, aimed at the pending seek target or AVPlayer's real position. The demuxer whose read just threw is marked suspect-dead so performRestart replaces it via the existing Piece-B seek-deadline recovery (#65 4.2.2) re-anchors producer to a stale coalesced seek target, not AVPlayer's rendered position → permanent wedge after a rapid scrub burst #79 fresh-demuxer path instead of seeking the connection that just failed. This recovers the trace's shape within seconds of the pump exit.
• Provider-side liveness escalation: a FINISHED pump skips the forward wait entirely and restarts immediately. A pump that never finishes (e.g. blocked in a read) escalates when a full backpressure wait elapsed with zero march-front progress for the same index; an advancing front resets the record and keeps the full Far-forward seek into an unproduced region (loopback-HLS VOD) #141/Loopback-HLS: a single backward seek on heavy 4K can wedge the segment producer (video stalls, audio continues -> A/V desync) #93 patience (slow WAN reads legitimately take 20-30 s). Both proofs bypass the producerCovers veto. This covers any way the march stops that produces no exit event.

Live sessions are untouched: they keep their pump watchdogs (#167/#177) and reopen machinery, and the escalation is gated !isLive.

Retest signatures (for the reporter)

• Pump exit case: #169 VOD pump died mid-session (readError N); rebuilding producer on a fresh demuxer at ... followed by seg719 producing and serving.
• Silent-stop case: seg719: #169 forward-wait march dead (front=718 frozen across a full wait); escalating to restart.

Verification

• New Issue169DeadProducerEscalationTests (7 tests): immediate restart on finished pump (including the producerCovers bypass), second-miss escalation on a frozen front, patience preserved while the front advances, live never escalates, pure-decision matrices for forwardWaitMarchDead and shouldReviveVODAfterReadError.
• Full suite 965/965 green; swift build -Xswiftc -strict-concurrency=complete clean; tvOS Simulator build green.

Refs #169 (leaving the issue to the reporter's device retest; not auto-closing).

🤖 Generated with Claude Code

https://claude.ai/code/session_01NAPCg4tCDdSqkK6tPkNptw