Skip to content

Add loadAllResult to surface corrupt records in collectionStore (#2728)#2816

Merged
atomantic merged 3 commits into
mainfrom
next/issue-2728
Jul 20, 2026
Merged

Add loadAllResult to surface corrupt records in collectionStore (#2728)#2816
atomantic merged 3 commits into
mainfrom
next/issue-2728

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Summary

collectionStore.loadAll() filtered out null records (a record whose index.json is missing, unreadable, unparseable, or rejected by sanitizeRecord), so every createCollectionStore consumer reasoned over a partial set with no way to tell "N records" from "N+K records, K corrupt." A silent undercount was indistinguishable from a complete read.

This adds a sibling loadAllResult() -> { records, failedIds } to the store, keeping the "which ids failed to load" signal. loadAll() becomes a back-compat flatten wrapper (loadAll = (await loadAllResult()).records), so no existing consumer changes. This mirrors the existing listOutcomesResult precedent and the CLAUDE.md "sentinel + validate — distinguish absent vs empty" convention.

The first consumer is tightened: layeredIntelligenceOutcomes.listOutcomesResult now reads via loadAllResult and reports read: false + partial: true + failedIds when any outcome record fails to load, instead of read: true with a silently-short list. Both existing consumers already treat read: false as unavailable:

  • routes/apps/taskTypes.js — renders the "couldn't load" dashboard state instead of a lie ("nothing filed yet").
  • autonomousJobs/layeredIntelligenceHooks.js — reports LI's merge rate as unavailable to the reasoner rather than feeding it a fabricated short history.

So a single corrupt outcome record now taints the whole read (we can't know which app's history it belonged to) rather than silently undercounting.

The other loadAll consumers (universeBuilder, pipeline issues/series, writersRoom, creativeCommissions, catalog user types, storyBuilder) are left on the loadAll back-compat wrapper unchanged.

Test plan

  • server/lib/collectionStore.test.js — new loadAllResult suite: clean read (empty failedIds), a corrupt record surfaced in failedIds while readable ones still return, sanitizer-rejected records reported in failedIds, and loadAll remaining a faithful flatten of loadAllResult().records.
  • server/services/layeredIntelligenceOutcomes.test.js — new partial-read test (one clean record + one corrupt record dir) asserting read: false / partial: true / failedIds and the surviving outcome; existing total-failure/non-array mocks retargeted from loadAll to loadAllResult.
  • cd server && npx vitest run collectionStore layeredIntelligenceOutcomes — 97 passing. Lib barrel/README invariant test passing.

Closes #2728

…ollectionStore

loadAll() silently filtered out null records (missing/unparseable/
sanitizer-rejected index.json), so every createCollectionStore consumer
reasoned over a partial set without being able to tell N records from
N+K-with-K-corrupt.

Add a sibling loadAllResult() -> { records, failedIds } that keeps the
which-ids-failed signal; loadAll becomes a back-compat flatten wrapper
(loadAll = (await loadAllResult()).records), so no existing caller changes.
Mirrors the listOutcomesResult precedent.

Tighten the first consumer: layeredIntelligenceOutcomes.listOutcomesResult
now reads via loadAllResult and reports read:false + partial:true + failedIds
when any outcome record fails to load, instead of read:true with a silently
short list. Both existing consumers (taskTypes route, LI hooks) already treat
read:false as unavailable, so a partial read no longer feeds a fabricated
merge rate to the reasoner.
@atomantic
atomantic merged commit c169c52 into main Jul 20, 2026
2 checks passed
@atomantic
atomantic deleted the next/issue-2728 branch July 20, 2026 02:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

collectionStore.loadAll silently drops corrupt records, so callers can't tell partial from complete

1 participant