Add loadAllResult to surface corrupt records in collectionStore (#2728)#2816
Merged
Conversation
…ollectionStore
loadAll() silently filtered out null records (missing/unparseable/
sanitizer-rejected index.json), so every createCollectionStore consumer
reasoned over a partial set without being able to tell N records from
N+K-with-K-corrupt.
Add a sibling loadAllResult() -> { records, failedIds } that keeps the
which-ids-failed signal; loadAll becomes a back-compat flatten wrapper
(loadAll = (await loadAllResult()).records), so no existing caller changes.
Mirrors the listOutcomesResult precedent.
Tighten the first consumer: layeredIntelligenceOutcomes.listOutcomesResult
now reads via loadAllResult and reports read:false + partial:true + failedIds
when any outcome record fails to load, instead of read:true with a silently
short list. Both existing consumers (taskTypes route, LI hooks) already treat
read:false as unavailable, so a partial read no longer feeds a fabricated
merge rate to the reasoner.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
collectionStore.loadAll()filtered outnullrecords (a record whoseindex.jsonis missing, unreadable, unparseable, or rejected bysanitizeRecord), so everycreateCollectionStoreconsumer reasoned over a partial set with no way to tell "N records" from "N+K records, K corrupt." A silent undercount was indistinguishable from a complete read.This adds a sibling
loadAllResult()->{ records, failedIds }to the store, keeping the "which ids failed to load" signal.loadAll()becomes a back-compat flatten wrapper (loadAll = (await loadAllResult()).records), so no existing consumer changes. This mirrors the existinglistOutcomesResultprecedent and the CLAUDE.md "sentinel + validate — distinguish absent vs empty" convention.The first consumer is tightened:
layeredIntelligenceOutcomes.listOutcomesResultnow reads vialoadAllResultand reportsread: false+partial: true+failedIdswhen any outcome record fails to load, instead ofread: truewith a silently-short list. Both existing consumers already treatread: falseas unavailable:routes/apps/taskTypes.js— renders the "couldn't load" dashboard state instead of a lie ("nothing filed yet").autonomousJobs/layeredIntelligenceHooks.js— reports LI's merge rate as unavailable to the reasoner rather than feeding it a fabricated short history.So a single corrupt outcome record now taints the whole read (we can't know which app's history it belonged to) rather than silently undercounting.
The other
loadAllconsumers (universeBuilder, pipeline issues/series, writersRoom, creativeCommissions, catalog user types, storyBuilder) are left on theloadAllback-compat wrapper unchanged.Test plan
server/lib/collectionStore.test.js— newloadAllResultsuite: clean read (emptyfailedIds), a corrupt record surfaced infailedIdswhile readable ones still return, sanitizer-rejected records reported infailedIds, andloadAllremaining a faithful flatten ofloadAllResult().records.server/services/layeredIntelligenceOutcomes.test.js— new partial-read test (one clean record + one corrupt record dir) assertingread: false/partial: true/failedIdsand the surviving outcome; existing total-failure/non-array mocks retargeted fromloadAlltoloadAllResult.cd server && npx vitest run collectionStore layeredIntelligenceOutcomes— 97 passing. Lib barrel/README invariant test passing.Closes #2728