[HV2] extractor quality: NPC dialogue-snippet mining + quest wrap-up capture (#1329)#1368
[HV2] extractor quality: NPC dialogue-snippet mining + quest wrap-up capture (#1329)#1368100yenadmin wants to merge 1 commit into
Conversation
…capture (#1329) The first live promotion batch (PR #1354) promoted only 1 of 11 artifacts: NPC extracts carried empty/thin dialogue_snippets and quests lacked narrative wrap-up. Raising extraction quality raises the promotable fraction. NPC dialogue (voice, not narration): dialogue_snippets now mines IN-VOICE quoted lines the NPC actually SPOKE — a quoted span (double/curly quotes) kept only when the NPC name AND a speech-attribution verb sit tight around it. The old matcher kept any narration block mentioning the name (combat rosters, scene-setting), which is 'described-not-voiced' and caps voice_distinctiveness under qa/rubric_artifact_npc.md. Mid-sentence fragments and possessive-apostrophe false quotes are filtered; a silent NPC returns [] (graceful). Quest wrap-up: quest payload gains additive 'description' (Quest.description, was dropped) and a 'resolution' object {status, evolves_to, callback_in_days, wrap_up[]} where wrap_up mines closing transcript beats naming the quest AND carrying resolution language — the outcome context qa/rubric_artifact_quest.md scores for consequence_weight. Schema (data/library/artifact_schema.json): additive-only, both new quest fields optional; envelope + provenance.source unchanged. Read-only on play-state (unchanged); extractor stays a sibling of export_scene_grid.py. Tests (-p no:xdist): 36 pass (was 15) — in-voice attribution, third-person/possessive exclusion, mid-sentence-fragment drop, dedup/cap, curly quotes, quest wrap-up cue matching + title-token cue masking, graceful empty cases. fast_gate T0: 257 pass. ruff clean. jsonschema: 43 real artifacts validate. Before/after on 5 real campaigns: quest wrap_up 0->4 (of 5), description 0->5; NPC snippets shift from combat-roster noise to genuine in-voice quotes. #1329
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 49 seconds Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
evaOS review status: in progressPR: #1368 - [HV2] extractor quality: NPC dialogue-snippet mining + quest wrap-up capture (#1329) evaOS review is running for this PR head. Automation note: agents should wait for this comment to reach PR URL: #1368 |
|
Architect review (verdict-as-comment): the in-voice mining is the RIGHT fix — quotes kept only when the NPC name + a speech verb co-occur in a tight window (kills the combat-roster false-positives that capped voice_distinctiveness ≤2); read-only on play-state preserved; schema additive-only (quest description + resolution.wrap_up both OPTIONAL, envelope unchanged); 5-campaign before/after shows wrap_up 0→4, description 0→5, snippets flip to genuine voice. 36 + 257 green, 43 artifacts re-validate. The two-NPC mis-speaker heuristic limit is acceptable (real speech, flagged). This raises the next promotion batch's yield. Merge on CI+bot green. |
1 participant
Why
The first LIVE promotion batch (PR #1354, merged) scored 11 extracted artifacts and promoted only 1. The rejections were legible: several NPC extracts carried empty/thin
dialogue_snippets, and quests lacked narrative wrap-up context — so they failed the promotion gate on those grounds. Raising extraction QUALITY directly raises the promotable fraction (the flywheel's throughput). Follow-up on epic #1329 / the HV3 first-promotion finding.What changed
1. NPC dialogue — extract the VOICE, not the narration.
dialogue_snippetsnow mines in-voice quoted lines the NPC actually SPOKE: a quoted span (straight/curly double quotes, curly single) is kept only when the NPC's name (first OR last token) AND a speech-attribution verb sit tight (~60 chars) around the quote. The old matcher kept any narration block mentioning the name — combat rosters (Party (Aldric 40/40, Maren 31/31)...), scene-setting — which is the "described-not-voiced" failure that forcesvoice_distinctiveness ≤ 2underqa/rubric_artifact_npc.md. Markdown emphasis is stripped, mid-sentence fragments and possessive-apostrophe false quotes are filtered, and a silent NPC returns[](graceful).2. Quest wrap-up / resolution context.
Quest payload gains additive
description(Quest.description, previously dropped) and aresolutionobject{status, evolves_to, callback_in_days, wrap_up[]}, wherewrap_upmines the closing transcript beats that name the quest AND carry resolution language (outcome/consequence cues, with the quest's own title tokens masked out so a title like The Price of Silence doesn't self-trigger thethe pricecue). This is the outcome contextqa/rubric_artifact_quest.mdscores forconsequence_weight/ narrative completeness.3. Schema — additive-only.
data/library/artifact_schema.json:quest_payloadgains optionaldescription+resolution; envelope, per-class required keys, andprovenance.source(HV1's optional field) are unchanged. HV2's merged schema stays canonical.Invariants
export_scene_grid.py; offline-testable (fabricated transcript fixtures, no live LLM).Before / after (5 real campaigns, re-extracted read-only)
The NPC count is ~flat, but the CONTENT flips from combat-roster noise (
Party confirmed: Aldric (40/40)...) to genuine in-voice lines ("I'd rather we pull together," Wyll/"Lord Wyll. Sir. Begging your pardon."/"So I don't want your gold..."). Quest wrap-up + description go from absent to present on nearly every quest — new signal the promotion panel can score. (glm-duo-h's 2→0 were false-positive combat mentions correctly dropped.)Tests
qa/test_export_campaign_artifacts.py: 36 passed (was 15) —-p no:xdist. In-voice attribution (pre/post), third-person + possessive-apostrophe exclusion, mid-sentence-fragment drop, dedup+cap, curly quotes, quest wrap-up cue matching + title-token cue masking, graceful empty/silent cases; golden fixture updated for the new fields.qa/fast_gate.sh: T0 PASS (257 passed).ruff check: clean.jsonschema: 43 real extracted artifacts validate against envelope + per-class payload defs (additive fields accepted).Do not merge — orchestrator runs the live promotion panel. Part of #1329.