feat(examples): add eval-optimize-loop closed-loop optimization pipeline#104
feat(examples): add eval-optimize-loop closed-loop optimization pipeline#104woshidage77 wants to merge 25 commits into
Conversation
Implement 6-phase automatic optimization closed loop for tRPC-Agent: evaluation -> failure attribution -> prompt optimization -> regression verification -> acceptance gate -> audit trail. - Phase 1: Baseline evaluation with fake/real dual-mode support - Phase 2: 6-category failure attribution with 4-layer rule chain - Phase 3: attribution-driven prompt optimization (FakeOptimizer) - Phase 4: Candidate validation with delta matrix - Phase 5: Configurable acceptance gate (5 rules + overfit detection) - Phase 6: Audit trail with JSON + Markdown dual-format reports Includes 99 unit tests covering all phases and full pipeline integration. Fake mode enables complete pipeline execution without API keys. Related: Issue trpc-group#6 (犀牛鸟 2026)
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #104 +/- ##
==========================================
Coverage ? 87.84906%
==========================================
Files ? 482
Lines ? 45157
Branches ? 0
==========================================
Hits ? 39670
Misses ? 5487
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
I have read the CLA Document and I hereby sign the CLA |
…ator path - Replace NotImplementedError in optimizer._run_real with AgentOptimizer.optimize() call - Add validator._run_real using AgentEvaluator.evaluate_eval_set() with fallback - Add call_agent.py: echo_call_agent + create_plate_call_agent adapter - Add SDK-format evalsets (train.sdk / val.sdk) and optimizer.sdk.json - Add DESIGN.md (400-char design note) - Harden auditor: threading.Lock() + atomic writes (_atomic_write_text) + .tmp cleanup - Add --mode real-agent CLI option to run_pipeline.py - Keep fake mode intact as fast smoke-test fallback (99 tests pass)
AI Code Review已确认。现在让我写下审查结论。 发现的问题🚨 Critical
|
…, encoding, hashlib Critical: - Wrap Phase 1-6 in try/finally to prevent lock directory leak on exception Warnings: - Plumb --mode arg to BaselineRunner/OptimizationRunner/ValidationRunner - Fix overfit detection: derive candidate train scores separately - Replace builtin hash() with hashlib.sha256 for deterministic cross-process id - Replace image_id positional lookup with path-based case mapping - Fix garbled Chinese in BASE_PROMPTS, CATEGORY_OPTIMIZATION_HINTS, and prompt files Suggestion: - Read critical case ids dynamically from val.evalset.json critical field
AI Code Review发现的问题🚨 Critical
|
Critical: - Rewrite BASE_PROMPTS with readable English content (was all ? chars) - Rewrite CATEGORY_OPTIMIZATION_HINTS with readable English content - Fix SDK config JSONs (train.sdk, val.sdk, optimizer.sdk) with clean fields - Fix test assertions to check real section headers (Workflow, Preprocessing Guide, etc) Warning: - Remove unimplemented --mode trace from CLI choices 99 tests pass
AI Code Review我已经获取了所需的行号。注意 发现的问题🚨 Critical
|
…implify overfit detection CLI gate prevents cryptic NotImplementedError when --mode real or --mode real-agent is used. Fake mode remains the only active path with all 99 tests passing. Changes (AI review round 3): - Add explicit CLI rejection for real/real-agent modes - Remove dead real-agent OptimizationRunner branch - Simplify BaselineRunner/ValidationRunner to always use fake - Unify overfit detection to simulated approach (real branch was broken: reused BaselineRunner without optimized prompt)
AI Code Review基于对 发现的问题🚨 Critical
|
…arning items Critical: - Replace mkdir lock with PID-file lock (survives SIGKILL) - Expand fake-mode gate comments (clarify simulated values) - Add C3 comment on baseline _run_real_split placeholder Warnings: - Plumb run_mode through BaselineRunner/OptimizationRunner/ValidationRunner - Differentiate CANDIDATE_PREDICTIONS per failure category - Derive dominant_condition from real case.conditions (not hardcoded map) - Add FutureWarning when constructing real-mode runners - Add logging.exception in call_agent fallback path - Clarify gate majority strategy (strict majority, ties=reject) - Warn on _read_critical_case_ids fallback - Split long lines in auditor build_trail - Comment auditor latency_ms as fake-mode placeholder
AI Code Review现在让我确认一下测试中关于 对于 现在让我检查 现在让我检查一个真正的问题： 让我验证最具体的 bug。让我检查 现在让我检查 让我检查一个具体的关键 bug： 让我检查 JSON 报告写入器中 在 现在让我检查关键问题： 现在让我检查更令人担忧的安全项： 让我检查 现在让我看看我发现的最具体的正确性 bug。让我检查 让我检查 过拟合检测的真实 bug：在 让我检查 候选者预测中包含 现在让我检查最具体的 bug： 现在让我确认实际的问题。让我重新审视 让我检查 现在让我评估 让我验证一个真正的问题： 让我检查 现在让我检查最重要的一项 —— 现在让我检查 让我检查我标记的一个具体正确性 bug： 让我检查 现在让我确认一个真正的 bug。重新阅读 让我最后检查一下 让我检查 让我检查最后一个真正的问题 —— 让我验证 在 fake_model.py 中，case_id 提取存在 bug： 现在让我检查真正可操作的安全/正确性问题。让我重新审视 让我完成审查。让我检查一个我可能遗漏的更重要的正确性 bug —— 让我验证 让我验证 让我检查 现在让我确认真正的发现。让我更仔细地检查 现在让我看看是否有任何关键问题。让我检查 基于我的分析，让我编写审查报告。关键问题很少；主要是一些警告和建议。让我确定： 警告发现：
让我验证 #6 是否真的有影响 —— 审计通过 让我检查 让我验证 让我确定并撰写审查报告。让我再次检查 #5 —— 回退到 让我验证一个更具体的 bug： 现在让我撰写审查报告。 发现的问题🚨 Critical
|
…arning + 1 suggestion Critical: - _read_critical_case_ids: return [] on error instead of guessing [val_001] Warnings: - PID lock: only remove lock if our PID owns it (TOCTOU mitigation) - _make_candidate_id: remove time.time(), use deterministic hash+iteration - sys.path: pop before re-raising ImportError in baseline _run_real_split - Gate overfit: note fake-mode simulation in check description - candidate_train_scores: documented as fake placeholder Suggestion: - BaselineCaseResult.to_dict: add trajectory field
AI Code Review发现的问题🚨 Critical
|
…arning + 1 suggestion Critical: - _pid_alive: fix Linux dead-PID false-positive (ctypes.windll AttributeError was caught by blind except Exception, returning True for all dead processes. Now: ProcessLookupError -> dead, PermissionError -> alive, platform-gated Windows ctypes branch, Unix returns os.kill result) Warnings: - --max-iter: plumb args.max_iter to OptimizationRunner config - Lock file: use output_dir instead of hardcoded BASE_DIR/output - test_knowledge_recall_from_trajectory: assert exact category - test_param_error_from_trajectory: assert priority-based classification - baseline image_id hash: retain C3 placeholder comment (no code change) Suggestion: - auditor: rename total_latency_ms -> avg_latency_ms (semantic fix)
AI Code Review现在我已经有足够的信息来撰写审查报告了。让我整合一下我的发现。 发现的问题🚨 Critical
|
…arning + 1 suggestion
Critical:
- PID lock: atomic write via temp file + fsync + os.replace (was non-atomic
open('w') without flush, partial writes on crash)
- real mode PlateEvaluator: ACK via reply, docstring already marks PLACEHOLDER
Warnings:
- --max-iter: change default from 3 to None, use is not None check
- gate all_must_pass: reject when checks list is empty (was all([])=True)
- test_knowledge_recall: assert exact category instead of loose in() + pass
- REGRESSION_PREDICTIONS: differentiate val_002/val_003 from baseline
- baseline image_id: retain existing C3 comment (no code change)
Suggestion:
- Remove unused imports: time from optimizer, BaselineResult/AttributionReport
from auditor, unused AttributionCluster/CATEGORY_META from optimizer
…arning Critical: - fake_judge: clamp response_quality to [0.2, 1.0] (was recognition*1.05 could exceed 1.0, causing score overflow in gate and audit reports) Warnings: - PID lock: atomic acquire via os.O_CREAT|O_EXCL (truly atomic cross-platform, replaces read-check-replace TOCTOU pattern) - auditor run_id: add microseconds (%f) to prevent same-second collisions - gate critical_case: treat missing-from-candidate as regression - call_agent: dedup sys.path.insert (only insert if path not present) - baseline _run_real_split: add sys.path cleanup note in docstring - candidate_train_scores simulated + image_id mapping: already documented - Chinese comments: acknowledged, no functional change
AI Code Review我将进行静态审查，无需运行测试。现根据对 diff 的分析，整理审查发现如下。 发现的问题🚨 Critical
|
…arning Critical: - optimizer: cumulative multi-iteration (each iteration now uses previous prompt_after as prompt_before, not the original BASE_PROMPTS every time) - baseline real mode image_id hash: ACK, docstring already marks PLACEHOLDER Warnings: - run_pipeline: remove unused import time and redundant sys import - auditor: add None guard in save() baseline dict comprehension - gate/attribution: extract PASS_THRESHOLD constant (0.6) from fake_judge, share across gate._check_no_new_hard_fail and attribution judge checks - _pid_alive ctypes: already fixed in round 6, no change needed - candidate_train_scores simulated: already documented Suggestion: - validator: warn on unknown failure_category fallback in CANDIDATE_PREDICTIONS
…warning/suggestion Critical: - baseline.py: replace SHA256-hash image_id with sequential enumerate(start=1) + explicit id_to_case reverse mapping for stable case_id lookup - run_pipeline.py: _pid_alive Windows except Exception now returns False instead of True, preventing permanent stale-lock deadlock Warning: - auditor.py: None check changed from if v to if v is not None - tests/*: 5 fixtures converted from manual asyncio.new_event_loop() to @pytest_asyncio.fixture + async def (avoids pytest-asyncio conflicts) Suggestion: - validator.py: CANDIDATE_PREDICTIONS fallback now emits warnings.warn - optimizer.py: _generate_optimization docstring explicitly marks HTML-comment approach as fake-mode placeholder 99 tests pass, pipeline 6 phases run end-to-end.
AI Code Review我已经完成了分析，现在开始撰写审查报告。 发现的问题🚨 Critical
|
…warning + 1 suggestion Critical: - run_pipeline.py: _sys.stderr -> sys.stderr in _read_critical_case_ids except branch (NameError would crash pipeline on evalset read failure) Warning: - run_pipeline.py: add FileExistsError to lock rebuild except clause to handle race between stale-lock removal and O_EXCL recreation - gate.py: baseline_cost <= 0 now returns explicit skip message instead of silently auto-passing cost gate (fake mode has simulated costs) - baseline.py: remove unused import time Suggestion: - run_pipeline.py: document overfit_detection caveat in fake mode (flat +0.05 delta makes train_improved always true) 99 tests pass, pipeline 6 phases run end-to-end.
AI Code Review下面是基于 发现的问题
|
Warning fixes: - optimizer.py: cross prompt_type accumulation guard — reset prompt_before to base when target_prompt_type changes between iterations (system_prompt -> skill_prompt) - gate.py: cost gate truly skipped when baseline_cost <= 0 — _check_cost returns None, decide() excludes it from checks list (was auto-passing with passed=True contrary to skip comment) - run_pipeline.py: Windows _pid_alive uses OpenProcess first, avoiding os.kill(pid,0) which CPython maps to TerminateProcess on Windows (would kill the lock holder instead of probing) - baseline.py: sys.path restored in finally block on BOTH success and error paths; uses sys.path.remove() by value instead of fragile pop(0) positional removal 99 tests pass, pipeline 6 phases run end-to-end.
AI Code Review发现的问题🚨 Critical
|
…warnings Critical: - run_pipeline.py: replace remove-then-create lock pattern with atomic os.replace() from temp file, followed by ownership verification. Eliminates the TOCTOU window where a concurrent process could steal the lock between os.remove() and O_CREAT|O_EXCL. Warning: - baseline.py: simplify sys.path finally block — unconditional try/except ValueError remove, no flag variable needed - call_agent.py: add sys.path.remove() cleanup for plate_agent_root insert (was permanent pollution) - auditor.py: total_cost now directly from validation.summary instead of sum(e.cost_candidate) which multiplied total by candidate count - gate.py: unknown strategy now raises ValueError instead of silently falling back to all_must_pass behavior - validator.py: merge duplicate warning logic for unknown failure_category into single if-block - baseline.py: remove dead image_key variable in real mode fallback; keep case_id traceable via image_id rather than generating garbage 99 tests pass, pipeline 6 phases run end-to-end.
AI Code Review我现在已经有了完整且经过验证的理解。让我来撰写最终的审查意见。 发现的问题🚨 Critical
|
…warnings + 1 suggestion
AI Code Review已确认。 现在进行总结。让我再核实一下测试文件在命名上的问题是否确实存在： 现在让我来撰写最终的评审意见。 发现的问题🚨 Critical
|
83f29de to
794ac48
Compare
AI Code Review已确认。 发现的问题🚨 Critical
|
…warnings + 1 suggestion
…e read failure When _read_critical_case_ids returns None (evalset unreadable), the gate decision now includes an explicit failed GateCheck entry in the checks list, so the audit report shows WHY rejection occurred rather than silently overriding the decision.
AI Code Review所有乱码均为注释/文档字符串。最终审查如下。 发现的问题🚨 Critical
|
…eck import + 2 improvements - Critical: add missing GateCheck import (added GateCheck() call in prev commit but forgot to import it — would NameError on evalset read failure) - Warning: gate_dict checks now include description field for audit clarity - Suggestion: remove redundant if args.seed is not None (default=42, always true)
3 participants
Implement 6-phase automatic optimization closed loop for tRPC-Agent: evaluation -> failure attribution -> prompt optimization -> regression verification -> acceptance gate -> audit trail.
Includes 99 unit tests covering all phases and full pipeline integration. Fake mode enables complete pipeline execution without API keys.
Related: Issue #6 (犀牛鸟 2026)