Property portfolio: requireInvariant discipline, 6-lens coverage checklist, multi-call techniques#43
Open
shellygr wants to merge 6 commits into
Open
Property portfolio: requireInvariant discipline, 6-lens coverage checklist, multi-call techniques#43shellygr wants to merge 6 commits into
shellygr wants to merge 6 commits into
Conversation
…chnique advice - cvl_guidelines.j2: guideline 24 (bare state requires are a spec smell; default to invariant + requireInvariant, justify exceptions in a comment); guideline 23 carve-out for hooks/ghosts tracking information never stored in contract state. - property_analysis_prompt.j2: 6-lens coverage checklist (unit / valid-state / variable-transition / state-transition / multi-call / risk) after the 3-category block, explicitly a brainstorming discipline, not a quota; task steps ask the agent to record swept lenses in its reasoning. No PropertyType schema change. - cvl_additions.j2: three author-only advice entries — lastStorage/at storage snapshots with the additivity idiom, satisfy-witness companion rules for implication-shaped assertions, ghost counters via expression summaries. - tests/test_template_render.py: render smoke tests for the three templates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Expression summaries replace the callee's implementation and never apply to direct CVL invocations; the advice now says so, so agents only use the idiom for external-dependency calls. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"source" is not a member of Sort (Literal["greenfield", "existing", "update"] in composer/spec/service_host.py); "existing" exercises the same non-greenfield template branch with a value production can pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Non-persistent ghosts are rolled back by reverted calls and at-snapshot restores, so a counter combined with the lastStorage additivity idiom would silently reset. Marking counters persistent is consistent with the persistent-ghost guideline since they do not mirror contract state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…efense Three template-level fixes from the architecture review of the property portfolio prompts: - Guideline 24: solver/capacity bounds (e.g. require totalSupply() <= 2^128) are a recognized documented-trust-assumption class needing no invariant attempt, and preserved blocks defer to the stricter judge standard (requireInvariant or verifiable outside evidence) instead of the comment-suffices rule, removing the conflict with judge Criteria 4. - Author instruction: companion satisfy-witness rules and supporting invariants must be listed in property_rules under their parent property, since rules unreferenced by any property are dropped from the report. - Guideline 20: persistent counter ghosts and expression summaries on external-dependency calls are legitimate for information never stored in contract state, so the judge shares the vocabulary the author advice uses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
shellygr
commented
Jul 4, 2026
Contributor
Author
There was a problem hiding this comment.
remove this test. does nothing
shellygr
commented
Jul 5, 2026
jtoman
reviewed
Jul 6, 2026
Comment on lines
+47
to
+52
| 24. A bare `require` that constrains contract state inside a rule or `preserved` block is a spec smell: the assumption may not actually hold in reachable states, silently | ||
| weakening the proof. The default discipline is to state the assumption as an `invariant`, prove it, and assume it via `requireInvariant`. Bare requires over contract | ||
| state are acceptable only when accompanied by a comment referencing a justification or a documented trust assumption. Exception: capacity/overflow-avoidance bounds on | ||
| supplies, balances, or timestamps (e.g. `require totalSupply() <= 2^128`) exist to keep the solver tractable, not to model contract behavior — they are a recognized | ||
| documented-trust-assumption class; a comment naming the bound's purpose suffices, and no invariant attempt is expected because such bounds are typically not preserved | ||
| by the contract. In `preserved` blocks the stricter standard applies — `requireInvariant` or verifiable outside evidence; a comment alone is insufficient. |
Contributor
There was a problem hiding this comment.
we discussed this on zoom, but we already have guidance for both the judge and author to not use require as a "make the rule pass" magic. Repeating it a third time will not fix the problem imo. The instructions you provide here are almost verbatim in the author prompt. It would be useful if we could find the run where this happened and see what reasoning/thinking tricks the agent used to talk itself out of the rules it already knew about, rather than just repeating the rule one more time and hoping that one makes it stick.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some more guidance on property inference.
Part 5 of 6 (spec-quality series). Generated specs were ~80% unit-style rules; human specs concentrate on accounting invariants, multi-call sequences, and state-transition properties, and never assume state with bare requires.
What
cvl_guidelines.j2(read by author AND judge): new guideline 24 — a barerequireover contract state is a spec smell; default isinvariant+requireInvariant, bare requires only with a justification comment. Two carve-outs: solver-capacity bounds (e.g.require totalSupply() <= 2^128) are a recognized documented-trust-assumption class needing no invariant attempt, andpreservedblocks defer to the stricter judge standard (requireInvariantor verifiable outside evidence — a comment alone is insufficient). Guideline 23 amended with a ghost-counter carve-out (information never stored in contract state); guideline 20 now names persistent counter ghosts and expression summaries on external-dependency calls as legitimate for that same class.property_analysis_prompt.j2: 6-lens coverage checklist (unit / valid-state / variable-transition / state-transition / multi-call / risk) — explicitly a brainstorming discipline, not a quota; swept lenses recorded in reasoning. NoPropertyTypeschema change (caches survive).property_generation_prompt.j2: theproperty_rulesmapping instruction now requires supporting invariants and companion witness rules to be listed under the property they support — rules unreferenced by any property are silently dropped from the final report.cvl_additions.j2(author-only): three<advice>entries —storage s = lastStorage/f(e,args) at ssnapshot idiom (additivity/round-trip),satisfy-witness companion rules for implication antecedents (mapped inproperty_rulesunder the parent property), ghost counters via expression summaries (with persistent-ghost note and the caveats that summaries replace the callee and never apply to direct CVL calls).Design revisions after architecture review
preserved-block policy now defers to the stricter judge standard (Criteria 4), removing the internal conflict where a comment sufficed for the author but not the judge.satisfy-witness rules and supporting invariants must be mapped inproperty_rulesunder their parent property — orphan rules are dropped from the report (confirmed by reviewers).🤖 Generated with Claude Code