Skip to content

Property portfolio: requireInvariant discipline, 6-lens coverage checklist, multi-call techniques#43

Open
shellygr wants to merge 6 commits into
masterfrom
shelly/ap-property-portfolio
Open

Property portfolio: requireInvariant discipline, 6-lens coverage checklist, multi-call techniques#43
shellygr wants to merge 6 commits into
masterfrom
shelly/ap-property-portfolio

Conversation

@shellygr

@shellygr shellygr commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Some more guidance on property inference.


Part 5 of 6 (spec-quality series). Generated specs were ~80% unit-style rules; human specs concentrate on accounting invariants, multi-call sequences, and state-transition properties, and never assume state with bare requires.

What

  • cvl_guidelines.j2 (read by author AND judge): new guideline 24 — a bare require over contract state is a spec smell; default is invariant + requireInvariant, bare requires only with a justification comment. Two carve-outs: solver-capacity bounds (e.g. require totalSupply() <= 2^128) are a recognized documented-trust-assumption class needing no invariant attempt, and preserved blocks defer to the stricter judge standard (requireInvariant or verifiable outside evidence — a comment alone is insufficient). Guideline 23 amended with a ghost-counter carve-out (information never stored in contract state); guideline 20 now names persistent counter ghosts and expression summaries on external-dependency calls as legitimate for that same class.
  • property_analysis_prompt.j2: 6-lens coverage checklist (unit / valid-state / variable-transition / state-transition / multi-call / risk) — explicitly a brainstorming discipline, not a quota; swept lenses recorded in reasoning. No PropertyType schema change (caches survive).
  • property_generation_prompt.j2: the property_rules mapping instruction now requires supporting invariants and companion witness rules to be listed under the property they support — rules unreferenced by any property are silently dropped from the final report.
  • cvl_additions.j2 (author-only): three <advice> entries — storage s = lastStorage / f(e,args) at s snapshot idiom (additivity/round-trip), satisfy-witness companion rules for implication antecedents (mapped in property_rules under the parent property), ghost counters via expression summaries (with persistent-ghost note and the caveats that summaries replace the callee and never apply to direct CVL calls).

Design revisions after architecture review

  • Guideline 24 carve-out for capacity/overflow-avoidance bounds — they keep the solver tractable rather than model behavior, so no invariant attempt is expected.
  • Guideline 24 preserved-block policy now defers to the stricter judge standard (Criteria 4), removing the internal conflict where a comment sufficed for the author but not the judge.
  • Companion satisfy-witness rules and supporting invariants must be mapped in property_rules under their parent property — orphan rules are dropped from the report (confirmed by reviewers).
  • Ghost-counter/persistent-ghost defense made judge-visible via guideline 20, so the judge shares the vocabulary the author advice uses.

🤖 Generated with Claude Code

shellygr and others added 5 commits July 3, 2026 20:21
…chnique advice

- cvl_guidelines.j2: guideline 24 (bare state requires are a spec smell;
  default to invariant + requireInvariant, justify exceptions in a comment);
  guideline 23 carve-out for hooks/ghosts tracking information never stored
  in contract state.
- property_analysis_prompt.j2: 6-lens coverage checklist (unit / valid-state /
  variable-transition / state-transition / multi-call / risk) after the
  3-category block, explicitly a brainstorming discipline, not a quota; task
  steps ask the agent to record swept lenses in its reasoning. No PropertyType
  schema change.
- cvl_additions.j2: three author-only advice entries — lastStorage/at storage
  snapshots with the additivity idiom, satisfy-witness companion rules for
  implication-shaped assertions, ghost counters via expression summaries.
- tests/test_template_render.py: render smoke tests for the three templates.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Expression summaries replace the callee's implementation and never
apply to direct CVL invocations; the advice now says so, so agents
only use the idiom for external-dependency calls.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"source" is not a member of Sort (Literal["greenfield", "existing",
"update"] in composer/spec/service_host.py); "existing" exercises the
same non-greenfield template branch with a value production can pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Non-persistent ghosts are rolled back by reverted calls and at-snapshot
restores, so a counter combined with the lastStorage additivity idiom
would silently reset. Marking counters persistent is consistent with
the persistent-ghost guideline since they do not mirror contract state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…efense

Three template-level fixes from the architecture review of the property
portfolio prompts:

- Guideline 24: solver/capacity bounds (e.g. require totalSupply() <= 2^128)
  are a recognized documented-trust-assumption class needing no invariant
  attempt, and preserved blocks defer to the stricter judge standard
  (requireInvariant or verifiable outside evidence) instead of the
  comment-suffices rule, removing the conflict with judge Criteria 4.
- Author instruction: companion satisfy-witness rules and supporting
  invariants must be listed in property_rules under their parent property,
  since rules unreferenced by any property are dropped from the report.
- Guideline 20: persistent counter ghosts and expression summaries on
  external-dependency calls are legitimate for information never stored in
  contract state, so the judge shares the vocabulary the author advice uses.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this test. does nothing

@shellygr shellygr marked this pull request as ready for review July 4, 2026 16:33
@shellygr shellygr requested a review from jtoman July 5, 2026 19:35
Comment thread composer/templates/cvl_additions.j2 Outdated

@jtoman jtoman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my 2 c.

Comment on lines +47 to +52
24. A bare `require` that constrains contract state inside a rule or `preserved` block is a spec smell: the assumption may not actually hold in reachable states, silently
weakening the proof. The default discipline is to state the assumption as an `invariant`, prove it, and assume it via `requireInvariant`. Bare requires over contract
state are acceptable only when accompanied by a comment referencing a justification or a documented trust assumption. Exception: capacity/overflow-avoidance bounds on
supplies, balances, or timestamps (e.g. `require totalSupply() <= 2^128`) exist to keep the solver tractable, not to model contract behavior — they are a recognized
documented-trust-assumption class; a comment naming the bound's purpose suffices, and no invariant attempt is expected because such bounds are typically not preserved
by the contract. In `preserved` blocks the stricter standard applies — `requireInvariant` or verifiable outside evidence; a comment alone is insufficient.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we discussed this on zoom, but we already have guidance for both the judge and author to not use require as a "make the rule pass" magic. Repeating it a third time will not fix the problem imo. The instructions you provide here are almost verbatim in the author prompt. It would be useful if we could find the run where this happened and see what reasoning/thinking tricks the agent used to talk itself out of the rules it already knew about, rather than just repeating the rule one more time and hoping that one makes it stick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants