Answer stability round 1: 1 🔴, 2 🟡 (0 misread); require every 🔴 and 🟡 to be mechanical, keep the named test as found, recount the archive, and say which passes still rerun
This commit is contained in:
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- larv-review skips a stability round 2 whose every 🔴 named a surviving mutant or a failing test, once the tests and the mutation command show each one answered.
|
||||
- larv-review skips stability round 2 where round 1 reported a 🔴 and every 🔴 and 🟡 asked only that a named mutant be killed or a named failing test pass, once the tests and the mutation command run clean on the fixes.
|
||||
- `review-replay` now qualifies a rung whose share of a pass's archived findings is no more than 10 points below the mean of two replays on the project's rung, over at least 8 rounds, where it misses no 🔴 that both found. It used to demand every finding the project's rung found again, which the project's rung itself failed. It tries the project's model at each lower effort first, then the ladder below. Its run on the prose pass of Opus 5.5 projects qualified no rung: Opus 5.5 at low effort found 49% against a bar of 66%.
|
||||
- loop-claude records beside each verdict, interrupt and quoted answer the claude-config release and commit that ran it, the tokens the worker, each subagent and the judge spent, by model and kind, with the list prices in force, and any typed answer of yours the judge quotes as reversing a choice the worker took on its own recommendation; `score-outcomes` counts those reversals per finished chunk.
|
||||
- `larv-review prepare-scoring` makes a scoring run's worktree and writes each seat's prompt; `record-scoring` removes both once it has recorded the run.
|
||||
|
||||
+2
-2
@@ -237,7 +237,7 @@ Valid while no published standard sets either number, the reading-cost window st
|
||||
|
||||
2026-10-08, lilleman. Serves goal 1.4: run beside the full scan, `/security-review` started seconds apart from it each round and returned a near-identical report. Valid while the full scan covers what `/security-review` reports.
|
||||
|
||||
## review: a fix for a 🔴 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in `larv-review` §3's re-entry table the fix meets; from round 2 a pass reviews only the answers to its last round's findings and any other pass's fix that re-entered it; a fix for a 🟡, a 🟢, a ❓ or a prose finding reruns none; fixes for a stability round whose every 🔴 named a surviving mutant or a failing test rerun no Stability round once the tests and the mutation command show each answered
|
||||
## review: a fix for a 🔴 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in `larv-review` §3's re-entry table the fix meets; from round 2 a pass reviews only the answers to its last round's findings and any other pass's fix that re-entered it; a fix for a 🟡, a 🟢, a ❓ or a prose finding reruns none; the stability pass doesn't rerun on a round's fixes where every 🔴 and 🟡 of that round asked only that a named mutant be killed or a named failing test pass, and the tests and the mutation command run clean on the fixes
|
||||
|
||||
2026-10-08, lilleman. Serves goal 1.4. An audit of the four costliest archived chunks found nearly every finding that mattered in a pass's first round; when every fix, a nit's included, re-entered review, later rounds mostly re-checked the last fix, reworded what an earlier round wrote, or contradicted another round on the same line. A round 2 caught real bugs only where a fix changed behaviour, so stability still reruns on every fix that does, and security on every fix that opens an untrusted-input boundary or touches authn/authz, secrets or crypto. Valid while round-2 reviews of nit and prose fixes keep changing nothing a user would notice.
|
||||
|
||||
@@ -245,7 +245,7 @@ Valid while no published standard sets either number, the reading-cost window st
|
||||
|
||||
2026-10-29, lilleman. Serves goals 1.4 and 1.5. A 🟡 fix no longer reruns its pass. One project's chunk on 2026-10-29, a README paragraph and plan rows, took 10 review rounds in 45 minutes, and three of its four second rounds answered only 🟡 findings, such as the wording of a question's options. Round 2 found 12 🔴 in about 200 runs since 2026-10-21, mostly in round 1's fixes of 🔴 findings, so a 🔴 fix still reruns its pass. A 🟡 fix that breaks something merges unreviewed by its pass; another pass whose row it meets for a 🔴 fix in the same round still reviews it. Valid while 🔴 findings in later chunks, fixing code a 🟡 fix wrote, stay rare.
|
||||
|
||||
2026-11-12, lilleman through todo item 308. Serves goal 1.4. Since the worker runs the mutation command before each stability round, a 🔴 that named a surviving mutant or a failing test has an answer a run can check: the mutant is killed, or the test passes. The archive from 2026-09-30 to 2026-11-12 holds 325 stability reports of round 2 or later, from 13 projects, and 81 of them open with BLOCK. Matching the 🔴 lines of the round before each one on mutant and test-failure wording found 7, and reading those 7 left 3 in which every 🔴 named a failing test. All 3 approved. The 1 BLOCK among the 7 followed 🔴 findings about the mutation tool's own code, not about a surviving mutant. So none of the 81 BLOCKs fell in a round this skips. Fixes from such a round still rerun every other pass whose row they meet. Valid while later chunks' 🔴 findings rarely fix code that such an answer wrote.
|
||||
2026-11-12, lilleman through todo item 308. Serves goal 1.4. Since the worker runs the mutation command before each stability round, a finding that asks only that a named mutant be killed or a named failing test pass has an answer a run can check. The archive from 2026-09-30 to 2026-11-12 holds 325 stability reports of round 2 or later, from 13 projects, and 81 of those reports open with BLOCK. For each of the 325, the 🔴 and 🟡 lines of the round before were matched on mutant and test-failure wording. In 4 rounds before, every such line matched; read by hand, 1 of the 4 asked only that a failing test pass, and the round 2 after each of the 4 approved. So none of the 81 BLOCKs fell in a round this skips. The skip needs every 🟡 to be of that kind too: in one chunk on 2026-11-12, round 1's only 🔴 was a red suite, and round 2 blocked on a 🟡 whose answer was wrong, which no test or mutant showed. A fix may not edit the named failing test, since a test made to pass that way shows nothing. Fixes from such a round still rerun every other pass whose row they meet. Valid while code that such a skipped round's fixes wrote is rarely what a later chunk's 🔴 findings fix.
|
||||
|
||||
## review: a comment is prose in any file; a branch changing only Markdown no agent follows runs only the prose pass and, for README or docs an outside reader lands on, the product-owner pass; plan rows a chunk files from a finding, re-scores or moves get no review
|
||||
|
||||
|
||||
@@ -82,10 +82,10 @@ Give a reviewer without Bash `git ls-files`'s output in its prompt, since some h
|
||||
- **Round.** A round is one review by a pass's reviewer: the first and each re-entry. The rerun after the architect returns Stage 0 questions ("Architecture" below) and a relaunch one rung up ("Rung up" below) stay in their round.
|
||||
- **Auto-reviewer.** The external auto-reviewer (§4) is a pass of its own, each review it posts, up to the cap, a round. Its count is the number of its reviews on the PR, less its reviews of a `Record the comprehension check:` commit, and `round-count` doesn't list it. A commit that answers one of its rounds takes an ordinary title, not an `Answer <pass> round` one, and a round that didn't change anything gets no commit.
|
||||
- **Count.** Close each round of a pass other than the auto-reviewer with one commit that holds the round's fixes. Title it `Answer <pass> round <n>: <k> 🔴, <m> 🟡 (<j> misread); …`: `<pass>` is the pass's name in lower case, such as `full scan`, `<k>` and `<m>` are the 🔴 and 🟡 findings all the round's reviewers reported, and `<j>` is how many of those they marked `[misread]` (review-findings.md → "Misread"); a reviewer that marks none, such as `/security-review`, adds 0. Example: `Answer stability round 1: 1 🔴, 2 🟡 (1 misread); flag a deleted test`. A round that didn't change anything gets an empty commit (`git commit --allow-empty`). Push the commits when §4 runs next, where there is a remote. `~/.claude/bin/larv-review round-count` prints each pass's count and marks a pass at the cap.
|
||||
- **Done.** A pass is done when its latest round leaves no 🔴 or 🟡 standing: none was reported, or each was shown factually wrong, declined, filed, or named under the bar per "Findings", or settled in the PR summary per "Comprehension check" → "Places", or, for the prose pass and a pass at its cap, fixed, or, for a stability round of "Re-entry" → "Mechanical answers", fixed and shown killed or passing. Resolve that round's 🟢s and ❓s per "Findings" too.
|
||||
- **Done.** A pass is done when its latest round leaves no 🔴 or 🟡 standing: none was reported, or each was shown factually wrong, declined, filed, or named under the bar per "Findings", or settled in the PR summary per "Comprehension check" → "Places", or, for the prose pass and a pass at its cap, fixed. A stability pass whose round's fixes skip the next round per "Re-entry" → "Mechanical answers" is done once that rule's run is clean. Resolve that round's 🟢s and ❓s per "Findings" too.
|
||||
- **Cap.** A pass takes at most two rounds per chunk, and the release review's architecture and principles passes five; don't ask the user for more. Answer its last round's findings per "Findings": those fixes merge once the gate passes, unreviewed by that pass. rule-hook's `round-cap` refuses a round commit that would take a pass past its cap, and no override waives it. A review the auto-reviewer posts after its second isn't a round: fix none of it and rerun no pass for it, but file a security finding it raises per `~/.claude/CLAUDE.md` → "Working loop", and any other 🔴 in `todo.md`.
|
||||
|
||||
**Re-entry.** A fix is a change no pass has seen. A fix for a 🟡, a 🟢, a ❓ or a prose finding reruns no pass, whatever it changes. A round's fixes for 🔴 findings re-enter together: run every pass that a row below names for any of them, a done pass included, except as "Mechanical answers" below says, in the waves of "Waves", then §4. A pass at its cap doesn't run again ("Round cap" → "Cap").
|
||||
**Re-entry.** A fix is a change no pass has seen. A fix for a 🟡, a 🟢, a ❓ or a prose finding reruns no pass, whatever it changes. A round's fixes for 🔴 findings re-enter together: run every pass that a row below names for any of them, a done pass included, in the waves of "Waves", then §4; "Mechanical answers" below names the one exception. A pass at its cap doesn't run again ("Round cap" → "Cap").
|
||||
|
||||
| The fix | Reruns |
|
||||
|---------|--------|
|
||||
@@ -98,7 +98,7 @@ Give a reviewer without Bash `git ls-files`'s output in its prompt, since some h
|
||||
|
||||
A fix meets every row it fits. Prose has no row, so a comment or doc edit inside a fix doesn't rerun any pass. Example: a fix for a stability 🔴 that changes code and a comment reruns Principles and Stability.
|
||||
|
||||
**Mechanical answers.** Where every 🔴 of a stability round named a surviving mutant or a failing test, run §2's tests and mutation command on the round's fixes instead of a Stability round. Where each named mutant is now killed and each named test passes, the fixes rerun no Stability round, though they still meet the table's other rows. Where one isn't, they re-enter Stability as the table says.
|
||||
**Mechanical answers.** A stability round's fixes may skip the next stability round. That holds where the round reported a 🔴, and each of its 🔴 and 🟡 findings asked only that a named mutant be killed or a named failing test pass. Run §2's tests and mutation command on the fixes, and fix again per §2 until they leave no failing test and no surviving mutant that changes behaviour. Each named failing test must pass as the round found it, not edited or deleted. Then the stability pass doesn't run again, and the round's answer commit title ends `; tests and mutants clean at <commit>`. The fixes still rerun each other pass whose row they meet. Where the run can't be made clean, or a fix edits a named failing test, the stability pass runs again as the table says.
|
||||
|
||||
**What a round reviews.** An agent under `~/.claude/agents/` reviews the branch in round 1. From round 2 it reviews only the answers to its last round's findings, and each fix from another pass that re-entered its pass. Prompt it with the path of its last round's report and each finding's answer: the fix's diff, or the reason it was dismissed. Give it each other pass's fix as `git diff <the commit before that fix> <the fix's commit>`, since the fixes may be unpushed, with the finding it answered. It reports only an answer that is plainly wrong or worse than the change it answered, so its pass converges in its second round. Other reviewers review the branch as their tool does.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user