Skip a stability round 2 whose every 🔴 named a surviving mutant or a failing test
CI / tests (pull_request) Successful in 2m12s

This commit is contained in:
2026-11-12 11:27:46 +01:00
parent 02bde923e5
commit dd7e9b9929
5 changed files with 25 additions and 10 deletions
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- larv-review skips a stability round 2 whose every 🔴 named a surviving mutant or a failing test, once the tests and the mutation command show each one answered.
- `review-replay` now qualifies a rung whose share of a pass's archived findings is no more than 10 points below the mean of two replays on the project's rung, over at least 8 rounds, where it misses no 🔴 that both found. It used to demand every finding the project's rung found again, which the project's rung itself failed. It tries the project's model at each lower effort first, then the ladder below. Its run on the prose pass of Opus 5.5 projects qualified no rung: Opus 5.5 at low effort found 49% against a bar of 66%.
- loop-claude records beside each verdict, interrupt and quoted answer the claude-config release and commit that ran it, the tokens the worker, each subagent and the judge spent, by model and kind, with the list prices in force, and any typed answer of yours the judge quotes as reversing a choice the worker took on its own recommendation; `score-outcomes` counts those reversals per finished chunk.
- `larv-review prepare-scoring` makes a scoring run's worktree and writes each seat's prompt; `record-scoring` removes both once it has recorded the run.
+3 -1
View File
@@ -237,7 +237,7 @@ Valid while no published standard sets either number, the reading-cost window st
2026-10-08, lilleman. Serves goal 1.4: run beside the full scan, `/security-review` started seconds apart from it each round and returned a near-identical report. Valid while the full scan covers what `/security-review` reports.
## review: a fix for a 🔴 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in `larv-review` §3's re-entry table the fix meets; from round 2 a pass reviews only the answers to its last round's findings and any other pass's fix that re-entered it; a fix for a 🟡, a 🟢, a ❓ or a prose finding reruns none
## review: a fix for a 🔴 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in `larv-review` §3's re-entry table the fix meets; from round 2 a pass reviews only the answers to its last round's findings and any other pass's fix that re-entered it; a fix for a 🟡, a 🟢, a ❓ or a prose finding reruns none; fixes for a stability round whose every 🔴 named a surviving mutant or a failing test rerun no Stability round once the tests and the mutation command show each answered
2026-10-08, lilleman. Serves goal 1.4. An audit of the four costliest archived chunks found nearly every finding that mattered in a pass's first round; when every fix, a nit's included, re-entered review, later rounds mostly re-checked the last fix, reworded what an earlier round wrote, or contradicted another round on the same line. A round 2 caught real bugs only where a fix changed behaviour, so stability still reruns on every fix that does, and security on every fix that opens an untrusted-input boundary or touches authn/authz, secrets or crypto. Valid while round-2 reviews of nit and prose fixes keep changing nothing a user would notice.
@@ -245,6 +245,8 @@ Valid while no published standard sets either number, the reading-cost window st
2026-10-29, lilleman. Serves goals 1.4 and 1.5. A 🟡 fix no longer reruns its pass. One project's chunk on 2026-10-29, a README paragraph and plan rows, took 10 review rounds in 45 minutes, and three of its four second rounds answered only 🟡 findings, such as the wording of a question's options. Round 2 found 12 🔴 in about 200 runs since 2026-10-21, mostly in round 1's fixes of 🔴 findings, so a 🔴 fix still reruns its pass. A 🟡 fix that breaks something merges unreviewed by its pass; another pass whose row it meets for a 🔴 fix in the same round still reviews it. Valid while 🔴 findings in later chunks, fixing code a 🟡 fix wrote, stay rare.
2026-11-12, lilleman through todo item 308. Serves goal 1.4. Since the worker runs the mutation command before each stability round, a 🔴 that named a surviving mutant or a failing test has an answer a run can check: the mutant is killed, or the test passes. The archive from 2026-09-30 to 2026-11-12 holds 325 stability reports of round 2 or later, from 13 projects, and 81 of them open with BLOCK. Matching the 🔴 lines of the round before each one on mutant and test-failure wording found 7, and reading those 7 left 3 in which every 🔴 named a failing test. All 3 approved. The 1 BLOCK among the 7 followed 🔴 findings about the mutation tool's own code, not about a surviving mutant. So none of the 81 BLOCKs fell in a round this skips. Fixes from such a round still rerun every other pass whose row they meet. Valid while later chunks' 🔴 findings rarely fix code that such an answer wrote.
## review: a comment is prose in any file; a branch changing only Markdown no agent follows runs only the prose pass and, for README or docs an outside reader lands on, the product-owner pass; plan rows a chunk files from a finding, re-scores or moves get no review
2026-10-29, lilleman. Serves goals 1.4 and 1.5. One project's chunk on 2026-10-29 changed a README paragraph, `todo.md` and three lines of `gate.yml`, and the principles, security and stability passes all ran on it, each twice: 6 of the chunk's 10 rounds. A change to config is one that changes what runs, so a comment in it fires only the prose pass, as a comment in code already did. A branch that changes only Markdown no agent follows changes no code, config or tests, so no pass that reads those runs on it. A plan row a chunk files from a finding, re-scores or moves is read by the chunk that ships it, so reviewing it at filing reviewed it twice: that chunk spent a prose round of 4 minutes on its plan change. Markdown an agent follows, such as a skill, is config, since a change there changes what agents do. Valid while defects in prose-only chunks stay rare in later chunks' fixes.
+17
View File
@@ -1788,3 +1788,20 @@ Read most (opens and searches):
- `bin/subagent_rung/main.py` 1
- `bin/subagent_rung/test_subagent_rung.py` 1
- `ci.sh` 1
## Reading cost, chunk ae13d485e994, finished 2026-11-12T10:25:08Z
97469 new input tokens before the first commit.
Read most (opens and searches):
- `bin/review_replay/test_review_replay.py` 15
- `bin/review_replay/main.py` 11
- `docs/decisions.md` 9
- `todo.md` 7
- `README.md` 4
- `bin/model_routing/ladder.py` 4
- `bin/review_replay/rounds.py` 4
- `bin/review_replay/test_recall.py` 3
- `CHANGELOG.md` 2
- `skills/larv-review/SKILL.md` 2
+4 -2
View File
@@ -82,10 +82,10 @@ Give a reviewer without Bash `git ls-files`'s output in its prompt, since some h
- **Round.** A round is one review by a pass's reviewer: the first and each re-entry. The rerun after the architect returns Stage 0 questions ("Architecture" below) and a relaunch one rung up ("Rung up" below) stay in their round.
- **Auto-reviewer.** The external auto-reviewer (§4) is a pass of its own, each review it posts, up to the cap, a round. Its count is the number of its reviews on the PR, less its reviews of a `Record the comprehension check:` commit, and `round-count` doesn't list it. A commit that answers one of its rounds takes an ordinary title, not an `Answer <pass> round` one, and a round that didn't change anything gets no commit.
- **Count.** Close each round of a pass other than the auto-reviewer with one commit that holds the round's fixes. Title it `Answer <pass> round <n>: <k> 🔴, <m> 🟡 (<j> misread); …`: `<pass>` is the pass's name in lower case, such as `full scan`, `<k>` and `<m>` are the 🔴 and 🟡 findings all the round's reviewers reported, and `<j>` is how many of those they marked `[misread]` (review-findings.md → "Misread"); a reviewer that marks none, such as `/security-review`, adds 0. Example: `Answer stability round 1: 1 🔴, 2 🟡 (1 misread); flag a deleted test`. A round that didn't change anything gets an empty commit (`git commit --allow-empty`). Push the commits when §4 runs next, where there is a remote. `~/.claude/bin/larv-review round-count` prints each pass's count and marks a pass at the cap.
- **Done.** A pass is done when its latest round leaves no 🔴 or 🟡 standing: none was reported, or each was shown factually wrong, declined, filed, or named under the bar per "Findings", or settled in the PR summary per "Comprehension check" → "Places", or, for the prose pass and a pass at its cap, fixed. Resolve that round's 🟢s and ❓s per "Findings" too.
- **Done.** A pass is done when its latest round leaves no 🔴 or 🟡 standing: none was reported, or each was shown factually wrong, declined, filed, or named under the bar per "Findings", or settled in the PR summary per "Comprehension check" → "Places", or, for the prose pass and a pass at its cap, fixed, or, for a stability round of "Re-entry" → "Mechanical answers", fixed and shown killed or passing. Resolve that round's 🟢s and ❓s per "Findings" too.
- **Cap.** A pass takes at most two rounds per chunk, and the release review's architecture and principles passes five; don't ask the user for more. Answer its last round's findings per "Findings": those fixes merge once the gate passes, unreviewed by that pass. rule-hook's `round-cap` refuses a round commit that would take a pass past its cap, and no override waives it. A review the auto-reviewer posts after its second isn't a round: fix none of it and rerun no pass for it, but file a security finding it raises per `~/.claude/CLAUDE.md` → "Working loop", and any other 🔴 in `todo.md`.
**Re-entry.** A fix is a change no pass has seen. A fix for a 🟡, a 🟢, a ❓ or a prose finding reruns no pass, whatever it changes. A round's fixes for 🔴 findings re-enter together: run every pass that a row below names for any of them, a done pass included, in the waves of "Waves", then §4. A pass at its cap doesn't run again ("Round cap" → "Cap").
**Re-entry.** A fix is a change no pass has seen. A fix for a 🟡, a 🟢, a ❓ or a prose finding reruns no pass, whatever it changes. A round's fixes for 🔴 findings re-enter together: run every pass that a row below names for any of them, a done pass included, except as "Mechanical answers" below says, in the waves of "Waves", then §4. A pass at its cap doesn't run again ("Round cap" → "Cap").
| The fix | Reruns |
|---------|--------|
@@ -98,6 +98,8 @@ Give a reviewer without Bash `git ls-files`'s output in its prompt, since some h
A fix meets every row it fits. Prose has no row, so a comment or doc edit inside a fix doesn't rerun any pass. Example: a fix for a stability 🔴 that changes code and a comment reruns Principles and Stability.
**Mechanical answers.** Where every 🔴 of a stability round named a surviving mutant or a failing test, run §2's tests and mutation command on the round's fixes instead of a Stability round. Where each named mutant is now killed and each named test passes, the fixes rerun no Stability round, though they still meet the table's other rows. Where one isn't, they re-enter Stability as the table says.
**What a round reviews.** An agent under `~/.claude/agents/` reviews the branch in round 1. From round 2 it reviews only the answers to its last round's findings, and each fix from another pass that re-entered its pass. Prompt it with the path of its last round's report and each finding's answer: the fix's diff, or the reason it was dismissed. Give it each other pass's fix as `git diff <the commit before that fix> <the fix's commit>`, since the fixes may be unpushed, with the finding it answered. It reports only an answer that is plainly wrong or worse than the change it answered, so its pass converges in its second round. Other reviewers review the branch as their tool does.
**Rung up.** A reviewer rules the work below standard when its report holds systems-architect's BLOCK or a finding against `standards/quality.md`. A comprehension seat does when `record-ruling` counts one of its votes as worse.
-7
View File
@@ -14,7 +14,6 @@
|---|---|---|---|---|---|---|---|---|---|---|---|
| 340 | 0.9.0 | | question | **Decide whether a reviewer may start without `CLAUDE.md` where its replay misses no more findings than an unchanged replay does.** | 2 | 2 | 3 | 5 | 4 | 1.4 by 5 | 1.7 |
| 331 | 0.9.0 | | question | **Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.** | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.9 |
| 308 | 0.9.0 | | | **Skip a stability round 2 whose every must-fix finding named a surviving mutant or a failing test, both of which the worker now checks before review.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3, 1.5 by 2 | 1.6 |
| 319 | 0.9.0 | | | **Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 2, 1.4 by 2 | 1.4 |
| 309 | 0.9.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.3 |
| 299 | 0.9.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.2 |
@@ -88,12 +87,6 @@ The architect's 🔀 answer on PR #493 (item 292): four decisions expire "when a
Recommendation: reword all four as above. They are decisions the repo held before item 292, so the rewording is lilleman's call.
### 308. Skip a stability round 2 whose every must-fix finding named a surviving mutant or a failing test, both of which the worker now checks before review.
Rounds past the first cost $846 of the $9,154 the archive measured from 2026-09-30 to 2026-11-11; stability's were $281 over 440 runs. Now that the worker runs `mutate-changes` before each stability round (`skills/larv-review/SKILL.md` §2 → "Mutation tests"), a 🔴 that named a surviving mutant or a failing test has a mechanical answer: a run of `mutate-changes` on the fix shows the mutant killed, or the test passes. A round 2 whose every 🔴 and 🟡 is of that kind needs no reviewer.
Round 2 still matters elsewhere: 53 of 241 archived stability reports of round 2 or later open with BLOCK. So first count, from the archived round 2s, the BLOCKs that a run of `mutate-changes` and the tests on the fix would not have raised, and ship the skip only if none falls in a round it would skip. Evidence: one review round scored F1 0.376 and a second 0.26 to 0.30 with more false positives (arXiv 2603.16244, `claude -p` with Opus 4.6, 30 artifacts). Revise larv-review §3 → "Re-entry" and the decision `review: a fix for a 🔴 reruns the pass that found it…`.
### 319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
On SWE-bench Verified with Haiku 4.5 and Opus 4.7, handing the strong model the weak model's transcript solved 69.2% at $1.61 a task; starting the strong model fresh solved 79.2% at $0.90 (arXiv 2608.24358, The Handoff Tax). A model switch also rereads the whole context uncached (Claude Code prompt caching docs). A worker that steps from Sonnet 5.5 to Opus 5.5 by `--resume` is the raw handoff; an effort-only step within one model keeps the cache and isn't. The escalated subagent, which starts fresh with the trigger's evidence, already follows the pattern that won.