Rerun the pass that raised a 🟡 when its fix changes what a test asserts or whether it runs #592

Merged
lilleman merged 6 commits from item-399 into main 2026-11-18 10:14:49 +01:00
5 changed files with 24 additions and 17 deletions
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- larv-review reruns the pass that raised a 🟡 when the fix removes or loosens an assertion, changes an expected value, or skips, deletes or guards a test. Other 🟡 fixes still rerun no pass.
- Add `markdown-section <file> <fragment>`, which prints the section of a markdown file under the heading the fragment picks, to the next heading of its level or a higher one. It fails, listing the headings, where several hold the fragment.
- loop-claude and subagent-rung run an L 3 task one rung above the project's rung and an L 4 task two above it: on an Opus 5.5 project, Fable 5.2 at low and at medium.
- Add `larv-review round-title <pass>`, which prints a review round commit's title prefix with the 🔴, 🟡 and misread counts taken from the round's report. rule-hook refuses a round commit whose counts differ from that report, where the report exists and can be counted; a report must head each finding `### 🔴` or `### 🟡`, optionally followed by ` [misread]`.
+3 -1
View File
@@ -314,7 +314,7 @@ Valid while no published standard sets either number, the reading-cost window st
2026-10-08, lilleman. Serves goal 1.4: run beside the full scan, `/security-review` started seconds apart from it each round and returned a near-identical report. Valid while the full scan covers what `/security-review` reports.
## review: a fix for a 🔴 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in `larv-review` §3's re-entry table the fix meets; from round 2 a pass reviews only the answers to its last round's findings and any other pass's fix that re-entered it; a fix for a 🟡, a 🟢, a ❓ or a prose finding reruns none; the stability pass doesn't rerun on a round's fixes where every 🔴 and 🟡 of that round asked only that a named mutant be killed or a named failing test pass, and the tests and the mutation command run clean on the fixes
## review: a fix for a 🔴 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in `larv-review` §3's re-entry table the fix meets; from round 2 a pass reviews only the answers to its last round's findings and any other pass's fix that re-entered it; a fix for a 🟢, a ❓ or a prose finding doesn't rerun any pass, and a fix for a 🟡 reruns the pass that raised it only when the fix weakens, skips or guards a test; the stability pass doesn't rerun on a round's fixes where every 🔴 and 🟡 of that round asked only that a named mutant be killed or a named failing test pass, and the tests and the mutation command run clean on the fixes
2026-10-08, lilleman. Serves goal 1.4. An audit of the four costliest archived chunks found nearly every finding that mattered in a pass's first round; when every fix, a nit's included, re-entered review, later rounds mostly re-checked the last fix, reworded what an earlier round wrote, or contradicted another round on the same line. A round 2 caught real bugs only where a fix changed behaviour, so stability still reruns on every fix that does, and security on every fix that opens an untrusted-input boundary or touches authn/authz, secrets or crypto. Valid while round-2 reviews of nit and prose fixes change only what a user wouldn't notice.
@@ -324,6 +324,8 @@ Valid while no published standard sets either number, the reading-cost window st
2026-11-12, lilleman through todo item 308. Serves goal 1.4. Since the worker runs the mutation command before each stability round, a finding that asks only that a named mutant be killed or a named failing test pass has an answer a run can check. The archive from 2026-09-30 to 2026-11-12 holds 326 stability reports of round 2 or later, from 13 projects, and 81 of those reports open with BLOCK (`stability-skip-count/`). For each of the 326, the 🔴 and 🟡 lines of the report before it were matched on mutant and test-failure wording. Every such line matched in 5 of those earlier reports. Read by hand, 1 of the 5 asked only that a failing test pass, and the round 2 after each of the 5 approved. So none of the 81 BLOCKs fell in a round this skips. The skip needs every 🟡 to be of that kind too: in one chunk on 2026-11-12, round 1's only 🔴 was a red suite, and round 2 blocked on a 🟡 whose answer was wrong, which no test or mutant showed. A fix may not edit the named failing test, since a test made to pass that way doesn't show anything. Fixes from such a round still rerun every other pass whose row they meet. Valid while code that such a skipped round's fixes wrote is rarely what a later chunk's 🔴 findings fix.
2026-11-18, lilleman through todo item 399. Serves goals 1.1 and 1.4. The 2026-10-29 entry's validity condition broke once. In a client project's pull request, stability round 1 raised a 🟡: an end-to-end payroll test passes on a used database. It proposed asserting the created row counts. The answer commit added that assertion behind a `test.skip` that runs when every count is zero. A broken payroll also creates zero rows, so a regression became a skip. The gate and the mutation run don't catch it, since a skipped test counts as passing; only the forge's auto-reviewer did. So a 🟡 fix that removes or loosens an assertion, changes an expected value, or skips, deletes or guards a test or assertion reruns the pass that raised it, as a 🔴 fix does. A 🟡 fix that only kills a named mutant or makes a named failing test pass doesn't, since the tests and the mutation command check it, and a skipped test kills no mutant. Every other 🟡 fix keeps the 2026-10-29 rule: the 2026-10-08 audit found that reruns of fixes to minor findings mostly re-checked wording. The rerun reviews only the answers, and the cap of two rounds still ends the pass. Valid while fixes of this kind are a small share of 🟡 fixes, so the extra round stays cheap.
## review: a comment is prose in any file; a branch changing only Markdown no agent follows runs only the prose pass and, for README or docs an outside reader lands on, the product-owner pass; plan rows a chunk files from a finding, re-scores or moves get no review
2026-10-29, lilleman. Serves goals 1.4 and 1.5. One project's chunk on 2026-10-29 changed a README paragraph, `todo.md` and three lines of `gate.yml`, and the principles, security and stability passes all ran on it, each twice: 6 of the chunk's 10 rounds. A change to config is one that changes what runs, so a comment in it fires only the prose pass, as a comment in code already did. A branch that changes only Markdown no agent follows changes no code, config or tests, so no pass that reads those runs on it. A plan row a chunk files from a finding, re-scores or moves is read by the chunk that ships it, so reviewing it at filing reviewed it twice: that chunk spent a prose round of 4 minutes on its plan change. Markdown an agent follows, such as a skill, is config, since a change there changes what agents do. Valid while defects in prose-only chunks stay rare in later chunks' fixes.
+17
View File
@@ -2570,3 +2570,20 @@ Read most (opens and searches):
- `bin/failure.py` 1
- `bin/markdown_section/test_markdown_section.py` 1
- `bin/run-tests` 1
## Reading cost, chunk 4655673a30a9, finished 2026-11-17T11:46:22Z
111449 new input tokens before the first commit.
Read most (opens and searches):
- `bin/review_replay/test_review_replay.py` 6
- `README.md` 2
- `bin/review_replay/main.py` 2
- `bin/review_replay/replay.py` 2
- `docs/decisions.md` 2
- `CHANGELOG.md` 1
- `bin/review_replay/omit_comparison.py` 1
- `bin/rung_replay/archive.py` 1
- `bin/rung_replay/search.py` 1
- `bin/rung_replay/test_rung_replay.py` 1
+3 -2
View File
@@ -87,18 +87,19 @@ Give a reviewer without Bash `git ls-files`'s output in its prompt, since some h
- **Done.** A pass is done when its latest round leaves no 🔴 or 🟡 standing: none was reported, or each was shown factually wrong, declined, filed, or named under the bar per "Findings", or settled in the PR summary per "Comprehension check" → "Places", or, for the prose pass and a pass at its cap, fixed. A stability pass whose round's fixes skip the next round per "Re-entry" → "Mechanical answers" is done once that rule's run is clean. Resolve that round's 🟢s and ❓s per "Findings" too.
- **Cap.** A pass takes at most two rounds per chunk, and the release review's architecture and principles passes five; don't ask the user for more. Answer its last round's findings per "Findings": those fixes merge once the gate passes, unreviewed by that pass. rule-hook's `round-cap` refuses a round commit that would take a pass past its cap, and no override waives it. A review the auto-reviewer posts after its second isn't a round: fix none of it and rerun no pass for it, but file a security finding it raises per `~/.claude/CLAUDE.md` → "Working loop", and any other 🔴 in `todo.md`.
**Re-entry.** A fix is a change no pass has seen. A fix for a 🟡, a 🟢, a ❓ or a prose finding reruns no pass, whatever it changes. A round's fixes for 🔴 findings re-enter together: run every pass that a row below names for any of them, a done pass included, in the waves of "Waves", then §4; "Mechanical answers" below names the one exception. A pass at its cap doesn't run again ("Round cap" → "Cap").
**Re-entry.** A fix is a change no pass has seen. A fix for a 🟢, a ❓ or a prose finding doesn't rerun any pass, whatever it changes. A fix for a 🟡 meets only the 🟡 row below. A round's fixes for 🔴 findings, and its 🟡 fixes that meet that row, re-enter together: run every pass that a row below names for any of them, a done pass included, in the waves of "Waves", then §4; "Mechanical answers" below names the one exception. A pass at its cap doesn't run again ("Round cap" → "Cap").
| The fix | Reruns |
|---------|--------|
| answers a 🔴 of a pass other than prose | that pass |
| answers a 🟡 of a pass other than prose, and removes or loosens an assertion, changes an expected value, or skips, deletes or guards a test or assertion with a condition or a catch, unless it only kills a named mutant or makes a named failing test pass | that pass |
| carries an architectural decision, in the release review | Architecture |
| changes code, config or tests, in the release review | Principles |
| opens an untrusted-input boundary or touches authn/authz, secrets or crypto | Security, unless the full scan reruns on it |
| made rule-hook log an override | Overrides |
| changes behaviour: what the code or its tests do | Stability |
A fix meets every row it fits. Prose has no row, so a comment or doc edit inside a fix doesn't rerun any pass. Example: a fix for a stability 🔴 that changes code and a comment reruns Principles and Stability.
A fix meets every row it fits, except that a fix for a 🟡 meets only the 🟡 row. Such a fix meets no row once §2's tests and mutation command run clean on it and the named failing test is unedited ("Mechanical answers"); a skipped test kills no mutant. Prose has no row, so a comment or doc edit inside a fix doesn't rerun any pass. Example: a fix for a stability 🔴 that changes code and a comment reruns Principles and Stability.
**Mechanical answers.** A stability round's fixes may skip the next stability round. That holds where the round reported a 🔴, and each of its 🔴 and 🟡 findings asked only that a named mutant be killed or a named failing test pass. Run §2's tests and mutation command on the fixes, and fix again per §2 until they leave no failing test and no surviving mutant that changes behaviour. Each named failing test must pass as the round found it, not edited or deleted. Then the stability pass doesn't run again, and the round's answer commit title ends `; tests and mutants clean at <commit>`. The fixes still rerun each other pass whose row they meet. Where the run can't be made clean, or a fix edits a named failing test, the stability pass runs again as the table says.
-14
View File
@@ -13,7 +13,6 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 398 | 0.12.0 | | | **Make each reasonable fix an outside reviewer asks for on a PR trigger a reflection that changes the setup so review would have caught it, in this repo first.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 6, 1.2 by 2 | 2.9 |
| 399 | 0.12.0 | | | **Rerun the pass that raised a 🟡 when its fix changes what a test asserts or whether it runs.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 6 | 2.2 |
| 404 | 0.12.0 | | | **Read do-epic's machine-wide settings from `~/.claude-projects/do-epic/config.toml` and its prose repo rule from `repo-rule.md` beside it, and refuse to start while either is missing or misspelt.** | 2 | 1 | 1 | 2 | 3 | 1 by 5 | 1.9 |
| 407 | 0.12.0 | 404 | | **Find an epic repo's checkout and project config by the repo's name, clone a missing one from the host of the epic's other repos, and ask the human where unsure.** | 2 | 2 | 2 | 4 | 3 | 1 by 5 | 1.9 |
| 401 | 0.12.0 | | | **Move loop-claude's cycle, judge, interrupt, tmux and routing into a package a second entry point drives, with loop-claude's behaviour unchanged.** | 3 | 2 | 3 | 6 | 6 | 1 by 3, code health by 2 | 1.8 |
@@ -101,19 +100,6 @@ Touches:
- `skills/larv-review/SKILL.md`: `## 4. Gate` (small), `## 3. Passes` (small)
- `CLAUDE.md`: `## Working loop` (small)
### 399. Rerun the pass that raised a 🟡 when its fix changes what a test asserts or whether it runs.
In a client project's pull request, stability round 1 raised a 🟡: an end-to-end payroll test passes on a used database. It proposed asserting the created row counts. The answer commit added that assertion behind a `test.skip` that runs when every count is zero. A broken payroll also creates zero rows, so a regression became a skip. No pass reviewed the fix, because a 🟡 fix reruns none (`skills/larv-review/SKILL.md` §3 → "Re-entry"). The forge's auto-reviewer found it.
Neither the gate nor the mutation run catches a weakened test, since a skipped test counts as passing. The rerun reviews only the answers (`review-findings.md`), and the cap of two rounds still ends the pass.
Revise the decision `review: a fix for a 🔴 reruns the pass that found it…` in `docs/decisions.md`. Its 2026-10-29 entry holds only "while 🔴 findings in later chunks, fixing code a 🟡 fix wrote, stay rare".
Touches:
- `skills/larv-review/SKILL.md`: `**Re-entry.**` (small)
- `docs/decisions.md`: `## review: a fix for a 🔴 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in larv-review §3's re-entry table the fix meets`
### 404. Read do-epic's machine-wide settings from `~/.claude-projects/do-epic/config.toml` and its prose repo rule from `repo-rule.md` beside it, and refuse to start while either is missing or misspelt.
Settings: the Jira site, `env-file`, the folder repos are cloned into (`~/code` by default), the default language, and the model for a no-repo item (`docs/decisions.md` → `do-epic: drives one Jira epic …`). Reuse `project_config/settings.py`'s parsing helpers: `read`, `anywhere`, `Scalar`, `language_fault`, and the `env-file` check that keeps credentials outside a checkout.