Mutation-test the lines each chunk changes in the stability pass
This commit is contained in:
@@ -31,6 +31,7 @@ Decisions live in `docs/decisions.md`:
|
||||
- rule-hook: a PreToolUse hook enforces each rule a program can check, a `# rule-override <rule>: <reason>` marker overrides one, and larv-review's overrides pass rules on each override before merge
|
||||
- rules/: `CLAUDE.md` holds the worker's rules that no file path can scope, and each topic's rules for the worker live in a `rules/` file that Claude Code loads by its `paths:` frontmatter
|
||||
- worker: a chunk's worker writes the simplest change that solves its item, test first wherever a test can check the change, and reads a technical principle only where the item, a finding or its own record names it; larv-review's reviewers hold the change to the principles and to the standards in `skills/larv-review/standards/`
|
||||
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
|
||||
|
||||
## Working rules
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- The stability reviewer mutation-tests the lines each change adds or modifies: it makes small changes to that code, such as flipping a comparison, and reports each one that changes behaviour and that no test notices as a 🟡 naming the test to add or strengthen. It uses the project's documented mutation command, or else mutmut, Stryker, cargo-mutants or Infection. The technical principles replace "Test coverage should not decline" with that rule, and say what a valuable test is: it checks behaviour through the unit's interface, and fails when that behaviour breaks. README → "Tools" gives the mutmut command for `bin/`.
|
||||
- Each new `loop-claude` worker session's prompt now quotes the README's Goals and Audience. A line above them says that the chunk's `todo.md` items, or else the task, set its scope, and that the goals only choose how to deliver it. A session outside the loop reads Goals and Audience before starting a chunk, as `CLAUDE.md` now says.
|
||||
- A session no longer reads the technical principles before writing code. It writes the simplest change that solves its item, test first wherever a test can check the change, and reads a principle only when the item changes it, a finding names it, or a decision or `todo.md` item it writes cites it. larv-review's reviewers hold the change to the principles, and a finding against one names it. Expect more findings from the prose and stability reviewers, since sessions no longer load the rules for comments and READMEs.
|
||||
- The rules for comments and READMEs, "Quality" and the examples behind "Prose" moved to `skills/larv-review/standards/`, which the prose and stability reviewers and `loop-claude`'s judge read. `CLAUDE.md` keeps a short core of "Prose"; sorting where order carries no meaning lives only in the technical principles.
|
||||
|
||||
@@ -77,6 +77,19 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
|
||||
Test on the host, with `uv` on `PATH` to fetch mypy and the Python version Install names (`docs/decisions.md` → `bin/`): `python3 -m unittest discover -s bin -p 'test_*.py'`
|
||||
|
||||
Mutation-test a changed module, here `larv_review/rounds.py`, in a scratch copy of `bin/`, since mutmut writes files beside its config. Name the module and its tests in a `setup.cfg` there, and leave out each test that starts a tool's launcher: the launcher's Python has no mutmut.
|
||||
|
||||
```ini
|
||||
[mutmut]
|
||||
source_paths=larv_review/rounds.py
|
||||
also_copy=.
|
||||
pytest_add_cli_args_test_selection=larv_review/
|
||||
pytest_add_cli_args=-k
|
||||
not LarvReviewTool
|
||||
```
|
||||
|
||||
Then, in the copy: `uvx --with pytest mutmut run && uvx --with pytest mutmut results`, and `uvx --with pytest mutmut show <mutant>` for a mutant's line.
|
||||
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, and `rule_hook` imports `rank_todo.py`'s check. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
|
||||
## Tracking a new file
|
||||
|
||||
@@ -13,7 +13,8 @@ You are an elite stability-focused code reviewer with deep expertise in defensiv
|
||||
2. **Read every file under `~/.claude/principles/`**, the shared principles, and `~/.claude/skills/larv-review/standards/quality.md`. These are mandatory org guidelines; all code MUST comply with each that applies. A file's `Applies to` line says what it covers; one without the line covers every change. Re-read each time — they change. Flag violations prominently.
|
||||
3. **Read README.md, AGENTS.md and `docs/decisions.md`** in the current project for project-specific conventions, build instructions, tooling, and the accepted trade-offs a reviewer should honor rather than re-flag. Note the documented test, lint, and static-analysis commands — you will run them.
|
||||
4. **Verify, don't just read.** Run the test suite and static analysis/linters through the project's container tooling, unless a decision puts them on the host; tear down test containers afterwards. A full-suite run catches broken callers that diff-reading misses. When a suspected bug is cheap to confirm, reproduce it — a scratch script, a targeted test run — before reporting it. Put scratch work in a directory you create for this run, and delete it when done; delete nothing else, since a directory the prompt names may hold other agents' files. Leave the working tree exactly as you found it. Anything you could not run goes in Coverage & Verification, never silently skipped.
|
||||
5. **Two passes minimum, then hunt absences.**
|
||||
5. **Mutation-test the changed lines.** Run the project's documented mutation command, or else a widely deployed mutation tool for the language: mutmut for Python, Stryker for JavaScript and TypeScript, cargo-mutants for Rust, Infection for PHP. Limit it to the lines the change adds or modifies, through the tool's own option where it has one (cargo-mutants `--in-diff`, Infection `--git-diff-lines`, Stryker `--mutate <file>:<lines>`); otherwise mutate only the changed files and keep only the mutants on changed lines. Run it in your scratch directory, since the tools write files. Each surviving mutant that changes behaviour is a 🟡 naming the test to add or strengthen; drop a mutant that leaves behaviour unchanged.
|
||||
6. **Two passes minimum, then hunt absences.**
|
||||
- *Breadth:* every hunk in every changed file, no skimming. Track which files you have covered; batch large diffs, never truncate the review.
|
||||
- *Depth:* trace the riskiest data paths end-to-end (input → logic → persistence/response), the interactions between changed pieces, and between the change and config (timeout hierarchies, retries × idempotency, queue worker counts, env parity).
|
||||
- *Absences:* what should be in this diff but isn't — un-updated callers (grep every changed or renamed public symbol), missing migration for a model change, missing authz on a new path, missing tests, un-regenerated OpenAPI/docs, stale README/AGENTS.md, and deletions the change obsoletes but didn't make.
|
||||
@@ -119,7 +120,7 @@ Every PR gets at least this sweep.
|
||||
### 11. Tests
|
||||
- Does the PR ship tests for the new behavior? (principle: most code has automated tests, and tests come first.)
|
||||
- Do they exercise the risky and error paths, not just the happy path?
|
||||
- Does test coverage decline? (principle: coverage must not decline.)
|
||||
- Does a mutant of a changed line survive (First Steps, step 5)? (principle: every mutant of a changed line that changes behaviour fails a test.)
|
||||
- Were tests and implementation changed together in a way that could mask a regression — e.g. an assertion weakened to make a test pass? Flag it; these should be separate changes.
|
||||
- Would each test fail if the behavior regressed? Watch for assertion-free tests, tautological asserts, and mocks so broad the test exercises the mock instead of the code.
|
||||
- Tests are code too: prefer extending an existing test with an assertion over adding a near-duplicate; flag overlapping tests that pin the same behavior twice, and tests of framework behavior or trivial pass-throughs — maintenance cost without protection.
|
||||
|
||||
@@ -164,3 +164,11 @@ A finding against a principle names it, so the worker reads only that one when i
|
||||
The item also cut the rule that a session proposes new rules for `CLAUDE.md` and `rules/` and restructures them after each change: changes to these rules go through this repo's chunks and their reviews, like any other change.
|
||||
|
||||
Valid while review rounds per pass do not rise. A principle found late, such as functional core, can cost a rewrite within a pass's three rounds. `todo.md` item 105 compares rounds per pass and reading cost in loop-claude's archive for the five chunks before and after this one.
|
||||
|
||||
## review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
|
||||
|
||||
2026-10-13, lilleman through todo item 108, implemented by claude. Serves goals 1.1 and 1.4. Coverage counts the lines a test runs, so a test without a meaningful assertion still raises it. A surviving mutant shows a test that would let behaviour break. Every mutant of an untested changed line survives, so mutation testing of the changed lines also shows what a coverage drop on them would. A deleted test is in the diff, which the stability reviewer reads. Measuring coverage beside it would add cost without raising quality, so todo item 69 was dropped and `principles/technical-principles.md` replaced "Test coverage should not decline" with the mutant rule.
|
||||
|
||||
The reviewer limits each run to the changed lines, so its cost grows with the chunk. The tools are those the language's ecosystem most uses, each with its own way to limit a run to a diff: cargo-mutants `--in-diff` (https://mutants.rs/in-diff.html), Infection `--git-diff-lines` (https://infection.github.io/guide/command-line-options.html), Stryker's mutation ranges (https://stryker-mutator.io/docs/stryker-js/configuration/). mutmut has no such option, so the reviewer mutates the changed files and keeps the mutants on changed lines. On 2026-10-13 mutmut 3 ran on this repo's `larv_review/rounds.py` in under two seconds, as README → "Tools" describes.
|
||||
|
||||
Valid while a mutation run on a chunk's changed lines stays cheap beside the review itself.
|
||||
|
||||
@@ -40,10 +40,11 @@
|
||||
- DTOs, enums, and value objects are pure data — no logic. They stand in for the structs/records a language lacks.
|
||||
* Always use explicit state machines.
|
||||
* Most, if not all, code should have automated tests.
|
||||
- A valuable test checks behaviour through the unit's interface, and fails when that behaviour breaks.
|
||||
- Write tests first for the behaviour we want, then implement until they pass.
|
||||
- Changing tests and changing other code are two separate processes — be very careful about doing both at once.
|
||||
- Use properly randomised UUID v7s — never `aaaa-0000`-style placeholders.
|
||||
* Test coverage should not decline.
|
||||
* Every mutant of a changed line that changes behaviour fails a test. A mutant is a small change to the code, such as a flipped comparison; one no test notices marks behaviour the tests would let break.
|
||||
* Idempotent data writing.
|
||||
- Typical example: `PUT /resource/{id}` instead of `POST /resource` + Idempotent-Id header.
|
||||
* Handle errors explicitly, as values — avoid the exception path.
|
||||
|
||||
@@ -24,7 +24,6 @@
|
||||
|
||||
| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| 108 | 0.1.0 | | **Mutation-test the lines each chunk changes, and make each mutant no test catches a finding of the stability pass.** | 2 | 4 | 7 | 8 | 1.1 | 26.8 |
|
||||
| 98 | 0.1.0 | | **Score R and S from a measurement wherever one exists, and calibrate each guessed score against the finished chunk's measured outcome.** | 2 | 5 | 6 | 7 | 1.1, 1.4 | 22.6 |
|
||||
| 99 | 0.1.0 | | **Drop the comprehension panel as a merge gate, and keep recording its scores and the places in the code its seats found hard, for later measurement and for other reviewers and agents to improve.** | 4 | 5 | 7 | 7 | 1.1, 1.4, 1.5 | 22.6 |
|
||||
| 87 | 0.1.0 | | **Score every `todo.md` item's complexity C from 1 to 10, and have `rank_todo.py` fail on an item without one.** | 3 | 4 | 6 | 7 | 1.1, 1.4 | 21.9 |
|
||||
@@ -51,7 +50,6 @@
|
||||
| 106 | 0.3.0 | | **Make larv-review's "Do not over-read a clean result" cite a rule that says Renovate or Dependabot owns dependency CVEs, or drop the citation.** | 1 | 1 | 3 | 5 | 1.1 | 14.7 |
|
||||
| 66 | 0.3.0 | principle | **Hold systems-architect's method thresholds only while the comprehension panel finds the flagged code hard.** | 3 | 3 | 4 | 5 | 1.1 | 14.2 |
|
||||
| 101 | 0.3.0 | principle | **Have `larv_review/record.py`'s `read_seats` take the time, the commit check and the history as inputs, so it parses and decides without effects.** | 3 | 3 | 4 | 5 | 1.1 | 14.2 |
|
||||
| 69 | 0.3.0 | principle | **Measure test coverage, so a chunk that lowers it is seen.** | 3 | 4 | 4 | 5 | 1.1 | 13.9 |
|
||||
| 29 | 0.3.0 | | **Make the contributor a standing member of every project's audience, unless its `## Audience` says maintainability does not matter.** | 2 | 2 | 3 | 5 | 1.1, 1.4 | 13.4 |
|
||||
| 68 | 0.3.0 | principle | **Test `workflow-bench/compare.py`'s blinding and `bench.py`'s run plan.** | 2 | 4 | 3 | 5 | 1.1 | 12.9 |
|
||||
| 65 | 0.3.0 | | **Archive each judge reply the loop rejects.** | 3 | 4 | 4 | 5 | 3 | 12.5 |
|
||||
@@ -68,17 +66,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 108. Mutation-test the lines each chunk changes, and make each mutant no test catches a finding of the stability pass.
|
||||
|
||||
lilleman, 2026-10-13: tests should be as valuable as possible, never written to tick a box. Today the technical principles ask that most code has tests, written first, and that coverage not decline, which item 69 would measure. The stability reviewer asks, by reading, whether each test would fail if the behaviour broke. Coverage counts the lines a test runs, so a test with no meaningful assertion still raises it.
|
||||
|
||||
Mutation testing measures what coverage cannot: it makes small changes to the code, called mutants, such as flipping a comparison, and counts the mutants no test notices. A mutant that survives usually marks a bug the tests would miss; one that leaves behaviour unchanged is not.
|
||||
|
||||
- Run it only on the lines a chunk changes, so its cost grows with the chunk (goal 1.4). Use a widely deployed tool for each language, such as mutmut for Python and Stryker for TypeScript.
|
||||
- Make a surviving mutant in the changed code a finding of the stability pass, which names the test to add or strengthen. The reviewer drops a mutant that leaves behaviour unchanged.
|
||||
- Say in the principle "Most, if not all, code should have automated tests" what a valuable test is: it checks behaviour through the unit's interface, and fails when that behaviour breaks.
|
||||
- Decide whether this replaces item 69's coverage measure.
|
||||
|
||||
### 98. Score R and S from a measurement wherever one exists, and calibrate each guessed score against the finished chunk's measured outcome.
|
||||
|
||||
A score that can be computed from the code it touches, such as the files an item names, comes from that computation. Otherwise size and risk are estimated from finished work, the way reference-class forecasting estimates a project from earlier projects like it. When a chunk merges, record its measured outcome beside the item's guessed R and S: lines and files changed, review rounds, 🔴 and 🟡 findings, and fixes the code needed later. Show the scorer, when it files an item, the finished items whose guessed scores sat nearest and what they measured. Once enough chunks are recorded, report how far each guess was from its outcome, so a scorer whose S runs high can be corrected. Item 87's C is scored and calibrated the same way.
|
||||
@@ -215,10 +202,6 @@ KISS, in `principles/technical-principles.md`, which keeps a limit only while re
|
||||
|
||||
Found by the principles audit of PR #242, against "Functional core, imperative shell" (`principles/technical-principles.md`). `bin/larv_review/record.py:94` reads `time.time()`, then checks git (`:97`, `commit_exists`) and the history file (`:100`) inside the function that parses the seats file and decides. A test cannot fix the heading's time, and every test of `read_seats` touches the real checkout.
|
||||
|
||||
### 69. Measure test coverage, so a chunk that lowers it is seen.
|
||||
|
||||
"Test coverage should not decline", in `principles/technical-principles.md`: nothing reports coverage, so a chunk that deletes a test or adds untested code passes unseen.
|
||||
|
||||
### 29. Make the contributor a standing member of every project's audience, unless its `## Audience` says maintainability does not matter.
|
||||
|
||||
State that in `~/.claude/rules/goals-and-audience.md` → "Goals and Audience", and have larv-review's comprehension floor name the contributor as the reader it serves. The floor's block then applies to a project whose audience includes the contributor, and a project that drops them skips it.
|
||||
|
||||
Reference in New Issue
Block a user