Answer prose round 1: 0 🔴, 5 🟡; count the audit's shares of all 196, say why the expected-values rule is repeated, and fix item 165's mutation claim

This commit is contained in:
2026-10-16 05:44:37 +02:00
parent 8ec835e36b
commit 5725baefab
5 changed files with 14 additions and 14 deletions
+1 -1
View File
@@ -14,7 +14,7 @@ How work proceeds in any repo, unless the repo's `AGENTS.md` says otherwise. The
- **Work one chunk at a time, and make it as large as one purpose allows.** Every review has a fixed cost, so a chunk takes the next plan item plus every other item for which four things hold: one PR title states what they achieve together, without listing them; they name one release; they fire the same review passes (`larv-review` §3); and no item outside the chunk blocks one of them. The next plan item is the first that does not wait: an item waits while it is marked `question`, until the human answers, or while an item that blocks it is still in the plan. Tiny items, a few lines of change each, need neither a shared purpose nor the same passes: bundle up to ten that name one release, and that no item outside the chunk blocks, into one chunk. Split a milestone into items in `todo.md` before starting it. Fix a finding in the chunk's own change within the chunk. File a finding about anything else in `todo.md`.
- **Ask, don't guess.** First check whether the goals already answer the question. If they nearly do, propose a change to the goals and show how it answers this case: an answer settles one question, a goal settles every question like it. Write every answer once, where it belongs, never only in chat: a goal in `README.md`, a working rule in `AGENTS.md`, a standing choice in a comment at the one place it governs, or in `docs/decisions.md` once made in more than one place. When an answer holds a goal and a choice, write the goal in the README, and record as a choice only what the goal leaves open.
- **Bench a syntax with the people who write it.** Before a new or changed spelling of something users read or write ships — markdown, HTML, a config key, a CLI flag — a panel of fresh-context readers, seated as those users from the README's `## Audience`, picks it.
- **Implement the simplest change that solves the item, and ship it with `/larv-review`.** Whenever a test can check the change, write that test first, taking expected values from the item, never from the code, and see it fail. larv-review's reviewers hold the change to the technical principles, and a finding against one names it. Read a file under `~/.claude/principles/` only when the item changes it, a finding or a record you write names it, or you decline a finding.
- **Implement the simplest change that solves the item, and ship it with `/larv-review`.** Whenever a test can check the change, write that test first and see it fail. Take its expected values from the item, never from the code. larv-review's reviewers hold the change to the technical principles, and a finding against one names it. Read a file under `~/.claude/principles/` only when the item changes it, a finding or a record you write names it, or you decline a finding.
- **Remove a finished item's row and detail from `todo.md`, and revise any decision that cites it.** If the repo publishes something others consume — a package, an image, an API — reword the item for them and move it to `CHANGELOG.md` under `## Unreleased`. A changelog is for the public, never for the next agent. Otherwise delete it; git history is the record.
- **Releases are the human's.** The version bump and the publishing credentials are theirs, whatever another rule says. Prepare everything else, so cutting a release is one decision.
- **Open each `todo.md` item or ticket with a bold one-sentence tagline**, so the taglines alone say what a release holds. The tagline states the target state, checkable when done: an imperative verb plus what holds afterwards. A present-tense fact reads as the current state and says nothing about what to do. Add "today X" only where the direction is otherwise ambiguous. GOOD: `**Accept undefined in MetadataValue and drop such keys on output.**` BANNED: `MetadataValue accepts undefined; such keys are dropped on output`.
+1 -1
View File
@@ -123,7 +123,7 @@ Every PR gets at least this sweep.
- Does a mutant of a changed line survive (First Steps, step 5)?
- Was a test deleted while the behaviour it pinned remains? Flag it; restore the test.
- Were tests and implementation changed together in a way that could mask a regression — e.g. an assertion weakened to make a test pass? Flag it; these should be separate changes.
- Does each test earn its keep, as the technical principles' test bullet says? Look at where its expected values come from, what it asserts, what wording it pins, whether another test already pins the behaviour, and whether it sleeps or reads the clock.
- Does each test earn its keep, as the technical principles' test bullet says? Look at where its expected values come from, what it asserts, what it mocks, what wording it pins, whether another test already pins the behaviour, and whether it sleeps or reads the wall clock.
- Are there untestable side effects that should be refactored?
+6 -6
View File
@@ -200,15 +200,15 @@ Valid while review rounds per pass do not rise. A principle found late, such as
2026-10-16, lilleman asked for research on what makes tests valuable, then for these changes; claude drafted them. Serves goal 1.1, since a test that cannot catch a bug, or breaks without one, lowers quality and costs review rounds.
An audit of 196 tests sampled across eight projects found about 61% clearly valuable. Of the rest, 15% pinned wording that is not the contract, such as whole prompt sentences; 9% copied or policed the implementation, such as scanning the source with `ast`; 6% ran on real time or heavy resources; and 6% asserted something no bug could break, such as a list's length. The bullet follows these sources:
An audit of 196 tests sampled across eight projects found about 61% clearly valuable. Of all 196, 15% pinned wording that is not the contract, such as whole prompt sentences; 9% copied or policed the implementation, such as scanning the source with `ast`; 6% ran on real time or heavy resources; and 6% asserted something no bug could break, such as a list's length. The bullet follows these sources:
- Coverage barely predicts fault detection: Inozemtseva & Holmes, ICSE 2014. Surviving mutants do predict real faults: Just et al., FSE 2014; Google's mutation testing in code review, arXiv 2102.11378.
- A valuable test fails when behaviour breaks, passes through a refactor, asserts state over interactions and keeps its expected outcome narrow: *Software Engineering at Google*, chapters 11, 12 and 14; Fowler, "The Practical Test Pyramid"; Beck, "Test Desiderata"; Khorikov, *Unit Testing: Principles, Practices, and Patterns*, chapter 4.
- Agents add mocks to tests more often than people do: Hora & Robbes, MSR 2026, arXiv 2602.00409.
- Tests whose expected values come from the code's current output pass on a bug and fail on its fix: Charoosaei et al., arXiv 2610.11835, a preprint; Mathews & Nagappan, arXiv 2412.14137.
- A valuable test fails when behaviour breaks, passes after a refactor, asserts state over interactions and keeps its expected outcome narrow: *Software Engineering at Google*, chapters 11, 12 and 14; Fowler, "The Practical Test Pyramid"; Beck, "Test Desiderata"; Khorikov, *Unit Testing: Principles, Practices, and Patterns*, chapter 4.
- A test that never failed on its assertion may be tautological: Seemann, blog.ploeh.dk, 2019-10-14.
- Tests whose expected values come from the code's current output pass on a bug and fail on its fix: Charoosaei et al., arXiv 2610.11835, a preprint; Mathews & Nagappan, arXiv 2412.14137.
- Agents add mocks to tests more often than people do: Hora & Robbes, MSR 2026, arXiv 2602.00409.
- Coverage barely predicts fault detection: Inozemtseva & Holmes, ICSE 2014. Surviving mutants do predict real faults: Just et al., FSE 2014; Google's mutation testing in code review, arXiv 2102.11378.
The worker reads no principle unless something names it, under `worker: a chunk's worker writes the simplest change…`. So the rule on expected values, the costliest of these mistakes, also stands in `CLAUDE.md` → "Working loop", "Implement the simplest change that solves the item, and ship it with `/larv-review`", and the stability reviewer holds a change to the whole bullet.
The decision `worker: a chunk's worker writes the simplest change…` has the worker read a principle only where something names it. A reviewer cannot tell from a literal in a finished test whether it was copied from the code's output, as no reviewer can make a test come first. So the rule on expected values also stands in `CLAUDE.md` → "Working loop", "Implement the simplest change that solves the item, and ship it with `/larv-review`", and the stability reviewer holds a change to the whole bullet.
Valid while the worker writes the tests that larv-review's reviewers hold to the principles.
+5 -5
View File
@@ -40,15 +40,15 @@
- DTOs, enums, and value objects are pure data — no logic. They stand in for the structs/records a language lacks.
* Always use explicit state machines.
* Most, if not all, code should have automated tests, and each test earns its keep.
- A valuable test checks behaviour through the unit's interface, fails when that behaviour breaks, and passes through a refactor that keeps it.
- A valuable test checks behaviour through the unit's interface, fails when that behaviour breaks, and still passes after a refactor that keeps that behaviour.
- Write the test first and watch it fail on its assertion, not on an error, before implementing.
- Take expected values from the requirement — the item, ticket, spec or a hand-worked example — never from running the code or calling its helpers. A test that records today's output locks in today's bug. BANNED: `assert text.endswith(applications_text(applied))`. GOOD: the expected text written out for one small input.
- Assert the outcome — returned values, stored state, emitted output — not how the code reached it. Use real collaborators or fakes; mock only slow, nondeterministic or external edges. BANNED: asserting which internal function ran, or scanning the source with `ast` or a regex. GOOD: a fake clock, a fake HTTP transport.
- Take expected values from the requirement — the item, ticket, spec or a hand-worked example — never from running the code or calling its helpers. A test that records today's output locks in today's bug. BANNED: `assert text.endswith(applications_text(applied))`, where `applications_text` is the code's own helper. GOOD: the expected text written out for one small input.
- Assert the outcome — returned values, stored state, emitted output — not how the code reached it. Use real collaborators or fakes; mock only slow, nondeterministic or external edges. BANNED: asserting which internal function ran, or scanning the source with `ast` or a regex. GOOD: `assertEqual(saved_row, {...})` against a fake store; a fake clock.
- Pin wording only where the wording is the contract: a message a consumer parses, a published format. For prompts, help and error text, assert the facts the text must carry — an item ID, a model argument, an error code — not the sentence. BANNED: `assertIn("<a 600-character prompt sentence>", prompt)`. GOOD: `assertIn("model opus, effort high", prompt)`.
- One behaviour per test, named for it, tested once, at the lowest level that reproduces it. No loops or branches in a test body; parametrize the cases.
- Every escaped bug becomes a test that fails without its fix.
- Skip a test that no real bug could fail: a constant's value, a list's length, the framework's own behaviour.
- Tests take time, randomness and I/O through the same injected seams: no sleeps, no wall-clock asserts, seeded randomness, fixtures built per test.
- Write no test that no real bug could fail: a constant's value, a list's length, the framework's own behaviour.
- No sleeps or wall-clock asserts in a test, and seeded randomness.
- For pure logic over many inputs, prefer a property test: a round trip, an invariant.
- Changing tests and changing other code are two separate processes — be very careful about doing both at once.
- Use properly randomised UUID v7s — never `aaaa-0000`-style placeholders.
+1 -1
View File
@@ -233,7 +233,7 @@ Found by the goals audit of PR #245. `skills/larv-review/SKILL.md` §3 → "Secu
### 165. Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant.
A test audit across lilleman's projects on 2026-10-16 found 10 of 33 sampled tests in this repo pinning wording that is not the contract. Examples: `bin/loop_claude/test_loop_claude.py:654`, `:661`, `:667`, `:677`, `:689` and `:696` each assert prompt sentences of several hundred characters, and `:859` asserts help-text wording. `:869` and `:918` parse the source with `ast` and list each `Stop(...)` and `Failure(...)` call by function name. `bin/model_routing/test_ladder.py:13` asserts `len(LADDER) == 11`. Rewrite each to the technical principles' test bullet: assert the facts the text must carry, such as an item ID, a rung or a model argument, and delete a test no real bug could fail. Keep the mutation run on the changed lines green, so no rewrite loses a behaviour the old test held. `bin/loop_claude/cycle/prompts.py`'s `off_the_projects_rung` is a place the comprehension seats found hard; its tests are among those rewritten.
A test audit across lilleman's projects on 2026-10-16 found 10 of 33 sampled tests in this repo pinning wording that is not the contract. Examples of these and other weak tests: `bin/loop_claude/test_loop_claude.py:165`, `:271` and `:664` to `:687` assert prompt sentences of 340 to 775 characters, and `:864` asserts help-text wording. `:906` and `:936` parse the source with `ast` and list each `Stop(...)` and `Failure(...)` call by function name. `bin/model_routing/test_ladder.py:16` asserts `len(LADDER) == 11`. Rewrite each to the technical principles' test bullet: assert the facts the text must carry, such as an item ID, a rung or a model argument, and delete a test no real bug could fail. Before and after rewriting a test, run the mutation tool on the code it covers: every mutant the old test killed, the new one kills.
### 162. Script each step that 0.2.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.