Answer prose round 2: 0 🔴, 0 🟡; fix item 165's figures, say what a test does with the clock and randomness, and state why a reviewer cannot catch a copied value

This commit is contained in:
2026-10-16 05:47:27 +02:00
parent 5725baefab
commit d1c1e769b3
3 changed files with 3 additions and 3 deletions
+1 -1
View File
@@ -208,7 +208,7 @@ An audit of 196 tests sampled across eight projects found about 61% clearly valu
- Agents add mocks to tests more often than people do: Hora & Robbes, MSR 2026, arXiv 2602.00409.
- Coverage barely predicts fault detection: Inozemtseva & Holmes, ICSE 2014. Surviving mutants do predict real faults: Just et al., FSE 2014; Google's mutation testing in code review, arXiv 2102.11378.
The decision `worker: a chunk's worker writes the simplest change…` has the worker read a principle only where something names it. A reviewer cannot tell from a literal in a finished test whether it was copied from the code's output, as no reviewer can make a test come first. So the rule on expected values also stands in `CLAUDE.md` → "Working loop", "Implement the simplest change that solves the item, and ship it with `/larv-review`", and the stability reviewer holds a change to the whole bullet.
The decision `worker: a chunk's worker writes the simplest change…` has the worker read a principle only where something names it. A reviewer sees only the finished test, so cannot tell whether a literal in it was copied from the code's output. So the rule on expected values also stands in `CLAUDE.md` → "Working loop", "Implement the simplest change that solves the item, and ship it with `/larv-review`", and the stability reviewer holds a change to the whole bullet.
Valid while the worker writes the tests that larv-review's reviewers hold to the principles.
+1 -1
View File
@@ -48,7 +48,7 @@
- One behaviour per test, named for it, tested once, at the lowest level that reproduces it. No loops or branches in a test body; parametrize the cases.
- Every escaped bug becomes a test that fails without its fix.
- Write no test that no real bug could fail: a constant's value, a list's length, the framework's own behaviour.
- No sleeps or wall-clock asserts in a test, and seeded randomness.
- A test neither sleeps nor asserts on the wall clock, and seeds any randomness.
- For pure logic over many inputs, prefer a property test: a round trip, an invariant.
- Changing tests and changing other code are two separate processes — be very careful about doing both at once.
- Use properly randomised UUID v7s — never `aaaa-0000`-style placeholders.
+1 -1
View File
@@ -233,7 +233,7 @@ Found by the goals audit of PR #245. `skills/larv-review/SKILL.md` §3 → "Secu
### 165. Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant.
A test audit across lilleman's projects on 2026-10-16 found 10 of 33 sampled tests in this repo pinning wording that is not the contract. Examples of these and other weak tests: `bin/loop_claude/test_loop_claude.py:165`, `:271` and `:664` to `:687` assert prompt sentences of 340 to 775 characters, and `:864` asserts help-text wording. `:906` and `:936` parse the source with `ast` and list each `Stop(...)` and `Failure(...)` call by function name. `bin/model_routing/test_ladder.py:16` asserts `len(LADDER) == 11`. Rewrite each to the technical principles' test bullet: assert the facts the text must carry, such as an item ID, a rung or a model argument, and delete a test no real bug could fail. Before and after rewriting a test, run the mutation tool on the code it covers: every mutant the old test killed, the new one kills.
A test audit across lilleman's projects on 2026-10-16 found 10 of 33 sampled tests in this repo pinning wording that is not the contract. Examples of these and other weak tests: `bin/loop_claude/test_loop_claude.py:165`, `:271` and most asserts in `:664` to `:687` pin whole sentences of the question, archive or prompt text, up to about 750 characters, and `:864` asserts help-text wording. `:906` and `:936` parse the source with `ast` and list each `Stop(...)` and `Failure(...)` call by function name. `bin/model_routing/test_ladder.py:16` asserts `len(LADDER) == 11`. Rewrite each to the technical principles' test bullet: assert the facts the text must carry, such as an item ID, a rung or a model argument, and delete a test no real bug could fail. Before and after rewriting a test, run the mutation tool on the code it covers: every mutant the old test killed, the new one kills.
### 162. Script each step that 0.2.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.