Hold each test to earning its keep, and take a test's expected values from its item
This commit is contained in:
@@ -35,6 +35,7 @@ Decisions live in `docs/decisions.md`:
|
||||
- rule-hook: a PreToolUse hook enforces each rule a program can check, a `# rule-override <rule>: <reason>` marker overrides one, and larv-review's overrides pass rules on each override before merge
|
||||
- rules/: `CLAUDE.md` holds the worker's rules that no file path can scope, and each topic's rules for the worker live in a `rules/` file that Claude Code loads by its `paths:` frontmatter
|
||||
- worker: a chunk's worker writes the simplest change that solves its item, test first wherever a test can check the change, and reads a technical principle only where the item, a finding or its own record names it; larv-review's reviewers hold the change to the principles and to the standards in `skills/larv-review/standards/`
|
||||
- tests: each test earns its keep as the technical principles' test bullet says, and the worker takes a test's expected values from its item, never from the code
|
||||
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
|
||||
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
|
||||
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model
|
||||
|
||||
@@ -14,7 +14,7 @@ How work proceeds in any repo, unless the repo's `AGENTS.md` says otherwise. The
|
||||
- **Work one chunk at a time, and make it as large as one purpose allows.** Every review has a fixed cost, so a chunk takes the next plan item plus every other item for which four things hold: one PR title states what they achieve together, without listing them; they name one release; they fire the same review passes (`larv-review` §3); and no item outside the chunk blocks one of them. The next plan item is the first that does not wait: an item waits while it is marked `question`, until the human answers, or while an item that blocks it is still in the plan. Tiny items, a few lines of change each, need neither a shared purpose nor the same passes: bundle up to ten that name one release, and that no item outside the chunk blocks, into one chunk. Split a milestone into items in `todo.md` before starting it. Fix a finding in the chunk's own change within the chunk. File a finding about anything else in `todo.md`.
|
||||
- **Ask, don't guess.** First check whether the goals already answer the question. If they nearly do, propose a change to the goals and show how it answers this case: an answer settles one question, a goal settles every question like it. Write every answer once, where it belongs, never only in chat: a goal in `README.md`, a working rule in `AGENTS.md`, a standing choice in a comment at the one place it governs, or in `docs/decisions.md` once made in more than one place. When an answer holds a goal and a choice, write the goal in the README, and record as a choice only what the goal leaves open.
|
||||
- **Bench a syntax with the people who write it.** Before a new or changed spelling of something users read or write ships — markdown, HTML, a config key, a CLI flag — a panel of fresh-context readers, seated as those users from the README's `## Audience`, picks it.
|
||||
- **Implement the simplest change that solves the item, and ship it with `/larv-review`.** Whenever a test can check the change, write that test first and see it fail. larv-review's reviewers hold the change to the technical principles, and a finding against one names it. Read a file under `~/.claude/principles/` only when the item changes it, a finding or a record you write names it, or you decline a finding.
|
||||
- **Implement the simplest change that solves the item, and ship it with `/larv-review`.** Whenever a test can check the change, write that test first, taking expected values from the item, never from the code, and see it fail. larv-review's reviewers hold the change to the technical principles, and a finding against one names it. Read a file under `~/.claude/principles/` only when the item changes it, a finding or a record you write names it, or you decline a finding.
|
||||
- **Remove a finished item's row and detail from `todo.md`, and revise any decision that cites it.** If the repo publishes something others consume — a package, an image, an API — reword the item for them and move it to `CHANGELOG.md` under `## Unreleased`. A changelog is for the public, never for the next agent. Otherwise delete it; git history is the record.
|
||||
- **Releases are the human's.** The version bump and the publishing credentials are theirs, whatever another rule says. Prepare everything else, so cutting a release is one decision.
|
||||
- **Open each `todo.md` item or ticket with a bold one-sentence tagline**, so the taglines alone say what a release holds. The tagline states the target state, checkable when done: an imperative verb plus what holds afterwards. A present-tense fact reads as the current state and says nothing about what to do. Add "today X" only where the direction is otherwise ambiguous. GOOD: `**Accept undefined in MetadataValue and drop such keys on output.**` BANNED: `MetadataValue accepts undefined; such keys are dropped on output`.
|
||||
|
||||
@@ -123,8 +123,7 @@ Every PR gets at least this sweep.
|
||||
- Does a mutant of a changed line survive (First Steps, step 5)?
|
||||
- Was a test deleted while the behaviour it pinned remains? Flag it; restore the test.
|
||||
- Were tests and implementation changed together in a way that could mask a regression — e.g. an assertion weakened to make a test pass? Flag it; these should be separate changes.
|
||||
- Would each test fail if the behavior regressed? Watch for assertion-free tests, tautological asserts, and mocks so broad the test exercises the mock instead of the code.
|
||||
- Tests are code too: flag overlapping tests that pin the same behavior twice, and tests of framework behavior or trivial pass-throughs — maintenance cost without protection.
|
||||
- Does each test earn its keep, as the technical principles' test bullet says? Look at where its expected values come from, what it asserts, what wording it pins, whether another test already pins the behaviour, and whether it sleeps or reads the clock.
|
||||
- Are there untestable side effects that should be refactored?
|
||||
|
||||
|
||||
|
||||
@@ -196,6 +196,22 @@ The item also cut the rule that a session proposes new rules for `CLAUDE.md` and
|
||||
|
||||
Valid while review rounds per pass do not rise. A principle found late, such as functional core, can cost a rewrite within a pass's three rounds. `todo.md` item 105 compares rounds per pass and reading cost in loop-claude's archive for the five chunks before and after this one.
|
||||
|
||||
## tests: each test earns its keep as the technical principles' test bullet says, and the worker takes a test's expected values from its item, never from the code
|
||||
|
||||
2026-10-16, lilleman asked for research on what makes tests valuable, then for these changes; claude drafted them. Serves goal 1.1, since a test that cannot catch a bug, or breaks without one, lowers quality and costs review rounds.
|
||||
|
||||
An audit of 196 tests sampled across eight projects found about 61% clearly valuable. Of the rest, 15% pinned wording that is not the contract, such as whole prompt sentences; 9% copied or policed the implementation, such as scanning the source with `ast`; 6% ran on real time or heavy resources; and 6% asserted something no bug could break, such as a list's length. The bullet follows these sources:
|
||||
|
||||
- Coverage barely predicts fault detection: Inozemtseva & Holmes, ICSE 2014. Surviving mutants do predict real faults: Just et al., FSE 2014; Google's mutation testing in code review, arXiv 2102.11378.
|
||||
- A valuable test fails when behaviour breaks, passes through a refactor, asserts state over interactions and keeps its expected outcome narrow: *Software Engineering at Google*, chapters 11, 12 and 14; Fowler, "The Practical Test Pyramid"; Beck, "Test Desiderata"; Khorikov, *Unit Testing: Principles, Practices, and Patterns*, chapter 4.
|
||||
- Agents add mocks to tests more often than people do: Hora & Robbes, MSR 2026, arXiv 2602.00409.
|
||||
- Tests whose expected values come from the code's current output pass on a bug and fail on its fix: Charoosaei et al., arXiv 2610.11835, a preprint; Mathews & Nagappan, arXiv 2412.14137.
|
||||
- A test that never failed on its assertion may be tautological: Seemann, blog.ploeh.dk, 2019-10-14.
|
||||
|
||||
The worker reads no principle unless something names it, under `worker: a chunk's worker writes the simplest change…`. So the rule on expected values, the costliest of these mistakes, also stands in `CLAUDE.md` → "Working loop", "Implement the simplest change that solves the item, and ship it with `/larv-review`", and the stability reviewer holds a change to the whole bullet.
|
||||
|
||||
Valid while the worker writes the tests that larv-review's reviewers hold to the principles.
|
||||
|
||||
## review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
|
||||
|
||||
2026-10-13, lilleman through todo item 97, implemented by claude. Serves goals 1.4 and 1.5. Anthropic's Claude Code best practices: "A reviewer prompted to find gaps will usually report some, even when the work is sound… Chasing every finding leads to over-engineering." 320 of this repo's 876 commits answered a review round. So stability and product-owner drop a finding that is none of these, and stability leaves the style of comments to prose, which reads every comment a chunk touches. An architect finding that names no later change it makes harder cannot be checked, so it is dropped. The architecture pass and the comprehension check keep their whole-project view, which guards goal 1.1 as a project grows. Valid while the findings these rules drop keep changing nothing a user would notice.
|
||||
|
||||
@@ -39,12 +39,20 @@
|
||||
- A cast (`x as Foo`) asserts what the compiler could not check and stops it checking — so when the type later changes, nothing tells you. Derive the real type instead: infer it, name the return type, or parse the input once into a typed shape. Where a value genuinely arrives untyped (HTTP body, `JSON.parse`, DB row), the boundary owes a **type guard validating the fields it claims**; past that boundary everything is typed. `as unknown as T` and non-null `!` are the same defect wearing a hat. BANNED: `const user = body as User` — one rename and it is a silent lie. GOOD: `function isUser(v: unknown): v is User` checking each field, then `if (!isUser(body)) return { error: … }`.
|
||||
- DTOs, enums, and value objects are pure data — no logic. They stand in for the structs/records a language lacks.
|
||||
* Always use explicit state machines.
|
||||
* Most, if not all, code should have automated tests.
|
||||
- A valuable test checks behaviour through the unit's interface, and fails when that behaviour breaks.
|
||||
- Write tests first for the behaviour we want, then implement until they pass.
|
||||
* Most, if not all, code should have automated tests, and each test earns its keep.
|
||||
- A valuable test checks behaviour through the unit's interface, fails when that behaviour breaks, and passes through a refactor that keeps it.
|
||||
- Write the test first and watch it fail on its assertion, not on an error, before implementing.
|
||||
- Take expected values from the requirement — the item, ticket, spec or a hand-worked example — never from running the code or calling its helpers. A test that records today's output locks in today's bug. BANNED: `assert text.endswith(applications_text(applied))`. GOOD: the expected text written out for one small input.
|
||||
- Assert the outcome — returned values, stored state, emitted output — not how the code reached it. Use real collaborators or fakes; mock only slow, nondeterministic or external edges. BANNED: asserting which internal function ran, or scanning the source with `ast` or a regex. GOOD: a fake clock, a fake HTTP transport.
|
||||
- Pin wording only where the wording is the contract: a message a consumer parses, a published format. For prompts, help and error text, assert the facts the text must carry — an item ID, a model argument, an error code — not the sentence. BANNED: `assertIn("<a 600-character prompt sentence>", prompt)`. GOOD: `assertIn("model opus, effort high", prompt)`.
|
||||
- One behaviour per test, named for it, tested once, at the lowest level that reproduces it. No loops or branches in a test body; parametrize the cases.
|
||||
- Every escaped bug becomes a test that fails without its fix.
|
||||
- Skip a test that no real bug could fail: a constant's value, a list's length, the framework's own behaviour.
|
||||
- Tests take time, randomness and I/O through the same injected seams: no sleeps, no wall-clock asserts, seeded randomness, fixtures built per test.
|
||||
- For pure logic over many inputs, prefer a property test: a round trip, an invariant.
|
||||
- Changing tests and changing other code are two separate processes — be very careful about doing both at once.
|
||||
- Use properly randomised UUID v7s — never `aaaa-0000`-style placeholders.
|
||||
* Every mutant of a changed line fails a test, unless the mutant leaves behaviour unchanged. A mutant is a small change to the code, such as a flipped comparison. One that no test notices marks behaviour the tests would let break. The stability pass checks this with a mutation run on the chunk's changed lines.
|
||||
* Every mutant of a changed line fails a test, unless the mutant leaves behaviour unchanged. A mutant is a small change to the code, such as a flipped comparison. One that no test notices marks behaviour the tests would let break. The stability pass checks this with a mutation run on the chunk's changed lines. Coverage is no target: it finds untested code, not weak tests.
|
||||
* Idempotent data writing.
|
||||
- Typical example: `PUT /resource/{id}` instead of `POST /resource` + Idempotent-Id header.
|
||||
* Handle errors explicitly, as values — avoid the exception path.
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | R | S | C | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| 164 | 0.2.0 | | | **Rewrite the technical principles' test bullet so each test earns its keep, tell the worker to take expected values from the item, and point stability-reviewer's §11 at that bullet.** | 1 | 2 | 3 | 1.1 by 6 | 10.9 |
|
||||
| 163 | 0.2.0 | | | **Have `rule-hook` block an Agent call that launches a reviewer, a comprehension seat or the security pass's subagent below the rung loop-claude names for them, so a worker below the project's rung cannot hand a review to its own cheaper model.** | 2 | 3 | 3 | 1.1 by 5 | 6.8 |
|
||||
| 146 | 0.2.0 | | decision | **Score R as L + E + I − 2 from anchored levels, at least 7 at I5 and 5 at I4, with L, E and I in their own columns and `rank_todo.py` computing R, drop C, and route a task's model and effort on its R alone.** | 6 | 7 | 7 | 1.1 by 8, 1.2 by 8, 1.4 by 8 | 2.8 |
|
||||
| 152 | 0.2.0 | 146 | decision | **Anchor each score level in `rules/todo-items.md` with words, and each project's with two or three items it shipped, and have a scorer compare a new item against them.** | 2 | 4 | 3 | 1.1 by 7 | 6.4 |
|
||||
@@ -41,7 +40,7 @@
|
||||
| 154 | 0.3.0 | | defect | **Have `bin/shell_command.py`'s `segments` keep a command's words together across a redirect, so `git > out commit` reads as one git commit.** | 2 | 2 | 3 | 1.1 by 6 | 10.9 |
|
||||
| 157 | 0.3.0 | | | **Split `docs/decisions.md` → `todo.md: one table of items …`'s 2026-10-02 paragraph into one paragraph per choice.** | 1 | 2 | 1 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 106 | 0.3.0 | | | **Make larv-review's "Do not over-read a clean result" cite a rule that says Renovate or Dependabot owns dependency CVEs, or drop the citation.** | 1 | 1 | 1 | 1.1 by 5 | 10.9 |
|
||||
| 165 | 0.3.0 | 164 | | **Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant.** | 2 | 3 | 2 | 1.1 by 4, 1.4 by 3 | 7.5 |
|
||||
| 165 | 0.3.0 | | | **Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant.** | 2 | 3 | 2 | 1.1 by 4, 1.4 by 3 | 7.5 |
|
||||
| 162 | 0.3.0 | | owner | **Script each step that 0.2.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 3 | 1.4 by 4, 1.5 by 4 | 6.6 |
|
||||
| 139 | 0.4.0 | | | **Split `bin/routing_figures/measure.py` by concern, and keep the figures' measured cycle and each raised call as fields loop-claude reads back, not as the prose it renders.** | 2 | 3 | 3 | 1.1 by 5, 1.4 by 5 | 10.3 |
|
||||
| 158 | 0.4.0 | | decision | **Measure the fixed cost of a chunk from loop-claude's archive, in tokens and hours against its items' S, and set F in `Score = 100 * V / (2**S + F)` to it.** | 1 | 3 | 4 | 1.1 by 5, 1.4 by 5 | 10.3 |
|
||||
@@ -81,33 +80,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 164. Rewrite the technical principles' test bullet so each test earns its keep, tell the worker to take expected values from the item, and point stability-reviewer's §11 at that bullet.
|
||||
|
||||
lilleman asked, on 2026-10-16, for research on what makes tests valuable. An audit of 196 tests sampled across eight projects found about 61% clearly valuable. The rest: 15% pinned wording that is not the contract, 9% copied or policed the implementation, 6% ran on real time or heavy resources, 6% asserted something no bug could break. The research behind the bullet:
|
||||
|
||||
- Coverage barely predicts fault detection: Inozemtseva & Holmes, ICSE 2014.
|
||||
- Surviving mutants do predict real faults: Just et al., FSE 2014. Google's mutation testing in code review: arXiv 2102.11378.
|
||||
- What a valuable test is: *Software Engineering at Google*, chapters 11, 12 and 14; Fowler, "The Practical Test Pyramid"; Beck, "Test Desiderata"; Khorikov, *Unit Testing: Principles, Practices, and Patterns*, chapter 4.
|
||||
- Agents over-mock: Hora & Robbes, MSR 2026, arXiv 2602.00409.
|
||||
- LLM-written tests lock in bugs when their expected values come from the code's output: Charoosaei et al., arXiv 2610.11835 (a preprint), and Mathews & Nagappan, arXiv 2412.14137.
|
||||
- Watch a test fail on its assertion: Seemann, blog.ploeh.dk, 2019-10-14.
|
||||
|
||||
1. In `principles/technical-principles.md`, replace the "automated tests" bullet's lines on what a test checks and when to write it with the draft below. Keep its lines on changing tests apart from other code and on UUID v7s. Add to the mutation bullet: "Coverage is no target: it finds untested code, not weak tests."
|
||||
2. In `CLAUDE.md` → "Implement the simplest change…", add "taking expected values from the item, never from the code" to the test-first sentence. The worker reads no principle unless one is named, and expected values taken from the code are the costliest mistake agents make in tests.
|
||||
3. Point `agents/stability-reviewer.md` §11 at the bullet in place of its own restatement, as item 67 did for the rest of the principles.
|
||||
|
||||
* Most, if not all, code should have automated tests, and each test earns its keep.
|
||||
- A valuable test checks behaviour through the unit's interface, fails when that behaviour breaks, and passes through a refactor that keeps it.
|
||||
- Write the test first and watch it fail on its assertion, not on an error, before implementing.
|
||||
- Take expected values from the requirement — the item, ticket, spec or a hand-worked example — never from running the code or calling its helpers. A test that records today's output locks in today's bug. BANNED: `assert text.endswith(applications_text(applied))`. GOOD: the expected text written out for one small input.
|
||||
- Assert the outcome — returned values, stored state, emitted output — not how the code reached it. Use real collaborators or fakes; mock only slow, nondeterministic or external edges. BANNED: asserting which internal function ran, or scanning the source with `ast` or a regex. GOOD: a fake clock, a fake HTTP transport.
|
||||
- Pin wording only where the wording is the contract: a message a consumer parses, a published format. For prompts, help and error text, assert the facts the text must carry — an item ID, a model argument, an error code — not the sentence. BANNED: `assertIn("<a 600-character prompt sentence>", prompt)`. GOOD: `assertIn("model opus, effort high", prompt)`.
|
||||
- One behaviour per test, named for it, tested once, at the lowest level that reproduces it. No loops or branches in a test body; parametrize the cases.
|
||||
- Every escaped bug becomes a test that fails without its fix.
|
||||
- Skip a test that no real bug could fail: a constant's value, a list's length, the framework's own behaviour.
|
||||
- Tests take time, randomness and I/O through the same injected seams: no sleeps, no wall-clock asserts, seeded randomness, fixtures built per test.
|
||||
- For pure logic over many inputs, prefer a property test: a round trip, an invariant.
|
||||
|
||||
### 163. Have `rule-hook` block an Agent call that launches a reviewer, a comprehension seat or the security pass's subagent below the rung loop-claude names for them, so a worker below the project's rung cannot hand a review to its own cheaper model.
|
||||
|
||||
`docs/decisions.md` → `routing: a chunk's worker runs on the rung its item picks…` keeps every reviewer, comprehension seat and the security pass's subagent at or above the project's rung. Only loop-claude's prompt holds that today. No agent under `agents/` sets `model:`, so an Agent call that names no model runs on the caller's: a Haiku worker that leaves out `model` once gets a Haiku review, and nothing reports it. On 2026-10-16, item 67's chunk passed `model=opus, effort=medium` on every such call, so no review has run low yet.
|
||||
@@ -261,7 +233,7 @@ Found by the goals audit of PR #245. `skills/larv-review/SKILL.md` §3 → "Secu
|
||||
|
||||
### 165. Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant.
|
||||
|
||||
The 2026-10-16 test audit behind item 164 found 10 of 33 sampled tests in this repo pinning wording that is not the contract. Examples: `bin/loop_claude/test_loop_claude.py:654`, `:661`, `:667`, `:677`, `:689` and `:696` each assert prompt sentences of several hundred characters, and `:859` asserts help-text wording. `:869` and `:918` parse the source with `ast` and list each `Stop(...)` and `Failure(...)` call by function name. `bin/model_routing/test_ladder.py:13` asserts `len(LADDER) == 11`. Rewrite each to the bullet item 164 writes: assert the facts the text must carry, such as an item ID, a rung or a model argument, and delete a test no real bug could fail. Keep the mutation run on the changed lines green, so no rewrite loses a behaviour the old test held. `bin/loop_claude/cycle/prompts.py`'s `off_the_projects_rung` is a place the comprehension seats found hard; its tests are among those rewritten.
|
||||
A test audit across lilleman's projects on 2026-10-16 found 10 of 33 sampled tests in this repo pinning wording that is not the contract. Examples: `bin/loop_claude/test_loop_claude.py:654`, `:661`, `:667`, `:677`, `:689` and `:696` each assert prompt sentences of several hundred characters, and `:859` asserts help-text wording. `:869` and `:918` parse the source with `ast` and list each `Stop(...)` and `Failure(...)` call by function name. `bin/model_routing/test_ladder.py:13` asserts `len(LADDER) == 11`. Rewrite each to the technical principles' test bullet: assert the facts the text must carry, such as an item ID, a rung or a model argument, and delete a test no real bug could fail. Keep the mutation run on the changed lines green, so no rewrite loses a behaviour the old test held. `bin/loop_claude/cycle/prompts.py`'s `off_the_projects_rung` is a place the comprehension seats found hard; its tests are among those rewritten.
|
||||
|
||||
### 162. Script each step that 0.2.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user