Answer prose round 1: 0 🔴, 6 🟡 (0 misread); define a yardstick in the auditor, name the one auditor plainly in the decision's title, split and correct the trial paragraph, explain the old pass in the changelog, and name the split in item 242's tagline

This commit is contained in:
2026-10-28 10:16:21 +01:00
parent 3af346ef75
commit c31f367771
5 changed files with 12 additions and 10 deletions
+1 -1
View File
@@ -35,7 +35,7 @@ Decisions live in `docs/decisions.md`:
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
- review: the comprehension check records how hard a project is to read and blocks no merge: a paired ruling against the last measured commit where the architecture pass ran, and the four-seat scoring run, once when an item names the floor and twice on one head when a release's items run out
- review: the premises chunk's whole review is the human's approval
- review: the principles audit runs on every chunk that changes code, config or tests, reads the project whole with one auditor for every yardstick, and runs on no other chunk
- review: the principles audit runs on every chunk that changes code, config or tests, reads the project whole with one auditor checking the goals and every principles file that applies, and runs on no other chunk
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
- routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass; the seats and the security pass always on the project's
- routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and only where the E + I steps no review up and the worker runs no higher than the project's rung, then even below the worker's; no pass has such a rung
+1 -1
View File
@@ -2,7 +2,7 @@
## Unreleased
- larv-review's principles pass now launches one auditor per round, which checks the README's goals, code health and every principles file that applies, in place of one auditor for each. It reads the project once per round, where it used to read it once per yardstick. Its report is `principles-round-<n>.md`, like every other pass's.
- larv-review's principles pass now launches one auditor per round, which checks the README's goals, code health and every principles file that applies, so it reads the project once per round. It used to launch one auditor for the goals with code health and one for each principles file, each reading the whole project. Its report is `principles-round-<n>.md`, like every other pass's.
- A session running larv-review now enters the chunk's worktree with `EnterWorktree` before the first review pass, so `/security-review` and the full scan read the branch's diff. Run from the checkout root, they used to find it empty.
- `loop-claude` now stops with exit 2 when the judge rules one chunk unfinished 6 cycles running, so a worker and a judge that disagree can no longer cycle unattended without end. Rerunning resumes the chunk and gives it 6 more cycles.
- larv-review now launches a round's reviewers in up to three waves, each wave's reviewers in one message, in place of one reviewer at a time: first the architect, then the principles auditor with the security and overrides passes, then the stability, product-owner and prose passes. Each wave now lasts as long as its slowest reviewer, where a round used to last as long as all its reviewers one after another.
+2 -2
View File
@@ -4,11 +4,11 @@ description: "Use this agent for larv-review's principles pass: the project's go
tools: Read, Grep, Glob, Write
---
You check every yardstick you are handed against the whole project, which you read once for all of them: every goal under the project's `README.md` → `## Goals`, with code health (`~/.claude/CLAUDE.md` → "Shared goal") unless the README voids it, and every principle in each principles file.
You check every yardstick you are handed against the whole project, which you read once for all of them. A yardstick is either the project's goals, every goal under its `README.md` → `## Goals` with code health (`~/.claude/CLAUDE.md` → "Shared goal") unless the README voids it, or one principles file, every principle in it.
Verdict, severities and rounds: ~/.claude/skills/larv-review/review-findings.md. Read the yardsticks, `README.md`, `docs/decisions.md` and `todo.md` first, then all of the project's code, config, tests and CI.
- Per principle or goal: every place the project departs from it, with file:line, the defect in one sentence and its concrete consequence. Then one line: followed, mostly, not followed, or does not apply and why.
- A departure a `todo.md` item plans away is named with that item, not raised.
- A decision in `docs/decisions.md` working against a goal or a technical principle is a 🔴 on the decision, unless it names the `todo.md` item that ends it. Weigh it hard: that it was decided is no reason.
- Report under one heading per yardstick, so each finding names the goal or principle it breaks.
- Report under one heading per yardstick.
+5 -3
View File
@@ -90,7 +90,7 @@ The trials: four stability rounds and five principles rounds archived on 2026-10
- Code-intelligence tooling: the `LSP` tool, through a pyright plugin built from the marketplace manifest with `pyright-langserver` on PATH, answers a file's symbols, a definition or the references in a few hundred characters. Under a rule telling them to use it, the stability reviewers and the workers never called it; the auditors did, read as much or more, and missed the one 🔴, 0 of 1 against the baseline's 1 of 1. Each machine would also need a server per language (goal 2). Not worth an item.
- A cap on tool results. Claude Code caps a Bash result at about 30k characters already. A PostToolUse hook capping it at 12k, which `updatedToolOutput` allows, left the stability reviewers 3 of 6 findings, two lost in one round. Read results, 59.9M characters in subagents' threads with 58% of their characters in results over 12k, are out of such a hook's reach. Not worth an item.
- A scout: Haiku 5.5 reading raw content, whole files or long output, and returning facts as path:line and the ranges worth reading, so the agent's own context holds the summary and the chosen lines. In session, as an agent under `agents/`, its first request writes 37k tokens, $0.005, and a read of a 38KB file costs $0.008 in 8 seconds. As a slim headless `claude -p` with its own system prompt, no setting sources and no dynamic sections, which keeps the subscription login, it starts at 6k tokens and $0.0003 in 3 to 5 seconds, its prefix cached across calls, and the same read costs $0.004 in 7 seconds; a direct API call and `claude --bare` need an API key, so they are out. Stability reviewers with the agent scout found 5 of 6 findings and read 29% fewer characters at 6% less cost and 14% more time; with the CLI scout 4 of 6, 38% fewer, 17% less and 39% more; the auditors with the agent scout 3 of 6, the 🔴 among the lost, at the same cost and twice the time. Workers offered the scout never called it.
- What a reviewer reads: the auditors read the project whole by design. Of their 🔴 and 🟡 findings since the reset, 47% named a file the chunk edited and 51% another file, so reading only the change would drop half of them (`review: the principles audit runs on every chunk that changes code, config or tests, reads the project whole with one auditor for every yardstick, and runs on no other chunk`). One auditor now reads the project once per round for every yardstick, as that decision records.
- What a reviewer reads: the auditors read the project whole by design. Of their 🔴 and 🟡 findings since the reset, 47% named a file the chunk edited and 51% another file, so reading only the change would drop half of them (`review: the principles audit runs on every chunk that changes code, config or tests, reads the project whole with one auditor checking the goals and every principles file that applies, and runs on no other chunk`).
- The base context: every thread starts with `~/.claude/CLAUDE.md` (19k characters), `~/.claude/AGENTS.md` (11k, this repo's decisions, which Claude Code loads as user instructions into every project's sessions; `todo.md` item 52), the project's `AGENTS.md`, and, for the worker and general-purpose subagents, the skill listing (18k) and the agent listing (12k). Subagents' first messages wrote $97 of it. Trimming the listings would save about $20, 2%, and is not worth an item: `disableBundledSkills` would also drop `/security-review`, the security pass of record.
Quality. 19 chunks compacted between 2026-10-10 and 2026-10-22, at the compaction points Claude Code set then, 214k to 972k tokens. Per 100 worker messages they took a median 3.6 review rounds past each pass's first against 4.0 for the 68 uncompacted chunks of 100 messages or more, and 13.9 🔴 and 🟡 findings against 16.6. 4 of 19 had an unfinished verdict against 3 of 68, each 8 to 10 hours after the compaction and followed by a finished verdict within minutes; in those four verdicts the judge quoted the worker's own passages and the human's typed answers, never a summary. One cause was possible and is closed here: Claude Code writes the summary as a user entry, which `bin/claude_code/entries.py` read as the human's words, so the judge would see a `[human]` line of thousands of words and `/talk` would count it as typed; it now reads as Claude Code's. Whether compaction raises unfinished verdicts stays open until `todo.md` item 235 measures it. Two summaries read whole kept the chunk's intent, files, errors and next step in 12k to 20k characters, and the worker resumed without re-reading.
@@ -223,13 +223,15 @@ The base is the last measured commit: the default branch's newest merged change
2026-09-20, lilleman. Serves goal 1.1. A reviewer's finding on that chunk could only be declined by citing the goals the chunk exists to write. Valid while every decline must name a goal or a technical principle.
## review: the principles audit runs on every chunk that changes code, config or tests, reads the project whole with one auditor for every yardstick, and runs on no other chunk
## review: the principles audit runs on every chunk that changes code, config or tests, reads the project whole with one auditor checking the goals and every principles file that applies, and runs on no other chunk
2026-10-16, lilleman, answering the question on the audit's trigger that the plan held. Serves goal 1.1: a principle that no chunk checks drifts, as the stability reviewer's own copy of the principles drifted from them, until it applied `principles/technical-principles.md` as written. It costs goal 1.4: a chunk that changes code, config or tests launches an auditor, and a code fix re-enters it. The human accepted that cost. The cost shows in tokens per finished chunk and in review rounds per chunk. Valid while the audit's findings are drift a chunk would otherwise merge, not the same whole-project findings repeated chunk after chunk: a repeat is a question on the audit's scope for the human.
2026-10-27, lilleman, answering the question PR #376 raised: the audit keeps reading the project whole. Since 2026-10-21, 342 auditor runs cost $274 at list prices, and of their 176 🔴 and 🟡 findings that named a file, 90 named one the chunk did not edit, which reading only the change would drop; lilleman keeps them.
2026-10-28, claude, under the plan's item on the principles pass's cost: one auditor per round checks the README's goals, code health and every applicable principles file, reading the project once, in place of one auditor per yardstick, 3.8 auditor runs per chunk before. Serves goal 1.4. A trial checked that it keeps goal 1.1. It took the 6 latest first rounds that launched two or more auditors, at most two per project, from this repo, `log`, `visthus` and `adf-codec`, 15 auditors whose reports held 40 🔴 and 🟡 findings. Each round was replayed on the project's rung as `review-replay` replays it, once with its separate auditors and once with one auditor given every yardstick, the new definition handed through `--agents`. The separate auditors found 10 of the 40 again, and the one auditor 12, among them 8 of those 10. The trial of 2026-10-27 above found that a reviewer replayed finds about 4 of its own 6 findings, so a variant finding 8 of 10 does what the same reviewers do twice. The two it missed, a code-health 🟡 and a technical-principles 🟡, were both on an untested or weakly tested line, and it found 4 findings the separate auditors missed. So no yardstick keeps an auditor of its own. Valid until the auditor's definition changes or a new model joins the ladder: replay the pass then, as this trial did.
2026-10-28, claude, through todo item 233. Serves goal 1.4. One auditor per round checks the README's goals with code health, and each principles file that applies. It reads the project once, where one auditor per goal set or file ran 3.8 times per chunk before.
A trial checked that it keeps goal 1.1. It replayed the 6 latest first rounds that launched two or more auditors, at most two per project, from this repo, `log`, `visthus` and `adf-codec`: 15 auditors, whose reports held 40 🔴 and 🟡 findings. Each round ran on the project's rung as `review-replay` replays it, once with its separate auditors and once with one auditor handed everything they checked, its new definition passed through `--agents`. A replayed reviewer finds a different part of its findings each time: the separate auditors found only 10 of their own 40 again. The one auditor found 12, among them 8 of those 10. So it finds as much as the separate auditors do. It missed two 🟡s on a missing or weak test, and found 4 findings they missed. No principles file keeps an auditor of its own. Valid until the auditor's definition changes or a new model joins the ladder: replay the pass then, as this trial did.
## review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
+3 -3
View File
@@ -28,7 +28,7 @@
| 158 | 0.6.0 | | decision | **Measure the fixed cost of a chunk from loop-claude's archive, in tokens and hours against its items' S, and set F in `Score = 100 * V / (2**S + F)` to it.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 5, 1.4 by 5 | 10.3 |
| 215 | 0.6.0 | | | **Decode a reload's handed JSON with a decoder for `LoopRun` and its `Route`, in place of `decoded_as`'s generic one with its two sentinels and `Route.fits` hook.** | 2 | 1 | 3 | 4 | 3 | code health by 2 | 9.0 |
| 237 | 0.6.0 | | principle | **Give `judge` and `judge_once` in `bin/loop_claude/cycle/judge.py` a record of the inputs they share, in place of nine and eight parameters, or record why they stay.** | 2 | 1 | 2 | 3 | 3 | code health by 2 | 9.0 |
| 242 | 0.6.0 | | principle | **Run each case of a table-driven test in its own `subTest`, so one failing case does not hide the others and the failure names its case.** | 2 | 1 | 1 | 2 | 3 | code health by 2 | 9.0 |
| 242 | 0.6.0 | | principle | **Run each case of a table-driven test in its own `subTest`, and split each test that checks several behaviours, so one failure does not hide the others and names its case.** | 2 | 1 | 1 | 2 | 3 | code health by 2 | 9.0 |
| 49 | 0.6.0 | | principle | **Check that `AGENTS.md` lists exactly the titles of `docs/decisions.md`.** | 1 | 1 | 1 | 1 | 2 | 1.1 by 6 | 8.4 |
| 105 | 0.6.0 | | decision | **Compare review rounds per pass and reading cost for the five chunks before and after the `worker:` decision moved the principles out of the worker's context, and file an item if either rose.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 6 | 8.0 |
| 162 | 0.6.0 | | owner | **Script each step that 0.2.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 2 | 2 | 1 | 3 | 3 | 1.4 by 4, 1.5 by 4 | 7.8 |
@@ -145,9 +145,9 @@ Release 0.3.0's Mid A (PR #324 at c713910) named `bin/loop_claude/loop/reload.py
Found by the principles audit of PR #376. "About seven things per level" in `principles/technical-principles.md`: `judge` takes nine parameters and `judge_once` eight, past the four or five for things that act on each other, with no recorded reason. Both take root, task, premises, records, entries, escalated_at, rung and effects, which one record could carry.
### 242. Run each case of a table-driven test in its own `subTest`, so one failing case does not hide the others and the failure names its case.
### 242. Run each case of a table-driven test in its own `subTest`, and split each test that checks several behaviours, so one failure does not hide the others and names its case.
"Tests" in `principles/technical-principles.md`. The principles audit of PR #380 found test bodies that loop over their cases without `subTest`, some checking several behaviours at once: `bin/judge_replay/test_judge_replay.py:38`, `:139`, `:176`; `bin/subagent_rung/test_subagent_rung.py:32`, `:43`, `:45`; `bin/test_settings_filter.py:28`; `bin/test_statusline.py:102`; `bin/loop_claude/test_loop_claude.py:590`; `bin/machine_load/test_machine_load.py:19`, `:31`; `bin/rule_hook/test_rule_hook.py:480`. The first failing case stops the loop, and without a message the failure doesn't say which case broke. Split the tests that check several behaviours at once, `test_judge_replay.py:176` and `test_subagent_rung.py:43-46`, into one test per behaviour ("One behaviour per test").
"Tests" in `principles/technical-principles.md`. The principles audit of PR #380 found test bodies that loop over their cases without `subTest`, some checking several behaviours at once: `bin/judge_replay/test_judge_replay.py:38`, `:139`, `:176`; `bin/loop_claude/test_loop_claude.py:590`; `bin/machine_load/test_machine_load.py:19`, `:31`; `bin/rule_hook/test_rule_hook.py:480`; `bin/subagent_rung/test_subagent_rung.py:32`, `:43`, `:45`; `bin/test_settings_filter.py:28`; `bin/test_statusline.py:102`. The first failing case stops the loop, and without a message the failure doesn't say which case broke. Split the tests that check several behaviours at once, `test_judge_replay.py:176` and `test_subagent_rung.py:43-46`, into one test per behaviour ("One behaviour per test").
### 49. Check that `AGENTS.md` lists exactly the titles of `docs/decisions.md`.