Run one principles-auditor per round for the goals and every principles file that applies
This commit is contained in:
+2
-1
@@ -2,9 +2,10 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- larv-review's principles pass now launches one auditor per round, which checks the README's goals, code health and every principles file that applies, in place of one auditor for each. It reads the project once per round, where it used to read it once per yardstick. Its report is `principles-round-<n>.md`, like every other pass's.
|
||||
- A session running larv-review now enters the chunk's worktree with `EnterWorktree` before the first review pass, so `/security-review` and the full scan read the branch's diff. Run from the checkout root, they used to find it empty.
|
||||
- `loop-claude` now stops with exit 2 when the judge rules one chunk unfinished 6 cycles running, so a worker and a judge that disagree can no longer cycle unattended without end. Rerunning resumes the chunk and gives it 6 more cycles.
|
||||
- larv-review now launches a round's reviewers in up to three waves, each wave's reviewers in one message, in place of one reviewer at a time: first the architect, then the principles auditors with the security and overrides passes, then the stability, product-owner and prose passes. Each wave now lasts as long as its slowest reviewer, where a round used to last as long as all its reviewers one after another.
|
||||
- larv-review now launches a round's reviewers in up to three waves, each wave's reviewers in one message, in place of one reviewer at a time: first the architect, then the principles auditor with the security and overrides passes, then the stability, product-owner and prose passes. Each wave now lasts as long as its slowest reviewer, where a round used to last as long as all its reviewers one after another.
|
||||
- loop-claude's worker, and every subagent it launches, now run with a 200k auto-compact window, handed as `CLAUDE_CODE_AUTO_COMPACT_WINDOW`: Claude Code compacts their context at about 170k tokens, where it used to let it grow to about 967k. loop-claude's judge no longer reads a compaction summary as words you typed. `CLAUDE.md` has a new section, "Compact instructions", saying what a summary keeps.
|
||||
- On Linux, `run-heavy` now caps the memory of a container its command makes with `docker run` or `docker create`. A container that sets no limit gets a memory limit and a swap limit, both the slot's share: three quarters of the machine's memory, split across the heavy slots. A container that sets one of the two limits keeps it, and gets the other as large. This covers containers a project's own scripts start, which rule-hook can't see. A `docker compose` service, and a docker named by its full path, get no cap. rule-hook's `docker-run-memory` now lets a `docker run` under run-heavy pass.
|
||||
- rule-hook has a new rule, `docker-run-memory`. It blocks a `docker run` or `docker create` that lacks a memory limit or a swap limit, such as `--memory 4g --memory-swap 4g`, sized to what the work needs. `--memory-swap -1`, unlimited swap, counts as none. `CLAUDE.md` asks for the same limits on a container a script or a compose file starts, which rule-hook can't see. rule-hook's `run-heavy` rule also now reads docker's `--tlscacert`, `--tlscert`, `--tlskey` and buildx's `--builder` as options that take a value, and reads `docker container run` as a run: `docker --tlscacert ca.pem build .` was not blocked outside run-heavy before.
|
||||
|
||||
@@ -1,13 +1,14 @@
|
||||
---
|
||||
name: principles-auditor
|
||||
description: "Use this agent for larv-review's principles pass: one yardstick checked against the whole project."
|
||||
description: "Use this agent for larv-review's principles pass: the project's goals and every principles file handed to it, checked against the whole project."
|
||||
tools: Read, Grep, Glob, Write
|
||||
---
|
||||
|
||||
You check one yardstick against the whole project: every principle in the file you are handed, or every goal under the project's `README.md` → `## Goals`, with code health (`~/.claude/CLAUDE.md` → "Shared goal") unless the README voids it.
|
||||
You check every yardstick you are handed against the whole project, which you read once for all of them: every goal under the project's `README.md` → `## Goals`, with code health (`~/.claude/CLAUDE.md` → "Shared goal") unless the README voids it, and every principle in each principles file.
|
||||
|
||||
Verdict, severities and rounds: ~/.claude/skills/larv-review/review-findings.md. Read the yardstick, `README.md`, `docs/decisions.md` and `todo.md` first, then all of the project's code, config, tests and CI.
|
||||
Verdict, severities and rounds: ~/.claude/skills/larv-review/review-findings.md. Read the yardsticks, `README.md`, `docs/decisions.md` and `todo.md` first, then all of the project's code, config, tests and CI.
|
||||
|
||||
- Per principle or goal: every place the project departs from it, with file:line, the defect in one sentence and its concrete consequence. Then one line: followed, mostly, not followed, or does not apply and why.
|
||||
- A departure a `todo.md` item plans away is named with that item, not raised.
|
||||
- A decision in `docs/decisions.md` working against a goal or a technical principle is a 🔴 on the decision, unless it names the `todo.md` item that ends it. Weigh it hard: that it was decided is no reason.
|
||||
- Report under one heading per yardstick, so each finding names the goal or principle it breaks.
|
||||
|
||||
@@ -14,7 +14,7 @@ FINDING = re.compile(r"^[\s#>*+|-]*(?:\d+[.)]\s*)?[*_]*\s*(?:🔴|🟡)")
|
||||
# An absolute path in a prompt, up to the quote, bracket or punctuation that ends it.
|
||||
PATH = re.compile(r"/[^\s`'\"()<>\[\],;]+")
|
||||
LINE_OR_STOP = re.compile(r"(?::\d+(?:-\d+)?)?[.:]?$")
|
||||
# A round's report, as skills/larv-review/SKILL.md → "Reports" names it, such as stability-round-2.md or principles-goals-round-1-rerun.md.
|
||||
# A round's report, as skills/larv-review/SKILL.md → "Reports" names it, such as stability-round-2.md or principles-round-1-rerun.md.
|
||||
ROUND_REPORT = re.compile(r"-round-\d+(?:-rerun)?\.md$")
|
||||
|
||||
|
||||
|
||||
@@ -82,7 +82,7 @@ class ReviewedPath(unittest.TestCase):
|
||||
|
||||
class LostInputs(unittest.TestCase):
|
||||
def test_names_each_file_the_prompt_names_in_the_copy_that_it_lacks_other_than_a_round_report(self):
|
||||
prompt = "Read /c/branch.diff and /c/ls-files.txt:3. Earlier: /c/prose-round-1.md. Write /c/principles-goals-round-2-rerun.md; see /elsewhere/x.diff."
|
||||
prompt = "Read /c/branch.diff and /c/ls-files.txt:3. Earlier: /c/prose-round-1.md. Write /c/principles-round-2-rerun.md; see /elsewhere/x.diff."
|
||||
self.assertEqual(lost_inputs(prompt, "/c", {"ls-files.txt"}.__contains__), ["branch.diff"])
|
||||
self.assertEqual(lost_inputs(prompt, "/c", {"branch.diff", "ls-files.txt"}.__contains__), [])
|
||||
|
||||
|
||||
+4
-2
@@ -90,7 +90,7 @@ The trials: four stability rounds and five principles rounds archived on 2026-10
|
||||
- Code-intelligence tooling: the `LSP` tool, through a pyright plugin built from the marketplace manifest with `pyright-langserver` on PATH, answers a file's symbols, a definition or the references in a few hundred characters. Under a rule telling them to use it, the stability reviewers and the workers never called it; the auditors did, read as much or more, and missed the one 🔴, 0 of 1 against the baseline's 1 of 1. Each machine would also need a server per language (goal 2). Not worth an item.
|
||||
- A cap on tool results. Claude Code caps a Bash result at about 30k characters already. A PostToolUse hook capping it at 12k, which `updatedToolOutput` allows, left the stability reviewers 3 of 6 findings, two lost in one round. Read results, 59.9M characters in subagents' threads with 58% of their characters in results over 12k, are out of such a hook's reach. Not worth an item.
|
||||
- A scout: Haiku 5.5 reading raw content, whole files or long output, and returning facts as path:line and the ranges worth reading, so the agent's own context holds the summary and the chosen lines. In session, as an agent under `agents/`, its first request writes 37k tokens, $0.005, and a read of a 38KB file costs $0.008 in 8 seconds. As a slim headless `claude -p` with its own system prompt, no setting sources and no dynamic sections, which keeps the subscription login, it starts at 6k tokens and $0.0003 in 3 to 5 seconds, its prefix cached across calls, and the same read costs $0.004 in 7 seconds; a direct API call and `claude --bare` need an API key, so they are out. Stability reviewers with the agent scout found 5 of 6 findings and read 29% fewer characters at 6% less cost and 14% more time; with the CLI scout 4 of 6, 38% fewer, 17% less and 39% more; the auditors with the agent scout 3 of 6, the 🔴 among the lost, at the same cost and twice the time. Workers offered the scout never called it.
|
||||
- What a reviewer reads: the auditors read the project whole by design. Of their 🔴 and 🟡 findings since the reset, 47% named a file the chunk edited and 51% another file, so reading only the change would drop half of them (`review: the principles audit runs on every chunk that changes code, config or tests, reads the project whole, and runs on no other chunk`); `todo.md` item 233 has one auditor read the project once per round.
|
||||
- What a reviewer reads: the auditors read the project whole by design. Of their 🔴 and 🟡 findings since the reset, 47% named a file the chunk edited and 51% another file, so reading only the change would drop half of them (`review: the principles audit runs on every chunk that changes code, config or tests, reads the project whole, and runs on no other chunk`). One auditor now reads the project once per round for every yardstick, as that decision records.
|
||||
- The base context: every thread starts with `~/.claude/CLAUDE.md` (19k characters), `~/.claude/AGENTS.md` (11k, this repo's decisions, which Claude Code loads as user instructions into every project's sessions; `todo.md` item 52), the project's `AGENTS.md`, and, for the worker and general-purpose subagents, the skill listing (18k) and the agent listing (12k). Subagents' first messages wrote $97 of it. Trimming the listings would save about $20, 2%, and is not worth an item: `disableBundledSkills` would also drop `/security-review`, the security pass of record.
|
||||
|
||||
Quality. 19 chunks compacted between 2026-10-10 and 2026-10-22, at the compaction points Claude Code set then, 214k to 972k tokens. Per 100 worker messages they took a median 3.6 review rounds past each pass's first against 4.0 for the 68 uncompacted chunks of 100 messages or more, and 13.9 🔴 and 🟡 findings against 16.6. 4 of 19 had an unfinished verdict against 3 of 68, each 8 to 10 hours after the compaction and followed by a finished verdict within minutes; in those four verdicts the judge quoted the worker's own passages and the human's typed answers, never a summary. One cause was possible and is closed here: Claude Code writes the summary as a user entry, which `bin/claude_code/entries.py` read as the human's words, so the judge would see a `[human]` line of thousands of words and `/talk` would count it as typed; it now reads as Claude Code's. Whether compaction raises unfinished verdicts stays open until `todo.md` item 235 measures it. Two summaries read whole kept the chunk's intent, files, errors and next step in 12k to 20k characters, and the worker resumed without re-reading.
|
||||
@@ -225,10 +225,12 @@ The base is the last measured commit: the default branch's newest merged change
|
||||
|
||||
## review: the principles audit runs on every chunk that changes code, config or tests, reads the project whole, and runs on no other chunk
|
||||
|
||||
2026-10-16, lilleman, answering the question on the audit's trigger that the plan held. Serves goal 1.1: a principle that no chunk checks drifts, as the stability reviewer's own copy of the principles drifted from them, until it applied `principles/technical-principles.md` as written. It costs goal 1.4: a chunk that changes code, config or tests launches one auditor for the goals and one per applicable principles file, and a code fix re-enters them. The human accepted that cost. The cost shows in tokens per finished chunk and in review rounds per chunk. Valid while the audit's findings are drift a chunk would otherwise merge, not the same whole-project findings repeated chunk after chunk: a repeat is a question on the audit's scope for the human.
|
||||
2026-10-16, lilleman, answering the question on the audit's trigger that the plan held. Serves goal 1.1: a principle that no chunk checks drifts, as the stability reviewer's own copy of the principles drifted from them, until it applied `principles/technical-principles.md` as written. It costs goal 1.4: a chunk that changes code, config or tests launches an auditor, and a code fix re-enters it. The human accepted that cost. The cost shows in tokens per finished chunk and in review rounds per chunk. Valid while the audit's findings are drift a chunk would otherwise merge, not the same whole-project findings repeated chunk after chunk: a repeat is a question on the audit's scope for the human.
|
||||
|
||||
2026-10-27, lilleman, answering the question PR #376 raised: the audit keeps reading the project whole. Since 2026-10-21, 342 auditor runs cost $274 at list prices, and of their 176 🔴 and 🟡 findings that named a file, 90 named one the chunk did not edit, which reading only the change would drop; lilleman keeps them.
|
||||
|
||||
2026-10-28, claude, under the plan's item on the principles pass's cost: one auditor per round checks the README's goals, code health and every applicable principles file, reading the project once, in place of one auditor per yardstick, 3.8 auditor runs per chunk before. Serves goal 1.4. A trial checked that it keeps goal 1.1. It took the 6 latest first rounds that launched two or more auditors, at most two per project, from this repo, `log`, `visthus` and `adf-codec`, 15 auditors whose reports held 40 🔴 and 🟡 findings. Each round was replayed on the project's rung as `review-replay` replays it, once with its separate auditors and once with one auditor given every yardstick, the new definition handed through `--agents`. The separate auditors found 10 of the 40 again, and the one auditor 12, among them 8 of those 10. The trial of 2026-10-27 above found that a reviewer replayed finds about 4 of its own 6 findings, so a variant finding 8 of 10 does what the same reviewers do twice. The two it missed, a code-health 🟡 and a technical-principles 🟡, were both on an untested or weakly tested line, and it found 4 findings the separate auditors missed. So no yardstick keeps an auditor of its own. Valid while the one auditor's 🔴 and 🟡 findings per round stay at the separate auditors' level: replay the pass if they fall.
|
||||
|
||||
## review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
|
||||
|
||||
2026-10-13, lilleman through todo item 108, implemented by claude. Serves goals 1.1 and 1.4. Coverage counts the lines a test runs, so a test without a meaningful assertion still raises it. A surviving mutant shows a test that would let behaviour break. Every mutant of an untested changed line survives, so mutation testing of the changed lines also shows what a coverage drop on them would. Coverage would also catch a deleted test; the stability reviewer sees one in the diff without it. Measuring coverage beside it would add cost without raising quality, so todo item 69 was dropped and `principles/technical-principles.md` replaced "Test coverage should not decline" with the mutant rule.
|
||||
|
||||
@@ -48,14 +48,14 @@ A comment is prose, not code.
|
||||
**Waves.** Launch a round's passes in up to three waves. Launch each wave's reviewers in one message: Agent calls in one message run together, so a wave lasts as long as its slowest reviewer. Start a wave only once every pass of the waves before it is done ("Round cap" → "Done"): line-level polish on code the architect wants restructured, or on a call the security pass will rewrite, is wasted work. Skip a wave none of whose passes runs.
|
||||
|
||||
1. Architecture.
|
||||
2. Principles, every auditor, with Full scan or Security, and Overrides.
|
||||
2. Principles, with Full scan or Security, and Overrides.
|
||||
3. Stability, Product-owner and Prose.
|
||||
|
||||
Give a reviewer without Bash the branch diff and `git ls-files`'s output in its prompt: some harnesses grant it Read alone.
|
||||
|
||||
`~/.claude/bin/larv-review` holds this section's mechanical rules: the chunk's review folder, the round count, the paired ruling's base and vote count, the scoring run's trigger, and the history's format and latest places, and its count of how often seats agree. Run it in the chunk's worktree; `~/.claude/bin/larv-review --help` states each rule. On exit 1, repair what its message names and run it again; when you cannot, ask the user.
|
||||
|
||||
**Reports.** Run `~/.claude/bin/larv-review review-folder` in the chunk's worktree: it makes the chunk's review folder and prints its path. In each prompt to an agent under `~/.claude/agents/`, name the file in that folder for its full report: `<pass>-round-<n>.md`. For a principles-auditor, use `principles-<yardstick>-round-<n>.md`. For a relaunch one rung up, add `-rerun` before `.md`. Before you answer a 🔴 or 🟡, read its entry: Grep the report for the finding's place, with the lines after it, rather than Read the whole report. rule-hook refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`. Keep the folder after the merge: `review-replay` replays a round from the reports its prompt names.
|
||||
**Reports.** Run `~/.claude/bin/larv-review review-folder` in the chunk's worktree: it makes the chunk's review folder and prints its path. In each prompt to an agent under `~/.claude/agents/`, name the file in that folder for its full report: `<pass>-round-<n>.md`. For a relaunch one rung up, add `-rerun` before `.md`. Before you answer a 🔴 or 🟡, read its entry: Grep the report for the finding's place, with the lines after it, rather than Read the whole report. rule-hook refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`. Keep the folder after the merge: `review-replay` replays a round from the reports its prompt names.
|
||||
|
||||
**Precedent, then ask** — for the architecture pass and the full scan:
|
||||
|
||||
@@ -118,7 +118,7 @@ Launch systems-architect: review the system as a whole entered through this chan
|
||||
|
||||
### Principles
|
||||
|
||||
Launch principles-auditor on the trigger in §3: one for the README's goals and code health (`~/.claude/CLAUDE.md` → "Shared goal") unless the README voids it, and one per file in `~/.claude/principles/` whose `Applies to` line the project meets, judged from the README, `git ls-files` and the branch diff. A file without the line applies to every project. Unsure → launch it.
|
||||
Launch one principles-auditor on the trigger in §3, so the project is read once per round. Hand it the README's goals and code health (`~/.claude/CLAUDE.md` → "Shared goal") unless the README voids it, and each file in `~/.claude/principles/` whose `Applies to` line the project meets, judged from the README, `git ls-files` and the branch diff. A file without the line applies to every project. Unsure → hand it the file.
|
||||
|
||||
### Comprehension check
|
||||
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 233 | 0.4.0 | | decision | **Run one principles-auditor per round that checks the goals and every principles file that applies, in place of one auditor for each, so the principles pass reads the project once per round.** | 3 | 2 | 3 | 6 | 3 | 1.4 by 3, 1.5 by 2 | 4.9 |
|
||||
| 234 | 0.4.0 | | | **Report the routing figures in dollars, cache reads included, from the cost each archived run records.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 3.0 |
|
||||
| 239 | 0.4.0 | | owner | **Block a Read, `cat`, `sed` or `head` of a file range already in the session's context, with a note saying so, and let it through once a compaction dropped the range or the file changed.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 2.0 |
|
||||
| 226 | 0.5.0 | | question | **Answer whether rule-hook's `run-heavy` rule should also catch a mutation run or docker build behind `uv run`, `poetry run`, `python3 -m`, `sudo`, `docker image build` and `docker compose up --build`, or narrow the `machine load:` decision's wording.** | 2 | 2 | 3 | 5 | 3 | 1.2 by 2, 1.5 by 2 | 4.0 |
|
||||
@@ -77,10 +76,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 233. Run one principles-auditor per round that checks the goals and every principles file that applies, in place of one auditor for each, so the principles pass reads the project once per round.
|
||||
|
||||
Since 2026-10-21, 342 principles-auditor runs, 3.8 per chunk, cost $274 at list prices, 23% of all spend: $129 of 5-minute cache writes, $92 of output and $53 of cache reads. Each read the project anew. Today larv-review §3 → "Principles" launches one auditor for the README's goals and code health, and one per file in `~/.claude/principles/` that applies. The merged auditor must find what the separate ones found: replay archived rounds with `review-replay`, and keep separate auditors where the merged one misses a 🔴 or 🟡. Rename the report files, `principles-<yardstick>-round-<n>.md` today (§3 → "Reports"), to one per round.
|
||||
|
||||
### 234. Report the routing figures in dollars, cache reads included, from the cost each archived run records.
|
||||
|
||||
The routing figures count input, cache-written and output tokens, and leave out cache reads, which were 28% of the cost since 2026-10-21, $340 of $1,215 at list prices. So their tables rank the principles auditor above the worker, while in dollars the worker cost 1.4 times as much, $375 against $274. Each archived worker transcript ends with a `cost-state` entry holding `totalCostUSD` and each model's cost, and each message's usage attributes the cost to a subagent type. The judge's runs record no cost in the archive: count them too, or say why not. The change touches `bin/routing_figures/measure.py`, which the comprehension seats found hard (item 139): make the places it touches easier to read.
|
||||
|
||||
Reference in New Issue
Block a user