Finish item 284: file items 325 to 327 from the 0.7.0 audit, and the 0.8.0 audit as item 328
CI / tests (pull_request) Successful in 2m25s
CI / tests (pull_request) Successful in 2m25s
This commit was merged in pull request #489.
This commit is contained in:
@@ -1616,3 +1616,18 @@ Read most (opens and searches):
|
||||
- `bin/loop_claude/verdicts.py` 2
|
||||
- `bin/loop_claude/routing.py` 1
|
||||
- `docs/decisions.md` 1
|
||||
|
||||
## Reading cost, chunk 3fba05e72d95, finished 2026-11-11T21:05:07Z
|
||||
|
||||
86411 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `todo.md` 7
|
||||
- `skills/larv-review/SKILL.md` 4
|
||||
- `README.md` 2
|
||||
- `docs/decisions.md` 2
|
||||
- `CLAUDE.md` 1
|
||||
- `agents/stability-reviewer.md` 1
|
||||
- `bin/reading-cost` 1
|
||||
- `bin/rule_hook/heavy_command.py` 1
|
||||
|
||||
@@ -6,18 +6,18 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 325`
|
||||
`Next ID = 329`
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 284 | 0.8.0 | | owner | **Script each step that 0.7.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.6 |
|
||||
| 270 | 0.8.0 | | | **Move each check a review pass repeats across chunks, other than the mutation tests `bin/mutate-changes` runs, into the checks the worker runs before review, starting with the passes most often taken to a second round.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 3, 1.4 by 3 | 2.5 |
|
||||
| 292 | 0.8.0 | | owner | **Rebuild the routing ladder for Fable 5.2, placing each of its efforts by the Artificial Analysis Intelligence Index beside Opus 5.5's and Sonnet 5.5's, and replace Fable 5.1 wherever it is named.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 4, 1.4 by 3 | 2.4 |
|
||||
| 320 | 0.8.0 | | | **Bench a worker below the project's rung with the advisor tool against the same worker alone and the project's rung, and file the switch if it wins.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3, 1.1 by 2 | 2.0 |
|
||||
| 315 | 0.8.0 | | | **Start each reviewer under `agents/` without `CLAUDE.md`, and hand it the vocabulary, the prose rules and the project's decision titles.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 6 | 2.0 |
|
||||
| 312 | 0.8.0 | | | **Research the tooling agents read, search and run commands with, and file an item for each tool that cuts tokens a replay shows keep quality.** | 1 | 2 | 1 | 2 | 3 | 1.4 by 3, 1.5 by 2 | 2.0 |
|
||||
| 325 | 0.8.0 | | | **Give this repo's test suite one command that runs it as README → Test says, prints the failing tests and pytest's summary line, and saves the whole log to a file whose path it prints.** | 3 | 2 | 2 | 5 | 3 | 1.4 by 3, 1.5 by 2 | 2.0 |
|
||||
| 305 | 0.8.0 | | | **Record with each archived verdict the claude-config commit that ran it, the raw tokens each model spent by kind, and each choice of the worker's that the human reversed, so a change to the setup can be judged before and after.** | 3 | 2 | 3 | 6 | 5 | 1.4 by 4, 1.2 by 3 | 1.8 |
|
||||
| 318 | 0.8.0 | | | **Let `review-replay` qualify a rung whose recall is no lower than the project's rung's own replay, and replay Opus 5.5 at low effort on the prose pass.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 5 | 1.7 |
|
||||
| 308 | 0.8.0 | | | **Skip a stability round 2 whose every must-fix finding named a surviving mutant or a failing test, both of which the worker now checks before review.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3, 1.5 by 2 | 1.6 |
|
||||
@@ -30,12 +30,15 @@
|
||||
| 310 | 0.8.0 | | | **Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 1.0 |
|
||||
| 298 | 0.8.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 1.0 |
|
||||
| 313 | 0.8.0 | | | **Launch the first agent of each type in a parallel wave one message before the others of that type, so they read its prompt cache.** | 1 | 2 | 2 | 3 | 2 | 1.4 by 2 | 0.9 |
|
||||
| 326 | 0.8.0 | | | **Let `chunk-pr ready` take the pull request's final title and set it as it marks the pull request ready.** | 2 | 1 | 3 | 4 | 2 | 1.4 by 1, 1.5 by 1 | 0.9 |
|
||||
| 327 | 0.8.0 | | | **Have `larv-review` print a round commit's title prefix, counted from the round's report, and have rule-hook refuse a round title whose counts differ from it.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 1, 1.4 by 1 | 0.7 |
|
||||
| 317 | 0.8.0 | | | **Cut the descriptions of stability-reviewer, product-owner-reviewer and systems-architect to one line each.** | 1 | 1 | 1 | 1 | 1 | 1.4 by 1 | 0.5 |
|
||||
| 303 | 0.9.0 | | defect | **Archive each judge run's prompt and json reply beside the verdict it gave, so judge cost, replays and backups cover the judge as they cover the worker.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3, 3 by 2 | 4.8 |
|
||||
| 304 | 0.9.0 | 303 | defect | **Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 2, 3 by 2 | 4.4 |
|
||||
| 306 | 0.9.0 | | | **Set a strategy that keeps loop-claude's archive and larv-review's review folders backed up and small over the years, compressing older entries without losing any.** | 3 | 2 | 4 | 7 | 5 | 3 by 5 | 3.9 |
|
||||
| 302 | 0.9.0 | | | **Call a scoring run for an item whose tagline makes the comprehension floor its target, and not for one that only names the floor, such as in a quoted decision title.** | 2 | 2 | 3 | 5 | 3 | code health by 2 | 3.6 |
|
||||
| 287 | 0.9.0 | | | **Move the forge helpers `Forge`, `forge_of`, `runner`, `open_pull`, `merged_pull`, `pull_at` and `gitea` into one module that `chunk_preflight`, `chunk_pr`, `deploy_release` and `larv_review` all import from, so no tool imports another tool's internals for them.** | 3 | 1 | 3 | 5 | 5 | code health by 3 | 3.4 |
|
||||
| 328 | 0.9.0 | | owner | **Script each step that 0.8.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.6 |
|
||||
| 288 | 0.9.0 | | | **Point the two reading-cost decisions at code health and goal 1.4's counter-signal, in place of the goal 1.1 they cite.** | 1 | 2 | 1 | 2 | 1 | code health by 1, 1.4 by 1 | 2.6 |
|
||||
| 77 | 0.9.0 | | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 4 | 2 | 1 | 5 | 6 | 1.1 by 7, 1.4 by 7 | 2.4 |
|
||||
| 321 | 0.9.0 | 318 | | **Evaluate the review passes that run on a rung item 318 qualified, against the chunks before them, and revert any pass whose findings or later fixes worsened.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3, 1.4 by 2 | 2.3 |
|
||||
@@ -73,12 +76,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 284. Script each step that 0.7.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.
|
||||
|
||||
lilleman asked, on 2026-10-16, for a standing audit each release, so the loop keeps moving work a program does as well from the model to scripts (goals 1.4 and 1.5). This audit reads the archived transcripts of the chunks that shipped release 0.7.0 for every step that chunks repeated by hand, before, during or after review. It keeps the steps whose answer a program computes the same way each time, such as a lookup, a check, a prompt assembled for a reviewer, a cleanup. File one item per such step, with the time and tokens it cost, into the next release with items left. Leave a step that needs judgement with the model. Run item 283's tool first, once it has shipped.
|
||||
|
||||
The chunk that ships this item files a copy of this audit in the release after this item's own, to audit this item's release, rewording "0.7.0" to the release that copy audits, and marks the copy `owner`, so the audit recurs each release until lilleman drops it.
|
||||
|
||||
### 270. Move each check a review pass repeats across chunks, other than the mutation tests `bin/mutate-changes` runs, into the checks the worker runs before review, starting with the passes most often taken to a second round.
|
||||
|
||||
The review folders under `~/.local/state/larv-review/`, review replays left out, hold a stability round 2 beside 137 of 211 round-1 reports (65%), product-owner 73 of 173 (42%) and prose 8 of 277 (3%), on 2026-11-08. The architecture and principles passes run up to five rounds in a release review, so they don't compare. From 2026-11-02 to 2026-11-10, 223 of stability's 428 round-1 findings (52%) were test gaps, and 31 of its 50 round 2s held one; `bin/mutate-changes` now runs, before review, the mutation tests that find most of them. Check which kinds of finding the other round 2s repeat, such as an unhandled error or a product-owner wording fix. Move only a check a program or the worker can run before review; leave judgement with the reviewer.
|
||||
@@ -107,6 +104,16 @@ lilleman, 2026-11-11: besides an output ceiling, keep evaluating better tooling.
|
||||
|
||||
Run a research round like that of 2026-11-11: a lead on the project's rung plans the searches, cheaper subagents search and distil sourced findings, and the lead checks every claim it relies on against this archive's measured spend. File an item for each tool that a replay could test. lilleman, 2026-11-11: repeat the round once per claude-config release, not per project. When this item's chunk finishes, file the next round into the next claude-config release with items left, so the check comes back periodically.
|
||||
|
||||
The 0.8.0 audit (item 284) measured: over the 35 chunks that shipped 0.7.0, workers made 36 Edit calls but about 250 `python3` heredocs that replace a string in a file and write it back, about 120k command tokens of the 135k that all 375 `python3` calls took. They read files with `sed` in 493 Bash calls and `cat` in 188, for 351k and 328k result tokens, beside 691 Read calls. Edit refuses a file not read with the Read tool, so a worker that read a file with `sed` or `cat` probably edits it through python. Check that cause, and whether reading through Read, or an Edit that accepts such a file, cuts tokens and keeps quality.
|
||||
|
||||
### 325. Give this repo's test suite one command that runs it as README → Test says, prints the failing tests and pytest's summary line, and saves the whole log to a file whose path it prints.
|
||||
|
||||
The 0.8.0 audit (item 284) measured, over the 35 chunks that shipped 0.7.0: 163 to 176 Bash calls ran pytest, mutation runs excluded, in 29 chunks, for about 32k command tokens and 3,000 to 3,500 s of wall time. Each worker typed README → Test's incantation, with its pins, and cut the output with its own `| tail -N`: 14 spellings (tail sizes 1, 2, 3, 4, 5, 6, 8, 12, 15, 20, 25, 30 and 70, or no tail in 44 calls). A short tail cuts a failing run's evidence, and a long run without one floods the context.
|
||||
|
||||
Add one command, beside the README's Test section, that runs the suite as README → Test says, passes on any test paths or pytest options it is given, prints each failing test with its error and pytest's summary line, and saves the whole log to a file whose path it prints. Point README → Test at it, so the pins live in one place. Write its tests first: a passing run, a failing one, and a run that cannot start.
|
||||
|
||||
Left with the model: which tests to run, reading the failures and fixing them.
|
||||
|
||||
### 305. Record with each archived verdict the claude-config commit that ran it, the raw tokens each model spent by kind, and each choice of the worker's that the human reversed, so a change to the setup can be judged before and after.
|
||||
|
||||
Three records the archive lacks, asked for by lilleman on 2026-11-11 after the token research of that day.
|
||||
@@ -179,6 +186,22 @@ A cache entry is readable only once the first response that wrote it has started
|
||||
|
||||
Change `skills/comprehension-panel/SKILL.md` → "Parallel" and `skills/larv-review/SKILL.md` → "Waves": launch one agent of a type, and the others of that type in the next message. Quality doesn't change, since every agent gets the same prompt and model; the cost is one extra worker turn, about $0.02, and a few seconds. Check: the first-turn `cache_read_input_tokens` of seats two to four in the next scoring run.
|
||||
|
||||
### 326. Let `chunk-pr ready` take the pull request's final title and set it as it marks the pull request ready.
|
||||
|
||||
The 0.8.0 audit (item 284) measured, over the 35 chunks that shipped 0.7.0: 30 `tea pr edit` calls in 18 chunks, about 2.0k command tokens. 17 of them, in 15 chunks, retitled a draft pull request the worker had opened with a provisional title; the rest looked up `tea pr edit --help`, rewrote a description or toggled the draft. On Gitea, `chunk-pr ready` already rewrites the title, dropping `WIP: `, so setting the final title there is the same call.
|
||||
|
||||
Let `chunk-pr ready` take an optional final title and set it as it marks the pull request ready, on GitHub and on Gitea, and refuse a title of more than one line. `bin/chunk_pr/main.py:42` `forge` is a place the comprehension seats found hard; where the change touches it, make it easier to read, or leave it to item 287.
|
||||
|
||||
Left with the model: wording the title.
|
||||
|
||||
### 327. Have `larv-review` print a round commit's title prefix, counted from the round's report, and have rule-hook refuse a round title whose counts differ from it.
|
||||
|
||||
The 0.8.0 audit (item 284) measured, over the 35 chunks that shipped 0.7.0: 72 commit calls in 33 chunks titled `Answer <pass> round <n>: R 🔴, Y 🟡 (K misread); <what changed>`, about 11k command tokens, most of them the summary. Each worker counts R, Y and K by hand. All three are in the round's report, `<pass>-round-<n>.md` in the review folder: R and Y are its finding headings, and K counts the findings the reviewer marked `[misread]`, as `bin/review_rules/round_titles.py` says. That module parses the counts for `score-outcomes`, which reports them as goal 1.1's signal, and rule-hook's `round_cap.py` checks only the round number, so nothing checks today that the counts match the report. Of the 8 titles whose report the audit could match, none miscounted; one report wrote its markers at line start rather than in `###` headings, so the reports' format varies by reviewer.
|
||||
|
||||
Add a `larv-review` subcommand that prints a round title's prefix, up to and including the `;`, from the round's report, and have rule-hook refuse a round title whose counts differ from it. Pin the marker format in each reviewer's definition under `agents/`, or count every format the reports use today. Where the report is missing, say so and let the commit through.
|
||||
|
||||
Left with the model: the summary after the `;`, and answering the findings.
|
||||
|
||||
### 317. Cut the descriptions of stability-reviewer, product-owner-reviewer and systems-architect to one line each.
|
||||
|
||||
Their descriptions, 7.5 KB together, sit in the agent listing of every worker and general-purpose subagent in every project, about 0.7% of spend. larv-review launches them by name, so its routing doesn't depend on them. The one downside: a session outside larv-review relies on a description to decide when to launch an agent unasked, so a one-line description makes that less likely. Keep the line specific enough that a reader knows when to pick each. lilleman approved on 2026-11-11.
|
||||
@@ -221,6 +244,12 @@ Both comprehension seats of PR #453's paired ruling (2026-11-09) named this web
|
||||
|
||||
Put the seven helpers in a folder named by what it holds, such as `bin/forge/`, and import them from there. Keep each helper's behaviour and its tests. `chunk_preflight/choices.py` keeps `pulls_command`, which is preflight's own. Check `docs/decisions.md` for a mention of where a helper lives, and update it.
|
||||
|
||||
### 328. Script each step that 0.8.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.
|
||||
|
||||
lilleman asked, on 2026-10-16, for a standing audit each release, so the loop keeps moving work a program does as well from the model to scripts (goals 1.4 and 1.5). This audit reads the archived transcripts of the chunks that shipped release 0.8.0 for every step that chunks repeated by hand, before, during or after review. It keeps the steps whose answer a program computes the same way each time, such as a lookup, a check, a prompt assembled for a reviewer, a cleanup. File one item per such step, with the time and tokens it cost, into the next release with items left. Leave a step that needs judgement with the model. Run `command-patterns 0.8.0` first; its table names the patterns, and `read_threads` and `calls` in `bin/command_patterns/tally.py` open the calls of one pattern. The 0.7.0 audit found that most steps its chunks repeated were already scripted by that release's own tools, so a step a shipped item already covers is not filed.
|
||||
|
||||
The chunk that ships this item files a copy of this audit in release 0.10.0, the release after this item's own, to audit release 0.9.0, rewording "0.8.0" to "0.9.0", and marks the copy `owner`, so the audit recurs each release until lilleman drops it.
|
||||
|
||||
### 288. Point the two reading-cost decisions at code health and goal 1.4's counter-signal, in place of the goal 1.1 they cite.
|
||||
|
||||
Both `docs/decisions.md` entries under `reading cost: a rise is …` and `reading cost: larv-review measures each finished chunk …` open "Serves goal 1.1: comprehension …". No goal says that any more: code health (`CLAUDE.md` → "Shared goal") holds comprehension, and since PR #455 a rising reading cost is goal 1.4's first counter-signal. Raised by the stability review of PR #455 as a 🟢 outside that change.
|
||||
|
||||
Reference in New Issue
Block a user