Finish items 325, 326, 336, 295, 332 and 334
CI / tests (pull_request) Successful in 2m36s

This commit was merged in pull request #503.
This commit is contained in:
2026-11-12 09:35:12 +01:00
parent 1f8b33d485
commit 8dc82fcf95
2 changed files with 6 additions and 50 deletions
+6
View File
@@ -2,6 +2,12 @@
## Unreleased
- `larv-review prepare-scoring` makes a scoring run's worktree and writes each seat's prompt; `record-scoring` removes both once it has recorded the run.
- `deploy-release` moves a checkout whose `settings.json` differs from the release only by a `/model` pick or the key order Claude Code writes, and then says to pick the model again.
- `chunk-pr merge` waits for the pull request's checks itself, allowing a minute for them to report, and no longer takes `--gate-passed`. `wait-for <file> [<pattern>]` waits for a line in a background job's log or report.
- `chunk-pr sync` merges origin's default branch into the chunk's branch, unless the branch already holds it, then runs the rank check on `todo.md`.
- `chunk-pr ready <title>` sets the pull request's final title as it marks the pull request ready.
- `bin/run-tests` runs this repo's test suite as README.md → Test says, prints each failing test with its error and pytest's summary line, and saves the whole log to a file whose path it prints.
- `command-patterns [<version>] --calls <file>` saves each Bash call of the release's chunks as a JSON line, with its chunk, pattern, seconds, sizes and text.
- `todo-item show <ID>` prints an item's table row and detail from `todo.md`.
- A reviewer under `agents/` can start without `CLAUDE.md`: `omitClaudeMd: true` in its frontmatter drops the file, a `SubagentStart` hook hands it the "Shared goal", "Prose" and "Vocabulary" sections, and `larv-review reviewer-header` names the project's principles file. `review-replay <pass> <model> --omit-claude-md` tests whether a pass's reviewer finds as much without it; principles-auditor is the first to pass.
-50
View File
@@ -14,11 +14,9 @@
|---|---|---|---|---|---|---|---|---|---|---|---|
| 340 | 0.9.0 | | question | **Decide whether a reviewer may start without `CLAUDE.md` where its replay misses no more findings than an unchanged replay does.** | 2 | 2 | 3 | 5 | 4 | 1.4 by 5 | 1.7 |
| 331 | 0.9.0 | | question | **Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.** | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.9 |
| 325 | 0.9.0 | | | **Give this repo's test suite one command that runs it as README → Test says, prints the failing tests and pytest's summary line, and saves the whole log to a file whose path it prints.** | 3 | 2 | 2 | 5 | 3 | 1.4 by 3, 1.5 by 2 | 2.0 |
| 305 | 0.9.0 | | | **Record with each archived verdict the claude-config commit that ran it, the raw tokens each model spent by kind, and each choice of the worker's that the human reversed, so a change to the setup can be judged before and after.** | 3 | 2 | 3 | 6 | 5 | 1.4 by 4, 1.2 by 3 | 1.8 |
| 318 | 0.9.0 | | | **Let `review-replay` qualify a rung whose recall is no lower than the project's rung's own replay, and replay Opus 5.5 at low effort on the prose pass.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 5 | 1.7 |
| 308 | 0.9.0 | | | **Skip a stability round 2 whose every must-fix finding named a surviving mutant or a failing test, both of which the worker now checks before review.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3, 1.5 by 2 | 1.6 |
| 295 | 0.9.0 | | | **Have `chunk-pr merge` wait for the forge's required checks itself, and give workers one command that waits for a background log or report file.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 2, 1.5 by 2 | 1.6 |
| 319 | 0.9.0 | | | **Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 2, 1.4 by 2 | 1.4 |
| 309 | 0.9.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.3 |
| 299 | 0.9.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.2 |
@@ -28,10 +26,6 @@
| 341 | 0.9.0 | | | **Hand each worker's claude a lower Read limit through `CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS`, kept only where replays of the archived review rounds find what two unchanged replays find.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.2 |
| 310 | 0.9.0 | | | **Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 1.0 |
| 298 | 0.9.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 1.0 |
| 326 | 0.9.0 | | | **Let `chunk-pr ready` take the pull request's final title and set it as it marks the pull request ready.** | 2 | 1 | 3 | 4 | 2 | 1.4 by 1, 1.5 by 1 | 0.9 |
| 332 | 0.9.0 | | defect | **Let `deploy-release` move a checkout whose `settings.json` differs from the release only by a `/model` pick or the key order Claude Code writes.** | 2 | 2 | 2 | 4 | 2 | 1.5 by 2 | 0.9 |
| 336 | 0.9.0 | | | **Bring a chunk's branch level with origin's default branch, and print the rank check, with one `chunk-pr sync`.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 1, 1.5 by 1 | 0.8 |
| 334 | 0.9.0 | | | **Set up a scoring run with one `larv-review` command that makes the scoring worktree, writes each seat's prompt file, and removes both once `record-scoring` has run.** | 3 | 2 | 2 | 5 | 5 | 1.4 by 2, 1.5 by 1 | 0.7 |
| 327 | 0.9.0 | | | **Have `larv-review` print a round commit's title prefix, counted from the round's report, and have rule-hook refuse a round title whose counts differ from it.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 1, 1.4 by 1 | 0.7 |
| 303 | 0.10.0 | | defect | **Archive each judge run's prompt and json reply beside the verdict it gave, so judge cost, replays and backups cover the judge as they cover the worker.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3, 3 by 2 | 4.8 |
| 304 | 0.10.0 | 303 | defect | **Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 2, 3 by 2 | 4.4 |
@@ -96,14 +90,6 @@ The architect's 🔀 answer on PR #493 (item 292): four decisions expire "when a
Recommendation: reword all four as above. They are decisions the repo held before item 292, so the rewording is lilleman's call.
### 325. Give this repo's test suite one command that runs it as README → Test says, prints the failing tests and pytest's summary line, and saves the whole log to a file whose path it prints.
The 0.8.0 audit (item 284) measured, over the 35 chunks that shipped 0.7.0: 163 to 176 Bash calls ran pytest, mutation runs excluded, in 29 chunks, for about 32k command tokens and 3,000 to 3,500 s of wall time. Each worker typed README → Test's incantation, with its pins, and cut the output with its own `| tail -N`: 14 spellings (tail sizes 1, 2, 3, 4, 5, 6, 8, 12, 15, 20, 25, 30 and 70, or no tail in 44 calls). A short tail cuts a failing run's evidence, and a long run without one floods the context.
Add one command, beside the README's Test section, that runs the suite as README → Test says, passes on any test paths or pytest options it is given, prints each failing test with its error and pytest's summary line, and saves the whole log to a file whose path it prints. Point README → Test at it, so the pins live in one place. Write its tests first: a passing run, a failing one, and a run that cannot start.
Left with the model: which tests to run, reading the failures and fixing them.
### 305. Record with each archived verdict the claude-config commit that ran it, the raw tokens each model spent by kind, and each choice of the worker's that the human reversed, so a change to the setup can be judged before and after.
Three records the archive lacks, asked for by lilleman on 2026-11-11 after the token research of that day.
@@ -126,10 +112,6 @@ Rounds past the first cost $846 of the $9,154 the archive measured from 2026-09-
Round 2 still matters elsewhere: 53 of 241 archived stability reports of round 2 or later open with BLOCK. So first count, from the archived round 2s, the BLOCKs that a run of `mutate-changes` and the tests on the fix would not have raised, and ship the skip only if none falls in a round it would skip. Evidence: one review round scored F1 0.376 and a second 0.26 to 0.30 with more false positives (arXiv 2603.16244, `claude -p` with Opus 4.6, 30 artifacts). Revise larv-review §3 → "Re-entry" and the decision `review: a fix for a 🔴 reruns the pass that found it…`.
### 295. Have `chunk-pr merge` wait for the forge's required checks itself, and give workers one command that waits for a background log or report file.
In 0.7.0's 35 chunks, workers made 65 `tea api` calls in 26 chunks, mostly polling commit status and trying merge methods in turn, and ran 25 `until … sleep` loops that took 1,744 seconds. `chunk-pr merge` takes `--gate-passed` on the worker's word; waiting for the required checks itself makes that flag a fact, not a claim. This is the kind of step item 284 files, and can count as its first.
### 319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
On SWE-bench Verified with Haiku 4.5 and Opus 4.7, handing the strong model the weak model's transcript solved 69.2% at $1.61 a task; starting the strong model fresh solved 79.2% at $0.90 (arXiv 2608.24358, The Handoff Tax). A model switch also rereads the whole context uncached (Claude Code prompt caching docs). A worker that steps from Sonnet 5.5 to Opus 5.5 by `--resume` is the raw handoff; an effort-only step within one model keeps the cache and isn't. The escalated subagent, which starts fresh with the trigger's evidence, already follows the pattern that won.
@@ -184,38 +166,6 @@ Probe first that `claude -p "/compact" --resume <id>` compacts a session: the Ag
153 `Judge cost:` entries total $109.76, a median of about $0.76 a verdict at every rung, and the cost follows the size of the records, not the transcript: about 76k characters in log and 188k in claude-config. In 224 of 251 recent verdicts (89%) the evidence sits in the worker's final message. A missed interrupt is the costly error, so the bar is zero changed interrupts and zero new rejections across the replay. The judge is about 4.5% of dollars, so the saving is small; each cycle also gets faster.
### 326. Let `chunk-pr ready` take the pull request's final title and set it as it marks the pull request ready.
The 0.8.0 audit (item 284) measured, over the 35 chunks that shipped 0.7.0: 30 `tea pr edit` calls in 18 chunks, about 2.0k command tokens. 17 of them, in 15 chunks, retitled a draft pull request the worker had opened with a provisional title; the rest looked up `tea pr edit --help`, rewrote a description or toggled the draft. On Gitea, `chunk-pr ready` already rewrites the title, dropping `WIP: `, so setting the final title there is the same call.
Let `chunk-pr ready` take an optional final title and set it as it marks the pull request ready, on GitHub and on Gitea, and refuse a title of more than one line. `bin/chunk_pr/main.py:42` `forge` is a place the comprehension seats found hard; where the change touches it, make it easier to read, or leave it to item 287.
Left with the model: wording the title.
### 332. Let `deploy-release` move a checkout whose `settings.json` differs from the release only by a `/model` pick or the key order Claude Code writes.
On 2026-11-12, moving `~/.claude` from v0.7.0 to v0.8.0, `deploy-release` failed at `git checkout --quiet --detach v0.8.0`: "Your local changes to the following files would be overwritten by checkout: settings.json". Claude Code had rewritten the file with `"model": "opus"` and its keys in another order. The `settings` clean filter drops the model and sorts the keys, so `git diff` showed no change and the tool's own check passed, but checkout compares the file as it is on disk. `deploy-release --help` says a model picked with `/model` doesn't block the move.
The move worked after `git checkout -- settings.json`, with the model added back by hand. Have the tool set aside a change the clean filter removes, move, and restore the model pick, or tell the user to pick it again, as it already does when the move changes `settings.json`. Test it with a working file that differs from HEAD only by the model and the key order.
### 336. Bring a chunk's branch level with origin's default branch, and print the rank check, with one `chunk-pr sync`.
Found by item 328's audit of release 0.8.0. 8 of its 9 chunks synced with the default branch by hand before the merge, in 13 calls (about 2.4k tokens, 6 s): `git fetch -q origin && git merge-base --is-ancestor origin/main HEAD && echo holds-main`, then, where that failed, `git merge -q origin/main`, then `rank_todo.py --check todo.md`.
`chunk-pr sync` fetches origin, merges its default branch into the chunk's branch where the branch does not already hold it, and prints one line saying which it did. Then it runs `rank_todo.py --check` where the worktree has a `todo.md`, and prints its result. On a merge conflict it stops with the merge in progress and names the conflicted files. chunk-pr already finds origin's default branch for `open`, so `sync` reuses that; where the change touches `bin/chunk_pr/main.py:42` `forge`, which the comprehension seats found hard because its helpers live in other packages, make it easier to read. `chunk-pr --help` and `larv-review` §4 step 1, which merges origin's default branch in before the gate, name the command.
Left with the model: resolving a conflict, such as one in `todo.md`, and fixing what the rank check reports.
### 334. Set up a scoring run with one `larv-review` command that makes the scoring worktree, writes each seat's prompt file, and removes both once `record-scoring` has run.
Found by item 328's audit of release 0.8.0. Of its 9 chunks, only item 296's ran a scoring run, since scoring runs are rare, and it set the run up by hand in about 20 Bash calls: 7 reads of `skills/comprehension-panel/SKILL.md`, `seat-prompts.md` and the scoring-run values (ls, wc, sed; about 12k result tokens); `git worktree add --detach ../claude-config-scoring origin/main`, then `git checkout -b scoring-item-301`; one `mkdir /tmp/seat296` and three heredocs writing shared.txt, report-plain.txt and report-arch.txt (6.6k chars); two heredocs writing scores.md (4.5k and 3.6k chars); `larv-review record-scoring`; and `rm -rf /tmp/seat296`. About 16k tokens in all.
The prompt text is fixed: `seat-prompts.md` says to copy the opening, each seat's background, the shared brief and the report for its kind of seat verbatim, and lists each `{placeholder}`. A program fills most of them: `{checkout}`, `{branch}`, `{main}` and `{git}` from the worktree it makes, `{forbidden}` from fixed text, `{files}` from `git ls-files`, `{anchors}` and `{dimensions}` from `SKILL.md` → "The score", and `{language}`, `{kind}`, `{domain}`, `{domain docs}` and `{3am question}` from the project's scoring-run values (claude-config's in `seat-prompts.md` → "claude-config's values"). Where one needs judgement, such as `{preloaded}`, the command takes it as an argument. It refuses, writing nothing, where the project has no scoring-run values. Where the prompt files land, a folder under the chunk's review folder or the state folder, is the implementer's call.
The command makes the detached scoring worktree on origin's default branch, writes one prompt file per seat of the four `SKILL.md` → "The scoring run" names, and prints their paths. A second form, or `record-scoring` itself, removes the worktree and the prompt folder after recording.
Left with the model: launching the seats, reading their reports and writing the scores. Item 316 (the seats' own agent type) and item 309 (two seats in the second pass) change who reads these prompts and how many, not how they are built, so this item does not overlap them.
### 327. Have `larv-review` print a round commit's title prefix, counted from the round's report, and have rule-hook refuse a round title whose counts differ from it.
The 0.8.0 audit (item 284) measured, over the 35 chunks that shipped 0.7.0: 72 commit calls in 33 chunks titled `Answer <pass> round <n>: R 🔴, Y 🟡 (K misread); <what changed>`, about 11k command tokens, most of them the summary. Each worker counts R, Y and K by hand. All three are in the round's report, `<pass>-round-<n>.md` in the review folder: R and Y are its finding headings, and K counts the findings the reviewer marked `[misread]`, as `bin/review_rules/round_titles.py` says. That module parses the counts for `score-outcomes`, which reports them as goal 1.1's signal, and rule-hook's `round_cap.py` checks only the round number, so nothing checks today that the counts match the report. Of the 8 titles whose report the audit could match, none miscounted; one report wrote its markers at line start rather than in `###` headings, so the reports' format varies by reviewer.