Answer stability round 2: 0 🔴, 3 🟡 (0 misread); keep item 313 as a question for the human, restore the decision's expiry, and make the sweep list general
CI / tests (pull_request) Successful in 2m28s

This commit was merged in pull request #490.
This commit is contained in:
2026-11-11 22:35:08 +01:00
parent a0b8cf7882
commit aacfe7d3d3
3 changed files with 12 additions and 3 deletions
+2 -2
View File
@@ -358,11 +358,11 @@ Valid while a mutation run on a chunk's changed lines stays cheap beside the rev
Stability reviewers ran the whole suite in 52 of this repo's 63 reports from the last 60 days, and the worker's run on the same commit gives the same result. So the worker sweeps for each name, flag, path and number the change touches, runs the suite and analysis as the project's README documents them, and hands the reviewer the passing commit; the reviewer runs them only for another commit. The sweep is the worker's reading, not a program, since a stale claim has no fixed spelling.
Valid while the worker's run on a commit finds what a reviewer's run on it would, and while stale-doc findings in round 2 stay near the 21% measured.
Valid while the worker's run on a commit finds what a reviewer's run on it would, and while stale-doc findings in round 2 fall below the 21% measured.
## review: agents of one type launch in one message, not one a message before the others
2026-11-11, lilleman through todo item 313, dropped by claude after the stability review. Serves goals 1.4 and 1.5. Anthropic's prompt caching docs say a cache entry is readable only once the first response that wrote it has started, and item 313 measured 748 comprehension seats, 153 security passes and 130 general-purpose subagents that each wrote their own prefix, $99 from 2026-09-30 to 2026-11-11. Launching one agent of a type before the rest, to read its cache, doesn't pay: a foreground Agent call returns only when its agent has finished, and larv-review runs reviewers in the foreground, so each later launch group would wait one agent's whole run, a seat's median 1.6 minutes, to save a prefix of about 14k tokens. The stability reviewer also found, in the transcripts of 73 seats started within 90 seconds of an earlier seat, that each already read the shared prefix.
2026-11-11, claude, in the stability review of the chunk that took todo item 313, which stays in the plan as a question for lilleman. Serves goals 1.4 and 1.5. Anthropic's prompt caching docs say a cache entry is readable only once the first response that wrote it has started, and item 313 measured 748 comprehension seats, 153 security passes and 130 general-purpose subagents that each wrote their own prefix, $99 from 2026-09-30 to 2026-11-11. Launching one agent of a type before the rest, to read its cache, doesn't pay: a foreground Agent call returns only when its agent has finished, and larv-review runs reviewers in the foreground, so each later launch group would wait one agent's whole run, a seat's median 1.6 minutes, to save a prefix of about 14k tokens. The stability reviewer also found, in the transcripts of 73 seats started within 90 seconds of an earlier seat, that each already read the shared prefix.
Valid while a foreground Agent call returns only when its agent has finished.
+1 -1
View File
@@ -28,7 +28,7 @@ Follow `~/.claude/CLAUDE.md` → "Git, commits, and PRs". Never commit to the de
**Reading cost.** Run `~/.claude/bin/reading-cost` in the chunk's worktree, unless loop-claude's prompt shows that `chunk-preflight` already ran it there, and commit what it appends to `docs/reading-cost.md`; the commit re-enters no pass. Resolve a merge conflict in that file by putting the default branch's figures first and dropping a figure for a chunk the default branch already holds. On `Rising:`, file a plan item titled `**Lower the reading cost of the files a chunk reads most before its first commit.**`, scored per `~/.claude/rules/todo-items.md` with the tool's medians and files as its detail; where the plan already holds it, leave it as it is: `docs/reading-cost.md` holds the figures. The item names the next release, or the one after when this chunk ships the next release's last item. On exit 1, repair what its message names and run it again; when the message names no repair, file it as a `question` item, or refresh the question item an earlier run filed, and go on. Leave the plan as it is on any other output. The chunk that ships that item runs the tool with `--reset` instead, so the next rise is measured on chunks that finished after it.
**Tests and static analysis.** Where the branch changes code, config or tests, before each stability round: search `README.md`, `CHANGELOG.md`, `docs/`, `todo.md`, the Markdown under `agents/`, `rules/` and `skills/`, the help and error text in code, and the comments beside the code for each flag, name, path and number the change renames, removes or changes, and fix each line it made false. Commit, then run the project's test suite and static analysis in the chunk's worktree as its README documents them, and fix a failure before going on. Hand stability-reviewer the commit they last passed on; a fix from the mutation tests below moves the head, so run them again after it, and the reviewer doesn't run them again on that commit.
**Tests and static analysis.** Where the branch changes code, config or tests, before each stability round: search the project's Markdown, the help and error text in code, and the comments beside the code for each flag, name, path and number the change renames, removes or changes, and fix each line it made false. Commit, then run the project's test suite and static analysis in the chunk's worktree as its README documents them, and fix a failure before going on. Hand stability-reviewer the commit they last passed on; a fix from the mutation tests below moves the head, so run them again after it, and the reviewer doesn't run them again on that commit.
**Mutation tests.** Where the branch changes code, run the project's mutation command in the chunk's worktree, with every change committed, before each stability round: the one the project's README documents, else `~/.claude/bin/run-heavy ~/.claude/bin/mutate-changes --prefix '<how the project runs its tools>'`. `mutate-changes --help` says how to set `--prefix` and `--jobs`. Where a mutant of a changed line survives and changes behaviour, kill it: add or strengthen a test that fails on it, commit, and run the command again. Hand stability-reviewer what the last run printed, with the commit it ran on. On exit 1, repair what its message names and run it again; when you cannot, hand the reviewer the message.
+9
View File
@@ -12,6 +12,7 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 313 | 0.8.0 | | question | **Launch the first agent of each type in a parallel wave one message before the others of that type, so they read its prompt cache.** | 1 | 2 | 2 | 3 | 2 | 1.4 by 2 | 0.9 |
| 292 | 0.8.0 | | owner | **Rebuild the routing ladder for Fable 5.2, placing each of its efforts by the Artificial Analysis Intelligence Index beside Opus 5.5's and Sonnet 5.5's, and replace Fable 5.1 wherever it is named.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 4, 1.4 by 3 | 2.4 |
| 320 | 0.8.0 | | | **Bench a worker below the project's rung with the advisor tool against the same worker alone and the project's rung, and file the switch if it wins.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3, 1.1 by 2 | 2.0 |
| 315 | 0.8.0 | | | **Start each reviewer under `agents/` without `CLAUDE.md`, and hand it the vocabulary, the prose rules and the project's decision titles.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 6 | 2.0 |
@@ -73,6 +74,14 @@
## Details
### 313. Launch the first agent of each type in a parallel wave one message before the others of that type, so they read its prompt cache.
A cache entry is readable only once the first response that wrote it has started ("If you need cache hits for parallel requests, wait for the first response before sending subsequent requests", Anthropic's prompt caching docs). Between 2026-09-30 and 2026-11-11, 748 comprehension seats started within 90 seconds of another seat in the same chunk and each still wrote its own prefix, $73; so did 153 security passes ($17) and 130 general-purpose subagents ($9). Different agent types share no prefix, so only agents of one type gain.
Change `skills/comprehension-panel/SKILL.md` → "Parallel" and `skills/larv-review/SKILL.md` → "Waves": launch one agent of a type, and the others of that type in the next message. Quality doesn't change, since every agent gets the same prompt and model; the cost is one extra worker turn, about $0.02, and a few seconds. Check: the first-turn `cache_read_input_tokens` of seats two to four in the next scoring run.
Question for lilleman, from the stability review of the chunk that took this item (recommendation: drop it): a foreground Agent call returns only when its agent has finished, and larv-review runs reviewers in the foreground, so launching one agent of a type a message before the rest makes each later launch group wait one agent's whole run, a seat's median 1.6 minutes, to save a prefix about 14k tokens long. The review found in the transcripts of 73 seats that each already read the shared prefix. `docs/decisions.md` → `review: agents of one type launch in one message, not one a message before the others` holds the evidence. Launching the first agent in the background would avoid the wait, but larv-review forbids a background reviewer, since a headless run ends with its turn.
### 292. Rebuild the routing ladder for Fable 5.2, placing each of its efforts by the Artificial Analysis Intelligence Index beside Opus 5.5's and Sonnet 5.5's, and replace Fable 5.1 wherever it is named.
Fable 5.2 (`claude-fable-5-2`) has been released. The ladder in `bin/model_routing/ladder.py` tops out at `Rung("claude-fable-5-1", "max")`, and the decision `routing: one ladder of (model, effort) rungs` holds "while no newer model joins the catalog", so it is due for a rebuild.