Merge remote-tracking branch 'origin/main' into item-308-count
CI / tests (pull_request) Successful in 2m39s

This commit was merged in pull request #508.
This commit is contained in:
2026-11-12 11:53:55 +01:00
+20 -17
View File
@@ -12,8 +12,7 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 340 | 0.9.0 | | question | **Decide whether a reviewer may start without `CLAUDE.md` where its replay misses no more findings than an unchanged replay does.** | 2 | 2 | 3 | 5 | 4 | 1.4 by 5 | 1.7 |
| 331 | 0.9.0 | | question | **Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.** | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.9 |
| 340 | 0.9.0 | | owner | **Let the stability reviewer start without `CLAUDE.md`, as principles-auditor does, once a replay with a control has been run on every pass and reported to lilleman.** | 2 | 2 | 3 | 5 | 4 | 1.4 by 5 | 1.7 |
| 319 | 0.9.0 | | | **Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 2, 1.4 by 2 | 1.4 |
| 309 | 0.9.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.3 |
| 299 | 0.9.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.2 |
@@ -23,6 +22,7 @@
| 341 | 0.9.0 | | | **Hand each worker's claude a lower Read limit through `CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS`, kept only where replays of the archived review rounds find what two unchanged replays find.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.2 |
| 310 | 0.9.0 | | | **Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 1.0 |
| 298 | 0.9.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 1.0 |
| 331 | 0.9.0 | | | **Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.** | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.9 |
| 327 | 0.9.0 | | | **Have `larv-review` print a round commit's title prefix, counted from the round's report, and have rule-hook refuse a round title whose counts differ from it.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 1, 1.4 by 1 | 0.7 |
| 303 | 0.10.0 | | defect | **Archive each judge run's prompt and json reply beside the verdict it gave, so judge cost, replays and backups cover the judge as they cover the worker.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3, 3 by 2 | 4.8 |
| 304 | 0.10.0 | 303 | defect | **Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 2, 3 by 2 | 4.4 |
@@ -68,24 +68,16 @@
## Details
### 340. Decide whether a reviewer may start without `CLAUDE.md` where its replay misses no more findings than an unchanged replay does.
### 340. Let the stability reviewer start without `CLAUDE.md`, as principles-auditor does, once a replay with a control has been run on every pass and reported to lilleman.
Item 315's gate kept `omitClaudeMd` only where a replay without `CLAUDE.md` found every finding that two unchanged replays both found. Read so, five passes failed it on 2026-11-12: stability missed 1 of 11, prose 1 of 24, product-owner 1 of 4, architecture 4 of 27, and overrides had no archived round to replay. The decision `agents/: a reviewer starts without CLAUDE.md only where a replay of its pass missed none of the findings two unchanged replays both found` holds the figures.
Item 315's gate kept `omitClaudeMd` only where a replay without `CLAUDE.md` found every finding that two unchanged replays both found. Read so, five passes failed it on 2026-11-12: stability missed 1 of 11, prose 1 of 24, product-owner 1 of 4, architecture 4 of 27, and overrides had no archived round to replay. The decision `agents/: a reviewer starts without CLAUDE.md only where a replay of its pass missed none of the findings two unchanged replays both found` holds the figures. The unchanged reviewer finds only about 70% of its findings again, so that gate can't tell a loss from the reviewer's own noise.
The question for lilleman: does a miss that small show a loss, or the reviewer's own noise? The unchanged reviewer finds only about 70% of its findings again, so a third unchanged replay would also miss some that the other two found, and the gate as written can't tell the two apart.
lilleman's call, 2026-11-12: principles-auditor and stability-reviewer start without `CLAUDE.md`; the other reviewers keep it. The call is provisional until the data below is in.
Recommended: give `review-replay --omit-claude-md` a control. It replays each round three times unchanged and once without `CLAUDE.md`, and prints for each unchanged replay how many findings it missed that the other two found. The reviewer may start without `CLAUDE.md` where its own count is no higher than the highest of those three. Then run it for the five passes, and add `omitClaudeMd: true` to each agent that passes; the hook and the header already serve any agent that has it. Each run costs four replays of 10 rounds on the project's model.
### 331. Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.
The architect's 🔀 answer on PR #493 (item 292): four decisions expire "when a new model joins the ladder". Fable 5.2 joined above every project model's rung, and none of these replays tested a rung above the project's, so the event fired with nothing new to test. Every later ladder rebuild would fire all four again.
- `review: each pass keeps a reviewer of its own`: the merge trials ran on Opus 5.5 at medium. Reword the "Valid" line to name that rung.
- `review: the principles audit runs only in the release review…`: its trial ran on the project's rung. Reword the line to name that rung.
- `routing: a review pass runs below the project's rung only on a rung whose replay of its rounds finds a share of their 🔴 and 🟡 findings no more than a margin below the project's rung's own replay…`: `review-replay` tests only rungs below the project's rung. Have the line expire when those rungs change, the event item 196 names ("until a cheaper model joins the ladder").
- `routing: on a model judge-replay covered, the judge runs…`: the replays tested the project's rung and below, and each rung above it that a review climbs to. Have the line expire when those rungs change.
Recommendation: reword all four as above. They are decisions the repo held before item 292, so the rewording is lilleman's call.
1. Give `review-replay --omit-claude-md` a control. It replays each round three times unchanged and once without `CLAUDE.md`, and prints for each unchanged replay how many findings it missed that the other two found. Each run costs four replays of 10 rounds on the project's model.
2. Run it for every pass that has archived rounds: principles, stability, prose, product-owner and architecture.
3. Report each pass's figures to lilleman, saying whether they change the picture: a pass passes where its count without `CLAUDE.md` is no higher than the highest of its three unchanged counts. The picture changes where stability or principles fails, or another pass passes.
4. Where the picture holds, add `omitClaudeMd: true` to stability-reviewer, and record the figures and the call in the decision above. Where it changes, ask lilleman before changing any agent; lilleman makes the call again from the figures. The hook and the header already serve any agent that has the key.
### 319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
@@ -141,6 +133,17 @@ Probe first that `claude -p "/compact" --resume <id>` compacts a session: the Ag
153 `Judge cost:` entries total $109.76, a median of about $0.76 a verdict at every rung, and the cost follows the size of the records, not the transcript: about 76k characters in log and 188k in claude-config. In 224 of 251 recent verdicts (89%) the evidence sits in the worker's final message. A missed interrupt is the costly error, so the bar is zero changed interrupts and zero new rejections across the replay. The judge is about 4.5% of dollars, so the saving is small; each cycle also gets faster.
### 331. Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.
The architect's 🔀 answer on PR #493 (item 292): four decisions expire "when a new model joins the ladder". Fable 5.2 joined above every project model's rung, and none of these replays tested a rung above the project's, so the event fired with nothing new to test. Every later ladder rebuild would fire all four again.
- `review: each pass keeps a reviewer of its own`: the merge trials ran on Opus 5.5 at medium. Reword the "Valid" line to name that rung.
- `review: the principles audit runs only in the release review…`: its trial ran on the project's rung. Reword the line to name that rung.
- `routing: a review pass runs below the project's rung only on a rung whose replay of its rounds finds a share of their 🔴 and 🟡 findings no more than a margin below the project's rung's own replay…`: `review-replay` tests only rungs below the project's rung. Have the line expire when those rungs change, the event item 196 names ("until a cheaper model joins the ladder").
- `routing: on a model judge-replay covered, the judge runs…`: the replays tested the project's rung and below, and each rung above it that a review climbs to. Have the line expire when those rungs change.
lilleman approved rewording all four as above on 2026-11-12.
### 327. Have `larv-review` print a round commit's title prefix, counted from the round's report, and have rule-hook refuse a round title whose counts differ from it.
The 0.8.0 audit (item 284) measured, over the 35 chunks that shipped 0.7.0: 72 commit calls in 33 chunks titled `Answer <pass> round <n>: R 🔴, Y 🟡 (K misread); <what changed>`, about 11k command tokens, most of them the summary. Each worker counts R, Y and K by hand. All three are in the round's report, `<pass>-round-<n>.md` in the review folder: R and Y are its finding headings, and K counts the findings the reviewer marked `[misread]`, as `bin/review_rules/round_titles.py` says. That module parses the counts for `score-outcomes`, which reports them as goal 1.1's signal, and rule-hook's `round_cap.py` checks only the round number, so nothing checks today that the counts match the report. Of the 8 titles whose report the audit could match, none miscounted; one report wrote its markers at line start rather than in `###` headings, so the reports' format varies by reviewer.