Merge remote-tracking branch 'origin/main' into item-308-count
CI / tests (pull_request) Successful in 2m39s
CI / tests (pull_request) Successful in 2m39s
This commit was merged in pull request #508.
This commit is contained in:
@@ -12,8 +12,7 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 340 | 0.9.0 | | question | **Decide whether a reviewer may start without `CLAUDE.md` where its replay misses no more findings than an unchanged replay does.** | 2 | 2 | 3 | 5 | 4 | 1.4 by 5 | 1.7 |
|
||||
| 331 | 0.9.0 | | question | **Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.** | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.9 |
|
||||
| 340 | 0.9.0 | | owner | **Let the stability reviewer start without `CLAUDE.md`, as principles-auditor does, once a replay with a control has been run on every pass and reported to lilleman.** | 2 | 2 | 3 | 5 | 4 | 1.4 by 5 | 1.7 |
|
||||
| 319 | 0.9.0 | | | **Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 2, 1.4 by 2 | 1.4 |
|
||||
| 309 | 0.9.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.3 |
|
||||
| 299 | 0.9.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.2 |
|
||||
@@ -23,6 +22,7 @@
|
||||
| 341 | 0.9.0 | | | **Hand each worker's claude a lower Read limit through `CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS`, kept only where replays of the archived review rounds find what two unchanged replays find.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.2 |
|
||||
| 310 | 0.9.0 | | | **Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 1.0 |
|
||||
| 298 | 0.9.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 1.0 |
|
||||
| 331 | 0.9.0 | | | **Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.** | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.9 |
|
||||
| 327 | 0.9.0 | | | **Have `larv-review` print a round commit's title prefix, counted from the round's report, and have rule-hook refuse a round title whose counts differ from it.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 1, 1.4 by 1 | 0.7 |
|
||||
| 303 | 0.10.0 | | defect | **Archive each judge run's prompt and json reply beside the verdict it gave, so judge cost, replays and backups cover the judge as they cover the worker.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3, 3 by 2 | 4.8 |
|
||||
| 304 | 0.10.0 | 303 | defect | **Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 2, 3 by 2 | 4.4 |
|
||||
@@ -68,24 +68,16 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 340. Decide whether a reviewer may start without `CLAUDE.md` where its replay misses no more findings than an unchanged replay does.
|
||||
### 340. Let the stability reviewer start without `CLAUDE.md`, as principles-auditor does, once a replay with a control has been run on every pass and reported to lilleman.
|
||||
|
||||
Item 315's gate kept `omitClaudeMd` only where a replay without `CLAUDE.md` found every finding that two unchanged replays both found. Read so, five passes failed it on 2026-11-12: stability missed 1 of 11, prose 1 of 24, product-owner 1 of 4, architecture 4 of 27, and overrides had no archived round to replay. The decision `agents/: a reviewer starts without CLAUDE.md only where a replay of its pass missed none of the findings two unchanged replays both found` holds the figures.
|
||||
Item 315's gate kept `omitClaudeMd` only where a replay without `CLAUDE.md` found every finding that two unchanged replays both found. Read so, five passes failed it on 2026-11-12: stability missed 1 of 11, prose 1 of 24, product-owner 1 of 4, architecture 4 of 27, and overrides had no archived round to replay. The decision `agents/: a reviewer starts without CLAUDE.md only where a replay of its pass missed none of the findings two unchanged replays both found` holds the figures. The unchanged reviewer finds only about 70% of its findings again, so that gate can't tell a loss from the reviewer's own noise.
|
||||
|
||||
The question for lilleman: does a miss that small show a loss, or the reviewer's own noise? The unchanged reviewer finds only about 70% of its findings again, so a third unchanged replay would also miss some that the other two found, and the gate as written can't tell the two apart.
|
||||
lilleman's call, 2026-11-12: principles-auditor and stability-reviewer start without `CLAUDE.md`; the other reviewers keep it. The call is provisional until the data below is in.
|
||||
|
||||
Recommended: give `review-replay --omit-claude-md` a control. It replays each round three times unchanged and once without `CLAUDE.md`, and prints for each unchanged replay how many findings it missed that the other two found. The reviewer may start without `CLAUDE.md` where its own count is no higher than the highest of those three. Then run it for the five passes, and add `omitClaudeMd: true` to each agent that passes; the hook and the header already serve any agent that has it. Each run costs four replays of 10 rounds on the project's model.
|
||||
|
||||
### 331. Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.
|
||||
|
||||
The architect's 🔀 answer on PR #493 (item 292): four decisions expire "when a new model joins the ladder". Fable 5.2 joined above every project model's rung, and none of these replays tested a rung above the project's, so the event fired with nothing new to test. Every later ladder rebuild would fire all four again.
|
||||
|
||||
- `review: each pass keeps a reviewer of its own`: the merge trials ran on Opus 5.5 at medium. Reword the "Valid" line to name that rung.
|
||||
- `review: the principles audit runs only in the release review…`: its trial ran on the project's rung. Reword the line to name that rung.
|
||||
- `routing: a review pass runs below the project's rung only on a rung whose replay of its rounds finds a share of their 🔴 and 🟡 findings no more than a margin below the project's rung's own replay…`: `review-replay` tests only rungs below the project's rung. Have the line expire when those rungs change, the event item 196 names ("until a cheaper model joins the ladder").
|
||||
- `routing: on a model judge-replay covered, the judge runs…`: the replays tested the project's rung and below, and each rung above it that a review climbs to. Have the line expire when those rungs change.
|
||||
|
||||
Recommendation: reword all four as above. They are decisions the repo held before item 292, so the rewording is lilleman's call.
|
||||
1. Give `review-replay --omit-claude-md` a control. It replays each round three times unchanged and once without `CLAUDE.md`, and prints for each unchanged replay how many findings it missed that the other two found. Each run costs four replays of 10 rounds on the project's model.
|
||||
2. Run it for every pass that has archived rounds: principles, stability, prose, product-owner and architecture.
|
||||
3. Report each pass's figures to lilleman, saying whether they change the picture: a pass passes where its count without `CLAUDE.md` is no higher than the highest of its three unchanged counts. The picture changes where stability or principles fails, or another pass passes.
|
||||
4. Where the picture holds, add `omitClaudeMd: true` to stability-reviewer, and record the figures and the call in the decision above. Where it changes, ask lilleman before changing any agent; lilleman makes the call again from the figures. The hook and the header already serve any agent that has the key.
|
||||
|
||||
### 319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
|
||||
|
||||
@@ -141,6 +133,17 @@ Probe first that `claude -p "/compact" --resume <id>` compacts a session: the Ag
|
||||
|
||||
153 `Judge cost:` entries total $109.76, a median of about $0.76 a verdict at every rung, and the cost follows the size of the records, not the transcript: about 76k characters in log and 188k in claude-config. In 224 of 251 recent verdicts (89%) the evidence sits in the worker's final message. A missed interrupt is the costly error, so the bar is zero changed interrupts and zero new rejections across the replay. The judge is about 4.5% of dollars, so the saving is small; each cycle also gets faster.
|
||||
|
||||
### 331. Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.
|
||||
|
||||
The architect's 🔀 answer on PR #493 (item 292): four decisions expire "when a new model joins the ladder". Fable 5.2 joined above every project model's rung, and none of these replays tested a rung above the project's, so the event fired with nothing new to test. Every later ladder rebuild would fire all four again.
|
||||
|
||||
- `review: each pass keeps a reviewer of its own`: the merge trials ran on Opus 5.5 at medium. Reword the "Valid" line to name that rung.
|
||||
- `review: the principles audit runs only in the release review…`: its trial ran on the project's rung. Reword the line to name that rung.
|
||||
- `routing: a review pass runs below the project's rung only on a rung whose replay of its rounds finds a share of their 🔴 and 🟡 findings no more than a margin below the project's rung's own replay…`: `review-replay` tests only rungs below the project's rung. Have the line expire when those rungs change, the event item 196 names ("until a cheaper model joins the ladder").
|
||||
- `routing: on a model judge-replay covered, the judge runs…`: the replays tested the project's rung and below, and each rung above it that a review climbs to. Have the line expire when those rungs change.
|
||||
|
||||
lilleman approved rewording all four as above on 2026-11-12.
|
||||
|
||||
### 327. Have `larv-review` print a round commit's title prefix, counted from the round's report, and have rule-hook refuse a round title whose counts differ from it.
|
||||
|
||||
The 0.8.0 audit (item 284) measured, over the 35 chunks that shipped 0.7.0: 72 commit calls in 33 chunks titled `Answer <pass> round <n>: R 🔴, Y 🟡 (K misread); <what changed>`, about 11k command tokens, most of them the summary. Each worker counts R, Y and K by hand. All three are in the round's report, `<pass>-round-<n>.md` in the review folder: R and Y are its finding headings, and K counts the findings the reviewer marked `[misread]`, as `bin/review_rules/round_titles.py` says. That module parses the counts for `score-outcomes`, which reports them as goal 1.1's signal, and rule-hook's `round_cap.py` checks only the round number, so nothing checks today that the counts match the report. Of the 8 titles whose report the audit could match, none miscounted; one report wrote its markers at line start rather than in `###` headings, so the reports' format varies by reviewer.
|
||||
|
||||
Reference in New Issue
Block a user