Record the approved steps and the judge's rung, keep the reviewers' floor, and widen item 159 to every pass

This commit is contained in:
2026-10-16 11:53:36 +02:00
parent 614c17c3d1
commit b1b798371d
2 changed files with 10 additions and 12 deletions
+3 -3
View File
@@ -232,7 +232,7 @@ Valid while a mutation run on a chunk's changed lines stays cheap beside the rev
The rungs, least competent first: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models, read 2026-10-14), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). That cut drops Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index falls among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scores it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price.
A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. A model off the ladder routes nothing: claude runs it at the effort its settings give. A task's L steps its worker from the project's rung: 1 two rungs down, 2 one down, 3 none, 4 one up. Its E + I steps its reviewers up from the project's rung: 4 or less none, 5 one up, 6 two up, 7 three up, 8 four up. A step past either end stops there. A worker errs more the more it changes and the newer that is, which L scores; review must look harder the further a defect would escape and the more it would break, which E + I scores (goals 1.1 and 1.4). Size sets L, so a large change runs its worker higher. The steps spread the values this repo's plan held when rescored on 2026-10-16, L 1 to 4 and E + I 2 to 7, so a worker on most items runs at or below the project's rung, and a review climbs only where a defect would escape far or break much. Until then, from 2026-10-14, one band of R + C picked the worker's rung. The steps are a first guess: `score-outcomes` reports how well L, E and I each order finished chunks by review rounds and 🔴 findings, and the steps are re-tuned once 10 chunks have shipped under them. A change to the ladder or the steps is lilleman's.
A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. A model off the ladder routes nothing: claude runs it at the effort its settings give. A task's L steps its worker from the project's rung: 1 two rungs down, 2 one down, 3 none, 4 one up. Its E + I steps its reviewers up from the project's rung: 4 or less none, 5 one up, 6 two up, 7 three up, 8 four up. A step past either end stops there. A worker errs more the more it changes and the newer that is, which L scores; review must look harder the further a defect would escape and the more it would break, which E + I scores (goals 1.1 and 1.4). Size sets L, so a large change runs its worker higher. The steps spread the values this repo's plan held when rescored on 2026-10-16, L 1 to 4 and E + I 2 to 7, so a worker on most items runs at or below the project's rung, and a review climbs only where a defect would escape far or break much. Until then, from 2026-10-14, one band of R + C picked the worker's rung. lilleman approved these steps on 2026-10-16 as a first guess: `score-outcomes` reports how well L, E and I each order finished chunks by review rounds and 🔴 findings, and the steps are re-tuned once 10 chunks have shipped under them. A change to the ladder or the steps is lilleman's.
Valid while the index ranks these models, and while no newer model joins the catalog.
@@ -242,7 +242,7 @@ Valid while the index ranks these models, and while no newer model joins the cat
Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks would spend on work that never ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each reviewer, each comprehension seat and the security pass's subagent on the project's rung, through the Agent tool's model and effort. The worker runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. lilleman chose the project's rung, medium, knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers inherit the worker's higher rung. On 2026-10-15 lilleman dropped routing them by R + C instead: a worker climbs because its chunk proved harder than its score said, which is when a cheaper reviewer is the bigger risk. On 2026-10-16 lilleman routed them on their item's E + I as well, which may lift them above both rungs: how far a defect would escape, and what it would break, set how hard review must look (goal 1.1). loop-claude names that rung in the worker's prompt, and hands it to the worker's claude beside the project's and the worker's, so rule-hook holds each reviewer launch to it. A reviewer or seat that rules the work below standard may rerun one rung up: `review: a reviewer or comprehension seat that rules the work below standard`.
While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers inherit the worker's higher rung. On 2026-10-15 lilleman dropped routing them by R + C instead: a worker climbs because its chunk proved harder than its score said, which is when a cheaper reviewer is the bigger risk. On 2026-10-16 lilleman routed them on their item's E + I as well, which may lift them above both rungs: how far a defect would escape, and what it would break, set how hard review must look (goal 1.1). loop-claude names that rung in the worker's prompt, and hands it to the worker's claude beside the project's and the worker's, so rule-hook holds each reviewer launch to it. A low E + I never steps a reviewer below the project's rung: on 2026-10-16 lilleman chose to lower a pass only where a replay shows a lower rung finding every 🔴 and 🟡 the project's rung finds, which `todo.md` item 159 measures for each pass. A reviewer or seat that rules the work below standard may rerun one rung up: `review: a reviewer or comprehension seat that rules the work below standard`.
Valid until the trial on real chunks shows whether a worker below the project's rung holds quality.
@@ -300,7 +300,7 @@ On a model listed in `JUDGE_REPLAYED` in `bin/loop_claude/routing.py`, the judge
On 2026-10-15, `bin/judge-replay claude-opus-5-5` replayed the latest 15 archived verdicts of each kind from the Opus 5.5 projects, 39 in all. The reference judge, Opus 5.5 at the xhigh effort `settings.json` gave every judge before, reproduced 31 of them. The other 8 may rest on records or judge rules that have changed since; either way no rung is held to them. On the 31, Haiku 5.5 missed 7 at xhigh and 8 at max. That run tried each judge once. 4 of Haiku's misses, 1 at xhigh and 3 at max, misquoted the transcript, which the loop and `judge-replay` now retry; without them it missed 6 and 5. Sonnet 5.5 at high returned another kind on 3 interrupts, and Opus 5.5 at medium missed none. A missed interrupt lets the worker settle a decision that is lilleman's (goal 1.2), so the judge never runs below the project's rung.
Only Opus 5.5 was replayed, so the judge of a project on another model keeps the effort its settings give until `judge-replay` covers that model (goal 1.1). Every routed chunk on record had an item of R + C 9, so the replay could not show that the project's rung suffices for harder items, and the judge climbs with the routed item. Since 2026-10-16 it climbs with the item's E + I, as a review does: the judge checks the worker's decisions as the reviewers check its code, and a decision it lets through reaches lilleman as a defect review lets through reaches the users (goal 1.2). It runs above the project's rung only on an E + I of 5 or more (goal 1.4). An item the worker takes in beyond the routed one lifts the worker's rung but not the judge's, until `todo.md` item 153 lifts the judge too.
Only Opus 5.5 was replayed, so the judge of a project on another model keeps the effort its settings give until `judge-replay` covers that model (goal 1.1). Every routed chunk on record had an item of R + C 9, so the replay could not show that the project's rung suffices for harder items, and the judge climbs with the routed item. Since 2026-10-16, lilleman's call, it climbs with the item's E + I, as a review does: the judge checks the worker's decisions as the reviewers check its code, and a decision it lets through reaches lilleman as a defect review lets through reaches the users (goal 1.2). It runs above the project's rung only on an E + I of 5 or more (goal 1.4). An item the worker takes in beyond the routed one lifts the worker's rung but not the judge's, until `todo.md` item 153 lifts the judge too.
Valid while the archived verdicts show what a judge should return, and until a new model joins the ladder: replay then.
+7 -9
View File
@@ -19,8 +19,8 @@
| 153 | 0.2.0 | | decision | **Route the judge and the reviewers on the highest E + I in their chunk, an item the worker takes in included.** | 3 | 1 | 3 | 5 | 4 | 1.2 by 5 | 3.0 |
| 161 | 0.2.0 | | | **Have loop-claude run a preflight script before each fresh worker and put its findings in the worker's prompt, so the worker no longer spends tool calls finding the chunk's mechanical facts.** | 4 | 2 | 3 | 7 | 6 | 1.1 by 4, 1.4 by 6, 1.5 by 6 | 2.9 |
| 84 | 0.2.0 | | | **Check, at each release's scoring run, that seats in a paired ruling agree more often than two scoring runs on near-identical code do, as the paired-ruling decision assumes.** | 2 | 2 | 2 | 4 | 4 | 1.4 by 5 | 2.3 |
| 159 | 0.2.0 | | | **Run the prose pass on the lowest rung that a replay of archived prose rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.** | 3 | 2 | 2 | 5 | 5 | 1.4 by 5 | 1.4 |
| 73 | 0.2.0 | | | **Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.** | 3 | 2 | 3 | 6 | 5 | 1.4 by 4 | 1.1 |
| 159 | 0.2.0 | | decision | **Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.** | 4 | 2 | 3 | 7 | 6 | 1.4 by 7 | 1.1 |
| 80 | 0.3.0 | | defect | **State in README → Install that every machine taking this checkout must be a sandbox, as `settings.json`'s auto mode assumes.** | 1 | 2 | 1 | 2 | 1 | 2 by 4 | 17.5 |
| 21 | 0.3.0 | | defect | **Give `settings.json`'s model one writer, and sort its keys.** | 3 | 3 | 3 | 7 | 3 | 2 by 5 | 13.7 |
| 29 | 0.3.0 | | | **Make the contributor a standing member of every project's audience, unless its `## Audience` says maintainability does not matter.** | 2 | 2 | 3 | 5 | 2 | 1.1 by 5, 1.4 by 5 | 13.7 |
@@ -118,14 +118,6 @@ Compute these in one tool and give loop-claude's prompt its output as facts, so
Raised by the principles audit of PR #228: the `review: the comprehension check records …` entry in `docs/decisions.md` is valid only while that holds, and `docs/comprehension-history.md` keeps every vote, yet nothing reads the agreement rate. Scoring runs on near-identical code are rare now, so compare with those already in the history, such as PR #224's and PR #225's, or make a second run for the check.
### 159. Run the prose pass on the lowest rung that a replay of archived prose rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.
lilleman asked, on 2026-10-16, for the prose pass to run on the worker's rung or lower. Today `bin/loop_claude/cycle/prompts.py`'s `review_launches` keeps every reviewer at or above the project's rung while the worker runs below it, so a docs-only chunk such as item 140's, routed to Haiku 5.5, runs its prose reviewer on Opus 5.5.
A missed prose finding breaks no behaviour and a later change fixes it cheaply, so the prose pass is the likeliest reviewer to run cheaper without lowering quality (goal 1.4). Whether it does is measured, not assumed, as `docs/decisions.md` → `routing: the stability reviewer stays on the project's rung…` measures the stability reviewer: replay archived prose rounds on the code each reviewed, first on the project's rung, then on each rung below it, and take the lowest rung that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again. Generalise `bin/stability-replay` to take the pass, in place of a second replay tool. Record the run and the chosen rung in a `routing:` decision, valid until a new model joins the ladder or the prose reviewer's prompt changes.
The comprehension seats found the function `review_launches` replaced hard: give the prose reviewer a row of its own in `review_launches`' table of review kinds, rather than a branch.
### 73. Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.
2026-10-08, lilleman: rescored from A10/G8 to A4/G4 on the measurement below, so other items that lower token use go first.
@@ -139,6 +131,12 @@ The reports would live in the session's scratchpad, such as `<scratchpad>/larv-r
This risks goal 1.1: a worker fixing from a one-line summary can miss context it needs.
### 159. Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.
lilleman asked, on 2026-10-16, for the prose pass to run on the worker's rung or lower, and the same day, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. Today `bin/loop_claude/cycle/prompts.py`'s `review_launches` keeps every reviewer at or above the project's rung whatever the item's E + I, so a docs-only chunk such as item 140's, routed to Haiku 5.5, runs its reviewers on Opus 5.5.
Whether a pass runs cheaper without lowering quality (goal 1.4 against 1.1) is measured, not assumed, as `docs/decisions.md` → `routing: the stability reviewer stays on the project's rung…` measures the stability reviewer: replay a pass's archived rounds on the code each reviewed, first on the project's rung, then on each rung below it, and take the lowest rung that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again. A missed prose finding breaks no behaviour and a later change fixes it cheaply, so replay the prose pass first. Generalise `bin/stability-replay` to take the pass, in place of a replay tool per pass. Record each run and the rung it chose in a `routing:` decision, valid until a new model joins the ladder or the pass's prompt changes. Give each pass a row of its own in `review_launches`' table of review kinds, rather than a branch.
### 80. State in README → Install that every machine taking this checkout must be a sandbox, as `settings.json`'s auto mode assumes.
A defect: goal 2, keeping the setup shareable between computers, does not hold while a new machine is never told it must be a sandbox. Raised by the principles audit of PR #224. The `settings.json` decision is valid while every machine taking this checkout is a sandbox, but Install asks for a sandbox only for `loop-claude`.