Answer prose round 1: 0 🔴, 6 🟡; state the replayed rung's conditions in the decision's title, separate the seats in the reviewers' title, reword the validity clause, the lost-input error, the changelog and the module docstrings, and fix the nits

This commit is contained in:
2026-10-18 18:46:52 +02:00
parent cfa16c65b5
commit 49e86fdbb9
14 changed files with 30 additions and 31 deletions
+2 -2
View File
@@ -40,14 +40,14 @@ Decisions live in `docs/decisions.md`:
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
- routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass, the seats and the security pass always on the project's
- routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass; the seats and the security pass always on the project's
- routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, or when the judge reports a taken-in item that picks a higher rung or an escalation trigger the worker cites
- routing: loop-claude measures review rounds and unfinished verdicts per rung every 20 cycles, and calls for a routing change when, of two rungs with 5 or more chunks each, the lower took more of either than the one above, or when 5 or more subagents of one type had reports longer than the tool results they read
- routing: a worker delegates a task only where it needs just the task's conclusion, and each subagent runs on the rung its task's L picks from the project's rung, never below the project's rung when it files or scores todo.md items
- routing: a worker hands a task one rung up on each escalation trigger, up to two rungs above the rung its item picks unless the human agrees, and opens the subagent's prompt with the trigger so the routing figures count it
- routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, steps one rung up as the judge interrupts or quotes an answer, or next cycle, up to two rungs above the rung its item picks
- routing: on a model judge-replay covered, the judge runs on the rung a review would get from the highest E + I among its chunk's items, never below the project's rung
- routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and there even below the worker's; no pass has such a rung
- routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and only where the E + I steps no review up and the worker runs no higher than the project's rung, then even below the worker's; no pass has such a rung
## Working rules
+1 -1
View File
@@ -2,7 +2,7 @@
## Unreleased
- `stability-replay <model>` is now `review-replay <pass> <model>`, and replays any pass an agent under `agents/` reviews, such as `review-replay prose claude-opus-5-5`. It now tries every rung below the project's, down to the ladder's lowest. Its run on the prose pass of Opus 5.5 projects found no rung below the project's that finds every 🔴 and 🟡 the project's rung finds: Sonnet 5.5 at high missed 12 of 27. So every pass stays where it runs. A future run that finds such a rung lowers only that pass, below the worker's rung too, wherever the chunk's E + I steps no review up and the worker runs no higher than the project's rung.
- `stability-replay <model>` is now `review-replay <pass> <model>`, and replays any pass an agent under `agents/` reviews, such as `review-replay prose claude-opus-5-5`. It now tries every rung below the project's, down to the ladder's lowest. Its runs on the prose and stability passes of Opus 5.5 projects found no rung below the project's that finds every 🔴 and 🟡 the project's rung finds: Sonnet 5.5 at high missed 12 of 27 prose findings and 11 of 20 stability findings. So every pass stays where it runs. A rung a future run finds lowers only its pass, once `REPLAYED` in `bin/model_routing/ladder.py` names it, and only where the chunk's E + I steps no review up and the worker runs no higher than the project's rung; there it may run below the worker's rung.
- `chunk-preflight` names the release it finds finished and ready to cut, or to be planned first, wherever todo.md has named that release since the last cut.
- Each reviewer under `agents/` now writes its full report to a file that larv-review names in the reviewer's prompt, in a folder per chunk under `~/.local/state/larv-review/`. It returns only its verdict and one line per finding, so a review takes less of the worker's context. The worker reads a finding's full entry before it fixes or answers a 🔴 or 🟡. Each reviewer now holds Write. `rule-hook` refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`; no override waives this rule. Each chunk's folder stays after the merge, so `review-replay` can replay a round that reads an earlier round's report. It replays on a copy of the folder.
- `loop-claude` asks an interrupt's question on its own terminal and resumes the worker headless with your one-line answer, so the loop no longer waits for you to leave a claude session. Type `/talk` instead to open the worker's session in claude, as every interrupt did before, to read the diff or discuss first. A release ready to cut opens that session at once. Pressing Enter on an empty answer stops the loop, exit 2. Keys typed before the question shows are dropped, and a pasted answer keeps only its first line; type `/talk` for a longer one. The judge counts that answer as yours, like one typed in claude: the chunk stays open until the answer is recorded, where a later chunk could need it.
+1 -1
View File
@@ -110,7 +110,7 @@ pytest_add_cli_args=-k
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and `loop_claude` runs `chunk-preflight` as a process of its own. `judge_replay` and `review_replay` share `rung_replay/`: the rung search, and reading the arguments, the candidate rungs and the exits. It reads `loop_claude`'s retry count and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and `loop_claude` runs `chunk-preflight` as a process of its own. `judge_replay` and `review_replay` share `rung_replay/`: the rung search, the argument reader, the candidate rungs and the exits. `rung_replay/` reads `loop_claude`'s retry count and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude, `rule_hook`, both replay tools and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
## Tracking a new file
+1 -1
View File
@@ -60,7 +60,7 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
return failed(TOOL, f"no archived verdicts of projects on {anchor.model} in {archive_root(home, env)}; run loop-claude on a project on that model first")
cases = sample(cases, PER_KIND)
reference = Rung(anchor.model, REFERENCE_EFFORT)
# The candidates stop at the lowest rung a worker routes to: judge-replay has no route below it to change.
# The candidates stop at the lowest rung a worker routes to: no route uses a rung below it for the judge.
replay = replayed(cases, reference, down_from(anchor, lowest_worker_rung(anchor)), lambda found, rung: rule(found, rung, home / ".claude", effects), reproduced, "verdicts")
if isinstance(replay, Limited):
return limited_exit(TOOL, replay)
+3 -3
View File
@@ -10,7 +10,7 @@ QUOTE_SOURCES = listed(("README.md's Goals and Audience", *RECORD_FILES, *HUMAN_
# Where the worker may write an answer the human typed, so it counts as recorded.
ANSWER_DESTINATIONS = "README.md's Goals or Audience, " + ", ".join(RECORD_FILES) + ", todo.md or a comment"
ANSWERS_RECORDED = f"every answer the human typed in it that a later chunk could need is recorded in {ANSWER_DESTINATIONS} at the one place it governs"
# Each kind of review, as review_launches names those it launches: a row per pass, since a replay may lower one pass alone, while "each reviewer" names the passes it left.
# Each kind of review, as review_launches names those it launches: a row per pass, since a replay may lower one pass alone. "each reviewer" names every pass no replay lowered.
REVIEWS: tuple[tuple[str, ReviewKind], ...] = (*((f"each {agent}", kind) for kind, agent in REVIEWERS.items()), ("each comprehension seat", "seat"), ("the security pass's subagent", "security"))
@@ -55,7 +55,7 @@ def review_launches(route: Route, anchor: Rung | None) -> str:
def shared_launches(rungs: dict[ReviewKind, Rung], reviewers: Rung, route: Route, anchor: Rung) -> str:
"""A sentence for each rung other than the worker's that a review runs on, naming the passes no replay lowered, which run on reviewers, "each reviewer"."""
"""A sentence for each rung other than the worker's that a review runs on. "each reviewer" names every pass no replay lowered; those run on reviewers."""
together = {"each reviewer": reviewers} | {reviews: rungs[kind] for reviews, kind in REVIEWS if kind not in REVIEWERS}
launched: dict[Rung, list[str]] = {}
for reviews, rung in together.items():
@@ -65,7 +65,7 @@ def shared_launches(rungs: dict[ReviewKind, Rung], reviewers: Rung, route: Route
def lowered(rungs: dict[ReviewKind, Rung], reviewers: Rung) -> list[tuple[str, Rung]]:
"""How REVIEWS names each pass a replay lowered below reviewers, the rung of the passes it left, and the rung the replay chose."""
"""Each pass a replay put below reviewers, the rung of every other pass, as REVIEWS names it, with the rung the replay chose."""
return [(reviews, rungs[kind]) for reviews, kind in REVIEWS if kind in REVIEWERS and rungs[kind] != reviewers]
+2 -2
View File
@@ -69,8 +69,8 @@ bin/model_routing/ladder.py, below it for a low L; without such an item, or with
unscored, the chunk runs on the model's cheapest rung. The
item's E + I picks the reviewers' rung, stepping up from the cheapest rung by REVIEW_STEPS there,
never below the worker's rung. Where it steps none and the worker runs no higher than the cheapest
rung, a pass that a review-replay run lowered runs on the rung REPLAYED there names, even below the
worker's. The worker's first prompt names the rung and the item, and asks the
rung, a pass REPLAYED there names runs on the rung it names, the lowest a review-replay run found
finding what the cheapest rung finds, even below the worker's. The worker's first prompt names the rung and the item, and asks the
worker to name each item the chunk takes, with its L and E + I, and to count the item's size again
once it has read the code (prompts.recount). The prompt names the rung of each review that runs on
a rung other than the worker's: the reviewers' rung or the replay's for each pass's reviewer, and
+2 -2
View File
@@ -65,7 +65,7 @@ LADDER = (
WORKER_STEPS = {1: -2, 2: -1, 3: 0, 4: 1}
# The rungs a task's E + I steps a review up from the project's rung; an E + I under 5 steps none.
REVIEW_STEPS = {5: 1, 6: 2, 7: 3, 8: 4}
# Each pass an agent under agents/ reviews, in the order of that table in SKILL.md, and that agent.
# Each pass an agent under agents/ reviews, in the order of SKILL.md's pass table, and that agent.
REVIEWERS: dict[ReviewedPass, str] = {"architecture": "systems-architect", "principles": "principles-auditor", "overrides": "override-reviewer", "stability": "stability-reviewer", "product-owner": "product-owner-reviewer", "prose": "prose-reviewer"}
# The rung a pass's replay chose for the projects on a model: docs/decisions.md → `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds`.
REPLAYED: dict[tuple[ReviewedPass, str], Rung] = {}
@@ -104,7 +104,7 @@ def stepped_up(rung: Rung) -> Rung:
def review_rungs(project: Rung, worker: Rung, escape_and_impact: int) -> dict[ReviewKind, Rung]:
"""The rung each kind of review in a chunk runs on, where its worker runs on worker and its item has that E + I.
A pass's reviewer runs on reviewers_rung. Where the E + I steps no review up and the worker runs no higher than the project's rung, a pass whose replay chose a rung for the project's model runs on that rung instead, even below the worker's: the replay measured the project's rung, never a stepped one or a climbed worker's chunk. A seat and the security pass run on the project's rung, so the seats score every chunk on one rung, and the security pass spends no more than the project chose.
A pass's reviewer runs on reviewers_rung. Where the E + I steps no review up and the worker runs no higher than the project's rung, a pass whose replay chose a rung for the project's model runs on that rung instead, even below the worker's: the replay compared reviews on the project's rung only, never on a rung an E + I stepped up, nor of a chunk whose worker climbed above the project's rung. A seat and the security pass run on the project's rung, so the seats score every chunk on one rung, and the security pass spends no more than the project chose.
"""
rungs: dict[ReviewKind, Rung] = {**dict.fromkeys(REVIEWERS, reviewers_rung(project, worker, escape_and_impact)), "seat": project, "security": project}
if escape_and_impact not in REVIEW_STEPS and position(worker) <= position(project):
+1 -1
View File
@@ -1,4 +1,4 @@
"""The rungs loop-claude hands a worker's claude in its environment, so rule-hook holds each review launch to them: docs/decisions.md → `routing:`.
"""The rungs and the E + I loop-claude hands a worker's claude in its environment, from which rule-hook works out the rung of each review launch: docs/decisions.md → `routing:`.
loop-claude sets these variables for its worker and for the claude an interrupt resumes; a human's shell does not, and rule-hook then checks no review launch."""
import typing
+3 -4
View File
@@ -2,8 +2,7 @@
usage: review-replay <pass> <model>
<pass> is a pass an agent under agents/ reviews, named as skills/larv-review/SKILL.md → "Passes"
names it in lower case: architecture, principles, overrides, stability, product-owner or prose.
<pass> is one of architecture, principles, overrides, stability, product-owner or prose.
<model> is a project model on the routing ladder, such as claude-opus-5-5. The tool replays the
latest 10 archived rounds of the pass whose report holds a 🔴 or 🟡 finding, from the projects
that ~/.claude/loop-claude-models.json runs on the model. It replays each round on the code it
@@ -15,7 +14,7 @@ chunk has since removed. A round whose code it cannot find in this way is not re
names it and why. The prompt also names a copy of the chunk's review folder in place of that
folder, holding only the reports written after the chunk began and before the round did. A replay
finds nothing where its prompt still names the review folder, or names a file there, other than a
round's report, that was written again after the round began.
round's report, that the folder did not hold from the chunk's start until the round's.
It first replays every round on the project's rung, the reference reviewer. Findings on record
that the reference reviewer does not raise again may rest on code or rules that have changed
@@ -214,7 +213,7 @@ def reviewed_in(found: Round, code: Code, rung: Rung, copies: Copies, home: path
if isinstance(prompt, Failure):
return prompt
if lost := lost_inputs(prompt, copies.reports, lambda name: (pathlib.Path(copies.reports) / name).exists()):
return Failure(f"the prompt names {', '.join(lost)}, which the review folder holds only as written after the round began")
return Failure(f"the prompt names {', '.join(lost)}, which the review folder did not hold from the chunk's start until the round's")
if isinstance(cloned := clone_reviewed(found, code, copies.clone, effects), Failure):
return cloned
reply = claude_reply(effects.claude, ["--agent", found.reviewer, *rung.flags()], copies.clone, prompt, "claude")
+1 -1
View File
@@ -325,7 +325,7 @@ class Main(unittest.TestCase):
code, out, err = self.run_main(["stability", "claude-opus-5-5"], unused)
self.assertEqual(code, 0, err)
self.assertEqual(self.reviews, [])
self.assertIn(f"on {REFERENCE} found nothing: the prompt names branch.diff, which the review folder holds only as written after the round began", err)
self.assertIn(f"on {REFERENCE} found nothing: the prompt names branch.diff, which the review folder did not hold from the chunk's start until the round's", err)
def test_finds_nothing_and_leaves_no_folder_behind_when_it_cannot_copy_the_reports(self):
folder = self.review_folder_with_round_1()
+1 -1
View File
@@ -6,7 +6,7 @@ from rule_hook.rules import Breach
from rule_hook.call import Call
from review_folder import REPORTS
# A reviewer is an agent under agents/, as CLAUDE.md → "Vocabulary" defines it; read from the folder, so an agent the routing tables miss is still confined.
# Each agent under agents/, read from the folder, so an agent the routing tables miss is still confined.
REVIEWERS = frozenset(path.stem for path in (pathlib.Path(__file__).resolve().parents[2] / "agents").glob("*.md"))
+1 -1
View File
@@ -1,4 +1,4 @@
"""Refuse an Agent call that launches a review below the rung it needs, worked out from the rungs loop-claude hands its worker: docs/decisions.md → `routing:`."""
"""Refuse an Agent call that launches a review below the rung it needs, worked out from the rungs and the E + I loop-claude hands its worker: docs/decisions.md → `routing:`."""
import typing
from rule_hook.rules import Breach
+8 -8
View File
@@ -260,15 +260,15 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
Valid while the index ranks these models, and while no newer model joins the catalog.
## routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass, the seats and the security pass always on the project's
## routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass; the seats and the security pass always on the project's
2026-10-14, lilleman. Serves goals 1.1 and 1.4.
Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks would spend on work that never ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each comprehension seat and the security pass's subagent on the project's rung, and each reviewer there too unless the highest E + I among its items picks a higher rung, through the Agent tool's model and effort. The worker runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. lilleman chose the project's rung, medium, knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks would spend on work that never ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each comprehension seat and the security pass's subagent on the project's rung, and each reviewer there too unless the highest E + I among its items picks a higher rung or a replay lowered its pass, through the Agent tool's model and effort. The worker runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. lilleman chose the project's rung, medium, knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers never run below the worker's rung while it runs above the project's: a worker climbs because its chunk proved harder than its score said, which is when a cheaper reviewer is the bigger risk. A reviewer or seat that rules the work below standard may rerun one rung up: `review: a reviewer or comprehension seat that rules the work below standard`.
On 2026-10-16 lilleman routed the reviewers on their item's E + I as well, on either side of the project's rung, which may lift them above both the worker's rung and the project's: how far a defect would escape, and what it would break, set how hard review must look (goal 1.1). loop-claude names that rung in the worker's prompt. It hands the E + I to the worker's claude beside the project's rung and the worker's, so rule-hook works out each review's rung as the prompt does and holds each launch to it. A low E + I never steps a reviewer below the project's rung: only a replay lowers a pass, as `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and there even below the worker's` records. An item the worker takes in beyond the routed one counts too, since its E + I scores its defects as the routed item's scores theirs. The worker's prompt names the reviewers' rung each higher E + I picks, so the worker reviews a taken-in item on it in the same run. The judge reports the highest E + I the worker states for a taken-in item. From the next cycle loop-claude routes on that E + I, and archives it with the route so a rerun keeps it.
On 2026-10-16 lilleman routed the reviewers on their item's E + I as well, on either side of the project's rung, which may lift them above both the worker's rung and the project's: how far a defect would escape, and what it would break, set how hard review must look (goal 1.1). loop-claude names that rung in the worker's prompt. It hands the E + I to the worker's claude beside the project's rung and the worker's, so rule-hook works out each review's rung as the prompt does and holds each launch to it. A low E + I never steps a reviewer below the project's rung: only a replay lowers a pass, as `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds…` records. An item the worker takes in beyond the routed one counts too, since its E + I scores its defects as the routed item's scores theirs. The worker's prompt names the reviewers' rung each higher E + I picks, so the worker reviews a taken-in item on it in the same run. The judge reports the highest E + I the worker states for a taken-in item. From the next cycle loop-claude routes on that E + I, and archives it with the route so a rerun keeps it.
Valid until the trial on real chunks shows whether a worker below the project's rung holds quality.
@@ -334,13 +334,13 @@ Only Opus 5.5 was replayed, so the judge of a project on another model keeps the
Valid while the archived verdicts show what a judge should return, and until a new model joins the ladder: replay then.
## routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and there even below the worker's; no pass has such a rung
## routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and only where the E + I steps no review up and the worker runs no higher than the project's rung, then even below the worker's; no pass has such a rung
2026-10-15, lilleman proposed running the stability reviewer on the worker's rung, and the agent proposed the replay that gates it. On 2026-10-16 lilleman asked for the prose pass to run on the worker's rung or lower, and, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. Serves goal 1.4 held to goal 1.1: a cheaper reviewer saves money only where it misses nothing.
2026-10-15, lilleman proposed running the stability reviewer on the worker's rung, and the agent proposed the replay that gates it. On 2026-10-16 lilleman asked for the prose pass to run on the worker's rung or lower, and, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. Serves goal 1.4 without giving up goal 1.1: a cheaper reviewer saves money only where it misses nothing.
`bin/review-replay <pass> <model>` replays a pass's archived rounds on the code each reviewed. It replays them first on the project's rung, then on each rung below it, highest first, down to the ladder's lowest. It stops at the first rung that misses a finding the project's rung found again. A rung qualifies only where it and each rung above it find every such 🔴 and 🟡.
`REPLAYED` in `bin/model_routing/ladder.py` holds each qualifying rung, by pass and project model. Where the chunk's E + I steps no review up and the worker runs no higher than the project's rung, the pass runs on that rung, even below the worker's. Otherwise the pass runs where every other pass does: the replay measured only reviews on the project's rung, never a stepped one or a climbed worker's chunk. loop-claude hands the worker's claude the reviews' E + I in `LOOP_CLAUDE_ESCAPE_AND_IMPACT`, so rule-hook works out each pass's rung with `review_rungs`, as the worker's prompt does.
`REPLAYED` in `bin/model_routing/ladder.py` holds each qualifying rung, by pass and project model. Where the chunk's E + I steps no review up and the worker runs no higher than the project's rung, the pass runs on that rung, even below the worker's. Otherwise the pass runs where every other pass does: the replay compared reviews on the project's rung only, never reviews on a rung an E + I stepped up, nor of a chunk whose worker climbed above the project's rung.
The runs, each on the latest 10 rounds of Opus 5.5 projects that hold a 🔴 or 🟡:
@@ -348,6 +348,6 @@ The runs, each on the latest 10 rounds of Opus 5.5 projects that hold a 🔴 or
- 2026-10-18, prose: 39 findings. The project's rung found 27 of them again. Sonnet 5.5 at high missed 12 of those 27. No rung qualifies.
- 2026-10-18, stability again, since that day's change to the reviewer prompts expired the first run: 29 findings. The project's rung found 20 of them again. Sonnet 5.5 at high missed 11 of those 20, 1 of them 🔴. No rung qualifies.
The prose pass went first: a missed prose finding breaks no behaviour, and a later change fixes it cheaply. Sonnet 5.5 missed nearly half of what the project's rung found again, in the prose pass as in the stability pass. Nothing suggests the other passes' findings are easier to find. So the architecture, principles, overrides and product-owner passes are not replayed yet: a replay would spend money with little chance of a qualifying rung (goal 1.4). `todo.md` item 196 keeps their replays in the plan, so the ranking, not this chunk, decides when they run. Every pass runs where `routing: a chunk's worker runs on the rung its item's L picks` puts every reviewer.
The prose pass went first: a missed prose finding breaks no behaviour, and a later change fixes it cheaply. Sonnet 5.5 missed 44 to 55% of what the project's rung found again, in the prose pass as in both stability runs. Nothing suggests the other passes' findings are easier to find. So the architecture, principles, overrides and product-owner passes are not replayed yet: a replay would spend money with little chance of a qualifying rung (goal 1.4). `todo.md` item 196 keeps their replays in the plan. Every pass runs where `routing: a chunk's worker runs on the rung its item's L picks` puts every reviewer.
Valid until a new model joins the ladder, or a pass's reviewer prompt changes: replay that pass then, and once a new model joins, each pass not yet replayed.
Valid until a new model joins the ladder, which calls for a replay of every pass, or a pass's reviewer prompt changes, which calls for a replay of that pass.
+3 -3
View File
@@ -40,7 +40,7 @@
| 21 | 0.3.0 | | defect | **Give `settings.json`'s model one writer, and sort its keys.** | 3 | 3 | 3 | 7 | 3 | 2 by 5 | 6.6 |
| 190 | 0.3.0 | | | **Rewrite the `docs/decisions.md` entries `todo.md: one table …` and `loop-claude: a judge decides …` as the current state, so a reader no longer replays superseded paragraphs to learn what holds.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 3, 1.4 by 2 | 6.3 |
| 171 | 0.3.0 | | decision | **Group the routing figures' review rounds by the reviewers' rung as well as the worker's, so a call blames a rung only for rounds its own reviews raised.** | 3 | 2 | 2 | 5 | 4 | 1.2 by 3, 1.4 by 3 | 4.9 |
| 196 | 0.3.0 | | decision | **Replay the architecture, principles, overrides and product-owner passes with `review-replay` on Opus 5.5 projects, and record each run, and any rung it qualifies in `REPLAYED`, in the review-replay routing decision.** | 2 | 2 | 3 | 5 | 2 | 1.4 by 3 | 4.8 |
| 196 | 0.3.0 | | decision | **Replay the architecture, principles, overrides and product-owner passes with `review-replay` on Opus 5.5 projects, record each run in the review-replay routing decision, and add any rung a run qualifies to `REPLAYED`.** | 2 | 2 | 3 | 5 | 2 | 1.4 by 3 | 4.8 |
| 181 | 0.3.0 | | principle | **Have the test of a quoted answer's step-up assert the stepped rung's facts as literals, not through `prompts.rungs_held`.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 4.1 |
| 192 | 0.3.0 | | principle | **Sort `.gitignore`'s allowlist block, from `!.claude/` to `!workflow-bench/test_review_git.py`, by code point, so rule-hook's `ignorefile-sorted` checks later edits to it again.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 4.1 |
| 193 | 0.3.0 | | | **Narrow `readme_goals`'s mark to `Mark` by a typed lookup in place of `typing.cast`, so a mark added to the regex but not the lookup fails at run time, and one added to the lookup but not `Mark` fails mypy.** | 1 | 1 | 3 | 3 | 1 | 1.1 by 2 | 4.1 |
@@ -215,9 +215,9 @@ Found by the prose review of PR #312, which judged both entries to read as amend
Raised by the goals audit of PR #296. `bin/routing_figures/measure.py` groups a chunk's review rounds by the rung its worker started on, and calls for a routing change when a lower rung took more of them. Since PR #296 the reviewers may run above the worker's rung, on the one their item's E + I picks, and no figure groups rounds by it, though the archived worker line names the E + I it follows from. A cheap worker on an item of high E + I gets stronger reviewers, who may raise more rounds, and a call could then blame the worker's rung for them: an interrupt of lilleman that settles nothing (goal 1.2).
### 196. Replay the architecture, principles, overrides and product-owner passes with `review-replay` on Opus 5.5 projects, and record each run, and any rung it qualifies in `REPLAYED`, in the review-replay routing decision.
### 196. Replay the architecture, principles, overrides and product-owner passes with `review-replay` on Opus 5.5 projects, record each run in the review-replay routing decision, and add any rung a run qualifies to `REPLAYED`.
Item 159's chunk, PR #322, replayed only the prose and stability passes. Sonnet 5.5 at high missed nearly half of what the project's rung found again in both, so the agent judged a replay of the other four unlikely to qualify a rung, and the product-owner review of PR #322 asked that this be the human's call, kept in the plan, rather than the agent's. Each run costs hours of reviewer replays (goal 1.4), so the score keeps it low until a cheaper model joins the ladder. `docs/decisions.md` → `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds…` records the runs so far.
Item 159's chunk, PR #322, replayed only the prose and stability passes. Sonnet 5.5 at high missed 44 to 55% of what the project's rung found again in each, so the agent judged a replay of the other four unlikely to qualify a rung. The product-owner review of PR #322 asked that the plan keep them, so the ranking decides when they run. Each run costs hours of reviewer replays (goal 1.4), so the score keeps it low until a cheaper model joins the ladder. `docs/decisions.md` → `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds…` records the runs so far.
### 181. Have the test of a quoted answer's step-up assert the stepped rung's facts as literals, not through `prompts.rungs_held`.