Merge pull request 'Run each review pass on its own rung, lowered only where a replay of its rounds shows a lower rung finds what the project's rung finds' (#322) from item-159 into main

This commit was merged in pull request #322.
This commit is contained in:
2026-10-18 19:00:38 +02:00
31 changed files with 508 additions and 235 deletions
+3 -3
View File
@@ -51,7 +51,10 @@
!bin/reading_cost/*.py
!bin/repo_checks/
!bin/repo_checks/*.py
!bin/review-replay
!bin/review_folder.py
!bin/review_replay/
!bin/review_replay/*.py
!bin/round_titles.py
!bin/rule-hook
!bin/routing_figures/
@@ -64,9 +67,6 @@
!bin/score_outcomes/
!bin/score_outcomes/*.py
!bin/shell_command.py
!bin/stability-replay
!bin/stability_replay/
!bin/stability_replay/*.py
!bin/statusline.py
!bin/subagent-rung
!bin/subagent_rung/
+2 -2
View File
@@ -40,14 +40,14 @@ Decisions live in `docs/decisions.md`:
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
- routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, the seats and the security pass always on the project's
- routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass; the seats and the security pass always on the project's
- routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, or when the judge reports a taken-in item that picks a higher rung or an escalation trigger the worker cites
- routing: loop-claude measures review rounds and unfinished verdicts per rung every 20 cycles, and calls for a routing change when, of two rungs with 5 or more chunks each, the lower took more of either than the one above, or when 5 or more subagents of one type had reports longer than the tool results they read
- routing: a worker delegates a task only where it needs just the task's conclusion, and each subagent runs on the rung its task's L picks from the project's rung, never below the project's rung when it files or scores todo.md items
- routing: a worker hands a task one rung up on each escalation trigger, up to two rungs above the rung its item picks unless the human agrees, and opens the subagent's prompt with the trigger so the routing figures count it
- routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, steps one rung up as the judge interrupts or quotes an answer, or next cycle, up to two rungs above the rung its item picks
- routing: on a model judge-replay covered, the judge runs on the rung a review would get from the highest E + I among its chunk's items, never below the project's rung
- routing: the stability reviewer runs no lower than the project's rung while the worker runs below it, until `stability-replay` shows a lower rung finding every 🔴 and 🟡 the project's rung finds
- routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and only where the E + I steps no review up and the worker runs no higher than the project's rung, then even below the worker's; no pass has such a rung
## Working rules
+3 -2
View File
@@ -2,8 +2,9 @@
## Unreleased
- `stability-replay <model>` is now `review-replay <pass> <model>`, and replays any pass an agent under `agents/` reviews, such as `review-replay prose claude-opus-5-5`. It now tries every rung below the project's, down to the ladder's lowest. Its runs on the prose and stability passes of Opus 5.5 projects found no rung below the project's that finds every 🔴 and 🟡 the project's rung finds: Sonnet 5.5 at high missed 12 of 27 prose findings and 11 of 20 stability findings. So every pass stays where it runs. A rung a future run finds lowers only its pass, once `REPLAYED` in `bin/model_routing/ladder.py` names it, and only where the chunk's E + I steps no review up and the worker runs no higher than the project's rung; there it may run below the worker's rung.
- `chunk-preflight` names the release it finds finished and ready to cut, or to be planned first, wherever todo.md has named that release since the last cut.
- Each reviewer under `agents/` now writes its full report to a file that larv-review names in the reviewer's prompt, in a folder per chunk under `~/.local/state/larv-review/`. It returns only its verdict and one line per finding, so a review takes less of the worker's context. The worker reads a finding's full entry before it fixes or answers a 🔴 or 🟡. Each reviewer now holds Write. `rule-hook` refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`; no override waives this rule. Each chunk's folder stays after the merge, so `stability-replay` can replay a round that reads an earlier round's report. It replays on a copy of the folder.
- Each reviewer under `agents/` now writes its full report to a file that larv-review names in the reviewer's prompt, in a folder per chunk under `~/.local/state/larv-review/`. It returns only its verdict and one line per finding, so a review takes less of the worker's context. The worker reads a finding's full entry before it fixes or answers a 🔴 or 🟡. Each reviewer now holds Write. `rule-hook` refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`; no override waives this rule. Each chunk's folder stays after the merge, so `review-replay` can replay a round that reads an earlier round's report. It replays on a copy of the folder.
- `loop-claude` asks an interrupt's question on its own terminal and resumes the worker headless with your one-line answer, so the loop no longer waits for you to leave a claude session. Type `/talk` instead to open the worker's session in claude, as every interrupt did before, to read the diff or discuss first. A release ready to cut opens that session at once. Pressing Enter on an empty answer stops the loop, exit 2. Keys typed before the question shows are dropped, and a pasted answer keeps only its first line; type `/talk` for a longer one. The judge counts that answer as yours, like one typed in claude: the chunk stays open until the answer is recorded, where a later chunk could need it.
- `todo.md` may mark a renamed release in a `## Renamed` section, one line per rename, such as `- 0.2.1 renamed to 0.3.0`. `chunk-preflight` and `loop-claude` then skip that release when they pick the next one, and larv-review's scoring-run trigger calls no run for it. Without the mark, a release named since the last tag whose items all moved to other releases reads as finished: `chunk-preflight` says it is ready to cut and the loop stops for you, and the trigger calls a scoring run. `rank_todo.py` fails on a `## Renamed` line of another shape, and on an item filed under a release it marks. Before you move a machine to this release, add the line to the `todo.md` of every project that renamed a release since its last tag, or its loop stops for that release's cut.
- When a worker's run ends with a question and a request for a higher rung, `loop-claude` now steps the worker one rung up as the judge interrupts, or as it answers the question with a quote from `README.md` or another file it may quote. The judge must quote a passage showing that the escalation trigger the run cites fired. Before, the step was lost for an answered question, and the worker carried on from the answer on its old rung. The step has the same limits as the step on an unfinished verdict. An escalation trigger that fired before the worker's last step-up doesn't step it again, so one trigger buys one step. A trigger that fires after the step steps the worker again, in the same cycle or a later one. A chunk left open across the upgrade to this release starts with no step-up on record, so a trigger that already stepped its worker may step it once more. The archive's interrupt and quoted-answer outcomes now name the worker's rung.
@@ -13,7 +14,7 @@
- Reshape every project's `todo.md` to the new shape below: until it is reshaped, that is the project's only work, as it is while README.md lacks Goals and Audience, and `rank_todo.py` says so. First move every machine to this release: an older release rejects the new formula, so it fails on every reshaped `todo.md`. Then mark the README's goals `**Maintain:**` or `**Achieve:**`, score each item's L, E and I, and set a bar on the new scale below the score of every item to keep, since the old bar would drop most items. Until then, an old-shape `todo.md` still ranks by its own formula and weights.
- `todo.md` ranks items by value per cost: this setup's own plan uses `Score = 100 * V / (2**S + 8)`, where 8 is a chunk's fixed cost, about one size-3 change, and `rank_todo.py --help` shows it. A Moves column replaces A, G and Goals, naming each achieve goal an item moves and how far, such as `1.1 by 8, 1.2 by 5`. V sums each named goal's weight times its move. `rank_todo.py` weighs each achieve goal 0.97 times the one above it in README.md, so a goal far down the list still counts almost as much as the first and the goal weights table goes. The weights sum to 1: 5 achieve goals weigh 0.212 down to 0.188, 15 weigh 0.082 down to 0.053. An item names the most specific goal it moves, never a goal beside its own sub-goal. `rules/todo-items.md` and `rank_todo.py --help` show the new shape.
- L, E and I replace C, and set R. L is the chance the change holds a defect, from 1 to 4; E the chance a defect escapes tests and review, from 1 to 3; I the damage it would do, from 1 to 5, counting every project and session that runs the change. `rules/todo-items.md` defines each level in words, and `rank_todo.py` writes R as L + E + I − 2, at least 7 when I is 5 and at least 5 when I is 4. The table's score columns are `| L | E | I | R | S |`. `score-outcomes` takes `L E I S`, and reports how well L, E and I each order finished chunks by review rounds and 🔴 findings.
- `loop-claude` routes a chunk's worker on its item's L, from two rungs below the project's rung at L 1 to one above it at L 4. It routes the reviewers on the highest E + I among the chunk's items, counting each item the worker takes in: one rung above the project's for each point past 4, never below the worker's rung. It runs the judge of an Opus 5.5 project on the rung that highest E + I picks, never below the project's, and keeps it there when the worker steps up. The comprehension seats and the security pass stay on the project's rung. rule-hook holds each reviewer launch to the reviewers' rung, which `loop-claude` hands the worker in `LOOP_CLAUDE_REVIEWER_RUNG`. A delegated task's subagent takes the rung its L picks: `subagent-rung` takes L in place of R + C. Until a project reshapes its `todo.md`, every chunk there runs on the project's rung. Expect reviews of items whose defects would escape far or break much to cost more, and escalation from an item of L 4 to reach Fable 5.1 only with your yes.
- `loop-claude` routes a chunk's worker on its item's L, from two rungs below the project's rung at L 1 to one above it at L 4. It routes the reviewers on the highest E + I among the chunk's items, counting each item the worker takes in: one rung above the project's for each point past 4, never below the worker's rung. It runs the judge of an Opus 5.5 project on the rung that highest E + I picks, never below the project's, and keeps it there when the worker steps up. The comprehension seats and the security pass stay on the project's rung. rule-hook holds each reviewer launch to the reviewers' rung, which it works out from the E + I `loop-claude` hands the worker in `LOOP_CLAUDE_ESCAPE_AND_IMPACT`. A delegated task's subagent takes the rung its L picks: `subagent-rung` takes L in place of R + C. Until a project reshapes its `todo.md`, every chunk there runs on the project's rung. Expect reviews of items whose defects would escape far or break much to cost more, and escalation from an item of L 4 to reach Fable 5.1 only with your yes.
- An item that restores a maintain goal the project breaks today is exempt as `defect`, and its Moves may name that maintain goal, weighed as the heaviest achieve goal. A defect ranks by its score like any item. Only two kinds rank ahead of the rest of their release: questions first, since an answer may change other items, then security items.
- The stability reviewer applies `principles/technical-principles.md` as written, in place of its own copies of it. Its test-data check now asks for UUID v7, where it asked for randomised UUIDs.
- larv-review's principles audit is a pass of its own, and runs on every chunk that changes code, config or tests, not only on one with an architectural decision. A chunk of prose or planning alone skips it.
+2 -2
View File
@@ -89,9 +89,9 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
- `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format.
- `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure.
- `review-replay <pass> <model>` replays a review pass's archived rounds from the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again; `review-replay --help` states the replay. Run it only in a sandbox, like loop-claude: each replayed reviewer skips every permission prompt, stability-reviewer has Bash, and an archived prompt may name paths on the host.
- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit, Write and Agent call, and `rule-hook --help` states the rules and the override.
- `score-outcomes [L E I S]` lists the finished items whose guessed scores sit nearest the given ones, and the shipped items the project's `docs/score-anchors.md` names for each level of L, E and I, each with what its chunk measured; then the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings; `score-outcomes --help` states the measures.
- `stability-replay <model>` replays archived stability reviews of the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again; `stability-replay --help` states the replay. Run it only in a sandbox, like loop-claude: each replayed reviewer has Bash and skips every permission prompt, and an archived prompt may name paths on the host.
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
- `subagent-rung [--plan] <model> <L>` prints the Agent tool's model and effort for a subagent's task, picked by its L from the project's model, and with `--plan` never below the project's rung; `subagent-rung --help` states the rule.
@@ -110,7 +110,7 @@ pytest_add_cli_args=-k
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and `loop_claude` runs `chunk-preflight` as a process of its own. `judge_replay` and `stability_replay` share `rung_replay/`: the rung search, and the argument, candidate rungs and exits. It reads `loop_claude`'s retry count and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and `loop_claude` runs `chunk-preflight` as a process of its own. `judge_replay` and `review_replay` share `rung_replay/`: the rung search, the argument reader, the candidate rungs and the exits. `rung_replay/` reads `loop_claude`'s retry count and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude, `rule_hook`, both replay tools and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
## Tracking a new file
+11 -4
View File
@@ -28,28 +28,29 @@ import typing
from transcript_archive.location import archive_root, transcript_of
from transcript_archive.read import archived_outcomes
from rung_replay.command import arguments_or_exit, down_from, failed, limited_exit, projects_on
from judge_replay.replay import Case, case_from, Kind, report, reproduced, sample, verdict_kind
from project_key import checkout_of
from rung_replay.command import down_from, failed, limited_exit, project_rung_or_exit, projects_on
from loop_claude.effects import Effects
from failure import Failure
from rung_replay.search import first_reply, replayed
from loop_claude.cycle.judge import judge_once, Verdict
from claude_code.reply import Limited
from model_routing.ladder import lowest_worker_rung, project_rung, Rung
from loop_claude.project.premises import read_premises
from loop_claude.project.records import read_records
from claude_code.transcript import read_transcript
from model_routing.ladder import Rung
from loop_claude.stop import Stop
TOOL = "judge-replay"
USAGE = f"{TOOL} <model>, a model on the routing ladder"
PER_KIND = 15
# The effort settings.json gave every judge before loop-claude routed it; the archived verdicts came from it.
REFERENCE_EFFORT = "xhigh"
def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, effects: Effects) -> int:
anchor = project_rung_or_exit(TOOL, argv, __doc__)
anchor = arguments_or_exit(TOOL, argv, __doc__, USAGE, model_argument)
if isinstance(anchor, int):
return anchor
cases = archived_cases(anchor.model, archive_root(home, env), home)
@@ -59,13 +60,19 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
return failed(TOOL, f"no archived verdicts of projects on {anchor.model} in {archive_root(home, env)}; run loop-claude on a project on that model first")
cases = sample(cases, PER_KIND)
reference = Rung(anchor.model, REFERENCE_EFFORT)
replay = replayed(cases, reference, down_from(anchor), lambda found, rung: rule(found, rung, home / ".claude", effects), reproduced, "verdicts")
# The candidates stop at the lowest rung a worker routes to: no route uses a rung below it for the judge.
replay = replayed(cases, reference, down_from(anchor, lowest_worker_rung(anchor)), lambda found, rung: rule(found, rung, home / ".claude", effects), reproduced, "verdicts")
if isinstance(replay, Limited):
return limited_exit(TOOL, replay)
print(report(anchor.model, reference, cases, replay))
return 0
def model_argument(argv: list[str]) -> Rung | None:
"""The project's rung of the one model argv names; None for any other argv."""
return project_rung(argv[0]) if len(argv) == 1 else None
def archived_cases(model: str, root: pathlib.Path, home: pathlib.Path) -> list[Case] | Failure:
projects = projects_on(model, home)
if isinstance(projects, Failure):
+6
View File
@@ -141,6 +141,12 @@ class Main(unittest.TestCase):
# The replay shows the judge the whole transcript, with no step-up marked in it.
self.assertNotIn("[step]", self.prompts[0])
def test_tries_no_rung_below_the_lowest_a_worker_routes_to_where_every_rung_returns_every_verdict(self):
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
self.assertEqual(code, 0, err)
self.assertEqual([(model, effort) for model, effort, _ in self.calls[::2]][-1], ("claude-haiku-5-5", "max"))
self.assertTrue(out.endswith("returns every verdict the reference judge reproduces: claude-haiku-5-5 at max effort\n"), out)
def test_stops_at_the_projects_rung_and_names_none_when_it_misses(self):
code, out, _ = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted and rung == REFERENCE, f"said {INTERRUPTED if interrupted else FINISHED}"))
self.assertEqual(code, 0)
+39 -19
View File
@@ -1,4 +1,4 @@
from model_routing.ladder import climbed, LADDER, position, review_rungs, ReviewKind, Rung
from model_routing.ladder import climbed, LADDER, position, review_rungs, REVIEWERS, reviewers_rung, ReviewKind, Rung
from model_routing.route import ESCALATION_MARKER, HIGHEST_ESCAPE_AND_IMPACT, Route
from loop_claude.routing import escalation_rungs
from loop_claude.project.records import HUMAN_RULE_NAMES, RECORD_FILES
@@ -10,8 +10,8 @@ QUOTE_SOURCES = listed(("README.md's Goals and Audience", *RECORD_FILES, *HUMAN_
# Where the worker may write an answer the human typed, so it counts as recorded.
ANSWER_DESTINATIONS = "README.md's Goals or Audience, " + ", ".join(RECORD_FILES) + ", todo.md or a comment"
ANSWERS_RECORDED = f"every answer the human typed in it that a later chunk could need is recorded in {ANSWER_DESTINATIONS} at the one place it governs"
# Each kind of review, as review_launches names those it launches.
REVIEWS: tuple[tuple[str, ReviewKind], ...] = (("each reviewer", "reviewer"), ("each comprehension seat", "seat"), ("the security pass's subagent", "security"))
# Each kind of review, as review_launches names those it launches: a row per pass, since a replay may lower one pass alone. "each reviewer" names every pass no replay lowered.
REVIEWS: tuple[tuple[str, ReviewKind], ...] = (*((f"each {agent}", kind) for kind, agent in REVIEWERS.items()), ("each comprehension seat", "seat"), ("the security pass's subagent", "security"))
def fresh(task: str, premises: str, route: Route, anchor: Rung | None, facts: str) -> str:
@@ -41,21 +41,32 @@ def preflight_failed(reason: str) -> str:
def review_launches(route: Route, anchor: Rung | None) -> str:
"""Which reviews to launch on a rung other than the worker's, and on which; below the project's rung, anchor, also where to launch a subagent that files or scores todo.md items; then taken_in_launches, the reviewers' rung each higher E + I picks. Empty off the ladder.
"""Which reviews to launch on a rung other than the worker's, and on which, then the rung of each pass a replay lowered; below the project's rung, anchor, also where to launch a subagent that files or scores todo.md items; then taken_in_launches, the reviewers' rung each higher E + I picks. Empty off the ladder.
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project, routed by item 3 of E + I 5, it opens: ` Launch each reviewer with the Agent tool set to model opus, effort high, the rung item 3's E + I of 5 picks. Launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.`
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project, routed by item 3 of E + I 5, it opens: ` Launch each reviewer with the Agent tool set to model opus, effort high, the rung item 3's E + I of 5 picks. Launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.` Where a replay lowered the prose pass to Haiku 5.5 at low, and the E + I steps no review up, a sentence follows: ` Launch each prose-reviewer instead with the Agent tool set to model haiku, effort low, the rung the replay of its pass chose.`
"""
if anchor is None:
return ""
rungs = review_rungs(anchor, route.rung, route.reviewed_on())
launched: dict[Rung, list[str]] = {}
for reviews, kind in REVIEWS:
if rungs[kind] != route.rung:
launched.setdefault(rungs[kind], []).append(reviews)
launches = "".join(f" Launch {listed(reviews, 'and')} with the Agent tool set to {rung.agent_args()}, {rung_named(rung, route, anchor)}." for rung, reviews in launched.items())
rungs, reviewers = review_rungs(anchor, route.rung, route.reviewed_on()), reviewers_rung(anchor, route.rung, route.reviewed_on())
lowered_launches = "".join(f" Launch {reviews} instead with the Agent tool set to {rung.agent_args()}, the rung the replay of its pass chose." for reviews, rung in lowered(rungs, reviewers))
scan = " Run a full scan only from such a subagent, since the scan's agents run on the model that starts them." if rungs["security"] != route.rung else ""
planning = f" Your rung is below the project's: launch each subagent that files or scores todo.md items on the rung `~/.claude/bin/subagent-rung --plan {anchor.model} <L>` prints." if position(route.rung) < position(anchor) else ""
return f"{launches}{scan}{planning}{taken_in_launches(route, anchor)}"
return f"{shared_launches(rungs, reviewers, route, anchor)}{lowered_launches}{scan}{planning}{taken_in_launches(route, anchor)}"
def shared_launches(rungs: dict[ReviewKind, Rung], reviewers: Rung, route: Route, anchor: Rung) -> str:
"""A sentence for each rung other than the worker's that a review runs on. "each reviewer" names every pass no replay lowered; those run on reviewers."""
together = {"each reviewer": reviewers} | {reviews: rungs[kind] for reviews, kind in REVIEWS if kind not in REVIEWERS}
launched: dict[Rung, list[str]] = {}
for reviews, rung in together.items():
if rung != route.rung:
launched.setdefault(rung, []).append(reviews)
return "".join(f" Launch {listed(reviews, 'and')} with the Agent tool set to {rung.agent_args()}, {rung_named(rung, route, anchor)}." for rung, reviews in launched.items())
def lowered(rungs: dict[ReviewKind, Rung], reviewers: Rung) -> list[tuple[str, Rung]]:
"""Each pass a replay put below reviewers, the rung of every other pass, as REVIEWS names it, with the rung the replay chose."""
return [(reviews, rungs[kind]) for reviews, kind in REVIEWS if kind in REVIEWERS and rungs[kind] != reviewers]
def taken_in_launches(route: Route, anchor: Rung) -> str:
@@ -63,12 +74,12 @@ def taken_in_launches(route: Route, anchor: Rung) -> str:
For a worker on Opus 5.5 at medium, routed by an item of E + I 6, it reads: ` Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8.`
"""
reviewers = review_rungs(anchor, route.rung, route.reviewed_on())["reviewer"]
now = review_rungs(anchor, route.rung, route.reviewed_on())
higher: dict[Rung, int] = {}
for escape_and_impact in range(route.reviewed_on() + 1, HIGHEST_ESCAPE_AND_IMPACT + 1):
rung = review_rungs(anchor, route.rung, escape_and_impact)["reviewer"]
if position(rung) > position(reviewers):
higher.setdefault(rung, escape_and_impact)
# An E + I that lifts any pass puts every pass on reviewers_rung, since a replay lowers a pass only where the E + I steps no review up.
if review_rungs(anchor, route.rung, escape_and_impact) != now:
higher.setdefault(reviewers_rung(anchor, route.rung, escape_and_impact), escape_and_impact)
if not higher:
return ""
picks = [f"`{rung.agent_args()}` from {escape_and_impact}" for rung, escape_and_impact in higher.items()]
@@ -108,11 +119,20 @@ def escalation(route: Route, anchor: Rung | None) -> str:
def reruns(route: Route, anchor: Rung | None) -> str:
"""The prompt's sentence naming the rung one above a reviewer's and a seat's, on which each reruns when it asks; empty off the ladder, and leaving out one already on the top rung."""
"""The prompt's sentences naming the rung one above a review's, on which it reruns when it asks: shared_reruns' for a reviewer and a seat, then one for each pass a replay lowered. Empty off the ladder.
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project whose prose pass a replay lowered to Haiku 5.5 at low, routed by an item of E + I 4, they read: ` A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`. Rerun each prose-reviewer that asks with the Agent tool set to `model haiku, effort medium` instead.`
"""
if anchor is None or route.rung not in LADDER:
return ""
rungs = review_rungs(anchor, route.rung, route.reviewed_on())
launched = (("reviewer", rungs["reviewer"]), ("comprehension seat", rungs["seat"]))
rungs, reviewers = review_rungs(anchor, route.rung, route.reviewed_on()), reviewers_rung(anchor, route.rung, route.reviewed_on())
lowered_reruns = "".join(f" Rerun {reviews} that asks with the Agent tool set to `{climbed(rung, 1).agent_args()}` instead." for reviews, rung in lowered(rungs, reviewers))
return f"{shared_reruns(reviewers, rungs['seat'])}{lowered_reruns}"
def shared_reruns(reviewers: Rung, seats: Rung) -> str:
"""The sentence naming the rung one above reviewers, the rung of each pass no replay lowered, and one above seats, leaving out one already on the top rung."""
launched = (("reviewer", reviewers), ("comprehension seat", seats))
rerun = {name: climbed(rung, 1) for name, rung in launched if rung != LADDER[-1]}
if not rerun:
return ""
+8 -6
View File
@@ -68,19 +68,21 @@ picks the worker's rung, stepping from the model's cheapest rung by WORKER_STEPS
bin/model_routing/ladder.py, below it for a low L; without such an item, or with its L, E or I
unscored, the chunk runs on the model's cheapest rung. The
item's E + I picks the reviewers' rung, stepping up from the cheapest rung by REVIEW_STEPS there,
never below the worker's rung. The worker's first prompt names the rung and the item, and asks the
never below the worker's rung. Where it steps none and the worker runs no higher than the cheapest
rung, a pass REPLAYED there names runs on the rung it names, the lowest a review-replay run found
finding what the cheapest rung finds, even below the worker's. The worker's first prompt names the rung and the item, and asks the
worker to name each item the chunk takes, with its L and E + I, and to count the item's size again
once it has read the code (prompts.recount). The prompt names the rung of each review that runs on
a rung other than the worker's: the reviewers' rung for the reviewers, and the cheapest rung for
each comprehension seat and the security pass. It also names the reviewers' rung
a rung other than the worker's: the reviewers' rung or the replay's for each pass's reviewer, and
the cheapest rung for each comprehension seat and the security pass. It also names the reviewers' rung
each higher E + I picks, so the worker reviews an item it takes in on that rung in the same run.
When the worker runs below the cheapest rung, the prompt has it launch each subagent that files or
scores todo.md items on the rung subagent-rung --plan prints. The prompt after
each unfinished verdict says this again, since a step-up may have moved the worker. A reviewer or
seat that rules the work below standard may ask to rerun one rung up; each prompt names that rung.
loop-claude hands the worker's claude its own rung, the project's and the reviewers', in
LOOP_CLAUDE_WORKER_RUNG, LOOP_CLAUDE_PROJECT_RUNG and LOOP_CLAUDE_REVIEWER_RUNG, and rule-hook
refuses a review launched below the rung named for it. The judge reports the highest L and the
loop-claude hands the worker's claude its own rung, the project's and the reviews' E + I, in
LOOP_CLAUDE_WORKER_RUNG, LOOP_CLAUDE_PROJECT_RUNG and LOOP_CLAUDE_ESCAPE_AND_IMPACT, and rule-hook
refuses a review launched below the rung those pick for it. The judge reports the highest L and the
highest E + I the worker states for an item it takes in. The next cycle steps the worker up to the
rung that L picks, never down, and the reviewers to the rung picked by the higher of two: that
E + I, or the routed item's (Route.reviewed_on in bin/model_routing/route.py). A level off its
+18 -2
View File
@@ -12,6 +12,7 @@ import sys
import tempfile
import time
import unittest
import unittest.mock
import uuid
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
@@ -24,6 +25,7 @@ from claude_code import entries, reply, session
from failure import Failure
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished, verdict_from
from loop_claude.cycle import interrupt, judge, prompts, retry, run, states, worker
from model_routing import ladder
from model_routing.ladder import LADDER, Rung
from claude_code.reply import Limited
from loop_claude.loop import main, reload, routing_check
@@ -220,11 +222,11 @@ class Cycles(unittest.TestCase):
self.assertIn("- A preflight fact.", fresh)
self.assertNotIn("- A preflight fact.", resumed)
def test_a_worker_is_handed_the_projects_rung_its_own_and_its_reviewers_in_its_environment(self):
def test_a_worker_is_handed_the_projects_rung_its_own_and_its_reviews_e_plus_i_in_its_environment(self):
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("finished")])
cycle = dataclasses.replace(self.cycle(fake), anchor=Rung("claude-opus-5-5", "medium"), route=Route(Rung("claude-haiku-5-5", "max"), 163, 1, 5))
self.assertEqual(run.run_cycle(cycle).verdict, Finished("working"))
self.assertEqual(fake.worker_environments, [{"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": "claude-opus-5-5 at high effort"}])
self.assertEqual(fake.worker_environments, [{"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5"}])
def test_the_claude_an_interrupt_resumes_is_handed_the_same_rungs_as_the_worker(self):
def answer(args):
@@ -1143,6 +1145,20 @@ class WorkerArgs(unittest.TestCase):
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus))
self.assertNotIn("Where the hard part", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 3, 4), opus))
def test_launches_each_pass_a_replay_lowered_on_its_replays_rung_and_reruns_it_one_above(self):
opus, haiku = Rung("claude-opus-5-5", "medium"), Rung("claude-haiku-5-5", "low")
with unittest.mock.patch.dict(ladder.REPLAYED, {("prose", "claude-opus-5-5"): haiku}):
below = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 4), opus, self.FACTS)
on_it = prompts.resumed("", Route(haiku, 3, 1, 4), opus)
launched = " Launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung. Launch each prose-reviewer instead with the Agent tool set to model haiku, effort low, the rung the replay of its pass chose."
rerun = " A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`. Rerun each prose-reviewer that asks with the Agent tool set to `model haiku, effort medium` instead."
self.assertIn(f"whether or not it is item 3.{launched}", below)
self.assertIn(" Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5,", below)
self.assertIn(f"{rerun} After your first read", below)
self.assertIn(f"loop-claude runs you on claude-haiku-5-5 at low effort.{launched}", on_it)
self.assertIn(" Where an item you take into the chunk has an E + I of 5 or more,", on_it)
self.assertTrue(on_it.endswith(rerun))
def test_names_the_rung_a_reviewer_or_seat_reruns_on_one_above_its_own(self):
opus = Rung("claude-opus-5-5", "medium")
both = " A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`."
+19 -8
View File
@@ -5,8 +5,10 @@ import typing
# How Rung.__str__ spells a rung that names an effort, and Rung.parse reads it back.
WORDING = re.compile(r"(?P<model>\S+) at (?P<effort>\S+) effort")
# What a review launch reviews: a reviewer, a comprehension seat or the security pass.
ReviewKind = typing.Literal["reviewer", "seat", "security"]
# A pass an agent under agents/ reviews, named as skills/larv-review/SKILL.md → "Passes" names it in lower case.
ReviewedPass = typing.Literal["architecture", "principles", "overrides", "stability", "product-owner", "prose"]
# What a review launch reviews: such a pass, a comprehension seat or the security pass.
ReviewKind = ReviewedPass | typing.Literal["seat", "security"]
@dataclasses.dataclass(frozen=True)
@@ -63,6 +65,10 @@ LADDER = (
WORKER_STEPS = {1: -2, 2: -1, 3: 0, 4: 1}
# The rungs a task's E + I steps a review up from the project's rung; an E + I under 5 steps none.
REVIEW_STEPS = {5: 1, 6: 2, 7: 3, 8: 4}
# Each pass an agent under agents/ reviews, in the order of SKILL.md's pass table, and that agent.
REVIEWERS: dict[ReviewedPass, str] = {"architecture": "systems-architect", "principles": "principles-auditor", "overrides": "override-reviewer", "stability": "stability-reviewer", "product-owner": "product-owner-reviewer", "prose": "prose-reviewer"}
# The rung a pass's replay chose for the projects on a model: docs/decisions.md → `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds`.
REPLAYED: dict[tuple[ReviewedPass, str], Rung] = {}
def project_rung(model: str) -> Rung | None:
@@ -96,14 +102,19 @@ def stepped_up(rung: Rung) -> Rung:
def review_rungs(project: Rung, worker: Rung, escape_and_impact: int) -> dict[ReviewKind, Rung]:
"""The rung each kind of review in a chunk runs on, where its worker runs on worker and its item has that E + I. The reviewers run on the higher of the worker's rung and review_floor's."""
reviewers = highest(worker, review_floor(project, escape_and_impact))
return {kind: review_rung(kind, project, reviewers) for kind in typing.get_args(ReviewKind)}
"""The rung each kind of review in a chunk runs on, where its worker runs on worker and its item has that E + I.
A pass's reviewer runs on reviewers_rung. Where the E + I steps no review up and the worker runs no higher than the project's rung, a pass whose replay chose a rung for the project's model runs on that rung instead, even below the worker's: the replay compared reviews on the project's rung only, never on a rung an E + I stepped up, nor of a chunk whose worker climbed above the project's rung. A seat and the security pass run on the project's rung, so the seats score every chunk on one rung, and the security pass spends no more than the project chose.
"""
rungs: dict[ReviewKind, Rung] = {**dict.fromkeys(REVIEWERS, reviewers_rung(project, worker, escape_and_impact)), "seat": project, "security": project}
if escape_and_impact not in REVIEW_STEPS and position(worker) <= position(project):
rungs |= {kind: rung for (kind, model), rung in REPLAYED.items() if model == project.model}
return rungs
def review_rung(kind: ReviewKind, project: Rung, reviewers: Rung) -> Rung:
"""The rung a review of kind runs on: a reviewer on the reviewers' rung; a seat or the security pass on the project's, so the seats score every chunk on one rung, and the security pass spends no more than the project chose."""
return reviewers if kind == "reviewer" else project
def reviewers_rung(project: Rung, worker: Rung, escape_and_impact: int) -> Rung:
"""The rung of each pass's reviewer that no replay lowered: the higher of the worker's rung and review_floor's."""
return highest(worker, review_floor(project, escape_and_impact))
def highest(*rungs: Rung) -> Rung:
+32 -12
View File
@@ -1,10 +1,13 @@
import pathlib
import sys
import typing
import unittest
import unittest.mock
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from model_routing.ladder import highest, LADDER, lowest_worker_rung, planning_rung, project_rung, review_floor, review_rung, review_rungs, Rung, stepped_up, worker_rung, WORKER_STEPS
from model_routing.ladder import highest, LADDER, lowest_worker_rung, planning_rung, position, project_rung, review_floor, review_rungs, ReviewedPass, REVIEWERS, Rung, stepped_up, worker_rung, WORKER_STEPS
from model_routing import ladder
from todo_table import LEVELS
OPUS = Rung("claude-opus-5-5", "medium")
@@ -94,20 +97,37 @@ class Ladder(unittest.TestCase):
def test_puts_reviewers_on_the_highest_of_the_projects_rung_the_workers_and_the_one_e_plus_i_picks(self):
haiku, fable = LADDER[4], LADDER[-1]
self.assertEqual(review_rungs(OPUS, haiku, 4)["reviewer"], OPUS)
self.assertEqual(review_rungs(OPUS, haiku, 5)["reviewer"], Rung("claude-opus-5-5", "high"))
self.assertEqual(review_rungs(OPUS, fable, 5)["reviewer"], fable)
self.assertEqual(review_rungs(OPUS, Rung("claude-opus-5-5", "high"), 7)["reviewer"], Rung("claude-opus-5-5", "max"))
self.assertEqual(review_rungs(OPUS, haiku, 4)["stability"], OPUS)
self.assertEqual(review_rungs(OPUS, haiku, 5)["stability"], Rung("claude-opus-5-5", "high"))
self.assertEqual(review_rungs(OPUS, fable, 5)["stability"], fable)
self.assertEqual(review_rungs(OPUS, Rung("claude-opus-5-5", "high"), 7)["stability"], Rung("claude-opus-5-5", "max"))
def test_keeps_the_seats_and_the_security_pass_on_the_projects_rung_whatever_the_worker_or_e_plus_i(self):
self.assertEqual(review_rungs(OPUS, LADDER[-1], 8), {"reviewer": LADDER[-1], "seat": OPUS, "security": OPUS})
self.assertEqual(review_rungs(OPUS, LADDER[-1], 8), {**dict.fromkeys(REVIEWERS, LADDER[-1]), "seat": OPUS, "security": OPUS})
def test_holds_a_reviewer_to_the_reviewers_rung_and_anything_else_to_the_projects(self):
fable = LADDER[-1]
self.assertEqual(review_rung("reviewer", OPUS, fable), fable)
for kind in ("seat", "security"):
with self.subTest(kind=kind):
self.assertEqual(review_rung(kind, OPUS, fable), OPUS)
def test_names_every_kind_of_review_once_each_pass_with_its_agent_under_agents(self):
self.assertEqual(list(REVIEWERS), list(typing.get_args(ReviewedPass)))
agents = pathlib.Path(__file__).resolve().parents[2] / "agents"
self.assertEqual(sorted(REVIEWERS.values()), sorted(path.stem for path in agents.glob("*.md")))
def test_runs_a_replayed_pass_on_its_replays_rung_below_the_workers_where_the_e_plus_i_steps_no_review_up_and_the_worker_runs_no_higher_than_the_project(self):
sonnet, haiku, opus_high = Rung("claude-sonnet-5-5", "high"), LADDER[0], Rung("claude-opus-5-5", "high")
with unittest.mock.patch.dict(ladder.REPLAYED, {("prose", "claude-opus-5-5"): haiku, ("stability", "claude-sonnet-5-5"): haiku}):
self.assertEqual(review_rungs(OPUS, LADDER[4], 4), {**dict.fromkeys(REVIEWERS, OPUS), "prose": haiku, "seat": OPUS, "security": OPUS})
self.assertEqual(review_rungs(OPUS, OPUS, 4)["prose"], haiku)
self.assertEqual(review_rungs(OPUS, opus_high, 4)["prose"], opus_high)
self.assertEqual(review_rungs(OPUS, sonnet, 5)["prose"], opus_high)
self.assertEqual(review_rungs(OPUS, LADDER[-1], 5)["prose"], LADDER[-1])
self.assertEqual((review_rungs(sonnet, sonnet, 4)["prose"], review_rungs(sonnet, sonnet, 4)["stability"]), (sonnet, haiku))
def test_holds_each_replayed_rung_below_its_models_project_rung_on_the_ladder(self):
for (kind, model), rung in ladder.REPLAYED.items():
with self.subTest(kind=kind, model=model):
project = project_rung(model)
self.assertIn(kind, REVIEWERS)
self.assertIsNotNone(project)
self.assertLess(position(rung), position(project) if project else 0)
self.assertIn(rung, LADDER)
if __name__ == "__main__":
unittest.main()
+15 -13
View File
@@ -9,20 +9,22 @@ class Environment(unittest.TestCase):
PROJECT = Rung("claude-opus-5-5", "medium")
WORKER = Rung("claude-haiku-5-5", "max")
def test_the_project_the_worker_and_the_reviewers_rungs_are_named_in_the_environment(self):
self.assertEqual(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5)), {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": "claude-opus-5-5 at high effort"})
self.assertEqual(environment(self.PROJECT, Route(self.WORKER))["LOOP_CLAUDE_REVIEWER_RUNG"], "claude-opus-5-5 at medium effort")
self.assertEqual(environment(self.PROJECT, Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 2))["LOOP_CLAUDE_REVIEWER_RUNG"], "claude-opus-5-5 at xhigh effort")
self.assertEqual(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5, escape_and_impact_taken_in=6))["LOOP_CLAUDE_REVIEWER_RUNG"], "claude-opus-5-5 at xhigh effort")
def test_the_project_and_the_workers_rungs_and_the_e_plus_i_reviews_take_are_named_in_the_environment(self):
self.assertEqual(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5)), {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5"})
self.assertEqual(environment(self.PROJECT, Route(self.WORKER))["LOOP_CLAUDE_ESCAPE_AND_IMPACT"], "0")
self.assertEqual(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5, escape_and_impact_taken_in=6))["LOOP_CLAUDE_ESCAPE_AND_IMPACT"], "6")
def test_a_project_off_the_ladder_is_handed_empty_rungs_so_none_is_inherited(self):
self.assertEqual(environment(None, Route(self.WORKER)), {"LOOP_CLAUDE_PROJECT_RUNG": "", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": ""})
def test_a_project_off_the_ladder_is_handed_an_empty_rung_so_none_is_inherited(self):
self.assertEqual(environment(None, Route(self.WORKER)), {"LOOP_CLAUDE_PROJECT_RUNG": "", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "0"})
def test_handed_reads_back_what_environment_named(self):
self.assertEqual(handed(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5))), (self.PROJECT, self.WORKER, Rung("claude-opus-5-5", "high")))
self.assertEqual(handed(environment(None, Route(self.WORKER))), (None, self.WORKER, None))
self.assertEqual(handed(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5))), (self.PROJECT, self.WORKER, 5))
self.assertEqual(handed(environment(None, Route(self.WORKER))), (None, self.WORKER, 0))
def test_a_missing_or_unknown_rung_reads_as_none(self):
self.assertEqual(handed({}), (None, None, None))
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 bogus"}), (None, None, None))
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at bogus effort"}), (None, None, None))
def test_a_missing_or_unknown_rung_reads_as_none_and_a_missing_or_unknown_e_plus_i_as_0(self):
self.assertEqual(handed({}), (None, None, 0))
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 bogus"}), (None, None, 0))
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at bogus effort"}), (None, None, 0))
for bogus in ("", "five", "-1", "5.0", "²"):
with self.subTest(bogus=bogus):
self.assertEqual(handed({"LOOP_CLAUDE_ESCAPE_AND_IMPACT": bogus})[2], 0)
+10 -9
View File
@@ -1,25 +1,26 @@
"""The rungs loop-claude hands a worker's claude in its environment, so rule-hook holds each review launch to them: docs/decisions.md → `routing:`.
"""The rungs and the E + I loop-claude hands a worker's claude in its environment, from which rule-hook works out the rung of each review launch: docs/decisions.md → `routing:`.
loop-claude sets these variables for its worker and for the claude an interrupt resumes; a human's shell does not, and rule-hook then checks no review launch."""
import typing
from model_routing.ladder import LADDER, review_rungs, Rung
from model_routing.ladder import LADDER, Rung
from model_routing.route import Route
PROJECT_RUNG = "LOOP_CLAUDE_PROJECT_RUNG"
WORKER_RUNG = "LOOP_CLAUDE_WORKER_RUNG"
REVIEWER_RUNG = "LOOP_CLAUDE_REVIEWER_RUNG"
# The E + I the chunk's reviews are routed on, from which review_rungs works out each review's rung.
ESCAPE_AND_IMPACT = "LOOP_CLAUDE_ESCAPE_AND_IMPACT"
def environment(project: Rung | None, route: Route) -> dict[str, str]:
"""The variables a worker's claude on route starts with. A project off the ladder gets empty project and reviewer rungs, so none is inherited from the environment."""
reviewers = review_rungs(project, route.rung, route.reviewed_on())["reviewer"] if project else None
return {PROJECT_RUNG: str(project) if project else "", WORKER_RUNG: str(route.rung), REVIEWER_RUNG: str(reviewers) if reviewers else ""}
"""The variables a worker's claude on route starts with. A project off the ladder gets an empty project rung, so none is inherited from the environment."""
return {PROJECT_RUNG: str(project) if project else "", WORKER_RUNG: str(route.rung), ESCAPE_AND_IMPACT: str(route.reviewed_on())}
def handed(environ: typing.Mapping[str, str]) -> tuple[Rung | None, Rung | None, Rung | None]:
"""The project's, the worker's and the reviewers' rungs, each None when unset or not on the ladder."""
return on_the_ladder(environ.get(PROJECT_RUNG)), on_the_ladder(environ.get(WORKER_RUNG)), on_the_ladder(environ.get(REVIEWER_RUNG))
def handed(environ: typing.Mapping[str, str]) -> tuple[Rung | None, Rung | None, int]:
"""The project's and the worker's rungs, each None when unset or not on the ladder, and the reviews' E + I, 0 when unset or not a whole number. An older loop-claude hands no E + I, and 0 steps no review up."""
escape_and_impact = environ.get(ESCAPE_AND_IMPACT, "")
return on_the_ladder(environ.get(PROJECT_RUNG)), on_the_ladder(environ.get(WORKER_RUNG)), int(escape_and_impact) if escape_and_impact.isdecimal() else 0
def on_the_ladder(value: str | None) -> Rung | None:
+11 -4
View File
@@ -5,20 +5,27 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from rule_hook.reviewer_rung import REVIEWERS, SEAT, SECURITY_OPENER
from round_titles import PASSES
from model_routing.ladder import REVIEWERS
from repo_checks.test_python_floor import ROOT
from rule_hook.reviewer_rung import SEAT, SECURITY_OPENER
OPENING = re.compile(r"^## Opening\n+```\n([^\n]+?\.)(?: |\n)", re.MULTILINE)
REVIEWER_NAME = re.compile(r"\b[a-z]+(?:-[a-z]+)*-(?:reviewer|architect|auditor)\b")
# A row of the pass table whose reviewer is an agent: its pass and that agent.
REVIEWED_ROW = re.compile(r"^\| ([A-Z][a-z-]+) \| ([a-z]+(?:-[a-z]+)*-(?:reviewer|architect|auditor)) \|", re.MULTILINE)
LARV_REVIEW = (ROOT / "skills" / "larv-review" / "SKILL.md").read_text(encoding="utf-8")
class ReviewerLaunches(unittest.TestCase):
"""rule-hook holds a review launch to its rung by recognising it; these fail when what it recognises drifts from what the skills launch."""
def test_the_reviewers_are_the_ones_larv_review_s_pass_table_names(self):
def test_the_reviewers_are_the_passes_and_agents_larv_review_s_pass_table_names_in_its_order(self):
passes = LARV_REVIEW.split("## 3. Passes", 1)[1].split("A comment is prose", 1)[0]
self.assertEqual(set(REVIEWER_NAME.findall(passes)), REVIEWERS)
self.assertEqual({name.lower(): agent for name, agent in REVIEWED_ROW.findall(passes)}, REVIEWERS)
self.assertEqual([name.lower() for name, _ in REVIEWED_ROW.findall(passes)], list(REVIEWERS))
def test_each_reviewed_pass_is_a_pass_the_round_titles_count(self):
self.assertEqual([name for name in PASSES if name in REVIEWERS], list(REVIEWERS))
def test_the_seat_opening_is_the_sentence_that_opens_every_seat_prompt(self):
opening = OPENING.search((ROOT / "skills" / "comprehension-panel" / "seat-prompts.md").read_text(encoding="utf-8"))
+1 -1
View File
@@ -15,5 +15,5 @@ class Types(unittest.TestCase):
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
if shutil.which("uv") is None:
self.fail("uv on PATH fetches mypy; see README.md's Test line")
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "chunk_preflight", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "rule_hook", "-p", "rung_replay", "-p", "score_outcomes", "-p", "stability_replay", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "chunk_preflight", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "review_replay", "-p", "rule_hook", "-p", "rung_replay", "-p", "score_outcomes", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
+1 -1
View File
@@ -5,7 +5,7 @@ import sys
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from stability_replay import main
from review_replay import main
from loop_claude.effects import system_effects
sys.exit(main.main(sys.argv[1:], os.environ, pathlib.Path.home(), system_effects()))
@@ -1,10 +1,11 @@
"""Replay archived stability rounds on the rungs below the project's, to find which rungs may run the stability reviewer: docs/decisions.md → `routing: the stability reviewer runs no lower than the project's rung`.
"""Replay a review pass's archived rounds on the rungs below the project's, to find which rungs may run its reviewer: docs/decisions.md → `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds`.
usage: stability-replay <model>
usage: review-replay <pass> <model>
<pass> is one of architecture, principles, overrides, stability, product-owner or prose.
<model> is a project model on the routing ladder, such as claude-opus-5-5. The tool replays the
latest 10 archived stability rounds whose report holds a 🔴 or 🟡 finding, from the projects that
~/.claude/loop-claude-models.json runs on the model. It replays each round on the code it
latest 10 archived rounds of the pass whose report holds a 🔴 or 🟡 finding, from the projects
that ~/.claude/loop-claude-models.json runs on the model. It replays each round on the code it
reviewed: in a throwaway clone of the project's checkout, at the newest commit of origin/HEAD
authored by the time the round began, on the branch the reviewer started on. The clone's default
branch stands at the newest commit authored before the round's chunk began, so a diff against it
@@ -12,18 +13,17 @@ shows the chunk's change. The prompt names the clone in place of the worktree it
chunk has since removed. A round whose code it cannot find in this way is not replayed; stderr
names it and why. The prompt also names a copy of the chunk's review folder in place of that
folder, holding only the reports written after the chunk began and before the round did. A replay
whose prompt still names the review folder finds nothing.
finds nothing where its prompt still names the review folder, or names a file there, other than a
round's report, that the folder did not hold from the chunk's start until the round's.
It first replays every round on the project's rung, the reference reviewer. Findings on record
that the reference reviewer does not raise again may rest on code or rules that have changed
since; no rung is held to them. It then replays the rounds where the reference raised any, on each
rung below, highest first, down to the lowest rung a worker routes to, and stops at the first rung
that misses one. For each replay, claude on the project's rung matches the report to the findings
on record, and counts a finding only where the report raises it as 🔴 or 🟡. The tool prints each
finding a rung missed and the lowest rung that, like each rung above it, missed none. That rung is
the evidence for running the stability reviewer on the worker's rung whenever the worker runs at
or above it; docs/decisions.md → `routing: the stability reviewer runs no lower than the project's rung`
records the run.
rung below, highest first, down to the ladder's lowest, and stops at the first rung that misses
one. For each replay, claude on the project's rung matches the report to the findings on record,
and counts a finding only where the report raises it as 🔴 or 🟡. The tool prints each finding a
rung missed and the lowest rung that, like each rung above it, missed none. That rung is the
evidence for running the pass's reviewer on it; the decision above records each run.
Like the loop, it tries claude three times running before it counts a reply as failed. A failed
review or match finds nothing. Exit 1, saying why, on bad arguments, an unreadable archive or
@@ -39,21 +39,21 @@ import tempfile
import typing
from transcript_archive.location import archive_root
from stability_replay.rounds import authored_by, report, reviewed_path, Round, round_from, sample, stamp_seconds
from rung_replay.command import arguments_or_exit, down_from, failed, limited_exit, projects_on
from review_replay.rounds import authored_by, lost_inputs, report, reviewed_path, Round, round_from, sample, stamp_seconds
from project_key import checkout_of
from claude_code.reply import claude_reply, Limited
from rung_replay.command import down_from, failed, limited_exit, project_rung_or_exit, projects_on
from loop_claude.effects import Effects
from failure import Failure
from rung_replay.search import first_reply, replayed
from claude_code.transcript import first_timestamped_line, read_transcript
from model_routing.ladder import LADDER, project_rung, ReviewedPass, REVIEWERS, Rung
from claude_code.entries import parse_entry, structured_output
from review_folder import REPORTS, review_folder
from model_routing.ladder import Rung
TOOL = "stability-replay"
TOOL = "review-replay"
USAGE = f"{TOOL} <pass> <model>, <pass> one of {', '.join(REVIEWERS)} and <model> a model on the routing ladder"
LATEST = 10
REVIEWER = "stability-reviewer"
MATCH_SCHEMA = {"type": "object", "properties": {"found": {"type": "array", "items": {"type": "integer"}}}, "required": ["found"], "additionalProperties": False}
@@ -67,30 +67,56 @@ class Code:
def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, effects: Effects) -> int:
anchor = project_rung_or_exit(TOOL, argv, __doc__)
if isinstance(anchor, int):
return anchor
rounds = archived_rounds(anchor.model, archive_root(home, env), home)
read = arguments_or_exit(TOOL, argv, __doc__, USAGE, pass_and_model)
if isinstance(read, int):
return read
reviewed, anchor = read
rounds = archived_rounds(REVIEWERS[reviewed], anchor.model, archive_root(home, env), home)
if isinstance(rounds, Failure):
return failed(TOOL, rounds.reason)
codes = codes_found(rounds, effects)
if not codes:
return failed(TOOL, f"no archived {reviewed} round of projects on {anchor.model} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find")
chosen = sample(list(codes), LATEST)
# The project's rung is the reference, so the candidates start below it. They reach the ladder's lowest rung, below the lowest a worker routes to, since a pass may run below the worker's rung.
candidates = down_from(anchor, LADDER[0])[1:]
def found_on(found: Round, rung: Rung) -> set[int] | Limited:
return findings_found(found, codes[found], rung, anchor, home, effects)
replay = replayed(chosen, anchor, candidates, found_on, reproduced, f"{reviewed} rounds")
if isinstance(replay, Limited):
return limited_exit(TOOL, replay)
print(report(reviewed, anchor.model, anchor, chosen, replay))
return 0
def pass_and_model(argv: list[str]) -> tuple[ReviewedPass, Rung] | None:
"""The pass and the project's rung of the model argv names, as `<pass> <model>`; None for any other argv."""
if len(argv) != 2:
return None
reviewed = next((kind for kind in REVIEWERS if kind == argv[0]), None)
anchor = project_rung(argv[1])
return (reviewed, anchor) if reviewed and anchor else None
def codes_found(rounds: list[Round], effects: Effects) -> dict[Round, Code]:
"""The code each round reviewed, as code_of finds it; stderr names each round left out, and why."""
codes = {}
for found in rounds:
if isinstance(code := code_of(found, effects), Failure):
print(f"{found.root} {found.at} is left out: {code.reason}", file=sys.stderr, flush=True)
else:
codes[found] = code
if not codes:
return failed(TOOL, f"no archived stability round of projects on {anchor.model} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find")
chosen = sample(list(codes), LATEST)
# The project's rung is the reference, so the candidates start below it.
replay = replayed(chosen, anchor, down_from(anchor)[1:], lambda found, rung: findings_found(found, codes[found], rung, anchor, home, effects), lambda _, found: sorted(found), "stability rounds")
if isinstance(replay, Limited):
return limited_exit(TOOL, replay)
print(report(anchor.model, anchor, chosen, replay))
return 0
return codes
def archived_rounds(model: str, root: pathlib.Path, home: pathlib.Path) -> list[Round] | Failure:
def reproduced(_: Round, found: set[int]) -> list[int]:
"""The indexes of the round's findings that a replay on the reference reviewer found, which each lower rung is held to."""
return sorted(found)
def archived_rounds(reviewer: str, model: str, root: pathlib.Path, home: pathlib.Path) -> list[Round] | Failure:
projects = projects_on(model, home)
if isinstance(projects, Failure):
return projects
@@ -98,13 +124,13 @@ def archived_rounds(model: str, root: pathlib.Path, home: pathlib.Path) -> list[
try:
for key in projects:
for chunk in sorted(path for path in (root / key).glob("*") if path.is_dir()):
found += chunk_rounds(checkout_of(key, home), chunk)
found += chunk_rounds(reviewer, checkout_of(key, home), chunk)
except (OSError, UnicodeDecodeError, ValueError) as failure:
return Failure(f"could not read the archive {root}: {failure}")
return found
def chunk_rounds(checkout: str, chunk: pathlib.Path) -> list[Round]:
def chunk_rounds(reviewer: str, checkout: str, chunk: pathlib.Path) -> list[Round]:
copies = sorted(chunk.glob("*.jsonl"))
began = parse_entry(first_timestamped_line(copies[0].read_bytes())) if copies else None
if not began or not isinstance(began.get("timestamp"), str):
@@ -112,10 +138,10 @@ def chunk_rounds(checkout: str, chunk: pathlib.Path) -> list[Round]:
found = []
for meta in sorted(chunk.glob("session/*/subagents/*.meta.json")):
described = json.loads(meta.read_text(encoding="utf-8"))
if not isinstance(described, dict) or described.get("agentType") != REVIEWER:
if not isinstance(described, dict) or described.get("agentType") != reviewer:
continue
entries = read_transcript(str(meta.with_name(meta.name.removesuffix(".meta.json") + ".jsonl")))
if not isinstance(entries, Failure) and (one := round_from(checkout, stamp_seconds(began["timestamp"]), entries)):
if not isinstance(entries, Failure) and (one := round_from(reviewer, checkout, stamp_seconds(began["timestamp"]), entries)):
found.append(one)
return found
@@ -162,28 +188,60 @@ def findings_found(found: Round, code: Code, rung: Rung, anchor: Rung, home: pat
return matched
@dataclasses.dataclass(frozen=True)
class Copies:
"""The throwaway copies a replay hands its reviewer: clone, of the project's checkout, and reports, of the chunk's review folder."""
clone: str
reports: str
# The shell makes and removes the clone itself, and is tested on a temporary home as judge-replay's is; rounds.py's core takes the filesystem as an argument.
def review(found: Round, code: Code, rung: Rung, home: pathlib.Path, effects: Effects) -> str | Failure | Limited:
try:
(home / REPORTS).mkdir(mode=0o700, parents=True, exist_ok=True)
with tempfile.TemporaryDirectory(prefix="stability-replay-", ignore_cleanup_errors=True) as clone, tempfile.TemporaryDirectory(prefix="stability-replay-", dir=home / REPORTS, ignore_cleanup_errors=True) as reports:
return reviewed_in(found, code, rung, home, effects, clone, reports)
with tempfile.TemporaryDirectory(prefix="review-replay-", ignore_cleanup_errors=True) as clone, tempfile.TemporaryDirectory(prefix="review-replay-", dir=home / REPORTS, ignore_cleanup_errors=True) as reports:
return reviewed_in(found, code, rung, Copies(clone, reports), home, effects)
except OSError as failure:
return Failure(f"could not prepare the replay: {failure}")
def reviewed_in(found: Round, code: Code, rung: Rung, home: pathlib.Path, effects: Effects, clone: str, reports: str) -> str | Failure | Limited:
"""The report of found's reviewer on rung, replayed on code cloned into clone, with its review folder copied into reports."""
# The round's own report, any later one, and one from an earlier attempt at the item in the same worktree would each hand the reviewer the findings it is to find.
def reviewed_in(found: Round, code: Code, rung: Rung, copies: Copies, home: pathlib.Path, effects: Effects) -> str | Failure | Limited:
"""The report of found's reviewer on rung, replayed on code cloned into copies.clone, with its review folder copied into copies.reports."""
folder = pathlib.Path(review_folder(str(home), code.path))
copy_reports(found, folder, copies.reports)
prompt = redirected(found.prompt, home, folder, code.path, copies)
if isinstance(prompt, Failure):
return prompt
if lost := lost_inputs(prompt, copies.reports, lambda name: (pathlib.Path(copies.reports) / name).exists()):
return Failure(f"the prompt names {', '.join(lost)}, which the review folder did not hold from the chunk's start until the round's")
if isinstance(cloned := clone_reviewed(found, code, copies.clone, effects), Failure):
return cloned
reply = claude_reply(effects.claude, ["--agent", found.reviewer, *rung.flags()], copies.clone, prompt, "claude")
if isinstance(reply, Failure | Limited):
return reply
return reply["result"] if isinstance(reply.get("result"), str) else Failure("the reviewer replied with no report")
def copy_reports(found: Round, folder: pathlib.Path, reports: str) -> None:
"""Copies into reports each report written to folder after found's chunk began and before its round did."""
# The round's own report, any later one, and one from an earlier attempt at the item in the same worktree would each hand the reviewer the findings it is to find.
for written in folder.iterdir() if folder.is_dir() else ():
if written.is_file() and found.began <= written.stat().st_mtime < found.seconds:
shutil.copy2(written, reports)
# Replace the folder's path first: outside home it contains the worktree's path, which replacing code.path would otherwise turn into the clone's.
prompt = found.prompt.replace(str(folder), reports).replace(f"~/{folder.relative_to(home)}", reports).replace(code.path, clone)
def redirected(prompt: str, home: pathlib.Path, folder: pathlib.Path, worktree: str, copies: Copies) -> str | Failure:
"""prompt naming copies.reports in place of the review folder, in either spelling, and copies.clone in place of the worktree; a Failure where it still names the review folder."""
# Replace the folder's path first: outside home it contains the worktree's path, which replacing the worktree's would otherwise turn into the clone's.
prompt = prompt.replace(str(folder), copies.reports).replace(f"~/{folder.relative_to(home)}", copies.reports).replace(worktree, copies.clone)
# A spelling the rewrite misses, such as $HOME/, would leave the reviewer the live folder and the findings in it.
if REPORTS in prompt.replace(reports, ""):
if REPORTS in prompt.replace(copies.reports, ""):
return Failure("the prompt names the review folder in a spelling the replay does not redirect")
return prompt
def clone_reviewed(found: Round, code: Code, clone: str, effects: Effects) -> Failure | None:
"""Clones found's checkout into clone, on the commit its reviewer read, with the default branch where it stood as the chunk began."""
for command in (
("clone", "--quiet", "--shared", "--no-checkout", found.root, clone),
("-C", clone, "update-ref", f"refs/remotes/origin/{code.default}", code.base),
@@ -192,11 +250,7 @@ def reviewed_in(found: Round, code: Code, rung: Rung, home: pathlib.Path, effect
):
if isinstance(ran := git(effects, found.root, *command), Failure):
return ran
args = ["--agent", REVIEWER, *rung.flags()]
reply = claude_reply(effects.claude, args, clone, prompt, "claude")
if isinstance(reply, Failure | Limited):
return reply
return reply["result"] if isinstance(reply.get("result"), str) else Failure("the reviewer replied with no report")
return None
def match(findings: tuple[str, ...], replay_report: str, root: str, rung: Rung, effects: Effects) -> set[int] | Failure | Limited:
@@ -1,4 +1,4 @@
"""Which archived stability rounds to replay, the code each reviewed, and the report: docs/decisions.md → `routing: the stability reviewer runs no lower than the project's rung`."""
"""Which archived rounds of a review pass to replay, the code each reviewed, and the report: docs/decisions.md → `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds`."""
import dataclasses
import datetime
import pathlib
@@ -14,11 +14,14 @@ FINDING = re.compile(r"^[\s#>*+|-]*(?:\d+[.)]\s*)?[*_]*\s*(?:🔴|🟡)")
# An absolute path in a prompt, up to the quote, bracket or punctuation that ends it.
PATH = re.compile(r"/[^\s`'\"()<>\[\],;]+")
LINE_OR_STOP = re.compile(r"(?::\d+(?:-\d+)?)?[.:]?$")
# A round's report, as skills/larv-review/SKILL.md → "Reports" names it, such as stability-round-2.md or principles-goals-round-1-rerun.md.
ROUND_REPORT = re.compile(r"-round-\d+(?:-rerun)?\.md$")
@dataclasses.dataclass(frozen=True)
class Round:
"""One archived stability review of the checkout at root, at the UTC stamp at, in a chunk that began at began: its prompt, and the 🔴 and 🟡 findings its report holds, each with its lines up to the next blank line or finding."""
"""One archived review by the agent reviewer of the checkout at root, at the UTC stamp at, in a chunk that began at began: its prompt, and the 🔴 and 🟡 findings its report holds, each with its lines up to the next blank line or finding."""
reviewer: str
root: str
branch: str
at: str
@@ -31,15 +34,15 @@ class Round:
return stamp_seconds(self.at)
def round_from(root: str, began: float, entries: list[dict]) -> Round | None:
"""The round a stability reviewer's transcript holds: its first entry's prompt, and its last text. None without a 🔴 or 🟡 finding to hold a rung to."""
def round_from(reviewer: str, root: str, began: float, entries: list[dict]) -> Round | None:
"""The round a reviewer's transcript holds: its first entry's prompt, and its last text. None without a 🔴 or 🟡 finding to hold a rung to."""
reports = [text for found in entries for part in assistant_parts(found) if (text := part_text(part))]
first = entries[0] if entries else {}
if not stamped(first) or not reports or not (findings := recorded_findings(reports[-1])):
return None
prompt = content(first)
prompt = prompt if isinstance(prompt, str) else "".join(part_text(part) or "" for part in parts(first))
return Round(root, first.get("gitBranch", ""), first["timestamp"], began, prompt, tuple(findings))
return Round(reviewer, root, first.get("gitBranch", ""), first["timestamp"], began, prompt, tuple(findings))
def stamped(entry: dict) -> bool:
@@ -84,6 +87,12 @@ def reviewed_path(prompt: str, root: str, exists: typing.Callable[[str], bool])
return root if not gone and root in named else None
def lost_inputs(prompt: str, reports: str, held: typing.Callable[[str], bool]) -> list[str]:
"""The files the prompt names in reports, the copy of a round's review folder, that the copy does not hold, other than round reports. A report the copy lacks is the round's own or a later one, left out on purpose; any other file it lacks, such as a branch diff written again after the round began, would leave the replay without an input the round had."""
named = [LINE_OR_STOP.sub("", path).removeprefix(reports + "/") for path in PATH.findall(prompt) if path.startswith(reports + "/")]
return sorted({name for name in named if not ROUND_REPORT.search(name) and not held(name)})
def authored_by(log: list[tuple[str, float]], seconds: float) -> str | None:
"""The newest commit in log, which lists newest first, authored at or before seconds. A rebase keeps author times, so a merged branch's commits still say what it held."""
return next((commit for commit, authored in log if authored <= seconds), None)
@@ -94,10 +103,10 @@ def sample(rounds: list[Round], latest: int) -> list[Round]:
return sorted(rounds, key=lambda found: (found.seconds, found.root))[-latest:]
def report(model: str, reference: Rung, rounds: list[Round], replay: Replay[Round, int, set[int]]) -> str:
def report(reviewed: str, model: str, reference: Rung, rounds: list[Round], replay: Replay[Round, int, set[int]]) -> str:
total = sum(len(found_round.findings) for found_round in rounds)
lines = [
f"Replayed {len(rounds)} stability rounds of {model} projects, holding {total} 🔴 and 🟡 findings.",
f"Replayed {len(rounds)} {reviewed} rounds of {model} projects, holding {total} 🔴 and 🟡 findings.",
f"The reference reviewer, {reference}, found {sum(len(found) for found in replay.on_reference)} of them.",
]
for tried in replay.tried:
@@ -14,14 +14,14 @@ import unittest.mock
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from stability_replay.rounds import authored_by, recorded_findings, report, reviewed_path, Round, round_from, sample, stamp_seconds
from review_replay.rounds import authored_by, lost_inputs, recorded_findings, report, reviewed_path, Round, round_from, sample, stamp_seconds
from loop_claude.effects import Effects, system_effects
from failure import Failure
from stability_replay import main
from review_replay import main
from rung_replay.search import Miss, Replay, Tried
from model_routing.ladder import Rung
from model_routing.ladder import REVIEWERS, Rung
TOOL = pathlib.Path(__file__).resolve().parents[1] / "stability-replay"
TOOL = pathlib.Path(__file__).resolve().parents[1] / "review-replay"
REFERENCE = Rung("claude-opus-5-5", "medium")
MIDDLE = Rung("claude-sonnet-5-5", "high")
CHEAP = Rung("claude-haiku-5-5", "max")
@@ -32,7 +32,7 @@ def entry(kind, text, **fields):
def a_round(at, findings=("🔴 a",)):
return Round("/code/p", "x", f"2026-10-14T10:00:0{at}Z", 0, "Review it.", tuple(findings))
return Round("stability-reviewer", "/code/p", "x", f"2026-10-14T10:00:0{at}Z", 0, "Review it.", tuple(findings))
class RecordedFindings(unittest.TestCase):
@@ -51,19 +51,19 @@ class RoundFrom(unittest.TestCase):
entry("assistant", "Reading."),
entry("assistant", "BLOCK\n\n🔴 a.py:1 breaks"),
]
self.assertEqual(round_from("/code/p", 5.0, entries), Round("/code/p", "x", "2026-10-14T10:00:00.500Z", 5.0, "Review branch x in /code/p-x.", ("🔴 a.py:1 breaks",)))
self.assertEqual(round_from("/code/p", 5.0, entries).seconds, 1791972000.5)
self.assertEqual(round_from("stability-reviewer", "/code/p", 5.0, entries), Round("stability-reviewer", "/code/p", "x", "2026-10-14T10:00:00.500Z", 5.0, "Review branch x in /code/p-x.", ("🔴 a.py:1 breaks",)))
self.assertEqual(round_from("stability-reviewer", "/code/p", 5.0, entries).seconds, 1791972000.5)
def test_reads_a_prompt_given_as_a_string(self):
start = {"type": "user", "message": {"content": "Review branch x in /code/p-x."}, "timestamp": "2026-10-14T10:00:00Z", "gitBranch": "x"}
self.assertEqual(round_from("/code/p", 5.0, [start, entry("assistant", "🔴 a")]).prompt, "Review branch x in /code/p-x.")
self.assertEqual(round_from("stability-reviewer", "/code/p", 5.0, [start, entry("assistant", "🔴 a")]).prompt, "Review branch x in /code/p-x.")
def test_skips_a_round_with_no_red_or_yellow_finding_or_no_report(self):
start = entry("user", "Review.", timestamp="2026-10-14T10:00:00Z", cwd="/code/p", gitBranch="x")
self.assertIsNone(round_from("/code/p", 5.0, [start, entry("assistant", "APPROVE\n\n🟢 nit")]))
self.assertIsNone(round_from("/code/p", 5.0, [start]))
self.assertIsNone(round_from("/code/p", 5.0, []))
self.assertIsNone(round_from("/code/p", 5.0, [{**start, "timestamp": "yesterday"}, entry("assistant", "🔴 a")]))
self.assertIsNone(round_from("stability-reviewer", "/code/p", 5.0, [start, entry("assistant", "APPROVE\n\n🟢 nit")]))
self.assertIsNone(round_from("stability-reviewer", "/code/p", 5.0, [start]))
self.assertIsNone(round_from("stability-reviewer", "/code/p", 5.0, []))
self.assertIsNone(round_from("stability-reviewer", "/code/p", 5.0, [{**start, "timestamp": "yesterday"}, entry("assistant", "🔴 a")]))
class ReviewedPath(unittest.TestCase):
@@ -79,6 +79,13 @@ class ReviewedPath(unittest.TestCase):
self.assertIsNone(reviewed_path("Review /code/p-x and /code/p-43.", "/code/p", exists))
class LostInputs(unittest.TestCase):
def test_names_each_file_the_prompt_names_in_the_copy_that_it_lacks_other_than_a_round_report(self):
prompt = "Read /c/branch.diff and /c/ls-files.txt:3. Earlier: /c/prose-round-1.md. Write /c/principles-goals-round-2-rerun.md; see /elsewhere/x.diff."
self.assertEqual(lost_inputs(prompt, "/c", {"ls-files.txt"}.__contains__), ["branch.diff"])
self.assertEqual(lost_inputs(prompt, "/c", {"branch.diff", "ls-files.txt"}.__contains__), [])
class AuthoredBy(unittest.TestCase):
def test_takes_the_newest_commit_authored_by_then(self):
log = [("c3", 30), ("c2", 20), ("c1", 10)]
@@ -96,7 +103,7 @@ class Sample(unittest.TestCase):
class Report(unittest.TestCase):
def test_counts_the_findings_and_names_each_miss_and_the_lowest_rung_that_missed_none(self):
rounds = [a_round(1, ("🔴 a\nmore", "🟡 b")), a_round(2)]
lines = report("claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, set()], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(rounds[0], 0, {1})])], MIDDLE)).splitlines()
lines = report("stability", "claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, set()], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(rounds[0], 0, {1})])], MIDDLE)).splitlines()
self.assertEqual(lines, [
"Replayed 2 stability rounds of claude-opus-5-5 projects, holding 3 🔴 and 🟡 findings.",
"The reference reviewer, claude-opus-5-5 at medium effort, found 2 of them.",
@@ -105,8 +112,8 @@ class Report(unittest.TestCase):
" /code/p 2026-10-14T10:00:01Z: 🔴 a",
"Lowest rung that finds every finding the reference reviewer finds: claude-sonnet-5-5 at high effort",
])
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, {0}], [Tried(MIDDLE, [Miss(rounds[1], 0, set())])], None)).splitlines()[-1], "Lowest rung that finds every finding the reference reviewer finds: none below the project's rung")
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay([set(), set()], [], None)).splitlines()[-1], "No rung tried: the reference reviewer found no finding to hold one to.")
self.assertEqual(report("stability", "claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, {0}], [Tried(MIDDLE, [Miss(rounds[1], 0, set())])], None)).splitlines()[-1], "Lowest rung that finds every finding the reference reviewer finds: none below the project's rung")
self.assertEqual(report("stability", "claude-opus-5-5", REFERENCE, rounds, Replay([set(), set()], [], None)).splitlines()[-1], "No rung tried: the reference reviewer found no finding to hold one to.")
class Match(unittest.TestCase):
@@ -174,6 +181,7 @@ class Main(unittest.TestCase):
for key in ("proj", "other"):
self.write_chunk(key, "c1", "2026-10-14T10:00:00Z")
self.reviews = []
self.agents = set()
def write_chunk(self, key, chunk, at, began="2026-10-14T09:15:00Z"):
"""A chunk that began at began, whose stability round began at at."""
@@ -207,6 +215,7 @@ class Main(unittest.TestCase):
if "--agent" in args:
shown = [subprocess.run(["git", *command], cwd=cwd, capture_output=True, text=True).stdout for command in (["show", "HEAD:a.txt"], ["branch", "--show-current"], ["show", "origin/main:a.txt"])]
self.reviews.append((rung, *shown, prompt, cwd))
self.agents.add(args[args.index("--agent") + 1])
result = review(rung)
else:
reviewed = Rung(*re.search(r"review on (\S+) (\S+)", prompt).groups())
@@ -221,7 +230,7 @@ class Main(unittest.TestCase):
return code, out.getvalue(), err.getvalue()
def test_replays_the_models_rounds_on_the_commit_reviewed_from_the_project_rung_down(self):
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0] if rung.model == "claude-haiku-5-5" else [0, 1]})
code, out, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0] if rung.model == "claude-haiku-5-5" else [0, 1]})
self.assertEqual(code, 0, err)
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE, MIDDLE, CHEAP])
_, reviewed, branch, base, prompt, cwd = self.reviews[0]
@@ -233,6 +242,18 @@ class Main(unittest.TestCase):
self.assertIn(f" {self.checkout} 2026-10-14T10:00:00Z: 🟡 slow", out)
self.assertTrue(out.endswith("finds: claude-sonnet-5-5 at high effort\n"))
def test_replays_the_prose_pass_with_the_prose_reviewer(self):
code, out, err = self.run_main(["prose", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
self.assertEqual(code, 0, err)
self.assertEqual(self.agents, {"prose-reviewer"})
self.assertTrue(out.startswith("Replayed 1 prose rounds"), out)
def test_tries_each_rung_down_to_the_ladders_lowest_below_the_lowest_a_worker_routes_to(self):
code, out, err = self.run_main(["prose", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
self.assertEqual(code, 0, err)
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE, MIDDLE, CHEAP, *(Rung("claude-haiku-5-5", effort) for effort in ("xhigh", "high", "medium", "low"))])
self.assertTrue(out.endswith("finds: claude-haiku-5-5 at low effort\n"), out)
def review_folder_with_round_1(self):
"""The chunk's review folder, holding round 1's report from before the replayed round began at 10:00, that round's own and a later pass's reports, written after, and an earlier attempt's report, from before the chunk began at 09:15."""
folder = self.home / ".local" / "state" / "larv-review" / "code" / "proj-x"
@@ -258,7 +279,7 @@ class Main(unittest.TestCase):
(named / "stability-round-2.md").write_text("replayed\n")
return {"type": "result", "result": f"review on {rung.model} {rung.effort}"}
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0, 1]}, review)
code, _, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0, 1]}, review)
self.assertEqual(code, 0, err)
named, copied, earlier = seen[0]
self.assertEqual(named.parent, folder.parents[1], "the copy sits in the reports folder, where rule-hook lets the reviewer write")
@@ -272,28 +293,47 @@ class Main(unittest.TestCase):
def test_points_the_review_folders_absolute_path_at_the_copy(self):
folder = self.review_folder_with_round_1()
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to {folder}/stability-round-2.md.")
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0, 1]})
code, _, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
self.assertEqual(code, 0, err)
prompt = self.reviews[0][4]
self.assertNotIn(str(folder), prompt)
self.assertRegex(prompt, rf"report to {folder.parents[1]}/stability-replay-\w+/stability-round-2\.md\.$")
self.assertRegex(prompt, rf"report to {folder.parents[1]}/review-replay-\w+/stability-round-2\.md\.$")
def test_replays_no_round_whose_prompt_still_names_the_review_folder_after_the_rewrite(self):
self.review_folder_with_round_1()
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to $HOME/.local/state/larv-review/code/proj-x/stability-round-2.md.")
code, out, err = self.run_main(["claude-opus-5-5"], unused)
code, out, err = self.run_main(["stability", "claude-opus-5-5"], unused)
self.assertEqual(code, 0, err)
self.assertEqual(self.reviews, [])
self.assertIn(f"on {REFERENCE} found nothing: the prompt names the review folder in a spelling the replay does not redirect", err)
self.assertTrue(out.endswith("No rung tried: the reference reviewer found no finding to hold one to.\n"))
def test_replays_a_round_whose_prompt_names_an_input_written_before_it_began_naming_the_copy(self):
folder = self.review_folder_with_round_1()
(folder / "branch.diff").write_text("the diff\n")
before = stamp_seconds("2026-10-14T09:50:00Z")
os.utime(folder / "branch.diff", (before, before))
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. The diff is ~/.local/state/larv-review/code/proj-x/branch.diff. Write your report to ~/.local/state/larv-review/code/proj-x/stability-round-2.md.")
code, _, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
self.assertEqual(code, 0, err)
self.assertRegex(self.reviews[0][4], rf"The diff is {folder.parents[1]}/review-replay-\w+/branch\.diff\.")
def test_replays_no_round_whose_prompt_names_an_input_the_folder_holds_only_as_rewritten_after_the_round_began(self):
folder = self.review_folder_with_round_1()
(folder / "branch.diff").write_text("a later diff\n")
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. The diff is ~/.local/state/larv-review/code/proj-x/branch.diff. Write your report to ~/.local/state/larv-review/code/proj-x/stability-round-2.md.")
code, out, err = self.run_main(["stability", "claude-opus-5-5"], unused)
self.assertEqual(code, 0, err)
self.assertEqual(self.reviews, [])
self.assertIn(f"on {REFERENCE} found nothing: the prompt names branch.diff, which the review folder did not hold from the chunk's start until the round's", err)
def test_finds_nothing_and_leaves_no_folder_behind_when_it_cannot_copy_the_reports(self):
folder = self.review_folder_with_round_1()
(folder / "stability-round-1.md").chmod(0)
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to ~/.local/state/larv-review/code/proj-x/stability-round-2.md.")
scratch = pathlib.Path(self.enterContext(tempfile.TemporaryDirectory()))
self.enterContext(unittest.mock.patch.object(tempfile, "tempdir", str(scratch)))
code, _, err = self.run_main(["claude-opus-5-5"], unused)
code, _, err = self.run_main(["stability", "claude-opus-5-5"], unused)
self.assertEqual(code, 0, err)
self.assertEqual(self.reviews, [])
self.assertIn(f"on {REFERENCE} found nothing: could not prepare the replay: ", err)
@@ -302,13 +342,13 @@ class Main(unittest.TestCase):
def test_replays_the_rounds_of_every_chunk(self):
self.write_chunk("proj", "c2", "2026-10-14T10:30:00Z")
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0, 1]})
code, out, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
self.assertEqual(code, 0, err)
self.assertTrue(out.startswith("Replayed 2 stability rounds"), out)
def test_checks_out_the_commit_detached_when_the_reviewer_started_on_the_default_branch(self):
self.set_branch("main")
self.assertEqual(self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0]})[0], 0)
self.assertEqual(self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0]})[0], 0)
self.assertEqual(self.reviews[0][1:3], ("1\n", ""))
def test_sets_the_default_branch_to_its_own_line_and_not_to_a_branch_merged_after_the_chunk_began(self):
@@ -320,20 +360,20 @@ class Main(unittest.TestCase):
run_git(self.checkout, "checkout", "-q", "main")
run_git(self.checkout, "merge", "-q", "--no-ff", "-m", "m", "side", at="2026-10-14T09:40:00Z")
run_git(self.checkout, "push", "-qf", "origin", "main")
self.assertEqual(self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0]})[0], 0)
self.assertEqual(self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0]})[0], 0)
self.assertEqual(self.reviews[0][1:4], ("s\n", "x\n", "0\n"))
def test_finds_nothing_in_a_round_whose_review_has_no_report(self):
code, out, err = self.run_main(["claude-opus-5-5"], unused, review=lambda rung: {"type": "result"})
code, out, err = self.run_main(["stability", "claude-opus-5-5"], unused, review=lambda rung: {"type": "result"})
self.assertEqual(code, 0, err)
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE] * 3)
self.assertIn(f" {self.checkout} 2026-10-14T10:00:00Z on {REFERENCE} found nothing: the reviewer replied with no report", err)
self.assertTrue(out.endswith("No rung tried: the reference reviewer found no finding to hold one to.\n"))
def assert_no_round(self):
code, _, err = self.run_main(["claude-opus-5-5"], unused)
code, _, err = self.run_main(["stability", "claude-opus-5-5"], unused)
self.assertEqual(code, 1)
self.assertEqual(err.splitlines(), [f"stability-replay: no archived stability round of projects on claude-opus-5-5 in {self.archive} holds a 🔴 or 🟡 on code it can find"])
self.assertEqual(err.splitlines(), [f"review-replay: no archived stability round of projects on claude-opus-5-5 in {self.archive} holds a 🔴 or 🟡 on code it can find"])
def test_skips_a_chunk_with_no_transcript_copy(self):
(self.archive / "proj" / "c1" / "t0.jsonl").unlink()
@@ -344,7 +384,7 @@ class Main(unittest.TestCase):
self.assert_no_round()
def assert_left_out(self, reason, root=None):
code, _, err = self.run_main(["claude-opus-5-5"], unused)
code, _, err = self.run_main(["stability", "claude-opus-5-5"], unused)
self.assertEqual(code, 1)
self.assertIn(f"{root or self.checkout} 2026-10-14T10:00:00Z is left out: {reason}", err)
@@ -366,23 +406,25 @@ class Main(unittest.TestCase):
self.assert_left_out("no commit was authored by then")
def test_exits_1_on_a_usage_limit_bad_arguments_or_no_round_to_replay(self):
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung: {"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
code, _, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
self.assertEqual(code, 1)
self.assertIn("usage limit: You've hit your limit", err)
for argv in ([], ["off-ladder"], ["claude-opus-5-5", "x"]):
self.assertEqual(self.run_main(argv, unused)[0], 1)
code, _, err = self.run_main(["claude-haiku-5-5"], unused)
for argv in ([], ["claude-opus-5-5"], ["full scan", "claude-opus-5-5"], ["stability", "off-ladder"], ["stability", "claude-opus-5-5", "x"]):
with self.subTest(argv=argv):
self.assertEqual(self.run_main(argv, unused)[0], 1)
code, _, err = self.run_main(["stability", "claude-haiku-5-5"], unused)
self.assertEqual(code, 1)
self.assertIn("no archived stability round of projects on claude-haiku-5-5", err)
def test_prints_the_help(self):
code, out, _ = self.run_main(["-h"], unused)
self.assertEqual((code, out.splitlines()[2]), (0, "usage: stability-replay <model>"))
self.assertEqual((code, out.splitlines()[2]), (0, "usage: review-replay <pass> <model>"))
self.assertIn(f"{', '.join(list(REVIEWERS)[:-1])} or {list(REVIEWERS)[-1]}.", " ".join(out.split()))
def test_the_launcher_prints_the_help(self):
ran = subprocess.run([sys.executable, str(TOOL), "--help"], capture_output=True, text=True)
self.assertEqual(ran.returncode, 0)
self.assertIn("usage: stability-replay <model>", ran.stdout)
self.assertIn("usage: review-replay <pass> <model>", ran.stdout)
if __name__ == "__main__":
+4 -1
View File
@@ -1,10 +1,13 @@
"""Refuse a reviewer's file call other than a Write of its report into the reports folder: skills/larv-review/review-findings.md → "Report"."""
import os
import pathlib
from rule_hook.rules import Breach
from rule_hook.call import Call
from review_folder import REPORTS
from rule_hook.reviewer_rung import REVIEWERS
# Each agent under agents/, read from the folder, so an agent the routing tables miss is still confined.
REVIEWERS = frozenset(path.stem for path in (pathlib.Path(__file__).resolve().parents[2] / "agents").glob("*.md"))
def report_breaches(call: Call, path: str, home: str) -> list[Breach]:
+10 -12
View File
@@ -1,14 +1,13 @@
"""Refuse an Agent call that launches a review below the rung it needs, worked out from the rungs loop-claude hands its worker: docs/decisions.md → `routing:`."""
import pathlib
"""Refuse an Agent call that launches a review below the rung it needs, worked out from the rungs and the E + I loop-claude hands its worker: docs/decisions.md → `routing:`."""
import typing
from rule_hook.rules import Breach
from rule_hook.call import Call, text
from model_routing.worker_rungs import handed
from model_routing.ladder import highest, position, review_rung, ReviewKind, Rung
from model_routing.ladder import position, review_rungs, REVIEWERS, ReviewKind, Rung
# A reviewer is an agent under agents/, as CLAUDE.md → "Vocabulary" defines it.
REVIEWERS = frozenset(path.stem for path in (pathlib.Path(__file__).resolve().parents[2] / "agents").glob("*.md"))
# The pass each agent under agents/ reviews.
PASS_OF = {agent: kind for kind, agent in REVIEWERS.items()}
# The opening of every comprehension seat's prompt: skills/comprehension-panel/seat-prompts.md → "Opening".
SEAT = "You are one seat of a comprehension panel."
# The opening of the security pass's prompt: skills/larv-review/SKILL.md → "Security".
@@ -18,23 +17,22 @@ SUBJECTS: dict[ReviewKind, str] = {"seat": "a comprehension seat", "security": "
def reviewer_breaches(call: Call, environ: typing.Mapping[str, str]) -> list[Breach]:
"""The breach of an Agent call that launches a review below its rung; none where loop-claude handed no rungs."""
project, worker, reviewers = handed(environ)
project, worker, escape_and_impact = handed(environ)
kind = review_kind(call.given)
if kind is None or project is None or worker is None:
return []
# A worker an older loop-claude started was handed no reviewers' rung; its reviewers are held to the rung that loop-claude gave them.
required = review_rung(kind, project, reviewers or highest(project, worker))
required = review_rungs(project, worker, escape_and_impact)[kind]
named = named_rung(call.given, worker)
if position(named) >= position(required):
return []
subject = text(call.given, "subagent_type") if kind == "reviewer" else SUBJECTS[kind]
subject = SUBJECTS.get(kind) or text(call.given, "subagent_type")
return [Breach("reviewer-rung", f"{subject} would run on {named}, below {required}: launch it with {required.agent_args()}", call.cwd)]
def review_kind(given: dict[str, typing.Any]) -> ReviewKind | None:
"""What a launch reviews, a reviewer, a comprehension seat or the security pass; None for any other subagent."""
if text(given, "subagent_type") in REVIEWERS:
return "reviewer"
"""What a launch reviews, a reviewer's pass, a comprehension seat or the security pass; None for any other subagent."""
if pass_reviewed := PASS_OF.get(text(given, "subagent_type")):
return pass_reviewed
opening = text(given, "prompt").lstrip()
if opening.startswith(SEAT):
return "seat"
+20 -8
View File
@@ -5,6 +5,7 @@ import subprocess
import sys
import tempfile
import unittest
import unittest.mock
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
@@ -14,8 +15,10 @@ from rule_hook.call import Call
from git_command import Commit, commit_message, commits
from test_rank_todo import DECISIONS, README, TODO
from test_env import isolated_env
from model_routing import ladder
from rule_hook.override_log import LogLine, parsed_entry, this_chunks
from rule_hook.reviewer_rung import reviewer_breaches
from model_routing.ladder import Rung
from shell_command import simple_commands
HOOK = pathlib.Path(__file__).resolve().parents[1] / "rule-hook"
@@ -167,9 +170,9 @@ class OverrideLog(unittest.TestCase):
class ReviewerRung(unittest.TestCase):
# A worker routed to Haiku 5.5 at max, on a project whose rung is Opus 5.5 at medium, with its reviewers on the project's rung.
ENVIRON = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": "claude-opus-5-5 at medium effort"}
# The same project, with a worker and its reviewers routed above it to Fable 5.1 at max.
ABOVE = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-fable-5-1 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": "claude-fable-5-1 at max effort"}
ENVIRON = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "4"}
# The same project, with a worker, and so its reviewers, routed above it to Fable 5.1 at max.
ABOVE = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-fable-5-1 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "4"}
def refused(self, given: dict, environ: dict[str, str] = ENVIRON) -> list[str]:
return rules(reviewer_breaches(Call("Agent", given, "/r"), environ))
@@ -200,20 +203,29 @@ class ReviewerRung(unittest.TestCase):
def test_an_effort_off_the_ladder_for_its_model_is_refused(self):
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "opus", "effort": "low"}), ["reviewer-rung"])
def test_a_reviewer_is_held_to_the_reviewers_rung_where_that_is_higher(self):
def test_a_reviewer_is_held_to_the_rung_its_e_plus_i_picks_where_that_is_higher(self):
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}, self.ABOVE), ["reviewer-rung"])
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "fable", "effort": "max"}, self.ABOVE), [])
lifted = self.ENVIRON | {"LOOP_CLAUDE_REVIEWER_RUNG": "claude-opus-5-5 at high effort"}
lifted = self.ENVIRON | {"LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5"}
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}, lifted), ["reviewer-rung"])
self.assertEqual(self.refused({"prompt": "You are one seat of a comprehension panel.", "model": "opus", "effort": "medium"}, lifted), [])
def test_holds_a_reviewer_to_the_higher_of_the_workers_and_the_projects_rung_where_loop_claude_handed_no_reviewers_rung(self):
without = {name: rung for name, rung in self.ABOVE.items() if name != "LOOP_CLAUDE_REVIEWER_RUNG"}
def test_holds_a_reviewer_to_the_higher_of_the_workers_and_the_projects_rung_where_loop_claude_handed_no_e_plus_i(self):
without = {name: rung for name, rung in self.ABOVE.items() if name != "LOOP_CLAUDE_ESCAPE_AND_IMPACT"}
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}, without), ["reviewer-rung"])
below = {name: rung for name, rung in self.ENVIRON.items() if name != "LOOP_CLAUDE_REVIEWER_RUNG"}
below = {name: rung for name, rung in self.ENVIRON.items() if name != "LOOP_CLAUDE_ESCAPE_AND_IMPACT"}
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "sonnet", "effort": "high"}, below), ["reviewer-rung"])
self.assertEqual(self.refused({"prompt": "You are one seat of a comprehension panel.", "model": "sonnet", "effort": "high"}, without), ["reviewer-rung"])
def test_a_replayed_passs_reviewer_is_held_to_its_replays_rung_below_the_workers_unless_the_e_plus_i_steps_reviews_up_or_the_worker_climbed(self):
lowered = {"subagent_type": "prose-reviewer", "model": "haiku", "effort": "medium"}
with unittest.mock.patch.dict(ladder.REPLAYED, {("prose", "claude-opus-5-5"): Rung("claude-haiku-5-5", "medium")}):
self.assertEqual(self.refused(lowered), [])
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "haiku", "effort": "low"}), ["reviewer-rung"])
self.assertEqual(self.refused({**lowered, "subagent_type": "stability-reviewer"}), ["reviewer-rung"])
self.assertEqual(self.refused(lowered, self.ENVIRON | {"LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5"}), ["reviewer-rung"])
self.assertEqual(self.refused(lowered, self.ABOVE), ["reviewer-rung"])
def test_a_seat_or_a_security_launch_is_held_to_the_project_rung_alone(self):
seat = {"prompt": "You are one seat of a comprehension panel. Tools: Read, Grep, Glob.", "model": "opus", "effort": "medium"}
self.assertEqual(self.refused(seat, self.ABOVE), [])
+16 -11
View File
@@ -1,22 +1,27 @@
"""What judge-replay's and stability-replay's commands share: the argument, the projects on its model, the candidate rungs and the exits."""
"""What judge-replay's and review-replay's commands share: reading the arguments, the projects on a model, the candidate rungs and the exits."""
import pathlib
import sys
import typing
from failure import Failure
from model_routing.ladder import LADDER, lowest_worker_rung, position, project_rung, Rung
from model_routing.ladder import LADDER, position, Rung
from claude_code.reply import Limited
from loop_claude.project.model_choice import read_models_file
from loop_claude.stop import Stop
def project_rung_or_exit(tool: str, argv: list[str], help_text: str) -> Rung | int:
"""The project's rung of the model argv names, or the exit code: 0 after printing help_text for -h or --help, 1 after saying why argv cannot be read."""
# What a tool reads from its arguments, such as the project's rung of the model they name.
Parsed = typing.TypeVar("Parsed")
def arguments_or_exit(tool: str, argv: list[str], help_text: str, usage: str, read: typing.Callable[[list[str]], Parsed | None]) -> Parsed | int:
"""What read makes of argv, or the exit code: 0 after printing help_text for -h or --help, 1 after saying, with usage, that argv cannot be read, where read gives None."""
if argv[:1] in (["-h"], ["--help"]):
print(help_text.strip())
return 0
anchor = project_rung(argv[0]) if len(argv) == 1 else None
if anchor is None:
return failed(tool, f"cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; usage: {tool} <model>, a model on the routing ladder")
return anchor
arguments = read(argv)
if arguments is None:
return failed(tool, f"cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; usage: {usage}")
return arguments
def projects_on(model: str, home: pathlib.Path) -> list[str] | Failure:
@@ -27,9 +32,9 @@ def projects_on(model: str, home: pathlib.Path) -> list[str] | Failure:
return sorted(key for key, on in models.items() if on == model)
def down_from(top: Rung) -> list[Rung]:
"""top, the project's rung, and each rung below it, highest first, down to the lowest rung a worker routes to: no route uses a rung below it, so a replay there could change no routing."""
return list(LADDER[position(lowest_worker_rung(top)):position(top) + 1])[::-1]
def down_from(top: Rung, floor: Rung) -> list[Rung]:
"""top, the project's rung, and each rung below it, highest first, down to floor."""
return list(LADDER[position(floor):position(top) + 1])[::-1]
def limited_exit(tool: str, limited: Limited) -> int:
+1 -1
View File
@@ -1,4 +1,4 @@
"""The rung search judge-replay and stability-replay share: replay archived cases on a reference rung, then hold each candidate rung to what the reference reproduced."""
"""The rung search judge-replay and review-replay share: replay archived cases on a reference rung, then hold each candidate rung to what the reference reproduced."""
import concurrent.futures
import dataclasses
import sys
+3 -2
View File
@@ -63,8 +63,9 @@ class FirstReply(unittest.TestCase):
class DownFrom(unittest.TestCase):
def test_runs_from_the_projects_rung_down_to_the_lowest_rung_a_worker_routes_to(self):
self.assertEqual(down_from(REFERENCE), [REFERENCE, MIDDLE, CHEAP])
def test_runs_from_the_projects_rung_down_to_the_floor_highest_first(self):
self.assertEqual(down_from(REFERENCE, CHEAP), [REFERENCE, MIDDLE, CHEAP])
self.assertEqual(down_from(REFERENCE, MIDDLE), [REFERENCE, MIDDLE])
if __name__ == "__main__":
+46
View File
@@ -2257,3 +2257,49 @@ Junior A, decided by:
- Self-sufficiency, Shape: `bin/loop_claude/cycle/judge.py:143` `Grounds`: each field documents what must quote it and what a misquote does, though `judge` and `judge_once` grow to nine positional parameters
- Shape: `bin/loop_claude/routing.py:84` `one_rung_up`: removes the name it shared with `Cycle.stepped_from`, which meant the opposite
- Self-sufficiency: `bin/loop_claude/cycle/prompts.py:87` `escalation`: a worked example of the exact sentences, as its neighbours already had
## 2026-10-18T16:57:38Z, PR #322 at 49e86fd, against 2911d3f
Ruling: same
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|---|---|---|---|---|---|
| Mid A | better | same | better | same | better |
| Inherited architect | same | worse | better | same | same |
| Maintainability senior | same | better | worse | same | same |
| Junior A | better | better | same | same | better |
Mid A, decided by:
- Navigation, Shape: `bin/review_replay/main.py:1` `review_replay`: the tool takes any pass and its name says so, with `REVIEWERS` listing the passes and the usage printing them, where the old name pointed at stability alone
- Shape: `bin/review_replay/main.py:208` `reviewed_in`: copy, rewrite, clone and run are each a named step with a docstring, where `bin/stability_replay/main.py:175` did all four in one body; `codes_found` takes the left-out loop out of `main`
- Locality: `bin/rule_hook/reviewer_rung.py:18` `reviewer_breaches`: asks `review_rungs(...)[kind]` directly, without the older loop-claude fallback or the filesystem glob
- Locality: `bin/loop_claude/cycle/prompts.py:72` `taken_in_launches`: compares whole `review_rungs` dicts, right only because of a condition inside `ladder.review_rungs` that a comment states
- Locality: `bin/model_routing/ladder.py:71` `REPLAYED`: an empty table that `review_rungs`, `lowered`, `shared_launches` and `reruns` all branch on, so a reader follows a path no data drives
- Shape: `bin/rule_hook/reviewer_report.py:10` `REVIEWERS`: a frozenset of agent files, while `ladder.py`'s `REVIEWERS` is a dict from pass to agent; one name for two shapes
- Shape: `bin/loop_claude/cycle/prompts.py:43` `review_launches`: fan-out grows by four units and "each reviewer" becomes a magic key, but stays bounded with docstring examples
Inherited architect, decided by:
- Locality: `bin/loop_claude/cycle/prompts.py:43` `review_launches`: one launch sentence now needs `REVIEWS`' per-pass rows, `shared_launches` skipping them for an `"each reviewer"` key, and `lowered` putting them back, where the base was one loop over three rows
- Locality: `bin/loop_claude/cycle/prompts.py:78` `taken_in_launches`: compares the whole `review_rungs` dict, correct only by a rule enforced in `ladder.review_rungs`
- Locality: `bin/model_routing/ladder.py:71` `REPLAYED`: a hand-edited table, empty today, that overrides a pass's rung below the worker's under a combined E + I and worker condition; a second source of truth
- Shape: `bin/review_replay/main.py:208` `reviewed_in`: split into `copy_reports`, `redirected` and `clone_reviewed`, each with a docstring; `codes_found` and `pass_and_model` take inline loops and lambdas out of `main`
- Shape: `bin/rung_replay/command.py:16` `arguments_or_exit`: argument parsing is no longer hard-wired to one model, and `down_from(top, floor)` makes judge-replay's floor show at its call site
- Shape: `bin/rule_hook/reviewer_report.py:10` `REVIEWERS`: shares its name with `ladder.py`'s hand-kept dict of a different shape
Maintainability senior, decided by:
- Shape: `bin/loop_claude/cycle/prompts.py:14` `REVIEWS`: six rows built from `REVIEWERS`, a separate `"each reviewer"` entry, a `kind not in REVIEWERS` filter and a `lowered` list re-joined in two places, all serving an empty `REPLAYED`
- Shape: `bin/model_routing/ladder.py:69` `REVIEWERS`: two modules on one rule-hook call path each hold a `REVIEWERS` of a different type and meaning, where the base had one
- Shape: `bin/model_routing/ladder.py:104` `review_rungs`: fans out to every pass then overlays `REPLAYED` under a condition only a dense docstring paragraph explains; `taken_in_launches` compares whole dicts
- Locality: `bin/review_replay/main.py:208` `reviewed_in`: copy, rewrite and clone are named units, and the lambdas passed to `replayed` are named functions
- Locality: `bin/rung_replay/command.py:35` `down_from`: takes the floor as an argument, so each caller states its own floor where the base hid judge-replay's policy in the shared helper
Junior A, decided by:
- Locality: `bin/rule_hook/reviewer_rung.py:24` `reviewer_breaches`: the required rung comes straight from `review_rungs`, the function the prompts use, where the base passed it through an environment variable and a fallback
- Locality: `bin/rule_hook/reviewer_report.py:10` `REVIEWERS`: report confinement reads `agents/` itself, no longer importing its set from a module about rungs
- Locality, Shape: `bin/review_replay/main.py:208` `reviewed_in`: four jobs and seven arguments became named units with docstrings, and `main`'s loop and lambdas are named
- Navigation: `README.md:113`: the import map now names every reader of `model_routing/` and what `rung_replay/` holds, and the tool's name leads to it for any pass
- Shape: `bin/loop_claude/cycle/prompts.py:57` `shared_launches`: five helpers, a made-up `"each reviewer"` key and paths an empty `REPLAYED` never runs, against `ReviewKind` naming the real passes
+19 -11
View File
@@ -228,11 +228,11 @@ Each reviewer holds Write for its report. Claude Code passes a subagent's type t
The folder is fixed, since loop-claude's headless sessions have no scratchpad. It sits outside every checkout, so no project sees a report. It survives a reboot, unlike `/tmp`, so a resumed chunk still reads its earlier rounds. It is in the user's home, so no other local user plants a symlink that widens it. A report may quote a secret, so `larv-review review-folder`, which makes the chunk's folder, leaves `~/.local/state/larv-review` readable by the user alone.
A chunk keeps its folder after the merge. `stability-replay` hands a replayed reviewer a copy holding only the reports written after the chunk began and before the round did. So the replay reads the earlier rounds its prompt names, never the findings it is to find. The chunk's own reports stay as they were.
A chunk keeps its folder after the merge. `review-replay` hands a replayed reviewer a copy holding only the reports written after the chunk began and before the round did. So the replay reads the earlier rounds its prompt names, never the findings it is to find. The chunk's own reports stay as they were.
Claude Code's own subagent prompt tells a subagent to return findings rather than write report files. `review-findings.md` → "Report" overrides that, since the file is the worker's input; a prose-reviewer probed on 2026-10-18 wrote the file its prompt named. A general-purpose wrapper that launches the reviewer was not tried: Claude Code's documentation says a subagent cannot start another.
Before this change, reports took a median 19k tokens per chunk, 8% of the worker's peak context, and lines carrying a severity mark were 9–24% of a report (2026-10-08, over 68 archived transcripts). `stability-replay` still counts each 🔴 and 🟡 by its line in the reviewer's reply, now a one-line summary. todo.md item 187 is to measure what the worker reads of the reports instead.
Before this change, reports took a median 19k tokens per chunk, 8% of the worker's peak context, and lines carrying a severity mark were 9–24% of a report (2026-10-08, over 68 archived transcripts). `review-replay` still counts each 🔴 and 🟡 by its line in the reviewer's reply, now a one-line summary. todo.md item 187 is to measure what the worker reads of the reports instead.
Valid while Claude Code passes `agent_type` to hooks inside a subagent. Without it, the rule refuses nothing, and a reviewer may write anywhere.
@@ -260,15 +260,15 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
Valid while the index ranks these models, and while no newer model joins the catalog.
## routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, the seats and the security pass always on the project's
## routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass; the seats and the security pass always on the project's
2026-10-14, lilleman. Serves goals 1.1 and 1.4.
Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks would spend on work that never ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each comprehension seat and the security pass's subagent on the project's rung, and each reviewer there too unless the highest E + I among its items picks a higher rung, through the Agent tool's model and effort. The worker runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. lilleman chose the project's rung, medium, knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks would spend on work that never ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each comprehension seat and the security pass's subagent on the project's rung, and each reviewer there too unless the highest E + I among its items picks a higher rung or a replay lowered its pass, through the Agent tool's model and effort. The worker runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. lilleman chose the project's rung, medium, knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers never run below the worker's rung: a worker climbs because its chunk proved harder than its score said, which is when a cheaper reviewer is the bigger risk. A reviewer or seat that rules the work below standard may rerun one rung up: `review: a reviewer or comprehension seat that rules the work below standard`.
While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers never run below the worker's rung while it runs above the project's: a worker climbs because its chunk proved harder than its score said, which is when a cheaper reviewer is the bigger risk. A reviewer or seat that rules the work below standard may rerun one rung up: `review: a reviewer or comprehension seat that rules the work below standard`.
On 2026-10-16 lilleman routed the reviewers on their item's E + I as well, on either side of the project's rung, which may lift them above both the worker's rung and the project's: how far a defect would escape, and what it would break, set how hard review must look (goal 1.1). loop-claude names that rung in the worker's prompt, and hands it to the worker's claude beside the project's and the worker's, so rule-hook holds each reviewer launch to it. A low E + I never steps a reviewer below the project's rung: lilleman chose to lower a pass only where a replay shows a lower rung finding every 🔴 and 🟡 the project's rung finds, which `todo.md` item 159 measures for each pass. An item the worker takes in beyond the routed one counts too, since its E + I scores its defects as the routed item's scores theirs. The worker's prompt names the reviewers' rung each higher E + I picks, so the worker reviews a taken-in item on it in the same run. The judge reports the highest E + I the worker states for a taken-in item. From the next cycle loop-claude routes on that E + I, and archives it with the route so a rerun keeps it.
On 2026-10-16 lilleman routed the reviewers on their item's E + I as well, on either side of the project's rung, which may lift them above both the worker's rung and the project's: how far a defect would escape, and what it would break, set how hard review must look (goal 1.1). loop-claude names that rung in the worker's prompt. It hands the E + I to the worker's claude beside the project's rung and the worker's, so rule-hook works out each review's rung as the prompt does and holds each launch to it. A low E + I never steps a reviewer below the project's rung: only a replay lowers a pass, as `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds…` records. An item the worker takes in beyond the routed one counts too, since its E + I scores its defects as the routed item's scores theirs. The worker's prompt names the reviewers' rung each higher E + I picks, so the worker reviews a taken-in item on it in the same run. The judge reports the highest E + I the worker states for a taken-in item. From the next cycle loop-claude routes on that E + I, and archives it with the route so a rerun keeps it.
Valid until the trial on real chunks shows whether a worker below the project's rung holds quality.
@@ -334,12 +334,20 @@ Only Opus 5.5 was replayed, so the judge of a project on another model keeps the
Valid while the archived verdicts show what a judge should return, and until a new model joins the ladder: replay then.
## routing: the stability reviewer runs no lower than the project's rung while the worker runs below it, until `stability-replay` shows a lower rung finding every 🔴 and 🟡 the project's rung finds
## routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and only where the E + I steps no review up and the worker runs no higher than the project's rung, then even below the worker's; no pass has such a rung
2026-10-15, lilleman proposed running the stability reviewer on the worker's rung; the replay that gates it is the agent's proposal. Serves goals 1.1 and 1.4: the stability pass is the only one that mutation-tests the changed lines, so a defect it misses may merge, and a cheaper reviewer saves money only where it misses nothing.
2026-10-15, lilleman proposed running the stability reviewer on the worker's rung, and the agent proposed the replay that gates it. On 2026-10-16 lilleman asked for the prose pass to run on the worker's rung or lower, and, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. Serves goal 1.4 without giving up goal 1.1: a cheaper reviewer saves money only where it misses nothing.
`bin/stability-replay <model>` replays archived stability rounds on the code each reviewed, first on the project's rung, then on each rung below it, highest first, and stops at the first that misses a finding the project's rung found again. A rung qualifies only where it finds every such 🔴 and 🟡.
`bin/review-replay <pass> <model>` replays a pass's archived rounds on the code each reviewed. It replays them first on the project's rung, then on each rung below it, highest first, down to the ladder's lowest. It stops at the first rung that misses a finding the project's rung found again. A rung qualifies only where it and each rung above it find every such 🔴 and 🟡.
On 2026-10-15, `bin/stability-replay claude-opus-5-5` replayed the latest 10 rounds holding a 🔴 or 🟡, 40 findings in all, 8 of the rounds from this repo. Opus 5.5 at medium, the project's rung, found 29 of them again. Sonnet 5.5 at high missed 13 of those 29, 2 of them 🔴, so no rung below the project's rung qualifies, and Haiku 5.5, below Sonnet on the ladder, was not tried. So the stability reviewer runs where `routing: a chunk's worker runs on the rung its item's L picks` puts every reviewer: never below the project's rung while the worker runs below it.
`REPLAYED` in `bin/model_routing/ladder.py` holds each qualifying rung, by pass and project model. Where the chunk's E + I steps no review up and the worker runs no higher than the project's rung, the pass runs on that rung, even below the worker's. Otherwise the pass runs where every other pass does: the replay compared reviews on the project's rung only, never reviews on a rung an E + I stepped up, nor of a chunk whose worker climbed above the project's rung.
Valid until a new model joins the ladder, or the archive holds rounds a new reviewer prompt produced: replay then.
The runs, each on the latest 10 rounds of Opus 5.5 projects that hold a 🔴 or 🟡:
- 2026-10-15, stability: 40 findings, 8 of the rounds from this repo. Opus 5.5 at medium, the project's rung, found 29 of them again. Sonnet 5.5 at high missed 13 of those 29, 2 of them 🔴. No rung qualifies.
- 2026-10-18, prose: 39 findings. The project's rung found 27 of them again. Sonnet 5.5 at high missed 12 of those 27. No rung qualifies.
- 2026-10-18, stability again, since that day's change to the reviewer prompts expired the first run: 29 findings. The project's rung found 20 of them again. Sonnet 5.5 at high missed 11 of those 20, 1 of them 🔴. No rung qualifies.
The prose pass went first: a missed prose finding breaks no behaviour, and a later change fixes it cheaply. Sonnet 5.5 missed 44 to 55% of what the project's rung found again, in the prose pass as in both stability runs. Nothing suggests the other passes' findings are easier to find. So the architecture, principles, overrides and product-owner passes are not replayed yet: a replay would spend money with little chance of a qualifying rung (goal 1.4). `todo.md` item 196 keeps their replays in the plan. Every pass runs where `routing: a chunk's worker runs on the rung its item's L picks` puts every reviewer.
Valid until a new model joins the ladder, which calls for a replay of every pass, or a pass's reviewer prompt changes, which calls for a replay of that pass.
+4
View File
@@ -665,3 +665,7 @@ Read most (opens and searches):
## Reading cost, chunk b897b24d5e70, finished 2026-10-18T11:15:04Z
87043 new input tokens before the first commit.
## Reading cost, chunk 9f9579dd30bb, finished 2026-10-18T12:17:04Z
163028 new input tokens before the first commit.
+1 -1
View File
@@ -47,7 +47,7 @@ Give a reviewer without Bash the branch diff and `git ls-files`'s output in its
`~/.claude/bin/larv-review` holds this section's mechanical rules: the chunk's review folder, the round count, the paired ruling's base and vote count, the scoring run's trigger, and the history's format and latest places, and its count of how often seats agree. Run it in the chunk's worktree; `~/.claude/bin/larv-review --help` states each rule. On exit 1, repair what its message names and run it again; when you cannot, ask the user.
**Reports.** Run `~/.claude/bin/larv-review review-folder` in the chunk's worktree: it makes the chunk's review folder and prints its path. In each prompt to an agent under `~/.claude/agents/`, name the file in that folder for its full report: `<pass>-round-<n>.md`. For a principles-auditor, use `principles-<yardstick>-round-<n>.md`. For a relaunch one rung up, add `-rerun` before `.md`. Before you answer a 🔴 or 🟡, read its entry: Grep the report for the finding's place, with the lines after it, rather than Read the whole report. rule-hook refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`. Keep the folder after the merge: `stability-replay` replays a round from the reports its prompt names.
**Reports.** Run `~/.claude/bin/larv-review review-folder` in the chunk's worktree: it makes the chunk's review folder and prints its path. In each prompt to an agent under `~/.claude/agents/`, name the file in that folder for its full report: `<pass>-round-<n>.md`. For a principles-auditor, use `principles-<yardstick>-round-<n>.md`. For a relaunch one rung up, add `-rerun` before `.md`. Before you answer a 🔴 or 🟡, read its entry: Grep the report for the finding's place, with the lines after it, rather than Read the whole report. rule-hook refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`. Keep the folder after the merge: `review-replay` replays a round from the reports its prompt names.
**Precedent, then ask** — for the architecture pass and the full scan:
+9 -11
View File
@@ -6,19 +6,18 @@
`Bar = 0.1`
`Next ID = 196`
`Next ID = 197`
## Items
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 159 | 0.2.0 | | decision | **Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.** | 4 | 2 | 3 | 7 | 6 | 1.4 by 7 | 1.9 |
| 189 | 0.2.0 | | | **State one achieve goal for the health of a project's code in `~/.claude`, which every project's ranking weighs, most while its comprehension panel's overall is under 7.0, unless its README voids it.** | 4 | 2 | 3 | 7 | 6 | 1.1 by 6 | 1.7 |
| 29 | 0.3.0 | | | **Make the contributor a standing member of every project's audience, unless its `## Audience` says maintainability does not matter.** | 2 | 2 | 3 | 5 | 2 | 1.1 by 5, 1.4 by 5 | 16.7 |
| 119 | 0.3.0 | | | **Retitle the comprehension-floor decision so it says a chunk lifting a project towards the floor works first on its lowest-scoring dimension, and that the floor blocks nothing.** | 1 | 2 | 1 | 2 | 1 | 1.1 by 4, 1.4 by 4 | 16.0 |
| 127 | 0.3.0 | | | **Define seat, place, paired ruling, scoring run and the judge's records in `CLAUDE.md` → "Vocabulary".** | 1 | 2 | 3 | 4 | 1 | 1.1 by 4, 1.4 by 4 | 16.0 |
| 129 | 0.3.0 | | | **Explain `bin/rule_hook/rules.py`'s `OVERRIDE` pattern beside it: why the reason is lazy, what the lookahead splits and why `-->` is stripped.** | 1 | 2 | 1 | 2 | 1 | 1.1 by 4, 1.4 by 4 | 16.0 |
| 114 | 0.3.0 | | decision | **Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost`, `stability_replay` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.** | 3 | 1 | 3 | 5 | 4 | 1.1 by 9, 1.4 by 9 | 15.0 |
| 114 | 0.3.0 | | decision | **Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost`, `review_replay` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.** | 3 | 1 | 3 | 5 | 4 | 1.1 by 9, 1.4 by 9 | 15.0 |
| 120 | 0.3.0 | | | **Rewrite larv-review §3's "Round cap" and "Re-entry" as one table that says, for each kind of fix, which passes rerun and what a round reviews.** | 2 | 2 | 3 | 5 | 4 | 1.1 by 6, 1.2 by 6, 1.4 by 6 | 15.0 |
| 117 | 0.3.0 | | | **Give `todo_table.items` one way to say a plan has no items table, so no caller tells it from an empty plan by truthiness.** | 1 | 1 | 3 | 3 | 2 | 1.1 by 4, 1.4 by 4 | 13.3 |
| 126 | 0.3.0 | | | **State when work has landed towards a release once, in `bin/larv_review/plan.py`, and have larv-review's help point to it.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 4, 1.4 by 4 | 13.3 |
@@ -41,6 +40,7 @@
| 21 | 0.3.0 | | defect | **Give `settings.json`'s model one writer, and sort its keys.** | 3 | 3 | 3 | 7 | 3 | 2 by 5 | 6.6 |
| 190 | 0.3.0 | | | **Rewrite the `docs/decisions.md` entries `todo.md: one table …` and `loop-claude: a judge decides …` as the current state, so a reader no longer replays superseded paragraphs to learn what holds.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 3, 1.4 by 2 | 6.3 |
| 171 | 0.3.0 | | decision | **Group the routing figures' review rounds by the reviewers' rung as well as the worker's, so a call blames a rung only for rounds its own reviews raised.** | 3 | 2 | 2 | 5 | 4 | 1.2 by 3, 1.4 by 3 | 4.9 |
| 196 | 0.3.0 | | decision | **Replay the architecture, principles, overrides and product-owner passes with `review-replay` on Opus 5.5 projects, record each run in the review-replay routing decision, and add any rung a run qualifies to `REPLAYED`.** | 2 | 2 | 3 | 5 | 2 | 1.4 by 3 | 4.8 |
| 181 | 0.3.0 | | principle | **Have the test of a quoted answer's step-up assert the stepped rung's facts as literals, not through `prompts.rungs_held`.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 4.1 |
| 192 | 0.3.0 | | principle | **Sort `.gitignore`'s allowlist block, from `!.claude/` to `!workflow-bench/test_review_git.py`, by code point, so rule-hook's `ignorefile-sorted` checks later edits to it again.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 4.1 |
| 193 | 0.3.0 | | | **Narrow `readme_goals`'s mark to `Mark` by a typed lookup in place of `typing.cast`, so a mark added to the regex but not the lookup fails at run time, and one added to the lookup but not `Mark` fails mypy.** | 1 | 1 | 3 | 3 | 1 | 1.1 by 2 | 4.1 |
@@ -91,12 +91,6 @@
## Details
### 159. Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.
lilleman asked, on 2026-10-16, for the prose pass to run on the worker's rung or lower, and the same day, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. Today `bin/loop_claude/cycle/prompts.py`'s `review_launches` keeps every reviewer at or above the project's rung whatever the item's E + I, so a docs-only chunk such as item 140's, routed to Haiku 5.5, runs its reviewers on Opus 5.5.
Whether a pass runs cheaper without lowering quality (goal 1.4 against 1.1) is measured, as `docs/decisions.md` → `routing: the stability reviewer runs no lower than the project's rung…` measures the stability reviewer: replay a pass's archived rounds on the code each reviewed, first on the project's rung, then on each rung below it, and take the lowest rung that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again. A missed prose finding breaks no behaviour and a later change fixes it cheaply, so replay the prose pass first. Generalise `bin/stability-replay` to take the pass, in place of a replay tool per pass. Record each run and the rung it chose in a `routing:` decision, valid until a new model joins the ladder or the pass's prompt changes. Give each pass a row of its own in `REVIEWS`, `review_launches`' table of review kinds in `bin/loop_claude/cycle/prompts.py`, rather than a branch.
### 189. State one achieve goal for the health of a project's code in `~/.claude`, which every project's ranking weighs, most while its comprehension panel's overall is under 7.0, unless its README voids it.
lilleman, 2026-10-18: every project shares the goal, so it lives in `~/.claude` once, and a project for which it is not relevant voids it explicitly in its README. It replaces each project's own readability goal, such as fejkdata's goal 3, visthus's goal 8 and adf-codec's goal 10: once it merges, file an item in each such project's `todo.md` to drop its own goal, since a project's Blocked by cannot name an item of this repo.
@@ -123,11 +117,11 @@ Two seats of PR #257's scoring run assembled these terms from four files: `bin/l
The maintainability senior of PR #257's scoring run named `bin/rule_hook/rules.py:49` `OVERRIDE`: nothing in the code explains the pattern. Make `OVERRIDE` easier to read.
### 114. Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost`, `stability_replay` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.
### 114. Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost`, `review_replay` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.
Shape is the lowest dimension, so this item moves goal 1.1 by 9, per `docs/decisions.md` → `larv-review: a project lifting itself to the comprehension floor starts at the lowest dimension`.
The comprehension seats on PR #254 found five git runners with different contracts: `git_repo.git`, `rule_hook/git.py`'s guarded one, `metric_bench/main.py`'s, `reading_cost/main.py`'s `located` and `loop_claude/effects.py`'s. PR #276 added a sixth, `stability_replay/main.py`'s `git`. The inherited architect of PR #257's scoring run named `bin/git_repo.py:8` `git` again. Make `git_repo.git` easier to read.
The comprehension seats on PR #254 found five git runners with different contracts: `git_repo.git`, `rule_hook/git.py`'s guarded one, `metric_bench/main.py`'s, `reading_cost/main.py`'s `located` and `loop_claude/effects.py`'s. PR #276 added a sixth, `review_replay/main.py`'s `git`. The inherited architect of PR #257's scoring run named `bin/git_repo.py:8` `git` again. Make `git_repo.git` easier to read.
### 120. Rewrite larv-review §3's "Round cap" and "Re-entry" as one table that says, for each kind of fix, which passes rerun and what a round reviews.
@@ -221,6 +215,10 @@ Found by the prose review of PR #312, which judged both entries to read as amend
Raised by the goals audit of PR #296. `bin/routing_figures/measure.py` groups a chunk's review rounds by the rung its worker started on, and calls for a routing change when a lower rung took more of them. Since PR #296 the reviewers may run above the worker's rung, on the one their item's E + I picks, and no figure groups rounds by it, though the archived worker line names the E + I it follows from. A cheap worker on an item of high E + I gets stronger reviewers, who may raise more rounds, and a call could then blame the worker's rung for them: an interrupt of lilleman that settles nothing (goal 1.2).
### 196. Replay the architecture, principles, overrides and product-owner passes with `review-replay` on Opus 5.5 projects, record each run in the review-replay routing decision, and add any rung a run qualifies to `REPLAYED`.
Item 159's chunk, PR #322, replayed only the prose and stability passes. Sonnet 5.5 at high missed 44 to 55% of what the project's rung found again in each, so the agent judged a replay of the other four unlikely to qualify a rung. The product-owner review of PR #322 asked that the plan keep them, so the ranking decides when they run. Each run costs hours of reviewer replays (goal 1.4), so the score keeps it low until a cheaper model joins the ladder. `docs/decisions.md` → `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds…` records the runs so far.
### 181. Have the test of a quoted answer's step-up assert the stepped rung's facts as literals, not through `prompts.rungs_held`.
`bin/loop_claude/test_loop_claude.py`'s `test_a_quoted_answer_that_cites_an_escalation_trigger_steps_the_worker_up_before_it_resumes_and_archives_the_step` builds its expected prompt from `prompts.rungs_held`, the code under test, so a bug there passes it. The technical principles' test bullet bans that shape. Assert the new rung and one review launch as literals, as the typed answer's step-up test does. Raised by the principles audit of PR #308.