Generalise stability-replay to review-replay of any pass, and route each pass on its own rung so a replay may lower one pass alone
This commit is contained in:
+3
-3
@@ -51,7 +51,10 @@
|
||||
!bin/reading_cost/*.py
|
||||
!bin/repo_checks/
|
||||
!bin/repo_checks/*.py
|
||||
!bin/review-replay
|
||||
!bin/review_folder.py
|
||||
!bin/review_replay/
|
||||
!bin/review_replay/*.py
|
||||
!bin/round_titles.py
|
||||
!bin/rule-hook
|
||||
!bin/routing_figures/
|
||||
@@ -64,9 +67,6 @@
|
||||
!bin/score_outcomes/
|
||||
!bin/score_outcomes/*.py
|
||||
!bin/shell_command.py
|
||||
!bin/stability-replay
|
||||
!bin/stability_replay/
|
||||
!bin/stability_replay/*.py
|
||||
!bin/statusline.py
|
||||
!bin/subagent-rung
|
||||
!bin/subagent_rung/
|
||||
|
||||
@@ -40,14 +40,14 @@ Decisions live in `docs/decisions.md`:
|
||||
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
|
||||
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
|
||||
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
|
||||
- routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, the seats and the security pass always on the project's
|
||||
- routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass, the seats and the security pass always on the project's
|
||||
- routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, or when the judge reports a taken-in item that picks a higher rung or an escalation trigger the worker cites
|
||||
- routing: loop-claude measures review rounds and unfinished verdicts per rung every 20 cycles, and calls for a routing change when, of two rungs with 5 or more chunks each, the lower took more of either than the one above, or when 5 or more subagents of one type had reports longer than the tool results they read
|
||||
- routing: a worker delegates a task only where it needs just the task's conclusion, and each subagent runs on the rung its task's L picks from the project's rung, never below the project's rung when it files or scores todo.md items
|
||||
- routing: a worker hands a task one rung up on each escalation trigger, up to two rungs above the rung its item picks unless the human agrees, and opens the subagent's prompt with the trigger so the routing figures count it
|
||||
- routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, steps one rung up as the judge interrupts or quotes an answer, or next cycle, up to two rungs above the rung its item picks
|
||||
- routing: on a model judge-replay covered, the judge runs on the rung a review would get from the highest E + I among its chunk's items, never below the project's rung
|
||||
- routing: the stability reviewer runs no lower than the project's rung while the worker runs below it, until `stability-replay` shows a lower rung finding every 🔴 and 🟡 the project's rung finds
|
||||
- routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and there even below the worker's; no pass has such a rung
|
||||
|
||||
## Working rules
|
||||
|
||||
|
||||
+3
-2
@@ -2,8 +2,9 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- `stability-replay <model>` is now `review-replay <pass> <model>`, and replays any pass an agent under `agents/` reviews, such as `review-replay prose claude-opus-5-5`. It now tries every rung below the project's, down to the ladder's lowest. Its run on the prose pass of Opus 5.5 projects found no rung below the project's that finds every 🔴 and 🟡 the project's rung finds: Sonnet 5.5 at high missed 12 of 27. So every pass stays where it runs. A future run that finds such a rung lowers only that pass, below the worker's rung too, wherever the chunk's E + I steps no review up.
|
||||
- `chunk-preflight` names the release it finds finished and ready to cut, or to be planned first, wherever todo.md has named that release since the last cut.
|
||||
- Each reviewer under `agents/` now writes its full report to a file that larv-review names in the reviewer's prompt, in a folder per chunk under `~/.local/state/larv-review/`. It returns only its verdict and one line per finding, so a review takes less of the worker's context. The worker reads a finding's full entry before it fixes or answers a 🔴 or 🟡. Each reviewer now holds Write. `rule-hook` refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`; no override waives this rule. Each chunk's folder stays after the merge, so `stability-replay` can replay a round that reads an earlier round's report. It replays on a copy of the folder.
|
||||
- Each reviewer under `agents/` now writes its full report to a file that larv-review names in the reviewer's prompt, in a folder per chunk under `~/.local/state/larv-review/`. It returns only its verdict and one line per finding, so a review takes less of the worker's context. The worker reads a finding's full entry before it fixes or answers a 🔴 or 🟡. Each reviewer now holds Write. `rule-hook` refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`; no override waives this rule. Each chunk's folder stays after the merge, so `review-replay` can replay a round that reads an earlier round's report. It replays on a copy of the folder.
|
||||
- `loop-claude` asks an interrupt's question on its own terminal and resumes the worker headless with your one-line answer, so the loop no longer waits for you to leave a claude session. Type `/talk` instead to open the worker's session in claude, as every interrupt did before, to read the diff or discuss first. A release ready to cut opens that session at once. Pressing Enter on an empty answer stops the loop, exit 2. Keys typed before the question shows are dropped, and a pasted answer keeps only its first line; type `/talk` for a longer one. The judge counts that answer as yours, like one typed in claude: the chunk stays open until the answer is recorded, where a later chunk could need it.
|
||||
- `todo.md` may mark a renamed release in a `## Renamed` section, one line per rename, such as `- 0.2.1 renamed to 0.3.0`. `chunk-preflight` and `loop-claude` then skip that release when they pick the next one, and larv-review's scoring-run trigger calls no run for it. Without the mark, a release named since the last tag whose items all moved to other releases reads as finished: `chunk-preflight` says it is ready to cut and the loop stops for you, and the trigger calls a scoring run. `rank_todo.py` fails on a `## Renamed` line of another shape, and on an item filed under a release it marks. Before you move a machine to this release, add the line to the `todo.md` of every project that renamed a release since its last tag, or its loop stops for that release's cut.
|
||||
- When a worker's run ends with a question and a request for a higher rung, `loop-claude` now steps the worker one rung up as the judge interrupts, or as it answers the question with a quote from `README.md` or another file it may quote. The judge must quote a passage showing that the escalation trigger the run cites fired. Before, the step was lost for an answered question, and the worker carried on from the answer on its old rung. The step has the same limits as the step on an unfinished verdict. An escalation trigger that fired before the worker's last step-up doesn't step it again, so one trigger buys one step. A trigger that fires after the step steps the worker again, in the same cycle or a later one. A chunk left open across the upgrade to this release starts with no step-up on record, so a trigger that already stepped its worker may step it once more. The archive's interrupt and quoted-answer outcomes now name the worker's rung.
|
||||
@@ -13,7 +14,7 @@
|
||||
- Reshape every project's `todo.md` to the new shape below: until it is reshaped, that is the project's only work, as it is while README.md lacks Goals and Audience, and `rank_todo.py` says so. First move every machine to this release: an older release rejects the new formula, so it fails on every reshaped `todo.md`. Then mark the README's goals `**Maintain:**` or `**Achieve:**`, score each item's L, E and I, and set a bar on the new scale below the score of every item to keep, since the old bar would drop most items. Until then, an old-shape `todo.md` still ranks by its own formula and weights.
|
||||
- `todo.md` ranks items by value per cost: this setup's own plan uses `Score = 100 * V / (2**S + 8)`, where 8 is a chunk's fixed cost, about one size-3 change, and `rank_todo.py --help` shows it. A Moves column replaces A, G and Goals, naming each achieve goal an item moves and how far, such as `1.1 by 8, 1.2 by 5`. V sums each named goal's weight times its move. `rank_todo.py` weighs each achieve goal 0.97 times the one above it in README.md, so a goal far down the list still counts almost as much as the first and the goal weights table goes. The weights sum to 1: 5 achieve goals weigh 0.212 down to 0.188, 15 weigh 0.082 down to 0.053. An item names the most specific goal it moves, never a goal beside its own sub-goal. `rules/todo-items.md` and `rank_todo.py --help` show the new shape.
|
||||
- L, E and I replace C, and set R. L is the chance the change holds a defect, from 1 to 4; E the chance a defect escapes tests and review, from 1 to 3; I the damage it would do, from 1 to 5, counting every project and session that runs the change. `rules/todo-items.md` defines each level in words, and `rank_todo.py` writes R as L + E + I − 2, at least 7 when I is 5 and at least 5 when I is 4. The table's score columns are `| L | E | I | R | S |`. `score-outcomes` takes `L E I S`, and reports how well L, E and I each order finished chunks by review rounds and 🔴 findings.
|
||||
- `loop-claude` routes a chunk's worker on its item's L, from two rungs below the project's rung at L 1 to one above it at L 4. It routes the reviewers on the highest E + I among the chunk's items, counting each item the worker takes in: one rung above the project's for each point past 4, never below the worker's rung. It runs the judge of an Opus 5.5 project on the rung that highest E + I picks, never below the project's, and keeps it there when the worker steps up. The comprehension seats and the security pass stay on the project's rung. rule-hook holds each reviewer launch to the reviewers' rung, which `loop-claude` hands the worker in `LOOP_CLAUDE_REVIEWER_RUNG`. A delegated task's subagent takes the rung its L picks: `subagent-rung` takes L in place of R + C. Until a project reshapes its `todo.md`, every chunk there runs on the project's rung. Expect reviews of items whose defects would escape far or break much to cost more, and escalation from an item of L 4 to reach Fable 5.1 only with your yes.
|
||||
- `loop-claude` routes a chunk's worker on its item's L, from two rungs below the project's rung at L 1 to one above it at L 4. It routes the reviewers on the highest E + I among the chunk's items, counting each item the worker takes in: one rung above the project's for each point past 4, never below the worker's rung. It runs the judge of an Opus 5.5 project on the rung that highest E + I picks, never below the project's, and keeps it there when the worker steps up. The comprehension seats and the security pass stay on the project's rung. rule-hook holds each reviewer launch to the reviewers' rung, which it works out from the E + I `loop-claude` hands the worker in `LOOP_CLAUDE_ESCAPE_AND_IMPACT`. A delegated task's subagent takes the rung its L picks: `subagent-rung` takes L in place of R + C. Until a project reshapes its `todo.md`, every chunk there runs on the project's rung. Expect reviews of items whose defects would escape far or break much to cost more, and escalation from an item of L 4 to reach Fable 5.1 only with your yes.
|
||||
- An item that restores a maintain goal the project breaks today is exempt as `defect`, and its Moves may name that maintain goal, weighed as the heaviest achieve goal. A defect ranks by its score like any item. Only two kinds rank ahead of the rest of their release: questions first, since an answer may change other items, then security items.
|
||||
- The stability reviewer applies `principles/technical-principles.md` as written, in place of its own copies of it. Its test-data check now asks for UUID v7, where it asked for randomised UUIDs.
|
||||
- larv-review's principles audit is a pass of its own, and runs on every chunk that changes code, config or tests, not only on one with an architectural decision. A chunk of prose or planning alone skips it.
|
||||
|
||||
@@ -89,9 +89,9 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
|
||||
- `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format.
|
||||
- `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure.
|
||||
- `review-replay <pass> <model>` replays a review pass's archived rounds from the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again; `review-replay --help` states the replay. Run it only in a sandbox, like loop-claude: each replayed reviewer skips every permission prompt, stability-reviewer has Bash, and an archived prompt may name paths on the host.
|
||||
- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit, Write and Agent call, and `rule-hook --help` states the rules and the override.
|
||||
- `score-outcomes [L E I S]` lists the finished items whose guessed scores sit nearest the given ones, and the shipped items the project's `docs/score-anchors.md` names for each level of L, E and I, each with what its chunk measured; then the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings; `score-outcomes --help` states the measures.
|
||||
- `stability-replay <model>` replays archived stability reviews of the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again; `stability-replay --help` states the replay. Run it only in a sandbox, like loop-claude: each replayed reviewer has Bash and skips every permission prompt, and an archived prompt may name paths on the host.
|
||||
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
|
||||
- `subagent-rung [--plan] <model> <L>` prints the Agent tool's model and effort for a subagent's task, picked by its L from the project's model, and with `--plan` never below the project's rung; `subagent-rung --help` states the rule.
|
||||
|
||||
@@ -110,7 +110,7 @@ pytest_add_cli_args=-k
|
||||
|
||||
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
|
||||
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and `loop_claude` runs `chunk-preflight` as a process of its own. `judge_replay` and `stability_replay` share `rung_replay/`: the rung search, and the argument, candidate rungs and exits. It reads `loop_claude`'s retry count and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and `loop_claude` runs `chunk-preflight` as a process of its own. `judge_replay` and `review_replay` share `rung_replay/`: the rung search and the exits, and judge-replay's argument and candidate rungs. It reads `loop_claude`'s retry count and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
|
||||
## Tracking a new file
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from model_routing.ladder import climbed, LADDER, position, review_rungs, ReviewKind, Rung
|
||||
from model_routing.ladder import climbed, LADDER, position, review_rungs, REVIEWERS, ReviewKind, Rung
|
||||
from model_routing.route import ESCALATION_MARKER, HIGHEST_ESCAPE_AND_IMPACT, Route
|
||||
from loop_claude.routing import escalation_rungs
|
||||
from loop_claude.project.records import HUMAN_RULE_NAMES, RECORD_FILES
|
||||
@@ -10,8 +10,8 @@ QUOTE_SOURCES = listed(("README.md's Goals and Audience", *RECORD_FILES, *HUMAN_
|
||||
# Where the worker may write an answer the human typed, so it counts as recorded.
|
||||
ANSWER_DESTINATIONS = "README.md's Goals or Audience, " + ", ".join(RECORD_FILES) + ", todo.md or a comment"
|
||||
ANSWERS_RECORDED = f"every answer the human typed in it that a later chunk could need is recorded in {ANSWER_DESTINATIONS} at the one place it governs"
|
||||
# Each kind of review, as review_launches names those it launches.
|
||||
REVIEWS: tuple[tuple[str, ReviewKind], ...] = (("each reviewer", "reviewer"), ("each comprehension seat", "seat"), ("the security pass's subagent", "security"))
|
||||
# Each kind of review, as review_launches names those it launches: a row per pass, since a replay may lower one pass alone.
|
||||
REVIEWS: tuple[tuple[str, ReviewKind], ...] = (*((f"each {agent}", kind) for kind, agent in REVIEWERS.items()), ("each comprehension seat", "seat"), ("the security pass's subagent", "security"))
|
||||
|
||||
|
||||
def fresh(task: str, premises: str, route: Route, anchor: Rung | None, facts: str) -> str:
|
||||
@@ -48,11 +48,11 @@ def review_launches(route: Route, anchor: Rung | None) -> str:
|
||||
if anchor is None:
|
||||
return ""
|
||||
rungs = review_rungs(anchor, route.rung, route.reviewed_on())
|
||||
launched: dict[Rung, list[str]] = {}
|
||||
for reviews, kind in REVIEWS:
|
||||
launched: dict[Rung, list[ReviewKind]] = {}
|
||||
for _, kind in REVIEWS:
|
||||
if rungs[kind] != route.rung:
|
||||
launched.setdefault(rungs[kind], []).append(reviews)
|
||||
launches = "".join(f" Launch {listed(reviews, 'and')} with the Agent tool set to {rung.agent_args()}, {rung_named(rung, route, anchor)}." for rung, reviews in launched.items())
|
||||
launched.setdefault(rungs[kind], []).append(kind)
|
||||
launches = "".join(f" Launch {listed(named(kinds), 'and')} with the Agent tool set to {rung.agent_args()}, {rung_named(rung, route, anchor)}." for rung, kinds in launched.items())
|
||||
scan = " Run a full scan only from such a subagent, since the scan's agents run on the model that starts them." if rungs["security"] != route.rung else ""
|
||||
planning = f" Your rung is below the project's: launch each subagent that files or scores todo.md items on the rung `~/.claude/bin/subagent-rung --plan {anchor.model} <L>` prints." if position(route.rung) < position(anchor) else ""
|
||||
return f"{launches}{scan}{planning}{taken_in_launches(route, anchor)}"
|
||||
@@ -63,11 +63,12 @@ def taken_in_launches(route: Route, anchor: Rung) -> str:
|
||||
|
||||
For a worker on Opus 5.5 at medium, routed by an item of E + I 6, it reads: ` Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8.`
|
||||
"""
|
||||
reviewers = review_rungs(anchor, route.rung, route.reviewed_on())["reviewer"]
|
||||
lowest = min(position(review_rungs(anchor, route.rung, route.reviewed_on())[kind]) for kind in REVIEWERS)
|
||||
higher: dict[Rung, int] = {}
|
||||
for escape_and_impact in range(route.reviewed_on() + 1, HIGHEST_ESCAPE_AND_IMPACT + 1):
|
||||
rung = review_rungs(anchor, route.rung, escape_and_impact)["reviewer"]
|
||||
if position(rung) > position(reviewers):
|
||||
# An E + I that steps reviews up puts every pass on one rung; one that does not leaves each pass where it is.
|
||||
rungs = {review_rungs(anchor, route.rung, escape_and_impact)[kind] for kind in REVIEWERS}
|
||||
if len(rungs) == 1 and position(rung := rungs.pop()) > lowest:
|
||||
higher.setdefault(rung, escape_and_impact)
|
||||
if not higher:
|
||||
return ""
|
||||
@@ -75,10 +76,18 @@ def taken_in_launches(route: Route, anchor: Rung) -> str:
|
||||
return f" Where an item you take into the chunk has an E + I of {min(higher.values())} or more, launch each reviewer with the Agent tool set to {listed(picks, 'and')}."
|
||||
|
||||
|
||||
def named(kinds: list[ReviewKind]) -> list[str]:
|
||||
"""How review_launches names kinds: by their rows in REVIEWS, with "each reviewer" for every pass where all are among them."""
|
||||
every = all(kind in kinds for kind in REVIEWERS)
|
||||
return [*(["each reviewer"] if every else []), *(name for name, kind in REVIEWS if kind in kinds and not (every and kind in REVIEWERS))]
|
||||
|
||||
|
||||
def rung_named(rung: Rung, route: Route, anchor: Rung) -> str:
|
||||
"""Why a review runs on rung: it is the project's, anchor, or the one the route's reviewed_on E + I picks, the item's or one taken in."""
|
||||
"""Why a review runs on rung: it is the project's, anchor, the one a pass's replay chose, which alone lies below anchor, or the one the route's reviewed_on E + I picks, the item's or one taken in."""
|
||||
if rung == anchor:
|
||||
return "the project's rung"
|
||||
if position(rung) < position(anchor):
|
||||
return "the rung the replay of its pass chose"
|
||||
if route.escape_and_impact_taken_in:
|
||||
return f"the rung picked by an item of E + I {route.escape_and_impact_taken_in} taken into the chunk"
|
||||
return f"the rung item {route.item}'s E + I of {route.escape_and_impact} picks"
|
||||
@@ -112,13 +121,21 @@ def reruns(route: Route, anchor: Rung | None) -> str:
|
||||
if anchor is None or route.rung not in LADDER:
|
||||
return ""
|
||||
rungs = review_rungs(anchor, route.rung, route.reviewed_on())
|
||||
launched = (("reviewer", rungs["reviewer"]), ("comprehension seat", rungs["seat"]))
|
||||
rerun = {name: climbed(rung, 1) for name, rung in launched if rung != LADDER[-1]}
|
||||
rerun: dict[Rung, list[ReviewKind]] = {}
|
||||
for _, kind in REVIEWS:
|
||||
if kind != "security" and rungs[kind] != LADDER[-1]:
|
||||
rerun.setdefault(climbed(rungs[kind], 1), []).append(kind)
|
||||
if not rerun:
|
||||
return ""
|
||||
if len(set(rerun.values())) == 1:
|
||||
return f" A {listed(list(rerun), 'or')} that asks for a rerun one rung up reruns with the Agent tool set to `{next(iter(rerun.values())).agent_args()}`."
|
||||
return f" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `{rerun['reviewer'].agent_args()}`, and a comprehension seat with `{rerun['comprehension seat'].agent_args()}`."
|
||||
(first, kinds), *others = rerun.items()
|
||||
withs = [f"a {listed(nouns(more), 'or')} with `{rung.agent_args()}`" for rung, more in others]
|
||||
rest = f", {'and ' if len(withs) == 1 else ''}{listed(withs, 'and')}" if withs else ""
|
||||
return f" A {listed(nouns(kinds), 'or')} that asks for a rerun one rung up reruns with the Agent tool set to `{first.agent_args()}`{rest}."
|
||||
|
||||
|
||||
def nouns(kinds: list[ReviewKind]) -> list[str]:
|
||||
"""named's names without their "each"."""
|
||||
return [name.removeprefix("each ") for name in named(kinds)]
|
||||
|
||||
|
||||
def resumed(unfinished_evidence: str, route: Route, anchor: Rung | None) -> str:
|
||||
|
||||
@@ -68,19 +68,20 @@ picks the worker's rung, stepping from the model's cheapest rung by WORKER_STEPS
|
||||
bin/model_routing/ladder.py, below it for a low L; without such an item, or with its L, E or I
|
||||
unscored, the chunk runs on the model's cheapest rung. The
|
||||
item's E + I picks the reviewers' rung, stepping up from the cheapest rung by REVIEW_STEPS there,
|
||||
never below the worker's rung. The worker's first prompt names the rung and the item, and asks the
|
||||
never below the worker's rung. Where it steps none, a pass that a review-replay run lowered runs on
|
||||
the rung REPLAYED there names, even below the worker's. The worker's first prompt names the rung and the item, and asks the
|
||||
worker to name each item the chunk takes, with its L and E + I, and to count the item's size again
|
||||
once it has read the code (prompts.recount). The prompt names the rung of each review that runs on
|
||||
a rung other than the worker's: the reviewers' rung for the reviewers, and the cheapest rung for
|
||||
each comprehension seat and the security pass. It also names the reviewers' rung
|
||||
a rung other than the worker's: the reviewers' rung or the replay's for each pass's reviewer, and
|
||||
the cheapest rung for each comprehension seat and the security pass. It also names the reviewers' rung
|
||||
each higher E + I picks, so the worker reviews an item it takes in on that rung in the same run.
|
||||
When the worker runs below the cheapest rung, the prompt has it launch each subagent that files or
|
||||
scores todo.md items on the rung subagent-rung --plan prints. The prompt after
|
||||
each unfinished verdict says this again, since a step-up may have moved the worker. A reviewer or
|
||||
seat that rules the work below standard may ask to rerun one rung up; each prompt names that rung.
|
||||
loop-claude hands the worker's claude its own rung, the project's and the reviewers', in
|
||||
LOOP_CLAUDE_WORKER_RUNG, LOOP_CLAUDE_PROJECT_RUNG and LOOP_CLAUDE_REVIEWER_RUNG, and rule-hook
|
||||
refuses a review launched below the rung named for it. The judge reports the highest L and the
|
||||
loop-claude hands the worker's claude its own rung, the project's and the reviews' E + I, in
|
||||
LOOP_CLAUDE_WORKER_RUNG, LOOP_CLAUDE_PROJECT_RUNG and LOOP_CLAUDE_ESCAPE_AND_IMPACT, and rule-hook
|
||||
refuses a review launched below the rung those pick for it. The judge reports the highest L and the
|
||||
highest E + I the worker states for an item it takes in. The next cycle steps the worker up to the
|
||||
rung that L picks, never down, and the reviewers to the rung picked by the higher of two: that
|
||||
E + I, or the routed item's (Route.reviewed_on in bin/model_routing/route.py). A level off its
|
||||
|
||||
@@ -12,6 +12,7 @@ import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import unittest.mock
|
||||
import uuid
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
@@ -24,6 +25,7 @@ from claude_code import entries, reply, session
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished, verdict_from
|
||||
from loop_claude.cycle import interrupt, judge, prompts, retry, run, states, worker
|
||||
from model_routing import ladder
|
||||
from model_routing.ladder import LADDER, Rung
|
||||
from claude_code.reply import Limited
|
||||
from loop_claude.loop import main, reload, routing_check
|
||||
@@ -220,11 +222,11 @@ class Cycles(unittest.TestCase):
|
||||
self.assertIn("- A preflight fact.", fresh)
|
||||
self.assertNotIn("- A preflight fact.", resumed)
|
||||
|
||||
def test_a_worker_is_handed_the_projects_rung_its_own_and_its_reviewers_in_its_environment(self):
|
||||
def test_a_worker_is_handed_the_projects_rung_its_own_and_its_reviews_e_plus_i_in_its_environment(self):
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("finished")])
|
||||
cycle = dataclasses.replace(self.cycle(fake), anchor=Rung("claude-opus-5-5", "medium"), route=Route(Rung("claude-haiku-5-5", "max"), 163, 1, 5))
|
||||
self.assertEqual(run.run_cycle(cycle).verdict, Finished("working"))
|
||||
self.assertEqual(fake.worker_environments, [{"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": "claude-opus-5-5 at high effort"}])
|
||||
self.assertEqual(fake.worker_environments, [{"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5"}])
|
||||
|
||||
def test_the_claude_an_interrupt_resumes_is_handed_the_same_rungs_as_the_worker(self):
|
||||
def answer(args):
|
||||
@@ -1143,6 +1145,17 @@ class WorkerArgs(unittest.TestCase):
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus))
|
||||
self.assertNotIn("Where the hard part", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 3, 4), opus))
|
||||
|
||||
def test_launches_each_pass_a_replay_lowered_on_its_replays_rung_and_reruns_it_one_above(self):
|
||||
opus, haiku = Rung("claude-opus-5-5", "medium"), Rung("claude-haiku-5-5", "low")
|
||||
with unittest.mock.patch.dict(ladder.REPLAYED, {("prose", "claude-opus-5-5"): haiku}):
|
||||
below = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 4), opus, self.FACTS)
|
||||
on_it = prompts.resumed("", Route(haiku, 3, 1, 4), opus)
|
||||
self.assertIn("whether or not it is item 3. Launch each systems-architect, each principles-auditor, each override-reviewer, each stability-reviewer, each product-owner-reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung. Launch each prose-reviewer with the Agent tool set to model haiku, effort low, the rung the replay of its pass chose.", below)
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5,", below)
|
||||
self.assertIn(" A systems-architect, principles-auditor, override-reviewer, stability-reviewer, product-owner-reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`, and a prose-reviewer with `model haiku, effort medium`.", below)
|
||||
self.assertNotIn("each prose-reviewer", on_it)
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 5 or more,", on_it)
|
||||
|
||||
def test_names_the_rung_a_reviewer_or_seat_reruns_on_one_above_its_own(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
both = " A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`."
|
||||
|
||||
@@ -5,8 +5,8 @@ import typing
|
||||
|
||||
# How Rung.__str__ spells a rung that names an effort, and Rung.parse reads it back.
|
||||
WORDING = re.compile(r"(?P<model>\S+) at (?P<effort>\S+) effort")
|
||||
# What a review launch reviews: a reviewer, a comprehension seat or the security pass.
|
||||
ReviewKind = typing.Literal["reviewer", "seat", "security"]
|
||||
# What a review launch reviews: a pass an agent under agents/ reviews, named as skills/larv-review/SKILL.md → "Passes" names it in lower case, a comprehension seat or the security pass.
|
||||
ReviewKind = typing.Literal["architecture", "principles", "overrides", "stability", "product-owner", "prose", "seat", "security"]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -63,6 +63,10 @@ LADDER = (
|
||||
WORKER_STEPS = {1: -2, 2: -1, 3: 0, 4: 1}
|
||||
# The rungs a task's E + I steps a review up from the project's rung; an E + I under 5 steps none.
|
||||
REVIEW_STEPS = {5: 1, 6: 2, 7: 3, 8: 4}
|
||||
# Each pass an agent under agents/ reviews, in the order of that table in SKILL.md, and that agent.
|
||||
REVIEWERS: dict[ReviewKind, str] = {"architecture": "systems-architect", "principles": "principles-auditor", "overrides": "override-reviewer", "stability": "stability-reviewer", "product-owner": "product-owner-reviewer", "prose": "prose-reviewer"}
|
||||
# The rung a pass's replay chose for the projects on a model: docs/decisions.md → `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds`.
|
||||
REPLAYED: dict[tuple[ReviewKind, str], Rung] = {}
|
||||
|
||||
|
||||
def project_rung(model: str) -> Rung | None:
|
||||
@@ -96,14 +100,13 @@ def stepped_up(rung: Rung) -> Rung:
|
||||
|
||||
|
||||
def review_rungs(project: Rung, worker: Rung, escape_and_impact: int) -> dict[ReviewKind, Rung]:
|
||||
"""The rung each kind of review in a chunk runs on, where its worker runs on worker and its item has that E + I. The reviewers run on the higher of the worker's rung and review_floor's."""
|
||||
"""The rung each kind of review in a chunk runs on, where its worker runs on worker and its item has that E + I.
|
||||
|
||||
A pass's reviewer runs on the higher of the worker's rung and review_floor's. Where the E + I steps no review up, a pass whose replay chose a rung for the project's model runs on that rung instead, even below the worker's: the replay measured the project's rung, never a stepped one. A seat and the security pass run on the project's rung, so the seats score every chunk on one rung, and the security pass spends no more than the project chose.
|
||||
"""
|
||||
reviewers = highest(worker, review_floor(project, escape_and_impact))
|
||||
return {kind: review_rung(kind, project, reviewers) for kind in typing.get_args(ReviewKind)}
|
||||
|
||||
|
||||
def review_rung(kind: ReviewKind, project: Rung, reviewers: Rung) -> Rung:
|
||||
"""The rung a review of kind runs on: a reviewer on the reviewers' rung; a seat or the security pass on the project's, so the seats score every chunk on one rung, and the security pass spends no more than the project chose."""
|
||||
return reviewers if kind == "reviewer" else project
|
||||
replayed = {} if escape_and_impact in REVIEW_STEPS else {kind: rung for (kind, model), rung in REPLAYED.items() if model == project.model}
|
||||
return {**dict.fromkeys(REVIEWERS, reviewers), **replayed, "seat": project, "security": project}
|
||||
|
||||
|
||||
def highest(*rungs: Rung) -> Rung:
|
||||
|
||||
@@ -1,10 +1,13 @@
|
||||
import pathlib
|
||||
import sys
|
||||
import typing
|
||||
import unittest
|
||||
import unittest.mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from model_routing.ladder import highest, LADDER, lowest_worker_rung, planning_rung, project_rung, review_floor, review_rung, review_rungs, Rung, stepped_up, worker_rung, WORKER_STEPS
|
||||
from model_routing.ladder import highest, LADDER, lowest_worker_rung, planning_rung, project_rung, review_floor, review_rungs, REVIEWERS, ReviewKind, Rung, stepped_up, worker_rung, WORKER_STEPS
|
||||
from model_routing import ladder
|
||||
from todo_table import LEVELS
|
||||
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
@@ -94,20 +97,26 @@ class Ladder(unittest.TestCase):
|
||||
|
||||
def test_puts_reviewers_on_the_highest_of_the_projects_rung_the_workers_and_the_one_e_plus_i_picks(self):
|
||||
haiku, fable = LADDER[4], LADDER[-1]
|
||||
self.assertEqual(review_rungs(OPUS, haiku, 4)["reviewer"], OPUS)
|
||||
self.assertEqual(review_rungs(OPUS, haiku, 5)["reviewer"], Rung("claude-opus-5-5", "high"))
|
||||
self.assertEqual(review_rungs(OPUS, fable, 5)["reviewer"], fable)
|
||||
self.assertEqual(review_rungs(OPUS, Rung("claude-opus-5-5", "high"), 7)["reviewer"], Rung("claude-opus-5-5", "max"))
|
||||
self.assertEqual(review_rungs(OPUS, haiku, 4)["stability"], OPUS)
|
||||
self.assertEqual(review_rungs(OPUS, haiku, 5)["stability"], Rung("claude-opus-5-5", "high"))
|
||||
self.assertEqual(review_rungs(OPUS, fable, 5)["stability"], fable)
|
||||
self.assertEqual(review_rungs(OPUS, Rung("claude-opus-5-5", "high"), 7)["stability"], Rung("claude-opus-5-5", "max"))
|
||||
|
||||
def test_keeps_the_seats_and_the_security_pass_on_the_projects_rung_whatever_the_worker_or_e_plus_i(self):
|
||||
self.assertEqual(review_rungs(OPUS, LADDER[-1], 8), {"reviewer": LADDER[-1], "seat": OPUS, "security": OPUS})
|
||||
self.assertEqual(review_rungs(OPUS, LADDER[-1], 8), {**dict.fromkeys(REVIEWERS, LADDER[-1]), "seat": OPUS, "security": OPUS})
|
||||
|
||||
def test_holds_a_reviewer_to_the_reviewers_rung_and_anything_else_to_the_projects(self):
|
||||
fable = LADDER[-1]
|
||||
self.assertEqual(review_rung("reviewer", OPUS, fable), fable)
|
||||
for kind in ("seat", "security"):
|
||||
with self.subTest(kind=kind):
|
||||
self.assertEqual(review_rung(kind, OPUS, fable), OPUS)
|
||||
def test_names_every_kind_of_review_once_each_pass_with_its_agent_under_agents(self):
|
||||
self.assertEqual([*REVIEWERS, "seat", "security"], list(typing.get_args(ReviewKind)))
|
||||
agents = pathlib.Path(__file__).resolve().parents[2] / "agents"
|
||||
self.assertEqual(sorted(REVIEWERS.values()), sorted(path.stem for path in agents.glob("*.md")))
|
||||
|
||||
def test_runs_a_replayed_pass_on_its_replays_rung_below_the_workers_where_the_e_plus_i_steps_no_review_up(self):
|
||||
sonnet, haiku = Rung("claude-sonnet-5-5", "high"), LADDER[0]
|
||||
with unittest.mock.patch.dict(ladder.REPLAYED, {("prose", "claude-opus-5-5"): haiku, ("stability", "claude-sonnet-5-5"): haiku}):
|
||||
self.assertEqual(review_rungs(OPUS, LADDER[-1], 4), {**dict.fromkeys(REVIEWERS, LADDER[-1]), "prose": haiku, "seat": OPUS, "security": OPUS})
|
||||
self.assertEqual(review_rungs(OPUS, sonnet, 5)["prose"], Rung("claude-opus-5-5", "high"))
|
||||
self.assertEqual(review_rungs(OPUS, LADDER[-1], 5)["prose"], LADDER[-1])
|
||||
self.assertEqual(review_rungs(sonnet, sonnet, 4)["prose"], sonnet)
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -9,20 +9,22 @@ class Environment(unittest.TestCase):
|
||||
PROJECT = Rung("claude-opus-5-5", "medium")
|
||||
WORKER = Rung("claude-haiku-5-5", "max")
|
||||
|
||||
def test_the_project_the_worker_and_the_reviewers_rungs_are_named_in_the_environment(self):
|
||||
self.assertEqual(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5)), {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": "claude-opus-5-5 at high effort"})
|
||||
self.assertEqual(environment(self.PROJECT, Route(self.WORKER))["LOOP_CLAUDE_REVIEWER_RUNG"], "claude-opus-5-5 at medium effort")
|
||||
self.assertEqual(environment(self.PROJECT, Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 2))["LOOP_CLAUDE_REVIEWER_RUNG"], "claude-opus-5-5 at xhigh effort")
|
||||
self.assertEqual(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5, escape_and_impact_taken_in=6))["LOOP_CLAUDE_REVIEWER_RUNG"], "claude-opus-5-5 at xhigh effort")
|
||||
def test_the_project_and_the_workers_rungs_and_the_e_plus_i_reviews_take_are_named_in_the_environment(self):
|
||||
self.assertEqual(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5)), {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5"})
|
||||
self.assertEqual(environment(self.PROJECT, Route(self.WORKER))["LOOP_CLAUDE_ESCAPE_AND_IMPACT"], "0")
|
||||
self.assertEqual(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5, escape_and_impact_taken_in=6))["LOOP_CLAUDE_ESCAPE_AND_IMPACT"], "6")
|
||||
|
||||
def test_a_project_off_the_ladder_is_handed_empty_rungs_so_none_is_inherited(self):
|
||||
self.assertEqual(environment(None, Route(self.WORKER)), {"LOOP_CLAUDE_PROJECT_RUNG": "", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": ""})
|
||||
def test_a_project_off_the_ladder_is_handed_an_empty_rung_so_none_is_inherited(self):
|
||||
self.assertEqual(environment(None, Route(self.WORKER)), {"LOOP_CLAUDE_PROJECT_RUNG": "", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "0"})
|
||||
|
||||
def test_handed_reads_back_what_environment_named(self):
|
||||
self.assertEqual(handed(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5))), (self.PROJECT, self.WORKER, Rung("claude-opus-5-5", "high")))
|
||||
self.assertEqual(handed(environment(None, Route(self.WORKER))), (None, self.WORKER, None))
|
||||
self.assertEqual(handed(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5))), (self.PROJECT, self.WORKER, 5))
|
||||
self.assertEqual(handed(environment(None, Route(self.WORKER))), (None, self.WORKER, 0))
|
||||
|
||||
def test_a_missing_or_unknown_rung_reads_as_none(self):
|
||||
self.assertEqual(handed({}), (None, None, None))
|
||||
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 bogus"}), (None, None, None))
|
||||
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at bogus effort"}), (None, None, None))
|
||||
def test_a_missing_or_unknown_rung_reads_as_none_and_a_missing_or_unknown_e_plus_i_as_0(self):
|
||||
self.assertEqual(handed({}), (None, None, 0))
|
||||
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 bogus"}), (None, None, 0))
|
||||
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at bogus effort"}), (None, None, 0))
|
||||
for bogus in ("", "five", "-1", "5.0", "²"):
|
||||
with self.subTest(bogus=bogus):
|
||||
self.assertEqual(handed({"LOOP_CLAUDE_ESCAPE_AND_IMPACT": bogus})[2], 0)
|
||||
|
||||
@@ -3,23 +3,24 @@
|
||||
loop-claude sets these variables for its worker and for the claude an interrupt resumes; a human's shell does not, and rule-hook then checks no review launch."""
|
||||
import typing
|
||||
|
||||
from model_routing.ladder import LADDER, review_rungs, Rung
|
||||
from model_routing.ladder import LADDER, Rung
|
||||
from model_routing.route import Route
|
||||
|
||||
PROJECT_RUNG = "LOOP_CLAUDE_PROJECT_RUNG"
|
||||
WORKER_RUNG = "LOOP_CLAUDE_WORKER_RUNG"
|
||||
REVIEWER_RUNG = "LOOP_CLAUDE_REVIEWER_RUNG"
|
||||
# The E + I the chunk's reviews are routed on, from which review_rungs works out each review's rung.
|
||||
ESCAPE_AND_IMPACT = "LOOP_CLAUDE_ESCAPE_AND_IMPACT"
|
||||
|
||||
|
||||
def environment(project: Rung | None, route: Route) -> dict[str, str]:
|
||||
"""The variables a worker's claude on route starts with. A project off the ladder gets empty project and reviewer rungs, so none is inherited from the environment."""
|
||||
reviewers = review_rungs(project, route.rung, route.reviewed_on())["reviewer"] if project else None
|
||||
return {PROJECT_RUNG: str(project) if project else "", WORKER_RUNG: str(route.rung), REVIEWER_RUNG: str(reviewers) if reviewers else ""}
|
||||
"""The variables a worker's claude on route starts with. A project off the ladder gets an empty project rung, so none is inherited from the environment."""
|
||||
return {PROJECT_RUNG: str(project) if project else "", WORKER_RUNG: str(route.rung), ESCAPE_AND_IMPACT: str(route.reviewed_on())}
|
||||
|
||||
|
||||
def handed(environ: typing.Mapping[str, str]) -> tuple[Rung | None, Rung | None, Rung | None]:
|
||||
"""The project's, the worker's and the reviewers' rungs, each None when unset or not on the ladder."""
|
||||
return on_the_ladder(environ.get(PROJECT_RUNG)), on_the_ladder(environ.get(WORKER_RUNG)), on_the_ladder(environ.get(REVIEWER_RUNG))
|
||||
def handed(environ: typing.Mapping[str, str]) -> tuple[Rung | None, Rung | None, int]:
|
||||
"""The project's and the worker's rungs, each None when unset or not on the ladder, and the reviews' E + I, 0 when unset or not a whole number. An older loop-claude hands no E + I, and 0 steps no review up."""
|
||||
escape_and_impact = environ.get(ESCAPE_AND_IMPACT, "")
|
||||
return on_the_ladder(environ.get(PROJECT_RUNG)), on_the_ladder(environ.get(WORKER_RUNG)), int(escape_and_impact) if escape_and_impact.isdecimal() else 0
|
||||
|
||||
|
||||
def on_the_ladder(value: str | None) -> Rung | None:
|
||||
|
||||
@@ -15,5 +15,5 @@ class Types(unittest.TestCase):
|
||||
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
|
||||
if shutil.which("uv") is None:
|
||||
self.fail("uv on PATH fetches mypy; see README.md's Test line")
|
||||
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "chunk_preflight", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "rule_hook", "-p", "rung_replay", "-p", "score_outcomes", "-p", "stability_replay", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
|
||||
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "chunk_preflight", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "review_replay", "-p", "rule_hook", "-p", "rung_replay", "-p", "score_outcomes", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
|
||||
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
|
||||
|
||||
@@ -5,7 +5,7 @@ import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
|
||||
from stability_replay import main
|
||||
from review_replay import main
|
||||
from loop_claude.effects import system_effects
|
||||
|
||||
sys.exit(main.main(sys.argv[1:], os.environ, pathlib.Path.home(), system_effects()))
|
||||
@@ -1,10 +1,12 @@
|
||||
"""Replay archived stability rounds on the rungs below the project's, to find which rungs may run the stability reviewer: docs/decisions.md → `routing: the stability reviewer runs no lower than the project's rung`.
|
||||
"""Replay a review pass's archived rounds on the rungs below the project's, to find which rungs may run its reviewer: docs/decisions.md → `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds`.
|
||||
|
||||
usage: stability-replay <model>
|
||||
usage: review-replay <pass> <model>
|
||||
|
||||
<pass> is a pass an agent under agents/ reviews, named as skills/larv-review/SKILL.md → "Passes"
|
||||
names it in lower case: architecture, principles, overrides, stability, product-owner or prose.
|
||||
<model> is a project model on the routing ladder, such as claude-opus-5-5. The tool replays the
|
||||
latest 10 archived stability rounds whose report holds a 🔴 or 🟡 finding, from the projects that
|
||||
~/.claude/loop-claude-models.json runs on the model. It replays each round on the code it
|
||||
latest 10 archived rounds of the pass whose report holds a 🔴 or 🟡 finding, from the projects
|
||||
that ~/.claude/loop-claude-models.json runs on the model. It replays each round on the code it
|
||||
reviewed: in a throwaway clone of the project's checkout, at the newest commit of origin/HEAD
|
||||
authored by the time the round began, on the branch the reviewer started on. The clone's default
|
||||
branch stands at the newest commit authored before the round's chunk began, so a diff against it
|
||||
@@ -17,13 +19,11 @@ whose prompt still names the review folder finds nothing.
|
||||
It first replays every round on the project's rung, the reference reviewer. Findings on record
|
||||
that the reference reviewer does not raise again may rest on code or rules that have changed
|
||||
since; no rung is held to them. It then replays the rounds where the reference raised any, on each
|
||||
rung below, highest first, down to the lowest rung a worker routes to, and stops at the first rung
|
||||
that misses one. For each replay, claude on the project's rung matches the report to the findings
|
||||
on record, and counts a finding only where the report raises it as 🔴 or 🟡. The tool prints each
|
||||
finding a rung missed and the lowest rung that, like each rung above it, missed none. That rung is
|
||||
the evidence for running the stability reviewer on the worker's rung whenever the worker runs at
|
||||
or above it; docs/decisions.md → `routing: the stability reviewer runs no lower than the project's rung`
|
||||
records the run.
|
||||
rung below, highest first, down to the ladder's lowest, and stops at the first rung that misses
|
||||
one. For each replay, claude on the project's rung matches the report to the findings on record,
|
||||
and counts a finding only where the report raises it as 🔴 or 🟡. The tool prints each finding a
|
||||
rung missed and the lowest rung that, like each rung above it, missed none. That rung is the
|
||||
evidence for running the pass's reviewer on it; the decision above records each run.
|
||||
|
||||
Like the loop, it tries claude three times running before it counts a reply as failed. A failed
|
||||
review or match finds nothing. Exit 1, saying why, on bad arguments, an unreadable archive or
|
||||
@@ -39,21 +39,20 @@ import tempfile
|
||||
import typing
|
||||
|
||||
from transcript_archive.location import archive_root
|
||||
from stability_replay.rounds import authored_by, report, reviewed_path, Round, round_from, sample, stamp_seconds
|
||||
from review_replay.rounds import authored_by, report, reviewed_path, Round, round_from, sample, stamp_seconds
|
||||
from project_key import checkout_of
|
||||
from claude_code.reply import claude_reply, Limited
|
||||
from rung_replay.command import down_from, failed, limited_exit, project_rung_or_exit, projects_on
|
||||
from loop_claude.effects import Effects
|
||||
from rung_replay.command import failed, limited_exit, projects_on
|
||||
from failure import Failure
|
||||
from rung_replay.search import first_reply, replayed
|
||||
from claude_code.transcript import first_timestamped_line, read_transcript
|
||||
from model_routing.ladder import LADDER, position, project_rung, REVIEWERS, Rung
|
||||
from claude_code.entries import parse_entry, structured_output
|
||||
from review_folder import REPORTS, review_folder
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
TOOL = "stability-replay"
|
||||
TOOL = "review-replay"
|
||||
LATEST = 10
|
||||
REVIEWER = "stability-reviewer"
|
||||
MATCH_SCHEMA = {"type": "object", "properties": {"found": {"type": "array", "items": {"type": "integer"}}}, "required": ["found"], "additionalProperties": False}
|
||||
|
||||
|
||||
@@ -67,10 +66,14 @@ class Code:
|
||||
|
||||
|
||||
def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, effects: Effects) -> int:
|
||||
anchor = project_rung_or_exit(TOOL, argv, __doc__)
|
||||
if isinstance(anchor, int):
|
||||
return anchor
|
||||
rounds = archived_rounds(anchor.model, archive_root(home, env), home)
|
||||
if argv[:1] in (["-h"], ["--help"]):
|
||||
print(__doc__.strip())
|
||||
return 0
|
||||
reviewed = next((kind for kind in REVIEWERS if kind == argv[0]), None) if len(argv) == 2 else None
|
||||
anchor = project_rung(argv[1]) if reviewed else None
|
||||
if reviewed is None or anchor is None:
|
||||
return failed(TOOL, f"cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; usage: {TOOL} <pass> <model>, <pass> one of {', '.join(REVIEWERS)} and <model> a model on the routing ladder")
|
||||
rounds = archived_rounds(REVIEWERS[reviewed], anchor.model, archive_root(home, env), home)
|
||||
if isinstance(rounds, Failure):
|
||||
return failed(TOOL, rounds.reason)
|
||||
codes = {}
|
||||
@@ -80,17 +83,18 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
else:
|
||||
codes[found] = code
|
||||
if not codes:
|
||||
return failed(TOOL, f"no archived stability round of projects on {anchor.model} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find")
|
||||
return failed(TOOL, f"no archived {reviewed} round of projects on {anchor.model} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find")
|
||||
chosen = sample(list(codes), LATEST)
|
||||
# The project's rung is the reference, so the candidates start below it.
|
||||
replay = replayed(chosen, anchor, down_from(anchor)[1:], lambda found, rung: findings_found(found, codes[found], rung, anchor, home, effects), lambda _, found: sorted(found), "stability rounds")
|
||||
# The project's rung is the reference, so the candidates start below it. They reach below the lowest rung a worker routes to, since a pass may run below the worker's rung.
|
||||
below = list(LADDER[:position(anchor)])[::-1]
|
||||
replay = replayed(chosen, anchor, below, lambda found, rung: findings_found(found, codes[found], rung, anchor, home, effects), lambda _, found: sorted(found), f"{reviewed} rounds")
|
||||
if isinstance(replay, Limited):
|
||||
return limited_exit(TOOL, replay)
|
||||
print(report(anchor.model, anchor, chosen, replay))
|
||||
print(report(reviewed, anchor.model, anchor, chosen, replay))
|
||||
return 0
|
||||
|
||||
|
||||
def archived_rounds(model: str, root: pathlib.Path, home: pathlib.Path) -> list[Round] | Failure:
|
||||
def archived_rounds(reviewer: str, model: str, root: pathlib.Path, home: pathlib.Path) -> list[Round] | Failure:
|
||||
projects = projects_on(model, home)
|
||||
if isinstance(projects, Failure):
|
||||
return projects
|
||||
@@ -98,13 +102,13 @@ def archived_rounds(model: str, root: pathlib.Path, home: pathlib.Path) -> list[
|
||||
try:
|
||||
for key in projects:
|
||||
for chunk in sorted(path for path in (root / key).glob("*") if path.is_dir()):
|
||||
found += chunk_rounds(checkout_of(key, home), chunk)
|
||||
found += chunk_rounds(reviewer, checkout_of(key, home), chunk)
|
||||
except (OSError, UnicodeDecodeError, ValueError) as failure:
|
||||
return Failure(f"could not read the archive {root}: {failure}")
|
||||
return found
|
||||
|
||||
|
||||
def chunk_rounds(checkout: str, chunk: pathlib.Path) -> list[Round]:
|
||||
def chunk_rounds(reviewer: str, checkout: str, chunk: pathlib.Path) -> list[Round]:
|
||||
copies = sorted(chunk.glob("*.jsonl"))
|
||||
began = parse_entry(first_timestamped_line(copies[0].read_bytes())) if copies else None
|
||||
if not began or not isinstance(began.get("timestamp"), str):
|
||||
@@ -112,10 +116,10 @@ def chunk_rounds(checkout: str, chunk: pathlib.Path) -> list[Round]:
|
||||
found = []
|
||||
for meta in sorted(chunk.glob("session/*/subagents/*.meta.json")):
|
||||
described = json.loads(meta.read_text(encoding="utf-8"))
|
||||
if not isinstance(described, dict) or described.get("agentType") != REVIEWER:
|
||||
if not isinstance(described, dict) or described.get("agentType") != reviewer:
|
||||
continue
|
||||
entries = read_transcript(str(meta.with_name(meta.name.removesuffix(".meta.json") + ".jsonl")))
|
||||
if not isinstance(entries, Failure) and (one := round_from(checkout, stamp_seconds(began["timestamp"]), entries)):
|
||||
if not isinstance(entries, Failure) and (one := round_from(reviewer, checkout, stamp_seconds(began["timestamp"]), entries)):
|
||||
found.append(one)
|
||||
return found
|
||||
|
||||
@@ -166,7 +170,7 @@ def findings_found(found: Round, code: Code, rung: Rung, anchor: Rung, home: pat
|
||||
def review(found: Round, code: Code, rung: Rung, home: pathlib.Path, effects: Effects) -> str | Failure | Limited:
|
||||
try:
|
||||
(home / REPORTS).mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory(prefix="stability-replay-", ignore_cleanup_errors=True) as clone, tempfile.TemporaryDirectory(prefix="stability-replay-", dir=home / REPORTS, ignore_cleanup_errors=True) as reports:
|
||||
with tempfile.TemporaryDirectory(prefix="review-replay-", ignore_cleanup_errors=True) as clone, tempfile.TemporaryDirectory(prefix="review-replay-", dir=home / REPORTS, ignore_cleanup_errors=True) as reports:
|
||||
return reviewed_in(found, code, rung, home, effects, clone, reports)
|
||||
except OSError as failure:
|
||||
return Failure(f"could not prepare the replay: {failure}")
|
||||
@@ -192,7 +196,7 @@ def reviewed_in(found: Round, code: Code, rung: Rung, home: pathlib.Path, effect
|
||||
):
|
||||
if isinstance(ran := git(effects, found.root, *command), Failure):
|
||||
return ran
|
||||
args = ["--agent", REVIEWER, *rung.flags()]
|
||||
args = ["--agent", found.reviewer, *rung.flags()]
|
||||
reply = claude_reply(effects.claude, args, clone, prompt, "claude")
|
||||
if isinstance(reply, Failure | Limited):
|
||||
return reply
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Which archived stability rounds to replay, the code each reviewed, and the report: docs/decisions.md → `routing: the stability reviewer runs no lower than the project's rung`."""
|
||||
"""Which archived rounds of a review pass to replay, the code each reviewed, and the report: docs/decisions.md → `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds`."""
|
||||
import dataclasses
|
||||
import datetime
|
||||
import pathlib
|
||||
@@ -18,7 +18,8 @@ LINE_OR_STOP = re.compile(r"(?::\d+(?:-\d+)?)?[.:]?$")
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Round:
|
||||
"""One archived stability review of the checkout at root, at the UTC stamp at, in a chunk that began at began: its prompt, and the 🔴 and 🟡 findings its report holds, each with its lines up to the next blank line or finding."""
|
||||
"""One archived review by the agent reviewer of the checkout at root, at the UTC stamp at, in a chunk that began at began: its prompt, and the 🔴 and 🟡 findings its report holds, each with its lines up to the next blank line or finding."""
|
||||
reviewer: str
|
||||
root: str
|
||||
branch: str
|
||||
at: str
|
||||
@@ -31,15 +32,15 @@ class Round:
|
||||
return stamp_seconds(self.at)
|
||||
|
||||
|
||||
def round_from(root: str, began: float, entries: list[dict]) -> Round | None:
|
||||
"""The round a stability reviewer's transcript holds: its first entry's prompt, and its last text. None without a 🔴 or 🟡 finding to hold a rung to."""
|
||||
def round_from(reviewer: str, root: str, began: float, entries: list[dict]) -> Round | None:
|
||||
"""The round a reviewer's transcript holds: its first entry's prompt, and its last text. None without a 🔴 or 🟡 finding to hold a rung to."""
|
||||
reports = [text for found in entries for part in assistant_parts(found) if (text := part_text(part))]
|
||||
first = entries[0] if entries else {}
|
||||
if not stamped(first) or not reports or not (findings := recorded_findings(reports[-1])):
|
||||
return None
|
||||
prompt = content(first)
|
||||
prompt = prompt if isinstance(prompt, str) else "".join(part_text(part) or "" for part in parts(first))
|
||||
return Round(root, first.get("gitBranch", ""), first["timestamp"], began, prompt, tuple(findings))
|
||||
return Round(reviewer, root, first.get("gitBranch", ""), first["timestamp"], began, prompt, tuple(findings))
|
||||
|
||||
|
||||
def stamped(entry: dict) -> bool:
|
||||
@@ -94,10 +95,10 @@ def sample(rounds: list[Round], latest: int) -> list[Round]:
|
||||
return sorted(rounds, key=lambda found: (found.seconds, found.root))[-latest:]
|
||||
|
||||
|
||||
def report(model: str, reference: Rung, rounds: list[Round], replay: Replay[Round, int, set[int]]) -> str:
|
||||
def report(reviewed: str, model: str, reference: Rung, rounds: list[Round], replay: Replay[Round, int, set[int]]) -> str:
|
||||
total = sum(len(found_round.findings) for found_round in rounds)
|
||||
lines = [
|
||||
f"Replayed {len(rounds)} stability rounds of {model} projects, holding {total} 🔴 and 🟡 findings.",
|
||||
f"Replayed {len(rounds)} {reviewed} rounds of {model} projects, holding {total} 🔴 and 🟡 findings.",
|
||||
f"The reference reviewer, {reference}, found {sum(len(found) for found in replay.on_reference)} of them.",
|
||||
]
|
||||
for tried in replay.tried:
|
||||
+57
-32
@@ -14,14 +14,14 @@ import unittest.mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from stability_replay.rounds import authored_by, recorded_findings, report, reviewed_path, Round, round_from, sample, stamp_seconds
|
||||
from review_replay.rounds import authored_by, recorded_findings, report, reviewed_path, Round, round_from, sample, stamp_seconds
|
||||
from loop_claude.effects import Effects, system_effects
|
||||
from failure import Failure
|
||||
from stability_replay import main
|
||||
from review_replay import main
|
||||
from rung_replay.search import Miss, Replay, Tried
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
TOOL = pathlib.Path(__file__).resolve().parents[1] / "stability-replay"
|
||||
TOOL = pathlib.Path(__file__).resolve().parents[1] / "review-replay"
|
||||
REFERENCE = Rung("claude-opus-5-5", "medium")
|
||||
MIDDLE = Rung("claude-sonnet-5-5", "high")
|
||||
CHEAP = Rung("claude-haiku-5-5", "max")
|
||||
@@ -32,7 +32,7 @@ def entry(kind, text, **fields):
|
||||
|
||||
|
||||
def a_round(at, findings=("🔴 a",)):
|
||||
return Round("/code/p", "x", f"2026-10-14T10:00:0{at}Z", 0, "Review it.", tuple(findings))
|
||||
return Round("stability-reviewer", "/code/p", "x", f"2026-10-14T10:00:0{at}Z", 0, "Review it.", tuple(findings))
|
||||
|
||||
|
||||
class RecordedFindings(unittest.TestCase):
|
||||
@@ -51,19 +51,19 @@ class RoundFrom(unittest.TestCase):
|
||||
entry("assistant", "Reading."),
|
||||
entry("assistant", "BLOCK\n\n🔴 a.py:1 breaks"),
|
||||
]
|
||||
self.assertEqual(round_from("/code/p", 5.0, entries), Round("/code/p", "x", "2026-10-14T10:00:00.500Z", 5.0, "Review branch x in /code/p-x.", ("🔴 a.py:1 breaks",)))
|
||||
self.assertEqual(round_from("/code/p", 5.0, entries).seconds, 1791972000.5)
|
||||
self.assertEqual(round_from("stability-reviewer", "/code/p", 5.0, entries), Round("stability-reviewer", "/code/p", "x", "2026-10-14T10:00:00.500Z", 5.0, "Review branch x in /code/p-x.", ("🔴 a.py:1 breaks",)))
|
||||
self.assertEqual(round_from("stability-reviewer", "/code/p", 5.0, entries).seconds, 1791972000.5)
|
||||
|
||||
def test_reads_a_prompt_given_as_a_string(self):
|
||||
start = {"type": "user", "message": {"content": "Review branch x in /code/p-x."}, "timestamp": "2026-10-14T10:00:00Z", "gitBranch": "x"}
|
||||
self.assertEqual(round_from("/code/p", 5.0, [start, entry("assistant", "🔴 a")]).prompt, "Review branch x in /code/p-x.")
|
||||
self.assertEqual(round_from("stability-reviewer", "/code/p", 5.0, [start, entry("assistant", "🔴 a")]).prompt, "Review branch x in /code/p-x.")
|
||||
|
||||
def test_skips_a_round_with_no_red_or_yellow_finding_or_no_report(self):
|
||||
start = entry("user", "Review.", timestamp="2026-10-14T10:00:00Z", cwd="/code/p", gitBranch="x")
|
||||
self.assertIsNone(round_from("/code/p", 5.0, [start, entry("assistant", "APPROVE\n\n🟢 nit")]))
|
||||
self.assertIsNone(round_from("/code/p", 5.0, [start]))
|
||||
self.assertIsNone(round_from("/code/p", 5.0, []))
|
||||
self.assertIsNone(round_from("/code/p", 5.0, [{**start, "timestamp": "yesterday"}, entry("assistant", "🔴 a")]))
|
||||
self.assertIsNone(round_from("stability-reviewer", "/code/p", 5.0, [start, entry("assistant", "APPROVE\n\n🟢 nit")]))
|
||||
self.assertIsNone(round_from("stability-reviewer", "/code/p", 5.0, [start]))
|
||||
self.assertIsNone(round_from("stability-reviewer", "/code/p", 5.0, []))
|
||||
self.assertIsNone(round_from("stability-reviewer", "/code/p", 5.0, [{**start, "timestamp": "yesterday"}, entry("assistant", "🔴 a")]))
|
||||
|
||||
|
||||
class ReviewedPath(unittest.TestCase):
|
||||
@@ -96,7 +96,7 @@ class Sample(unittest.TestCase):
|
||||
class Report(unittest.TestCase):
|
||||
def test_counts_the_findings_and_names_each_miss_and_the_lowest_rung_that_missed_none(self):
|
||||
rounds = [a_round(1, ("🔴 a\nmore", "🟡 b")), a_round(2)]
|
||||
lines = report("claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, set()], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(rounds[0], 0, {1})])], MIDDLE)).splitlines()
|
||||
lines = report("stability", "claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, set()], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(rounds[0], 0, {1})])], MIDDLE)).splitlines()
|
||||
self.assertEqual(lines, [
|
||||
"Replayed 2 stability rounds of claude-opus-5-5 projects, holding 3 🔴 and 🟡 findings.",
|
||||
"The reference reviewer, claude-opus-5-5 at medium effort, found 2 of them.",
|
||||
@@ -105,8 +105,8 @@ class Report(unittest.TestCase):
|
||||
" /code/p 2026-10-14T10:00:01Z: 🔴 a",
|
||||
"Lowest rung that finds every finding the reference reviewer finds: claude-sonnet-5-5 at high effort",
|
||||
])
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, {0}], [Tried(MIDDLE, [Miss(rounds[1], 0, set())])], None)).splitlines()[-1], "Lowest rung that finds every finding the reference reviewer finds: none below the project's rung")
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay([set(), set()], [], None)).splitlines()[-1], "No rung tried: the reference reviewer found no finding to hold one to.")
|
||||
self.assertEqual(report("stability", "claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, {0}], [Tried(MIDDLE, [Miss(rounds[1], 0, set())])], None)).splitlines()[-1], "Lowest rung that finds every finding the reference reviewer finds: none below the project's rung")
|
||||
self.assertEqual(report("stability", "claude-opus-5-5", REFERENCE, rounds, Replay([set(), set()], [], None)).splitlines()[-1], "No rung tried: the reference reviewer found no finding to hold one to.")
|
||||
|
||||
|
||||
class Match(unittest.TestCase):
|
||||
@@ -174,6 +174,7 @@ class Main(unittest.TestCase):
|
||||
for key in ("proj", "other"):
|
||||
self.write_chunk(key, "c1", "2026-10-14T10:00:00Z")
|
||||
self.reviews = []
|
||||
self.agents = set()
|
||||
|
||||
def write_chunk(self, key, chunk, at, began="2026-10-14T09:15:00Z"):
|
||||
"""A chunk that began at began, whose stability round began at at."""
|
||||
@@ -207,6 +208,7 @@ class Main(unittest.TestCase):
|
||||
if "--agent" in args:
|
||||
shown = [subprocess.run(["git", *command], cwd=cwd, capture_output=True, text=True).stdout for command in (["show", "HEAD:a.txt"], ["branch", "--show-current"], ["show", "origin/main:a.txt"])]
|
||||
self.reviews.append((rung, *shown, prompt, cwd))
|
||||
self.agents.add(args[args.index("--agent") + 1])
|
||||
result = review(rung)
|
||||
else:
|
||||
reviewed = Rung(*re.search(r"review on (\S+) (\S+)", prompt).groups())
|
||||
@@ -221,7 +223,7 @@ class Main(unittest.TestCase):
|
||||
return code, out.getvalue(), err.getvalue()
|
||||
|
||||
def test_replays_the_models_rounds_on_the_commit_reviewed_from_the_project_rung_down(self):
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0] if rung.model == "claude-haiku-5-5" else [0, 1]})
|
||||
code, out, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0] if rung.model == "claude-haiku-5-5" else [0, 1]})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE, MIDDLE, CHEAP])
|
||||
_, reviewed, branch, base, prompt, cwd = self.reviews[0]
|
||||
@@ -233,6 +235,29 @@ class Main(unittest.TestCase):
|
||||
self.assertIn(f" {self.checkout} 2026-10-14T10:00:00Z: 🟡 slow", out)
|
||||
self.assertTrue(out.endswith("finds: claude-sonnet-5-5 at high effort\n"))
|
||||
|
||||
def test_replays_the_rounds_of_the_pass_it_names_with_that_passs_reviewer(self):
|
||||
(self.archive / "proj" / "c1" / "session" / "s" / "subagents" / "agent-a.jsonl").write_text(json.dumps(entry("user", "Review.", timestamp="2026-10-14T10:00:00Z")) + "\n")
|
||||
code, out, err = self.run_main(["prose", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("no archived prose round of projects on claude-opus-5-5", err)
|
||||
self.assertEqual(self.agents, set())
|
||||
code, out, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual(self.agents, {"stability-reviewer"})
|
||||
self.assertTrue(out.startswith("Replayed 1 stability rounds"), out)
|
||||
|
||||
def test_replays_the_prose_pass_with_the_prose_reviewer(self):
|
||||
code, out, err = self.run_main(["prose", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual(self.agents, {"prose-reviewer"})
|
||||
self.assertTrue(out.startswith("Replayed 1 prose rounds"), out)
|
||||
|
||||
def test_tries_each_rung_down_to_the_ladders_lowest_below_the_lowest_a_worker_routes_to(self):
|
||||
code, out, err = self.run_main(["prose", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE, MIDDLE, CHEAP, *(Rung("claude-haiku-5-5", effort) for effort in ("xhigh", "high", "medium", "low"))])
|
||||
self.assertTrue(out.endswith("finds: claude-haiku-5-5 at low effort\n"), out)
|
||||
|
||||
def review_folder_with_round_1(self):
|
||||
"""The chunk's review folder, holding round 1's report from before the replayed round began at 10:00, that round's own and a later pass's reports, written after, and an earlier attempt's report, from before the chunk began at 09:15."""
|
||||
folder = self.home / ".local" / "state" / "larv-review" / "code" / "proj-x"
|
||||
@@ -258,7 +283,7 @@ class Main(unittest.TestCase):
|
||||
(named / "stability-round-2.md").write_text("replayed\n")
|
||||
return {"type": "result", "result": f"review on {rung.model} {rung.effort}"}
|
||||
|
||||
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0, 1]}, review)
|
||||
code, _, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0, 1]}, review)
|
||||
self.assertEqual(code, 0, err)
|
||||
named, copied, earlier = seen[0]
|
||||
self.assertEqual(named.parent, folder.parents[1], "the copy sits in the reports folder, where rule-hook lets the reviewer write")
|
||||
@@ -272,16 +297,16 @@ class Main(unittest.TestCase):
|
||||
def test_points_the_review_folders_absolute_path_at_the_copy(self):
|
||||
folder = self.review_folder_with_round_1()
|
||||
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to {folder}/stability-round-2.md.")
|
||||
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
code, _, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
self.assertEqual(code, 0, err)
|
||||
prompt = self.reviews[0][4]
|
||||
self.assertNotIn(str(folder), prompt)
|
||||
self.assertRegex(prompt, rf"report to {folder.parents[1]}/stability-replay-\w+/stability-round-2\.md\.$")
|
||||
self.assertRegex(prompt, rf"report to {folder.parents[1]}/review-replay-\w+/stability-round-2\.md\.$")
|
||||
|
||||
def test_replays_no_round_whose_prompt_still_names_the_review_folder_after_the_rewrite(self):
|
||||
self.review_folder_with_round_1()
|
||||
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to $HOME/.local/state/larv-review/code/proj-x/stability-round-2.md.")
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], unused)
|
||||
code, out, err = self.run_main(["stability", "claude-opus-5-5"], unused)
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual(self.reviews, [])
|
||||
self.assertIn(f"on {REFERENCE} found nothing: the prompt names the review folder in a spelling the replay does not redirect", err)
|
||||
@@ -293,7 +318,7 @@ class Main(unittest.TestCase):
|
||||
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to ~/.local/state/larv-review/code/proj-x/stability-round-2.md.")
|
||||
scratch = pathlib.Path(self.enterContext(tempfile.TemporaryDirectory()))
|
||||
self.enterContext(unittest.mock.patch.object(tempfile, "tempdir", str(scratch)))
|
||||
code, _, err = self.run_main(["claude-opus-5-5"], unused)
|
||||
code, _, err = self.run_main(["stability", "claude-opus-5-5"], unused)
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual(self.reviews, [])
|
||||
self.assertIn(f"on {REFERENCE} found nothing: could not prepare the replay: ", err)
|
||||
@@ -302,13 +327,13 @@ class Main(unittest.TestCase):
|
||||
|
||||
def test_replays_the_rounds_of_every_chunk(self):
|
||||
self.write_chunk("proj", "c2", "2026-10-14T10:30:00Z")
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
code, out, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertTrue(out.startswith("Replayed 2 stability rounds"), out)
|
||||
|
||||
def test_checks_out_the_commit_detached_when_the_reviewer_started_on_the_default_branch(self):
|
||||
self.set_branch("main")
|
||||
self.assertEqual(self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0]})[0], 0)
|
||||
self.assertEqual(self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0]})[0], 0)
|
||||
self.assertEqual(self.reviews[0][1:3], ("1\n", ""))
|
||||
|
||||
def test_sets_the_default_branch_to_its_own_line_and_not_to_a_branch_merged_after_the_chunk_began(self):
|
||||
@@ -320,20 +345,20 @@ class Main(unittest.TestCase):
|
||||
run_git(self.checkout, "checkout", "-q", "main")
|
||||
run_git(self.checkout, "merge", "-q", "--no-ff", "-m", "m", "side", at="2026-10-14T09:40:00Z")
|
||||
run_git(self.checkout, "push", "-qf", "origin", "main")
|
||||
self.assertEqual(self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0]})[0], 0)
|
||||
self.assertEqual(self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0]})[0], 0)
|
||||
self.assertEqual(self.reviews[0][1:4], ("s\n", "x\n", "0\n"))
|
||||
|
||||
def test_finds_nothing_in_a_round_whose_review_has_no_report(self):
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], unused, review=lambda rung: {"type": "result"})
|
||||
code, out, err = self.run_main(["stability", "claude-opus-5-5"], unused, review=lambda rung: {"type": "result"})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE] * 3)
|
||||
self.assertIn(f" {self.checkout} 2026-10-14T10:00:00Z on {REFERENCE} found nothing: the reviewer replied with no report", err)
|
||||
self.assertTrue(out.endswith("No rung tried: the reference reviewer found no finding to hold one to.\n"))
|
||||
|
||||
def assert_no_round(self):
|
||||
code, _, err = self.run_main(["claude-opus-5-5"], unused)
|
||||
code, _, err = self.run_main(["stability", "claude-opus-5-5"], unused)
|
||||
self.assertEqual(code, 1)
|
||||
self.assertEqual(err.splitlines(), [f"stability-replay: no archived stability round of projects on claude-opus-5-5 in {self.archive} holds a 🔴 or 🟡 on code it can find"])
|
||||
self.assertEqual(err.splitlines(), [f"review-replay: no archived stability round of projects on claude-opus-5-5 in {self.archive} holds a 🔴 or 🟡 on code it can find"])
|
||||
|
||||
def test_skips_a_chunk_with_no_transcript_copy(self):
|
||||
(self.archive / "proj" / "c1" / "t0.jsonl").unlink()
|
||||
@@ -344,7 +369,7 @@ class Main(unittest.TestCase):
|
||||
self.assert_no_round()
|
||||
|
||||
def assert_left_out(self, reason, root=None):
|
||||
code, _, err = self.run_main(["claude-opus-5-5"], unused)
|
||||
code, _, err = self.run_main(["stability", "claude-opus-5-5"], unused)
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn(f"{root or self.checkout} 2026-10-14T10:00:00Z is left out: {reason}", err)
|
||||
|
||||
@@ -366,23 +391,23 @@ class Main(unittest.TestCase):
|
||||
self.assert_left_out("no commit was authored by then")
|
||||
|
||||
def test_exits_1_on_a_usage_limit_bad_arguments_or_no_round_to_replay(self):
|
||||
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung: {"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
|
||||
code, _, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("usage limit: You've hit your limit", err)
|
||||
for argv in ([], ["off-ladder"], ["claude-opus-5-5", "x"]):
|
||||
for argv in ([], ["claude-opus-5-5"], ["full scan", "claude-opus-5-5"], ["stability", "off-ladder"], ["stability", "claude-opus-5-5", "x"]):
|
||||
self.assertEqual(self.run_main(argv, unused)[0], 1)
|
||||
code, _, err = self.run_main(["claude-haiku-5-5"], unused)
|
||||
code, _, err = self.run_main(["stability", "claude-haiku-5-5"], unused)
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("no archived stability round of projects on claude-haiku-5-5", err)
|
||||
|
||||
def test_prints_the_help(self):
|
||||
code, out, _ = self.run_main(["-h"], unused)
|
||||
self.assertEqual((code, out.splitlines()[2]), (0, "usage: stability-replay <model>"))
|
||||
self.assertEqual((code, out.splitlines()[2]), (0, "usage: review-replay <pass> <model>"))
|
||||
|
||||
def test_the_launcher_prints_the_help(self):
|
||||
ran = subprocess.run([sys.executable, str(TOOL), "--help"], capture_output=True, text=True)
|
||||
self.assertEqual(ran.returncode, 0)
|
||||
self.assertIn("usage: stability-replay <model>", ran.stdout)
|
||||
self.assertIn("usage: review-replay <pass> <model>", ran.stdout)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
@@ -3,13 +3,13 @@ import os
|
||||
|
||||
from rule_hook.rules import Breach
|
||||
from rule_hook.call import Call
|
||||
from rule_hook.reviewer_rung import PASS_OF
|
||||
from review_folder import REPORTS
|
||||
from rule_hook.reviewer_rung import REVIEWERS
|
||||
|
||||
|
||||
def report_breaches(call: Call, path: str, home: str) -> list[Breach]:
|
||||
"""The breach of a file call from a reviewer that is not a Write under REPORTS in home; none from any other agent."""
|
||||
if call.agent not in REVIEWERS:
|
||||
if call.agent not in PASS_OF:
|
||||
return []
|
||||
written = os.path.realpath(os.path.join(call.cwd, path))
|
||||
reports = os.path.realpath(os.path.join(home, REPORTS))
|
||||
|
||||
@@ -1,14 +1,13 @@
|
||||
"""Refuse an Agent call that launches a review below the rung it needs, worked out from the rungs loop-claude hands its worker: docs/decisions.md → `routing:`."""
|
||||
import pathlib
|
||||
import typing
|
||||
|
||||
from rule_hook.rules import Breach
|
||||
from rule_hook.call import Call, text
|
||||
from model_routing.worker_rungs import handed
|
||||
from model_routing.ladder import highest, position, review_rung, ReviewKind, Rung
|
||||
from model_routing.ladder import position, review_rungs, REVIEWERS, ReviewKind, Rung
|
||||
|
||||
# A reviewer is an agent under agents/, as CLAUDE.md → "Vocabulary" defines it.
|
||||
REVIEWERS = frozenset(path.stem for path in (pathlib.Path(__file__).resolve().parents[2] / "agents").glob("*.md"))
|
||||
# The pass each agent under agents/ reviews.
|
||||
PASS_OF = {agent: kind for kind, agent in REVIEWERS.items()}
|
||||
# The opening of every comprehension seat's prompt: skills/comprehension-panel/seat-prompts.md → "Opening".
|
||||
SEAT = "You are one seat of a comprehension panel."
|
||||
# The opening of the security pass's prompt: skills/larv-review/SKILL.md → "Security".
|
||||
@@ -18,23 +17,22 @@ SUBJECTS: dict[ReviewKind, str] = {"seat": "a comprehension seat", "security": "
|
||||
|
||||
def reviewer_breaches(call: Call, environ: typing.Mapping[str, str]) -> list[Breach]:
|
||||
"""The breach of an Agent call that launches a review below its rung; none where loop-claude handed no rungs."""
|
||||
project, worker, reviewers = handed(environ)
|
||||
project, worker, escape_and_impact = handed(environ)
|
||||
kind = review_kind(call.given)
|
||||
if kind is None or project is None or worker is None:
|
||||
return []
|
||||
# A worker an older loop-claude started was handed no reviewers' rung; its reviewers are held to the rung that loop-claude gave them.
|
||||
required = review_rung(kind, project, reviewers or highest(project, worker))
|
||||
required = review_rungs(project, worker, escape_and_impact)[kind]
|
||||
named = named_rung(call.given, worker)
|
||||
if position(named) >= position(required):
|
||||
return []
|
||||
subject = text(call.given, "subagent_type") if kind == "reviewer" else SUBJECTS[kind]
|
||||
subject = SUBJECTS.get(kind) or text(call.given, "subagent_type")
|
||||
return [Breach("reviewer-rung", f"{subject} would run on {named}, below {required}: launch it with {required.agent_args()}", call.cwd)]
|
||||
|
||||
|
||||
def review_kind(given: dict[str, typing.Any]) -> ReviewKind | None:
|
||||
"""What a launch reviews, a reviewer, a comprehension seat or the security pass; None for any other subagent."""
|
||||
if text(given, "subagent_type") in REVIEWERS:
|
||||
return "reviewer"
|
||||
"""What a launch reviews, a reviewer's pass, a comprehension seat or the security pass; None for any other subagent."""
|
||||
if pass_reviewed := PASS_OF.get(text(given, "subagent_type")):
|
||||
return pass_reviewed
|
||||
opening = text(given, "prompt").lstrip()
|
||||
if opening.startswith(SEAT):
|
||||
return "seat"
|
||||
|
||||
@@ -5,6 +5,7 @@ import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import unittest.mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
@@ -14,8 +15,10 @@ from rule_hook.call import Call
|
||||
from git_command import Commit, commit_message, commits
|
||||
from test_rank_todo import DECISIONS, README, TODO
|
||||
from test_env import isolated_env
|
||||
from model_routing import ladder
|
||||
from rule_hook.override_log import LogLine, parsed_entry, this_chunks
|
||||
from rule_hook.reviewer_rung import reviewer_breaches
|
||||
from model_routing.ladder import Rung
|
||||
from shell_command import simple_commands
|
||||
|
||||
HOOK = pathlib.Path(__file__).resolve().parents[1] / "rule-hook"
|
||||
@@ -167,9 +170,9 @@ class OverrideLog(unittest.TestCase):
|
||||
|
||||
class ReviewerRung(unittest.TestCase):
|
||||
# A worker routed to Haiku 5.5 at max, on a project whose rung is Opus 5.5 at medium, with its reviewers on the project's rung.
|
||||
ENVIRON = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": "claude-opus-5-5 at medium effort"}
|
||||
# The same project, with a worker and its reviewers routed above it to Fable 5.1 at max.
|
||||
ABOVE = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-fable-5-1 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": "claude-fable-5-1 at max effort"}
|
||||
ENVIRON = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "4"}
|
||||
# The same project, with a worker, and so its reviewers, routed above it to Fable 5.1 at max.
|
||||
ABOVE = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-fable-5-1 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "4"}
|
||||
|
||||
def refused(self, given: dict, environ: dict[str, str] = ENVIRON) -> list[str]:
|
||||
return rules(reviewer_breaches(Call("Agent", given, "/r"), environ))
|
||||
@@ -200,20 +203,28 @@ class ReviewerRung(unittest.TestCase):
|
||||
def test_an_effort_off_the_ladder_for_its_model_is_refused(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "opus", "effort": "low"}), ["reviewer-rung"])
|
||||
|
||||
def test_a_reviewer_is_held_to_the_reviewers_rung_where_that_is_higher(self):
|
||||
def test_a_reviewer_is_held_to_the_rung_its_e_plus_i_picks_where_that_is_higher(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}, self.ABOVE), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "fable", "effort": "max"}, self.ABOVE), [])
|
||||
lifted = self.ENVIRON | {"LOOP_CLAUDE_REVIEWER_RUNG": "claude-opus-5-5 at high effort"}
|
||||
lifted = self.ENVIRON | {"LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5"}
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}, lifted), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"prompt": "You are one seat of a comprehension panel.", "model": "opus", "effort": "medium"}, lifted), [])
|
||||
|
||||
def test_holds_a_reviewer_to_the_higher_of_the_workers_and_the_projects_rung_where_loop_claude_handed_no_reviewers_rung(self):
|
||||
without = {name: rung for name, rung in self.ABOVE.items() if name != "LOOP_CLAUDE_REVIEWER_RUNG"}
|
||||
def test_holds_a_reviewer_to_the_higher_of_the_workers_and_the_projects_rung_where_loop_claude_handed_no_e_plus_i(self):
|
||||
without = {name: rung for name, rung in self.ABOVE.items() if name != "LOOP_CLAUDE_ESCAPE_AND_IMPACT"}
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}, without), ["reviewer-rung"])
|
||||
below = {name: rung for name, rung in self.ENVIRON.items() if name != "LOOP_CLAUDE_REVIEWER_RUNG"}
|
||||
below = {name: rung for name, rung in self.ENVIRON.items() if name != "LOOP_CLAUDE_ESCAPE_AND_IMPACT"}
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "sonnet", "effort": "high"}, below), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"prompt": "You are one seat of a comprehension panel.", "model": "sonnet", "effort": "high"}, without), ["reviewer-rung"])
|
||||
|
||||
def test_a_replayed_passs_reviewer_is_held_to_its_replays_rung_below_the_workers_unless_the_e_plus_i_steps_reviews_up(self):
|
||||
haiku = {"model": "haiku", "effort": "low"}
|
||||
with unittest.mock.patch.dict(ladder.REPLAYED, {("prose", "claude-opus-5-5"): Rung("claude-haiku-5-5", "medium")}):
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "haiku", "effort": "medium"}, self.ABOVE), [])
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", **haiku}, self.ABOVE), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "haiku", "effort": "medium"}, self.ABOVE), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "opus", "effort": "medium"}, self.ABOVE | {"LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5"}), ["reviewer-rung"])
|
||||
|
||||
def test_a_seat_or_a_security_launch_is_held_to_the_project_rung_alone(self):
|
||||
seat = {"prompt": "You are one seat of a comprehension panel. Tools: Read, Grep, Glob.", "model": "opus", "effort": "medium"}
|
||||
self.assertEqual(self.refused(seat, self.ABOVE), [])
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""What judge-replay's and stability-replay's commands share: the argument, the projects on its model, the candidate rungs and the exits."""
|
||||
"""What judge-replay's and review-replay's commands share: the projects on a model and the exits, and judge-replay's argument and candidate rungs."""
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
@@ -28,7 +28,7 @@ def projects_on(model: str, home: pathlib.Path) -> list[str] | Failure:
|
||||
|
||||
|
||||
def down_from(top: Rung) -> list[Rung]:
|
||||
"""top, the project's rung, and each rung below it, highest first, down to the lowest rung a worker routes to: no route uses a rung below it, so a replay there could change no routing."""
|
||||
"""top, the project's rung, and each rung below it, highest first, down to the lowest rung a worker routes to: judge-replay has no route below it to change."""
|
||||
return list(LADDER[position(lowest_worker_rung(top)):position(top) + 1])[::-1]
|
||||
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""The rung search judge-replay and stability-replay share: replay archived cases on a reference rung, then hold each candidate rung to what the reference reproduced."""
|
||||
"""The rung search judge-replay and review-replay share: replay archived cases on a reference rung, then hold each candidate rung to what the reference reproduced."""
|
||||
import concurrent.futures
|
||||
import dataclasses
|
||||
import sys
|
||||
|
||||
+17
-10
@@ -228,11 +228,11 @@ Each reviewer holds Write for its report. Claude Code passes a subagent's type t
|
||||
|
||||
The folder is fixed, since loop-claude's headless sessions have no scratchpad. It sits outside every checkout, so no project sees a report. It survives a reboot, unlike `/tmp`, so a resumed chunk still reads its earlier rounds. It is in the user's home, so no other local user plants a symlink that widens it. A report may quote a secret, so `larv-review review-folder`, which makes the chunk's folder, leaves `~/.local/state/larv-review` readable by the user alone.
|
||||
|
||||
A chunk keeps its folder after the merge. `stability-replay` hands a replayed reviewer a copy holding only the reports written after the chunk began and before the round did. So the replay reads the earlier rounds its prompt names, never the findings it is to find. The chunk's own reports stay as they were.
|
||||
A chunk keeps its folder after the merge. `review-replay` hands a replayed reviewer a copy holding only the reports written after the chunk began and before the round did. So the replay reads the earlier rounds its prompt names, never the findings it is to find. The chunk's own reports stay as they were.
|
||||
|
||||
Claude Code's own subagent prompt tells a subagent to return findings rather than write report files. `review-findings.md` → "Report" overrides that, since the file is the worker's input; a prose-reviewer probed on 2026-10-18 wrote the file its prompt named. A general-purpose wrapper that launches the reviewer was not tried: Claude Code's documentation says a subagent cannot start another.
|
||||
|
||||
Before this change, reports took a median 19k tokens per chunk, 8% of the worker's peak context, and lines carrying a severity mark were 9–24% of a report (2026-10-08, over 68 archived transcripts). `stability-replay` still counts each 🔴 and 🟡 by its line in the reviewer's reply, now a one-line summary. todo.md item 187 is to measure what the worker reads of the reports instead.
|
||||
Before this change, reports took a median 19k tokens per chunk, 8% of the worker's peak context, and lines carrying a severity mark were 9–24% of a report (2026-10-08, over 68 archived transcripts). `review-replay` still counts each 🔴 and 🟡 by its line in the reviewer's reply, now a one-line summary. todo.md item 187 is to measure what the worker reads of the reports instead.
|
||||
|
||||
Valid while Claude Code passes `agent_type` to hooks inside a subagent. Without it, the rule refuses nothing, and a reviewer may write anywhere.
|
||||
|
||||
@@ -260,15 +260,15 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
|
||||
|
||||
Valid while the index ranks these models, and while no newer model joins the catalog.
|
||||
|
||||
## routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, the seats and the security pass always on the project's
|
||||
## routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass, the seats and the security pass always on the project's
|
||||
|
||||
2026-10-14, lilleman. Serves goals 1.1 and 1.4.
|
||||
|
||||
Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks would spend on work that never ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each comprehension seat and the security pass's subagent on the project's rung, and each reviewer there too unless the highest E + I among its items picks a higher rung, through the Agent tool's model and effort. The worker runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. lilleman chose the project's rung, medium, knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
|
||||
|
||||
While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers never run below the worker's rung: a worker climbs because its chunk proved harder than its score said, which is when a cheaper reviewer is the bigger risk. A reviewer or seat that rules the work below standard may rerun one rung up: `review: a reviewer or comprehension seat that rules the work below standard`.
|
||||
While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers never run below the worker's rung, unless a replay lowered their pass: a worker climbs because its chunk proved harder than its score said, which is when a cheaper reviewer is the bigger risk. A reviewer or seat that rules the work below standard may rerun one rung up: `review: a reviewer or comprehension seat that rules the work below standard`.
|
||||
|
||||
On 2026-10-16 lilleman routed the reviewers on their item's E + I as well, on either side of the project's rung, which may lift them above both the worker's rung and the project's: how far a defect would escape, and what it would break, set how hard review must look (goal 1.1). loop-claude names that rung in the worker's prompt, and hands it to the worker's claude beside the project's and the worker's, so rule-hook holds each reviewer launch to it. A low E + I never steps a reviewer below the project's rung: lilleman chose to lower a pass only where a replay shows a lower rung finding every 🔴 and 🟡 the project's rung finds, which `todo.md` item 159 measures for each pass. An item the worker takes in beyond the routed one counts too, since its E + I scores its defects as the routed item's scores theirs. The worker's prompt names the reviewers' rung each higher E + I picks, so the worker reviews a taken-in item on it in the same run. The judge reports the highest E + I the worker states for a taken-in item. From the next cycle loop-claude routes on that E + I, and archives it with the route so a rerun keeps it.
|
||||
On 2026-10-16 lilleman routed the reviewers on their item's E + I as well, on either side of the project's rung, which may lift them above both the worker's rung and the project's: how far a defect would escape, and what it would break, set how hard review must look (goal 1.1). loop-claude names that rung in the worker's prompt. It hands the E + I to the worker's claude beside the project's rung and the worker's, so rule-hook works out each review's rung as the prompt does and holds each launch to it. A low E + I never steps a reviewer below the project's rung: only a replay lowers a pass, as `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and there even below the worker's` records. An item the worker takes in beyond the routed one counts too, since its E + I scores its defects as the routed item's scores theirs. The worker's prompt names the reviewers' rung each higher E + I picks, so the worker reviews a taken-in item on it in the same run. The judge reports the highest E + I the worker states for a taken-in item. From the next cycle loop-claude routes on that E + I, and archives it with the route so a rerun keeps it.
|
||||
|
||||
Valid until the trial on real chunks shows whether a worker below the project's rung holds quality.
|
||||
|
||||
@@ -334,12 +334,19 @@ Only Opus 5.5 was replayed, so the judge of a project on another model keeps the
|
||||
|
||||
Valid while the archived verdicts show what a judge should return, and until a new model joins the ladder: replay then.
|
||||
|
||||
## routing: the stability reviewer runs no lower than the project's rung while the worker runs below it, until `stability-replay` shows a lower rung finding every 🔴 and 🟡 the project's rung finds
|
||||
## routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and there even below the worker's; no pass has such a rung
|
||||
|
||||
2026-10-15, lilleman proposed running the stability reviewer on the worker's rung; the replay that gates it is the agent's proposal. Serves goals 1.1 and 1.4: the stability pass is the only one that mutation-tests the changed lines, so a defect it misses may merge, and a cheaper reviewer saves money only where it misses nothing.
|
||||
2026-10-15, lilleman proposed running the stability reviewer on the worker's rung, and the agent proposed the replay that gates it. On 2026-10-16 lilleman asked for the prose pass to run on the worker's rung or lower, and, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. Serves goal 1.4 held to goal 1.1: a cheaper reviewer saves money only where it misses nothing.
|
||||
|
||||
`bin/stability-replay <model>` replays archived stability rounds on the code each reviewed, first on the project's rung, then on each rung below it, highest first, and stops at the first that misses a finding the project's rung found again. A rung qualifies only where it finds every such 🔴 and 🟡.
|
||||
`bin/review-replay <pass> <model>` replays a pass's archived rounds on the code each reviewed. It replays them first on the project's rung, then on each rung below it, highest first, down to the ladder's lowest. It stops at the first rung that misses a finding the project's rung found again. A rung qualifies only where it and each rung above it find every such 🔴 and 🟡.
|
||||
|
||||
On 2026-10-15, `bin/stability-replay claude-opus-5-5` replayed the latest 10 rounds holding a 🔴 or 🟡, 40 findings in all, 8 of the rounds from this repo. Opus 5.5 at medium, the project's rung, found 29 of them again. Sonnet 5.5 at high missed 13 of those 29, 2 of them 🔴, so no rung below the project's rung qualifies, and Haiku 5.5, below Sonnet on the ladder, was not tried. So the stability reviewer runs where `routing: a chunk's worker runs on the rung its item's L picks` puts every reviewer: never below the project's rung while the worker runs below it.
|
||||
`REPLAYED` in `bin/model_routing/ladder.py` holds each qualifying rung, by pass and project model. Where the chunk's E + I steps no review up, the pass runs on that rung, even below the worker's. Where the E + I steps reviews up, the pass runs on the stepped rung like every other pass, since the replay measured only the project's rung. loop-claude hands the worker's claude the reviews' E + I in `LOOP_CLAUDE_ESCAPE_AND_IMPACT`, so rule-hook works out each pass's rung with `review_rungs`, as the worker's prompt does.
|
||||
|
||||
Valid until a new model joins the ladder, or the archive holds rounds a new reviewer prompt produced: replay then.
|
||||
The runs, each on the latest 10 rounds of Opus 5.5 projects that hold a 🔴 or 🟡:
|
||||
|
||||
- 2026-10-15, stability: 40 findings, 8 of the rounds from this repo. Opus 5.5 at medium, the project's rung, found 29 of them again. Sonnet 5.5 at high missed 13 of those 29, 2 of them 🔴. No rung qualifies.
|
||||
- 2026-10-18, prose: 39 findings. The project's rung found 27 of them again. Sonnet 5.5 at high missed 12 of those 27. No rung qualifies.
|
||||
|
||||
The prose pass went first: a missed prose finding breaks no behaviour, and a later change fixes it cheaply. Sonnet 5.5 missed nearly half of what the project's rung found there too. So the architecture, principles, overrides and product-owner passes are not replayed: their misses cost more, so a replay would spend money with little chance of a qualifying rung (goal 1.4). Every pass runs where `routing: a chunk's worker runs on the rung its item's L picks` puts every reviewer.
|
||||
|
||||
Valid until a new model joins the ladder, or a pass's reviewer prompt changes: replay that pass then, and once a new model joins, each pass not yet replayed.
|
||||
|
||||
@@ -665,3 +665,7 @@ Read most (opens and searches):
|
||||
## Reading cost, chunk b897b24d5e70, finished 2026-10-18T11:15:04Z
|
||||
|
||||
87043 new input tokens before the first commit.
|
||||
|
||||
## Reading cost, chunk 9f9579dd30bb, finished 2026-10-18T12:17:04Z
|
||||
|
||||
163028 new input tokens before the first commit.
|
||||
|
||||
@@ -47,7 +47,7 @@ Give a reviewer without Bash the branch diff and `git ls-files`'s output in its
|
||||
|
||||
`~/.claude/bin/larv-review` holds this section's mechanical rules: the chunk's review folder, the round count, the paired ruling's base and vote count, the scoring run's trigger, and the history's format and latest places, and its count of how often seats agree. Run it in the chunk's worktree; `~/.claude/bin/larv-review --help` states each rule. On exit 1, repair what its message names and run it again; when you cannot, ask the user.
|
||||
|
||||
**Reports.** Run `~/.claude/bin/larv-review review-folder` in the chunk's worktree: it makes the chunk's review folder and prints its path. In each prompt to an agent under `~/.claude/agents/`, name the file in that folder for its full report: `<pass>-round-<n>.md`. For a principles-auditor, use `principles-<yardstick>-round-<n>.md`. For a relaunch one rung up, add `-rerun` before `.md`. Before you answer a 🔴 or 🟡, read its entry: Grep the report for the finding's place, with the lines after it, rather than Read the whole report. rule-hook refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`. Keep the folder after the merge: `stability-replay` replays a round from the reports its prompt names.
|
||||
**Reports.** Run `~/.claude/bin/larv-review review-folder` in the chunk's worktree: it makes the chunk's review folder and prints its path. In each prompt to an agent under `~/.claude/agents/`, name the file in that folder for its full report: `<pass>-round-<n>.md`. For a principles-auditor, use `principles-<yardstick>-round-<n>.md`. For a relaunch one rung up, add `-rerun` before `.md`. Before you answer a 🔴 or 🟡, read its entry: Grep the report for the finding's place, with the lines after it, rather than Read the whole report. rule-hook refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`. Keep the folder after the merge: `review-replay` replays a round from the reports its prompt names.
|
||||
|
||||
**Precedent, then ask** — for the architecture pass and the full scan:
|
||||
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 159 | 0.2.0 | | decision | **Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.** | 4 | 2 | 3 | 7 | 6 | 1.4 by 7 | 1.9 |
|
||||
| 189 | 0.2.0 | | | **State one achieve goal for the health of a project's code in `~/.claude`, which every project's ranking weighs, most while its comprehension panel's overall is under 7.0, unless its README voids it.** | 4 | 2 | 3 | 7 | 6 | 1.1 by 6 | 1.7 |
|
||||
| 29 | 0.3.0 | | | **Make the contributor a standing member of every project's audience, unless its `## Audience` says maintainability does not matter.** | 2 | 2 | 3 | 5 | 2 | 1.1 by 5, 1.4 by 5 | 16.7 |
|
||||
| 119 | 0.3.0 | | | **Retitle the comprehension-floor decision so it says a chunk lifting a project towards the floor works first on its lowest-scoring dimension, and that the floor blocks nothing.** | 1 | 2 | 1 | 2 | 1 | 1.1 by 4, 1.4 by 4 | 16.0 |
|
||||
@@ -91,12 +90,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 159. Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.
|
||||
|
||||
lilleman asked, on 2026-10-16, for the prose pass to run on the worker's rung or lower, and the same day, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. Today `bin/loop_claude/cycle/prompts.py`'s `review_launches` keeps every reviewer at or above the project's rung whatever the item's E + I, so a docs-only chunk such as item 140's, routed to Haiku 5.5, runs its reviewers on Opus 5.5.
|
||||
|
||||
Whether a pass runs cheaper without lowering quality (goal 1.4 against 1.1) is measured, as `docs/decisions.md` → `routing: the stability reviewer runs no lower than the project's rung…` measures the stability reviewer: replay a pass's archived rounds on the code each reviewed, first on the project's rung, then on each rung below it, and take the lowest rung that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again. A missed prose finding breaks no behaviour and a later change fixes it cheaply, so replay the prose pass first. Generalise `bin/stability-replay` to take the pass, in place of a replay tool per pass. Record each run and the rung it chose in a `routing:` decision, valid until a new model joins the ladder or the pass's prompt changes. Give each pass a row of its own in `REVIEWS`, `review_launches`' table of review kinds in `bin/loop_claude/cycle/prompts.py`, rather than a branch.
|
||||
|
||||
### 189. State one achieve goal for the health of a project's code in `~/.claude`, which every project's ranking weighs, most while its comprehension panel's overall is under 7.0, unless its README voids it.
|
||||
|
||||
lilleman, 2026-10-18: every project shares the goal, so it lives in `~/.claude` once, and a project for which it is not relevant voids it explicitly in its README. It replaces each project's own readability goal, such as fejkdata's goal 3, visthus's goal 8 and adf-codec's goal 10: once it merges, file an item in each such project's `todo.md` to drop its own goal, since a project's Blocked by cannot name an item of this repo.
|
||||
|
||||
Reference in New Issue
Block a user