Merge pull request 'Replay archived stability rounds on the rungs below the project's, and keep the stability reviewer on the project's rung, as no cheaper rung found every finding' (#276) from plan-145 into main

This commit was merged in pull request #276.
This commit is contained in:
2026-10-15 21:39:45 +02:00
29 changed files with 1106 additions and 206 deletions
+5
View File
@@ -53,10 +53,15 @@
!bin/routing_figures/*.py
!bin/rule_hook/
!bin/rule_hook/*.py
!bin/rung_replay/
!bin/rung_replay/*.py
!bin/score-outcomes
!bin/score_outcomes/
!bin/score_outcomes/*.py
!bin/shell_command.py
!bin/stability-replay
!bin/stability_replay/
!bin/stability_replay/*.py
!bin/statusline.py
!bin/subagent-rung
!bin/subagent_rung/
+1
View File
@@ -38,6 +38,7 @@ Decisions live in `docs/decisions.md`:
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model, and a chunk's worker may run below that rung, launching its reviewers on that rung; the worker routes each subagent it delegates to by the R + C of the subagent's task
- routing: a worker hands a task one rung up on each escalation trigger, up to two rungs above the rung its item picks unless the human agrees, and opens the subagent's prompt with the trigger so the routing figures count it
- routing: on a model judge-replay covered, the judge runs on the rung its chunk's item picks, never below the project's rung
- routing: the stability reviewer stays on the project's rung while the worker runs below it, until `stability-replay` shows a lower rung finding every 🔴 and 🟡 the project's rung finds
## Working rules
+2 -1
View File
@@ -2,11 +2,12 @@
## Unreleased
- The new `bin/stability-replay <model>` replays archived stability reviews of the projects run on the model, each on the code it reviewed, first on the project's rung and then on each rung below, and prints the lowest rung that, like each rung above it, finds every 🔴 and 🟡 finding the project's rung finds. It shares its search with `judge-replay`, which now tries rungs from the project's rung down, where it tried them from the cheapest up. Its first run, on Opus 5.5 projects, found none: Sonnet 5.5 at high missed 13 of the 29 findings the project's rung found again. So the stability reviewer stays on the project's rung while the worker runs below it.
- A `todo.md` item you tell a session to keep is marked `owner` in its Exempt cell, and stays whatever its score.
- A `loop-claude` worker hands a task that proves too hard for its rung to a subagent one rung up, edits included, when a trigger fires: a re-score of the item on the project's rung or above picks a higher rung, the same failing test or finding survives two fix attempts, or a review pass reports a 🔴 in round 2 or later. Each further trigger climbs one more rung, to at most two rungs above the rung the chunk's `todo.md` item picks, unless you agree. `CLAUDE.md` states the rule, and `loop-claude`'s prompt names the rungs a trigger opens. The worker opens the subagent's prompt with `Escalation trigger <n>:`, and `routing-figures.md` counts the escalations by trigger. Expect a chunk that gets stuck to spend more on the stuck part, and fewer unfinished verdicts.
- A reviewer that rules a chunk's work below standard, with systems-architect's BLOCK or a finding against larv-review's `standards/quality.md`, may ask to rerun one rung up, as may a comprehension seat whose vote in a paired ruling `record-ruling` counts as worse. The worker relaunches it once, on the rung `loop-claude`'s prompt names. A relaunched reviewer gets the report that asked and confirms or dismisses each of its findings. The relaunch's report stands in place of the one that asked, in the same round, so it adds no round. The ruling that stands is a fourth escalation trigger in `CLAUDE.md`. Expect a chunk whose work falls below standard to spend more on its review and its fix.
- A session or subagent below the project's rung, the cheapest rung of the project's model, no longer scores or reshapes `todo.md`: it drafts each new item and each change to a filed item, and before its next commit one subagent on the project's rung or above applies and scores the drafts and sets their release. It may still remove finished items. `rules/todo-items.md` states the rule. `CLAUDE.md` has a session launch a subagent that files or scores items on the rung the new `bin/subagent-rung --plan <model> <R+C>` prints, never below the project's rung, and `loop-claude`'s prompt to a worker below that rung names the command. Expect one extra subagent in a chunk that files items while its worker runs below the project's rung.
- `loop-claude` runs the judge of an Opus 5.5 project on the rung the chunk's `todo.md` item picks, never below Opus 5.5 at medium effort, the project's rung. Before, the judge ran at the effort `settings.json` sets, xhigh. A replay of 39 archived verdicts found every cheaper rung missing verdicts. The judge keeps its rung when the worker steps up or takes in another item, and runs on the project's rung when no item routed the chunk. A project on another model keeps its judge at its settings' effort. Each verdict in the archive now names the judge's rung on a `Judge:` line. The new `bin/judge-replay <model>` replays the judge on archived verdicts and prints the cheapest rung that returns them. Expect the judge to cost less on items of R + C 12 or less, and more on items of R + C 15 or more.
- `loop-claude` runs the judge of an Opus 5.5 project on the rung the chunk's `todo.md` item picks, never below Opus 5.5 at medium effort, the project's rung. Before, the judge ran at the effort `settings.json` sets, xhigh. A replay of 39 archived verdicts found every cheaper rung missing verdicts. The judge keeps its rung when the worker steps up or takes in another item, and runs on the project's rung when no item routed the chunk. A project on another model keeps its judge at its settings' effort. Each verdict in the archive now names the judge's rung on a `Judge:` line. The new `bin/judge-replay <model>` replays the judge on archived verdicts and prints the lowest rung that, like each rung above it, returns them. Expect the judge to cost less on items of R + C 12 or less, and more on items of R + C 15 or more.
- Every 20 cycles across all projects, `loop-claude` measures every finished chunk in its archive, with no model run, and writes the figures to `routing-figures.md` at the archive's root: tokens by project, subagent type, rung and tool, and, for each chunk a `todo.md` item routed, grouped by the rung it started on: its review rounds past each pass's first, its unfinished verdicts and its subagents' rungs. It interrupts you, with `claude` on the project's model on the terminal in `~/.claude`, when chunks routed onto a rung took more review rounds or unfinished verdicts than chunks routed onto the next rung up, each rung with 5 chunks or more, or when 5 or more subagents of one type wrote reports longer than the tool results they read. Each call is raised once, then listed in `routing-calls-raised.md` beside the figures; the next chunk of this repo files a `todo.md` item for it. Off a terminal, the calls are said on stderr and the loop goes on. They are raised at the next measurement on a terminal. After a routing change, a `- <UTC time> Reset` line in that file makes the figures count only chunks finished after it, and lets its calls be raised again. Expect a pause of about ten seconds at each measurement.
- A session delegates a task to a subagent only when it needs just the task's conclusion, such as a search across many files, and keeps edits in its own thread. It scores the task's R and C as for a `todo.md` item, and launches the subagent on the model and effort that the new `bin/subagent-rung <model> <R+C>` picks from the project's model, such as `model sonnet, effort high`. `CLAUDE.md` states the rule. Expect searches to run on cheaper models, and less of each session's context to fill with search output.
- `loop-claude` routes each chunk's worker on a ladder of model and effort rungs, from Haiku 5.5 at low effort to Fable 5.1 at max, when the project's model is on it. As a chunk starts, the R + C of `todo.md`'s top item in the next release picks the rung: 11–12 picks one rung above the project's rung, so Opus 5.5 at high for an Opus 5.5 project, and so on up to four rungs. An item of R + C 10 or less runs the worker at or below the project's model's cheapest rung, which for Opus 5.5 is medium effort, below the xhigh `settings.json` sets. While the worker runs below the project's model's cheapest rung, it launches every reviewer, comprehension seat and security pass, the full scan included, in a subagent on that rung. While it runs above that rung, it launches each comprehension seat and the security pass on that rung. It steps up one rung after every second unfinished verdict on a chunk, and to a higher rung when it takes in an item of higher R + C. Each cycle's header, and each finished or unfinished verdict, stop and step-up in the archive, name the worker's rung; a rerun resumes an open chunk on the latest rung archived for it. `docs/decisions.md` → `routing:` gives the ladder, its source and the bands. Expect a chunk on an easy item to cost much less than under xhigh, and perhaps to take more review rounds, and one on an item of R + C 15 or more to cost more.
+3 -2
View File
@@ -67,7 +67,7 @@ Rename `## Unreleased` in `CHANGELOG.md` to the version, commit, tag that commit
Install puts only `loop-claude` on `PATH`; call the others by their path, such as `~/.claude/bin/metric-bench`, from the project's checkout.
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints the cheapest rung that returns them; `judge-replay --help` states the replay.
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints the lowest rung that, like each rung above it, returns them; `judge-replay --help` states the replay.
- `larv-review <command>` holds larv-review's mechanical rules: the paired ruling's base and vote count, each pass's round count, the scoring run's trigger, and the comprehension history's format and latest places; `larv-review --help` states them.
- `loop-claude <task>` runs a project's next chunk cycle after cycle; `loop-claude --help` states its contract.
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
@@ -75,6 +75,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure.
- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit and Write call, and `rule-hook --help` states the rules and the override.
- `score-outcomes [R S C]` lists the finished items whose guessed R, S and C sit nearest the given scores, with what their chunks measured, and each score's median outcome; `score-outcomes --help` states the measures.
- `stability-replay <model>` replays archived stability reviews of the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again; `stability-replay --help` states the replay. Run it only in a sandbox, like loop-claude: each replayed reviewer has Bash and skips every permission prompt, and an archived prompt may name paths on the host.
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
- `subagent-rung [--plan] <model> <R+C>` prints the Agent tool's model and effort for a subagent's task, picked by its R + C from the project's model, and with `--plan` never below the project's rung; `subagent-rung --help` states the rule.
@@ -93,7 +94,7 @@ pytest_add_cli_args=-k
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, and `judge_replay` imports `loop_claude`'s judge. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge. `judge_replay` and `stability_replay` share `rung_replay/`: the rung search, and the argument, candidate rungs and exits. It reads `loop_claude`'s retry count and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
## Tracking a new file
+38
View File
@@ -0,0 +1,38 @@
"""One `claude -p --output-format json` run, as loop-claude's judge and the replays start it."""
import dataclasses
import subprocess
import typing
from failure import Failure
from claude_code.entries import hit_limit, limit_said, parse_entry
# Effects.claude: claude's arguments, its working directory and the prompt it reads on stdin.
Claude = typing.Callable[[list[str], str, str], subprocess.CompletedProcess[str]]
@dataclasses.dataclass(frozen=True)
class Limited:
"""claude hit a usage limit; said is claude's message about it, and resets_at is when the limit resets, in seconds since the epoch, or None when unknown."""
resets_at: int | None
said: str
# The json reply claude_reply parses; no persistence, so its runs stay out of the session picker; no permission prompt, as no one answers it.
HEADLESS = ["--no-session-persistence", "--output-format", "json", "--permission-mode", "bypassPermissions"]
def claude_reply(claude: Claude, args: list[str], cwd: str, prompt: str, who: str) -> dict | Failure | Limited:
"""claude's json reply to `claude -p <args>` and HEADLESS, or why there is none; who, such as "the judge", names the run in the message for a non-zero exit or a reply that is no json."""
try:
proc = claude(["-p", *args, *HEADLESS], cwd, prompt)
except OSError as failure:
return Failure(f"could not start claude: {failure.strerror}")
reply = parse_entry(proc.stdout)
# The json output names the reset only in prose, such as "resets 7:40pm (Europe/Stockholm)", so its limit carries no reset time.
if reply is not None and hit_limit(reply):
return Limited(None, limit_said(reply))
if proc.returncode != 0:
return Failure(proc.stderr.strip() or f"{who} exited {proc.returncode}")
if reply is None:
return Failure(f"{who} printed no json reply; it printed: {proc.stdout.strip() or 'nothing'}")
return reply
+34 -1
View File
@@ -1,11 +1,13 @@
import json
import pathlib
import subprocess
import sys
import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from claude_code import entries, transcript, trust
from claude_code import entries, reply, transcript, trust
from failure import Failure
def assistant(*parts):
@@ -80,6 +82,37 @@ class Parse(unittest.TestCase):
self.assertIsNone(entries.parse_entry("[1, 2]"))
class ClaudeReply(unittest.TestCase):
@staticmethod
def claude(*outcomes):
def run(args, cwd, prompt):
outcome = outcomes[0]
if isinstance(outcome, OSError):
raise outcome
return subprocess.CompletedProcess(args, *outcome)
return run
def test_runs_claude_headless_with_the_args_in_cwd_on_the_prompt(self):
asked = []
def run(args, cwd, prompt):
asked.append((args, cwd, prompt))
return subprocess.CompletedProcess(args, 0, "{}", "")
reply.claude_reply(run, ["--tools", ""], "/r", "p", "the judge")
self.assertEqual(asked, [(["-p", "--tools", "", "--no-session-persistence", "--output-format", "json", "--permission-mode", "bypassPermissions"], "/r", "p")])
def test_returns_the_json_reply_a_usage_limit_whatever_the_exit_code_or_why_there_is_none(self):
self.assertEqual(reply.claude_reply(self.claude((0, '{"result": "ok"}', "")), [], "/r", "p", "the judge"), {"result": "ok"})
self.assertEqual(reply.claude_reply(self.claude((0, "Warning: hook said hi\n", "")), [], "/r", "p", "the judge"), Failure("the judge printed no json reply; it printed: Warning: hook said hi"))
self.assertEqual(reply.claude_reply(self.claude((0, "", "")), [], "/r", "p", "the judge"), Failure("the judge printed no json reply; it printed: nothing"))
limit = json.dumps({"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
self.assertEqual(reply.claude_reply(self.claude((1, limit, "")), [], "/r", "p", "the judge"), reply.Limited(None, "You've hit your limit"))
self.assertEqual(reply.claude_reply(self.claude((2, "", " ")), [], "/r", "p", "the judge"), Failure("the judge exited 2"))
self.assertEqual(reply.claude_reply(self.claude((2, "", "boom\n")), [], "/r", "p", "the judge"), Failure("boom"))
self.assertEqual(reply.claude_reply(self.claude(FileNotFoundError(2, "No such file or directory")), [], "/r", "p", "claude"), Failure("could not start claude: No such file or directory"))
class MarkTrusted(unittest.TestCase):
def test_returns_the_updated_config_and_leaves_the_one_it_was_handed(self):
config = {"other": 1, "projects": {"/a": {"hasTrustDialogAccepted": True}, "/b": {"kept": True}}}
+28 -65
View File
@@ -1,4 +1,4 @@
"""Replay loop-claude's judge on archived verdicts, to find the cheapest rung that returns them: docs/decisions.md → `routing: on a model judge-replay covered`.
"""Replay loop-claude's judge on archived verdicts, to find the lowest rung that returns them: docs/decisions.md → `routing: on a model judge-replay covered`.
usage: judge-replay <model>
@@ -11,104 +11,78 @@ with the task, premises and records each project holds now.
It first replays every verdict on the reference judge: the model at xhigh effort, the rung the
judge ran on before loop-claude routed it. A verdict the reference judge returns another kind on
may rest on records or judge rules that have changed since; either way no rung is held to it. It
then replays the rest on each candidate rung, cheapest first: from the rung the lowest R + C picks
up to the model's cheapest rung. It stops at the first rung that returns all of them. It prints
each verdict the reference judge did not reproduce, each verdict a rung missed, and the cheapest
rung that missed none. A rung it names is the evidence to add the model to JUDGE_REPLAYED in
bin/loop_claude/routing.py, which routes that model's judge, and to record the run in
docs/decisions.md → `routing: on a model judge-replay covered`.
then replays the rest on each candidate rung, highest first: from the model's cheapest rung down
to the rung the lowest R + C picks. It stops at the first rung that misses one. It prints each
verdict the reference judge did not reproduce, each verdict a rung missed, and the lowest rung
that, like each rung above it, missed none. A rung it names is the evidence to add the model to
JUDGE_REPLAYED in bin/loop_claude/routing.py, which routes that model's judge, and to record the
run in docs/decisions.md → `routing: on a model judge-replay covered`.
Like the loop, it tries a judge three times running before it counts the reply as failed. A
verdict whose project or transcript cannot be read replays as failed too. Exit 1, saying why, on
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
or a usage limit.
"""
import concurrent.futures
import pathlib
import sys
import typing
from transcript_archive.location import archive_root, transcript_of
from transcript_archive.read import archived_outcomes
from judge_replay.replay import Case, case_from, Kind, replayed, report, sample, verdict_kind
from judge_replay.replay import Case, case_from, Kind, report, reproduced, sample, verdict_kind
from project_key import checkout_of
from rung_replay.command import down_from, failed, limited_exit, project_rung_or_exit, projects_on
from loop_claude.effects import Effects
from failure import Failure
from loop_claude.cycle.retry import FAILURES_RUNNING, Limited
from rung_replay.search import first_reply, replayed
from loop_claude.cycle.judge import judge_once, Verdict
from model_routing.ladder import LADDER, picked, position, project_rung, Rung
from loop_claude.project.model_choice import read_models_file
from claude_code.reply import Limited
from loop_claude.project.premises import read_premises
from loop_claude.project.records import read_records
from claude_code.transcript import read_transcript, render
from model_routing.ladder import Rung
from loop_claude.stop import Stop
TOOL = "judge-replay"
PER_KIND = 15
# The effort settings.json gave every judge before loop-claude routed it; the archived verdicts came from it.
REFERENCE_EFFORT = "xhigh"
# Judges replayed at once.
AT_ONCE = 4
LOWEST_R_PLUS_C = 2
USAGE = "usage: judge-replay <model>, a model on the routing ladder"
def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, effects: Effects) -> int:
if argv[:1] in (["-h"], ["--help"]):
print(__doc__.strip())
return 0
anchor = project_rung(argv[0]) if len(argv) == 1 else None
if anchor is None:
print(f"judge-replay: cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; {USAGE}", file=sys.stderr)
return 1
cases = archived_cases(argv[0], archive_root(home, env), home)
anchor = project_rung_or_exit(TOOL, argv, __doc__)
if isinstance(anchor, int):
return anchor
cases = archived_cases(anchor.model, archive_root(home, env), home)
if isinstance(cases, Failure):
print(f"judge-replay: {cases.reason}", file=sys.stderr)
return 1
return failed(TOOL, cases.reason)
if not cases:
print(f"judge-replay: no archived verdicts of projects on {argv[0]} in {archive_root(home, env)}; run loop-claude on a project on that model first", file=sys.stderr)
return 1
return failed(TOOL, f"no archived verdicts of projects on {anchor.model} in {archive_root(home, env)}; run loop-claude on a project on that model first")
cases = sample(cases, PER_KIND)
reference = Rung(argv[0], REFERENCE_EFFORT)
candidates = list(LADDER[position(picked(anchor, LOWEST_R_PLUS_C)):position(anchor) + 1])
replay = replayed(cases, reference, candidates, lambda on, rung: rule_all(on, rung, home / ".claude", effects))
reference = Rung(anchor.model, REFERENCE_EFFORT)
replay = replayed(cases, reference, down_from(anchor), lambda found, rung: rule(found, rung, home / ".claude", effects), reproduced, "verdicts")
if isinstance(replay, Limited):
print(f"judge-replay: claude hit a usage limit: {replay.said}; replay again once it resets", file=sys.stderr)
return 1
print(report(argv[0], reference, cases, replay))
return limited_exit(TOOL, replay)
print(report(anchor.model, reference, cases, replay))
return 0
def archived_cases(model: str, root: pathlib.Path, home: pathlib.Path) -> list[Case] | Failure:
models = read_models_file(home / ".claude" / "loop-claude-models.json")
if isinstance(models, Stop):
return Failure(models.reason)
projects = projects_on(model, home)
if isinstance(projects, Failure):
return projects
listed = archived_outcomes(root)
if isinstance(listed, Failure):
return listed
found = []
try:
for key, outcomes in listed:
case = case_from(checkout_of(key, home), outcomes.read_text(encoding="utf-8"), transcript_of(outcomes)) if models.get(key) == model else None
case = case_from(checkout_of(key, home), outcomes.read_text(encoding="utf-8"), transcript_of(outcomes)) if key in projects else None
found += [case] if case else []
except (OSError, UnicodeDecodeError) as failure:
return Failure(f"could not read the archive {root}: {failure}")
return found
def rule_all(cases: list[Case], rung: Rung, claude_dir: pathlib.Path, effects: Effects) -> list[Kind] | Limited:
print(f"replaying {len(cases)} verdicts on {rung}", file=sys.stderr, flush=True)
with concurrent.futures.ThreadPoolExecutor(AT_ONCE) as pool:
running = [pool.submit(rule, found, rung, claude_dir, effects) for found in cases]
kinds = []
for future in running:
kind = future.result()
if isinstance(kind, Limited):
pool.shutdown(cancel_futures=True)
return kind
kinds.append(kind)
return kinds
def rule(case: Case, rung: Rung, claude_dir: pathlib.Path, effects: Effects) -> Kind | Limited:
premises, records, entries = read_premises(case.root), read_records(case.root, claude_dir), read_transcript(str(case.transcript))
if isinstance(premises, Stop):
@@ -117,16 +91,5 @@ def rule(case: Case, rung: Rung, claude_dir: pathlib.Path, effects: Effects) ->
return verdict_kind(Failure(records.reason))
if isinstance(entries, Failure):
return verdict_kind(entries)
verdict = first_reply(lambda: judge_once(case.root, case.task, premises, records, render(entries), rung, effects))
verdict: Verdict | Failure | Limited = first_reply(lambda: judge_once(case.root, case.task, premises, records, render(entries), rung, effects))
return verdict if isinstance(verdict, Limited) else verdict_kind(verdict)
def first_reply(attempt: typing.Callable[[], Verdict | Failure | Limited]) -> Verdict | Failure | Limited:
"""The first reply that is no Failure, or the last of FAILURES_RUNNING failures.
It counts failures as the loop's retried does, but neither waits between attempts nor waits out a usage limit, which ends the replay.
"""
for _ in range(FAILURES_RUNNING - 1):
if not isinstance(verdict := attempt(), Failure):
return verdict
return attempt()
+12 -40
View File
@@ -1,4 +1,4 @@
"""Which archived verdicts to replay, the cheapest rung that returns them, and the report: docs/decisions.md → `routing: on a model judge-replay covered`."""
"""Which archived verdicts to replay, what a replay reproduces, and the report: docs/decisions.md → `routing: on a model judge-replay covered`."""
import collections
import dataclasses
import pathlib
@@ -7,7 +7,7 @@ import typing
from transcript_archive.outcomes import ANSWERED, FINISHED, first_verdict, INTERRUPTED, task_of, UNFINISHED
from failure import Failure
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict
from loop_claude.cycle.retry import Limited
from rung_replay.search import Replay
from model_routing.ladder import Rung
# claude's reply as the replay compares it: a verdict as the archive words it, such as FINISHED, or a failure with its reason.
@@ -24,21 +24,6 @@ class Case:
on_record: Kind
@dataclasses.dataclass(frozen=True)
class Tried:
"""A rung, and each case it returned another kind on, with that kind."""
rung: Rung
missed: list[tuple[Case, Kind]]
@dataclasses.dataclass(frozen=True)
class Replay:
"""unreproduced holds each case the reference judge returned another kind on; tried, each rung tried, cheapest first; cheapest, the first rung that missed none, or None."""
unreproduced: list[tuple[Case, Kind]]
tried: list[Tried]
cheapest: Rung | None
def case_from(root: str, outcomes: str, transcript: pathlib.Path) -> Case | None:
"""The first verdict an outcomes file holds: the judge ruled on the transcript copy beside it as the copy stands. None when it holds none.
@@ -77,39 +62,26 @@ def verdict_kind(verdict: Verdict | Failure) -> Kind:
typing.assert_never(verdict)
def replayed(cases: list[Case], reference: Rung, candidates: list[Rung], rule_all: typing.Callable[[list[Case], Rung], list[Kind] | Limited]) -> Replay | Limited:
"""Replays the cases on the reference rung, then only those it reproduces on each candidate, cheapest first, until one returns every verdict."""
ruled = rule_all(cases, reference)
if isinstance(ruled, Limited):
return ruled
reproduced = [found for found, kind in zip(cases, ruled) if kind == found.on_record]
unreproduced = [(found, kind) for found, kind in zip(cases, ruled) if kind != found.on_record]
tried: list[Tried] = []
# With no verdict to hold a rung to, every rung would miss none.
for rung in candidates if reproduced else []:
ruled = rule_all(reproduced, rung)
if isinstance(ruled, Limited):
return ruled
tried.append(Tried(rung, [(found, kind) for found, kind in zip(reproduced, ruled) if kind != found.on_record]))
if not tried[-1].missed:
return Replay(unreproduced, tried, rung)
return Replay(unreproduced, tried, None)
def reproduced(case: Case, kind: Kind) -> list[Kind]:
"""The kind on record, where the reply returns it: a rung is held only to a verdict the reference judge reproduces."""
return [kind] if kind == case.on_record else []
def report(model: str, reference: Rung, cases: list[Case], replay: Replay) -> str:
def report(model: str, reference: Rung, cases: list[Case], replay: Replay[Case, Kind, Kind]) -> str:
kinds = collections.Counter(found.on_record for found in cases)
unreproduced = [(found, kind) for found, kind in zip(cases, replay.on_reference) if not reproduced(found, kind)]
lines = [
f"Replayed {len(cases)} verdicts of {model} projects: {', '.join(f'{count} {kind}' for kind, count in sorted(kinds.items()))}.",
f"The reference judge, {reference}, reproduced {len(cases) - len(replay.unreproduced)}{'. It returned another kind on:' if replay.unreproduced else '.'}",
*missed_lines(replay.unreproduced),
f"The reference judge, {reference}, reproduced {len(cases) - len(unreproduced)}{'. It returned another kind on:' if unreproduced else '.'}",
*missed_lines(unreproduced),
]
for tried in replay.tried:
lines += [f"{tried.rung} missed {len(tried.missed)}{':' if tried.missed else '.'}", *missed_lines(tried.missed)]
lines += [f"{tried.rung} missed {len(tried.missed)}{':' if tried.missed else '.'}", *missed_lines([(miss.case, miss.reply) for miss in tried.missed])]
if not replay.tried:
lines.append("No rung tried: the reference judge reproduced no verdict to hold one to.")
else:
cheapest = replay.cheapest or "none up to the project's rung"
lines.append(f"Cheapest rung that returns every verdict the reference judge reproduces: {cheapest}")
lowest = replay.lowest or "none, the project's rung misses one"
lines.append(f"Lowest rung that, like each rung above it, returns every verdict the reference judge reproduces: {lowest}")
return "\n".join(lines)
+22 -43
View File
@@ -10,12 +10,12 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINISHED
from judge_replay.replay import Case, case_from, Replay, replayed, report, sample, Tried, verdict_kind
from judge_replay.replay import Case, case_from, report, reproduced, sample, verdict_kind
from loop_claude.effects import Effects
from failure import Failure
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished
from loop_claude.cycle.retry import Limited
from judge_replay import main
from rung_replay.search import Miss, Replay, Tried
from model_routing.ladder import Rung
TOOL = pathlib.Path(__file__).resolve().parents[1] / "judge-replay"
@@ -59,52 +59,29 @@ class VerdictKind(unittest.TestCase):
self.assertEqual(verdict_kind(Failure("the judge quoted x")), "failed: the judge quoted x")
class Replayed(unittest.TestCase):
def test_tries_rungs_cheapest_first_on_the_verdicts_the_reference_reproduces_and_stops_at_the_first_that_returns_all(self):
cases = [case("2026-10-14T10:00:00Z", FINISHED), case("2026-10-14T10:00:01Z", INTERRUPTED), case("2026-10-14T10:00:02Z", FINISHED)]
replies = {REFERENCE: [FINISHED, INTERRUPTED, UNFINISHED], CHEAP: [FINISHED, FINISHED], MIDDLE: [FINISHED, INTERRUPTED]}
asked = []
def rule_all(on, rung):
asked.append((len(on), rung))
return replies[rung]
replay = replayed(cases, REFERENCE, [CHEAP, MIDDLE, Rung("claude-opus-5-5", "medium")], rule_all)
self.assertEqual(replay, Replay([(cases[2], UNFINISHED)], [Tried(CHEAP, [(cases[1], FINISHED)]), Tried(MIDDLE, [])], MIDDLE))
self.assertEqual(asked, [(3, REFERENCE), (2, CHEAP), (2, MIDDLE)])
def test_names_no_rung_when_every_candidate_misses_a_verdict(self):
cases = [case("2026-10-14T10:00:00Z", FINISHED)]
replay = replayed(cases, REFERENCE, [CHEAP], lambda on, rung: [FINISHED] if rung == REFERENCE else [INTERRUPTED])
self.assertEqual(replay, Replay([], [Tried(CHEAP, [(cases[0], INTERRUPTED)])], None))
def test_tries_no_rung_when_the_reference_reproduces_no_verdict(self):
cases = [case("2026-10-14T10:00:00Z", FINISHED)]
self.assertEqual(replayed(cases, REFERENCE, [CHEAP], lambda on, rung: [INTERRUPTED] * len(on)), Replay([(cases[0], INTERRUPTED)], [], None))
def test_stops_at_a_usage_limit(self):
limit = Limited(None, "limit")
self.assertEqual(replayed([case("2026-10-14T10:00:00Z", FINISHED)], REFERENCE, [CHEAP], lambda on, rung: limit), limit)
self.assertEqual(replayed([case("2026-10-14T10:00:00Z", FINISHED)], REFERENCE, [CHEAP], lambda on, rung: [FINISHED] if rung == REFERENCE else limit), limit)
class Reproduced(unittest.TestCase):
def test_holds_a_rung_to_a_verdict_only_where_the_reference_returned_the_kind_on_record(self):
self.assertEqual(reproduced(case("2026-10-14T10:00:00Z", FINISHED), FINISHED), [FINISHED])
self.assertEqual(reproduced(case("2026-10-14T10:00:00Z", FINISHED), INTERRUPTED), [])
class Report(unittest.TestCase):
def test_counts_the_verdicts_and_names_each_miss_and_the_cheapest_rung(self):
def test_counts_the_verdicts_and_names_each_miss_and_the_lowest_rung(self):
cases = [case("2026-10-14T10:00:00Z", FINISHED), case("2026-10-14T10:00:01Z", INTERRUPTED)]
replay = Replay([(cases[0], UNFINISHED)], [Tried(CHEAP, [(cases[1], FINISHED)]), Tried(MIDDLE, [])], MIDDLE)
replay = Replay([UNFINISHED, INTERRUPTED], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(cases[1], INTERRUPTED, FINISHED)])], MIDDLE)
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, replay).splitlines(), [
"Replayed 2 verdicts of claude-opus-5-5 projects: 1 Verdict: finished, 1 Verdict: interrupt.",
"The reference judge, claude-opus-5-5 at xhigh effort, reproduced 1. It returned another kind on:",
" /code/p 2026-10-14T10:00:00Z: on record Verdict: finished, replayed Verdict: unfinished",
"claude-sonnet-5-5 at high effort missed 0.",
"claude-haiku-5-5 at xhigh effort missed 1:",
" /code/p 2026-10-14T10:00:01Z: on record Verdict: interrupt, replayed Verdict: finished",
"claude-sonnet-5-5 at high effort missed 0.",
"Cheapest rung that returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort",
"Lowest rung that, like each rung above it, returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort",
])
missed_all = report("claude-opus-5-5", REFERENCE, cases, Replay([], [Tried(CHEAP, [(cases[1], FINISHED)])], None)).splitlines()
missed_all = report("claude-opus-5-5", REFERENCE, cases, Replay([FINISHED, INTERRUPTED], [Tried(MIDDLE, [Miss(cases[1], INTERRUPTED, FINISHED)])], None)).splitlines()
self.assertEqual(missed_all[1], "The reference judge, claude-opus-5-5 at xhigh effort, reproduced 2.")
self.assertEqual(missed_all[-1], "Cheapest rung that returns every verdict the reference judge reproduces: none up to the project's rung")
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, Replay([(cases[0], UNFINISHED), (cases[1], FINISHED)], [], None)).splitlines()[-1], "No rung tried: the reference judge reproduced no verdict to hold one to.")
self.assertEqual(missed_all[-1], "Lowest rung that, like each rung above it, returns every verdict the reference judge reproduces: none, the project's rung misses one")
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, Replay([UNFINISHED, FINISHED], [], None)).splitlines()[-1], "No rung tried: the reference judge reproduced no verdict to hold one to.")
def unused(*_):
@@ -148,25 +125,25 @@ class Main(unittest.TestCase):
output = {"answer": "", "bundled": 0, "evidence": evidence, "finished": not interrupt, "interrupt": interrupt, "question": "Which?" if interrupt else ""}
return {"type": "result", "structured_output": output}
def test_replays_the_models_projects_on_the_reference_then_cheapest_first_and_prints_the_rung_that_returns_all(self):
# Haiku misses the interrupt; Sonnet returns both.
def test_replays_the_models_projects_on_the_reference_then_from_the_projects_rung_down_and_prints_the_lowest_that_returns_all(self):
# Haiku misses the interrupt; Sonnet and Opus return both.
def reply(rung, interrupted):
return self.verdict(interrupted and rung.model != "claude-haiku-5-5", f"said {INTERRUPTED if interrupted else FINISHED}")
code, out, err = self.run_main(["claude-opus-5-5"], reply)
self.assertEqual(code, 0, err)
self.assertEqual({cwd for _, _, cwd in self.calls}, {str(self.home / "code" / "proj")})
self.assertEqual([(model, effort) for model, effort, _ in self.calls[::2]], [("claude-opus-5-5", "xhigh"), ("claude-haiku-5-5", "xhigh"), ("claude-haiku-5-5", "max"), ("claude-sonnet-5-5", "high")])
self.assertEqual([(model, effort) for model, effort, _ in self.calls[::2]], [("claude-opus-5-5", "xhigh"), ("claude-opus-5-5", "medium"), ("claude-sonnet-5-5", "high"), ("claude-haiku-5-5", "max")])
self.assertIn("The reference judge, claude-opus-5-5 at xhigh effort, reproduced 2.", out)
self.assertTrue(out.endswith("returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort\n"))
for said in ("The task an agent was given:\nDo it.", "1. Ship.", "From ~/.claude/CLAUDE.md:\nBe kind."):
self.assertIn(said, self.prompts[0])
def test_tries_rungs_up_to_the_models_cheapest_and_names_none_when_all_miss(self):
def test_stops_at_the_projects_rung_and_names_none_when_it_misses(self):
code, out, _ = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted and rung == REFERENCE, f"said {INTERRUPTED if interrupted else FINISHED}"))
self.assertEqual(code, 0)
self.assertEqual(self.calls[-1][:2], ("claude-opus-5-5", "medium"))
self.assertTrue(out.endswith(": none up to the project's rung\n"))
self.assertEqual({(model, effort) for model, effort, _ in self.calls[2:]}, {("claude-opus-5-5", "medium")})
self.assertTrue(out.endswith(": none, the project's rung misses one\n"))
def test_replays_a_verdict_whose_transcript_is_gone_as_failed(self):
(self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1" / "t0.jsonl").unlink()
@@ -174,14 +151,16 @@ class Main(unittest.TestCase):
self.assertEqual(code, 0)
self.assertIn("replayed failed: could not read the transcript", out)
def test_tries_up_to_the_models_cheapest_rung_and_retries_a_failed_reply_as_the_loop_does(self):
def test_tries_the_projects_rung_first_and_retries_a_failed_reply_as_the_loop_does(self):
def reply(rung, interrupted):
return self.verdict(interrupted, "not in the transcript" if rung.model != "claude-opus-5-5" else f"said {INTERRUPTED if interrupted else FINISHED}")
code, out, _ = self.run_main(["claude-opus-5-5"], reply)
self.assertEqual(code, 0)
self.assertEqual(sum(model == "claude-sonnet-5-5" for model, _, _ in self.calls), 6)
self.assertNotIn("claude-haiku-5-5", {model for model, _, _ in self.calls})
self.assertIn("claude-opus-5-5 at medium effort missed 0.", out)
self.assertTrue(out.endswith(": claude-opus-5-5 at medium effort\n"))
def test_exits_1_on_a_usage_limit_bad_arguments_or_an_unreadable_models_file(self):
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: {"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
+8 -18
View File
@@ -4,13 +4,14 @@ import re
import typing
from loop_claude.cycle.prompts import ANSWER_DESTINATIONS, ANSWERS_RECORDED, either
from claude_code.reply import claude_reply, Limited
from loop_claude.effects import Effects
from failure import Failure
from claude_code.entries import hit_limit, limit_said, parse_entry, structured_output
from loop_claude.project.records import HUMAN_RULE_NAMES
from loop_claude.cycle.retry import Limited, retried
from loop_claude.cycle.retry import retried
from model_routing.ladder import Rung
from loop_claude.stop import Stop
from claude_code.entries import structured_output
from todo_table import TOP_SCORE
# At 3 to 4 characters a token, 100,000 to 133,000 tokens, which leaves a 200,000-token context room for the reply; the largest session rendered here was 364,000 characters on 2026-10-03.
@@ -132,32 +133,21 @@ def judge_once(root: str, task: str, premises: str, records: str, rendered_trans
def judge_args(rung: Rung) -> list[str]:
# --json-schema has claude check its reply against the schema and return it as the json result's structured_output.
# No tools, so the verdict rests on the transcript grounded() checks it against; no persistence, so its runs stay out of the session picker.
# No tools, so the verdict rests on the transcript grounded() checks it against.
return [
"-p",
"--json-schema", json.dumps(JUDGE_SCHEMA),
*rung.flags(),
"--no-session-persistence",
"--output-format", "json",
"--permission-mode", "bypassPermissions",
"--tools", "",
]
def run_judge(root: str, args: list[str], prompt: str, shown: str, quotable: str, effects: Effects) -> Verdict | Failure | Limited:
try:
proc = effects.claude(args, root, prompt)
except OSError as failure:
return Failure(f"could not start claude: {failure.strerror}")
reply = parse_entry(proc.stdout) or {}
# The json output names the reset only in prose, such as "resets 7:40pm (Europe/Stockholm)", so the judge's limit carries no reset time.
if hit_limit(reply):
return Limited(None, limit_said(reply))
if proc.returncode != 0:
return Failure(proc.stderr.strip() or f"the judge exited {proc.returncode}")
reply = claude_reply(effects.claude, args, root, prompt, "the judge")
if isinstance(reply, Failure | Limited):
return reply
verdict = verdict_from(structured_output(reply))
if isinstance(verdict, Failure):
return Failure(f"{verdict.reason}; it replied: {proc.stdout.strip() or 'nothing'}")
return Failure(f"{verdict.reason}; it replied: {json.dumps(reply, ensure_ascii=False)}")
if not grounded(verdict.evidence, shown):
return Failure(f"the judge quoted {verdict.evidence!r}, which is not in the transcript" + (f", for the question: {verdict.question}" if isinstance(verdict, Interrupt | QuotedAnswer) else ""))
# Not a Failure: a misquote would otherwise exit an unattended loop. Never elided: stitched fragments can say what the records do not.
+1 -8
View File
@@ -1,9 +1,9 @@
import dataclasses
import sys
import typing
from loop_claude.effects import Effects
from failure import Failure
from claude_code.reply import Limited
from loop_claude.stop import Stop
T = typing.TypeVar("T")
@@ -14,13 +14,6 @@ RETRY_DELAY = 60
UNKNOWN_RESET_POLL = 600
@dataclasses.dataclass(frozen=True)
class Limited:
"""claude hit a usage limit; said is claude's message about it, and resets_at is when the limit resets, in seconds since the epoch, or None when unknown."""
resets_at: int | None
said: str
def retried(who: str, attempt: typing.Callable[[], T | Failure | Limited], effects: Effects) -> T | Stop:
failures = 0
while True:
+1 -1
View File
@@ -1,7 +1,7 @@
from claude_code.entries import assistant_parts, ends_stream, hit_limit, limit_said, parse_entry, rejected_until
from loop_claude.effects import Effects
from failure import Failure
from loop_claude.cycle.retry import Limited
from claude_code.reply import Limited
from claude_code.transcript import part_line
from model_routing.ladder import Rung
from claude_code.session import Session
+4 -4
View File
@@ -102,10 +102,10 @@ Exit 1, saying why, when:
handover
- the worker or the judge fails three times running, retried a minute apart within the cycle. A
worker fails when claude cannot start, exits non-zero, is killed, or ends without a result event;
a judge when claude cannot start or exits non-zero, or its reply misses or mistypes a field,
interrupts or quotes an answer without a question, gives an unfinished verdict an R + C outside
0 to 20, or quotes a passage the transcript does not hold; it holds an elided passage when each
fragment appears in order.
a judge when claude cannot start, exits non-zero or prints no json reply, or its reply misses or
mistypes a field, interrupts or quotes an answer without a question, gives an unfinished verdict
an R + C outside 0 to 20, or quotes a passage the transcript does not hold; it holds an elided
passage when each fragment appears in order.
A usage limit is not a failure, and restarts the count of three: the worker or the judge is
retried 2 minutes after the limit resets, or every 10 minutes when the reset time is unknown or
more than 2 minutes past. The judge's reply gives its reset time only as text, so the judge is
+1 -1
View File
@@ -455,7 +455,7 @@ class Cycles(unittest.TestCase):
self.assertEqual(run.roles(), ["worker", "judge", "judge", "judge"])
self.assertEqual(run.sleeps, [60, 60])
self.assertIn("retrying the judge in a minute: judge down", run.stderr)
self.assertIn("retrying the judge in a minute: the judge returned no reply; it replied: nothing\n", run.stderr)
self.assertIn("retrying the judge in a minute: the judge printed no json reply; it printed: nothing\n", run.stderr)
self.assertEqual(run.stderr.strip().splitlines()[-1], 'the judge failed three times running: the judge\'s reply has no answer of type string; it replied: {"structured_output": {"interrupt": false}}')
def test_waits_out_a_usage_limit_which_ends_a_run_of_failures(self):
+13 -9
View File
@@ -19,12 +19,12 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from routing_figures import archive
from loop_claude.project import checkout, model_choice, records
from loop_claude.effects import Effects
from claude_code import entries, session
from claude_code import entries, reply, session
from failure import Failure
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished, verdict_from
from loop_claude.cycle import interrupt, judge, prompts, retry, run, states, worker
from model_routing.ladder import LADDER, Rung
from loop_claude.cycle.retry import Limited
from claude_code.reply import Limited
from loop_claude.loop import main, reload, routing_check
from transcript_archive import outcomes, write
from loop_claude import paths, routing
@@ -885,21 +885,25 @@ class ExitContract(unittest.TestCase):
class JudgeFailures(unittest.TestCase):
def test_every_failed_judge_run_has_its_clause_in_help(self):
started, field, unasked = "claude cannot start or exits non-zero", "its reply misses or mistypes a field", "interrupts or quotes an answer without a question"
# Every Failure the judge's run returns, in source order: path, function, start of reason, its --help clause.
started, exited, unparsed = "claude cannot start", "exits non-zero", "prints no json reply"
field, unasked = "its reply misses or mistypes a field", "interrupts or quotes an answer without a question"
# Every Failure the judge's run returns, in source order of judge.py, then of the claude run it calls: function, start of reason, its --help clause.
clauses = [
("verdict_from", "'the judge returned no reply'", field),
("verdict_from", 'f"the judge\'s reply has no {name}', field),
("verdict_from", "'the judge interrupted without a question'", unasked),
("verdict_from", "'the judge quoted an answer without a question'", unasked),
("verdict_from", 'f"the judge\'s bundled R + C', "gives an unfinished verdict an R + C outside 0 to 20"),
("run_judge", "f'could not start claude:", started),
("run_judge", "proc.stderr.strip() or f'the judge exited", started),
("run_judge", 'f"{verdict.reason}; it replied:', field),
("run_judge", "f'{verdict.reason}; it replied:", field),
("run_judge", "f'the judge quoted {verdict.evidence!r}, which is not in the transcript", "quotes a passage the transcript does not hold"),
("claude_reply", "f'could not start claude:", started),
("claude_reply", "proc.stderr.strip() or f'{who} exited", exited),
("claude_reply", 'f"{who} printed no json reply', unparsed),
]
functions = [node for node in ast.walk(ast.parse(pathlib.Path(judge.__file__).read_text())) if isinstance(node, ast.FunctionDef)]
found = sorted((call.lineno, function.name, ast.unparse(call.args[0])) for function in functions for call in ast.walk(function) if isinstance(call, ast.Call) and getattr(call.func, "id", None) == "Failure")
found = []
for module in (judge, reply):
functions = [node for node in ast.walk(ast.parse(pathlib.Path(module.__file__).read_text())) if isinstance(node, ast.FunctionDef)]
found += sorted((call.lineno, function.name, ast.unparse(call.args[0])) for function in functions for call in ast.walk(function) if isinstance(call, ast.Call) and getattr(call.func, "id", None) == "Failure")
self.assertEqual(len(found), len(clauses), found)
for (_, function, reason), (want_function, reason_start, _) in zip(found, clauses):
self.assertEqual(function, want_function)
+1 -1
View File
@@ -15,5 +15,5 @@ class Types(unittest.TestCase):
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
if shutil.which("uv") is None:
self.fail("uv on PATH fetches mypy; see README.md's Test line")
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "rule_hook", "-p", "score_outcomes", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "rule_hook", "-p", "rung_replay", "-p", "score_outcomes", "-p", "stability_replay", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
View File
+44
View File
@@ -0,0 +1,44 @@
"""What judge-replay's and stability-replay's commands share: the argument, the projects on its model, the candidate rungs and the exits."""
import pathlib
import sys
from failure import Failure
from model_routing.ladder import LADDER, picked, position, project_rung, Rung
from claude_code.reply import Limited
from loop_claude.project.model_choice import read_models_file
from loop_claude.stop import Stop
LOWEST_R_PLUS_C = 2
def project_rung_or_exit(tool: str, argv: list[str], help_text: str) -> Rung | int:
"""The project's rung of the model argv names, or the exit code: 0 after printing help_text for -h or --help, 1 after saying why argv cannot be read."""
if argv[:1] in (["-h"], ["--help"]):
print(help_text.strip())
return 0
anchor = project_rung(argv[0]) if len(argv) == 1 else None
if anchor is None:
return failed(tool, f"cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; usage: {tool} <model>, a model on the routing ladder")
return anchor
def projects_on(model: str, home: pathlib.Path) -> list[str] | Failure:
"""The keys of the projects ~/.claude/loop-claude-models.json runs on model, sorted."""
models = read_models_file(home / ".claude" / "loop-claude-models.json")
if isinstance(models, Stop):
return Failure(models.reason)
return sorted(key for key, on in models.items() if on == model)
def down_from(top: Rung) -> list[Rung]:
"""top and each rung below it, down to the rung the lowest R + C picks from top, highest first."""
return list(LADDER[position(picked(top, LOWEST_R_PLUS_C)):position(top) + 1])[::-1]
def limited_exit(tool: str, limited: Limited) -> int:
return failed(tool, f"claude hit a usage limit: {limited.said}; replay again once it resets")
def failed(tool: str, reason: str) -> int:
print(f"{tool}: {reason}", file=sys.stderr)
return 1
+101
View File
@@ -0,0 +1,101 @@
"""The rung search judge-replay and stability-replay share: replay archived cases on a reference rung, then hold each candidate rung to what the reference reproduced."""
import concurrent.futures
import dataclasses
import sys
import typing
from failure import Failure
from loop_claude.cycle.retry import FAILURES_RUNNING
from claude_code.reply import Limited
from model_routing.ladder import Rung
# Cases replayed at once.
AT_ONCE = 4
Case = typing.TypeVar("Case")
# What of a case's record a reply reproduces, such as a verdict's kind or a finding's index.
Item = typing.TypeVar("Item")
Reply = typing.TypeVar("Reply")
T = typing.TypeVar("T")
@dataclasses.dataclass(frozen=True)
class Miss(typing.Generic[Case, Item, Reply]):
"""An item of case that the reference reproduced and a rung's reply missed."""
case: Case
item: Item
reply: Reply
@dataclasses.dataclass(frozen=True)
class Tried(typing.Generic[Case, Item, Reply]):
rung: Rung
missed: list[Miss[Case, Item, Reply]]
@dataclasses.dataclass(frozen=True)
class Replay(typing.Generic[Case, Item, Reply]):
"""on_reference holds the reference's reply on each case; tried, each candidate tried, highest first; lowest, the lowest rung that, like each candidate above it, missed nothing, or None."""
on_reference: list[Reply]
tried: list[Tried[Case, Item, Reply]]
lowest: Rung | None
def replayed(
cases: list[Case],
reference: Rung,
candidates: list[Rung],
replay_one: typing.Callable[[Case, Rung], Reply | Limited],
reproduced: typing.Callable[[Case, Reply], list[Item]],
what: str,
) -> Replay[Case, Item, Reply] | Limited:
"""Replays the cases on the reference rung, then each candidate, highest first, on the cases where the reference reproduced anything, until one misses an item it reproduced.
Going down, every rung at or above the one named was tried and missed nothing, so the result needs no assumption that a higher rung does at least as well; and a miss high up spares the cheaper rungs their replays.
"""
on_reference = replayed_on(cases, reference, replay_one, what)
if isinstance(on_reference, Limited):
return on_reference
held = [(case, aim) for case, reply in zip(cases, on_reference) if (aim := reproduced(case, reply))]
tried: list[Tried[Case, Item, Reply]] = []
lowest = None
# With nothing to hold a rung to, every rung would miss nothing.
for rung in candidates if held else []:
replies = replayed_on([case for case, _ in held], rung, replay_one, what)
if isinstance(replies, Limited):
return replies
misses: list[Miss[Case, Item, Reply]] = []
for (case, aim), reply in zip(held, replies):
got = reproduced(case, reply)
misses += [Miss(case, item, reply) for item in aim if item not in got]
tried.append(Tried(rung, misses))
if misses:
break
lowest = rung
return Replay(on_reference, tried, lowest)
def replayed_on(cases: list[Case], rung: Rung, replay_one: typing.Callable[[Case, Rung], Reply | Limited], what: str) -> list[Reply] | Limited:
"""Each case's reply on rung, AT_ONCE at a time; a usage limit cancels the rest."""
print(f"replaying {len(cases)} {what} on {rung}", file=sys.stderr, flush=True)
with concurrent.futures.ThreadPoolExecutor(AT_ONCE) as pool:
running = [pool.submit(replay_one, case, rung) for case in cases]
replies: list[Reply] = []
for future in running:
reply = future.result()
if isinstance(reply, Limited):
pool.shutdown(cancel_futures=True)
return reply
replies.append(reply)
return replies
def first_reply(attempt: typing.Callable[[], T | Failure | Limited]) -> T | Failure | Limited:
"""The first reply that is no Failure, or the last of FAILURES_RUNNING failures.
It counts failures as loop-claude's retried does, but neither waits between attempts nor waits out a usage limit, which ends a replay.
"""
for _ in range(FAILURES_RUNNING - 1):
if not isinstance(reply := attempt(), Failure):
return reply
return attempt()
+71
View File
@@ -0,0 +1,71 @@
import contextlib
import io
import pathlib
import sys
import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from rung_replay.command import down_from
from failure import Failure
from rung_replay.search import first_reply, Miss, Replay, replayed, Tried
from claude_code.reply import Limited
from model_routing.ladder import Rung
REFERENCE = Rung("claude-opus-5-5", "medium")
MIDDLE = Rung("claude-sonnet-5-5", "high")
CHEAP = Rung("claude-haiku-5-5", "max")
LOWER = Rung("claude-haiku-5-5", "xhigh")
def search(cases, candidates, replies):
"""replayed on cases named by strings, where replies[rung][case] is the set of items a rung reproduces in the case."""
asked = []
def replay_one(case, rung):
asked.append((case, rung))
reply = replies[rung]
return reply if isinstance(reply, Limited) else reply[case]
with contextlib.redirect_stderr(io.StringIO()):
return replayed(cases, REFERENCE, candidates, replay_one, lambda _, reply: sorted(reply), "cases"), asked
class Replayed(unittest.TestCase):
def test_holds_each_candidate_highest_first_to_what_the_reference_reproduced_and_stops_at_the_first_that_misses(self):
replies = {REFERENCE: {"a": {0, 1}, "b": set()}, MIDDLE: {"a": {0, 1}}, CHEAP: {"a": {1}}}
replay, asked = search(["a", "b"], [MIDDLE, CHEAP, LOWER], replies)
self.assertEqual(replay, Replay([{0, 1}, set()], [Tried(MIDDLE, []), Tried(CHEAP, [Miss("a", 0, {1})])], MIDDLE))
self.assertEqual(sorted(asked, key=lambda pair: (pair[1] != REFERENCE, pair[0])), [("a", REFERENCE), ("b", REFERENCE), ("a", MIDDLE), ("a", CHEAP)])
def test_names_no_rung_when_the_first_misses_and_tries_none_when_the_reference_reproduced_nothing(self):
self.assertEqual(search(["a"], [MIDDLE, CHEAP], {REFERENCE: {"a": {0}}, MIDDLE: {"a": set()}})[0], Replay([{0}], [Tried(MIDDLE, [Miss("a", 0, set())])], None))
self.assertEqual(search(["a"], [MIDDLE], {REFERENCE: {"a": set()}}), (Replay([set()], [], None), [("a", REFERENCE)]))
def test_counts_a_miss_in_any_case_not_only_the_last(self):
replies = {REFERENCE: {"a": {0}, "b": {0}}, MIDDLE: {"a": set(), "b": {0}}}
self.assertEqual(search(["a", "b"], [MIDDLE], replies)[0], Replay([{0}, {0}], [Tried(MIDDLE, [Miss("a", 0, set())])], None))
def test_stops_at_a_usage_limit(self):
limit = Limited(None, "limit")
self.assertEqual(search(["a"], [MIDDLE], {REFERENCE: limit})[0], limit)
self.assertEqual(search(["a"], [MIDDLE, CHEAP], {REFERENCE: {"a": {0}}, MIDDLE: limit})[0], limit)
class FirstReply(unittest.TestCase):
def test_tries_three_times_and_returns_the_first_reply_that_is_no_failure(self):
replies = [Failure("one"), Failure("two"), "done"]
self.assertEqual(first_reply(lambda: replies.pop(0)), "done")
replies = [Failure("one"), Limited(None, "limit")]
self.assertEqual(first_reply(lambda: replies.pop(0)), Limited(None, "limit"))
replies = [Failure("one"), Failure("two"), Failure("three"), "unasked"]
self.assertEqual(first_reply(lambda: replies.pop(0)), Failure("three"))
class DownFrom(unittest.TestCase):
def test_runs_from_the_projects_rung_down_to_the_rung_the_lowest_r_plus_c_picks(self):
self.assertEqual(down_from(REFERENCE), [REFERENCE, MIDDLE, CHEAP, LOWER])
if __name__ == "__main__":
unittest.main()
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env python3
import os
import pathlib
import sys
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from stability_replay import main
from loop_claude.effects import system_effects
sys.exit(main.main(sys.argv[1:], os.environ, pathlib.Path.home(), system_effects()))
View File
+207
View File
@@ -0,0 +1,207 @@
"""Replay archived stability rounds on the rungs below the project's, to find which rungs may run the stability reviewer: docs/decisions.md → `routing: the stability reviewer stays on the project's rung`.
usage: stability-replay <model>
<model> is a project model on the routing ladder, such as claude-opus-5-5. The tool replays the
latest 10 archived stability rounds whose report holds a 🔴 or 🟡 finding, from the projects that
~/.claude/loop-claude-models.json runs on the model. It replays each round on the code it
reviewed: in a throwaway clone of the project's checkout, at the newest commit of origin/HEAD
authored by the time the round began, on the branch the reviewer started on. The clone's default
branch stands at the newest commit authored before the round's chunk began, so a diff against it
shows the chunk's change. The prompt names the clone in place of the worktree it named, which the
chunk has since removed. A round whose code it cannot find in this way is not replayed; stderr
names it and why.
It first replays every round on the project's rung, the reference reviewer. Findings on record
that the reference reviewer does not raise again may rest on code or rules that have changed
since; no rung is held to them. It then replays the rounds where the reference raised any, on each
rung below, highest first, down to the rung the lowest R + C picks, and stops at the first rung
that misses one. For each replay, claude on the project's rung matches the report to the findings
on record, and counts a finding only where the report raises it as 🔴 or 🟡. The tool prints each
finding a rung missed and the lowest rung that, like each rung above it, missed none. That rung is
the evidence for running the stability reviewer on the worker's rung whenever the worker runs at
or above it; docs/decisions.md → `routing: the stability reviewer stays on the project's rung`
records the run.
Like the loop, it tries claude three times running before it counts a reply as failed. A failed
review or match finds nothing. Exit 1, saying why, on bad arguments, an unreadable archive or
models file, no archived round to replay, or a usage limit.
"""
import dataclasses
import json
import os
import pathlib
import shutil
import sys
import tempfile
import typing
from transcript_archive.location import archive_root
from stability_replay.rounds import authored_by, report, reviewed_path, Round, round_from, sample, stamp_seconds
from project_key import checkout_of
from claude_code.reply import claude_reply, Limited
from rung_replay.command import down_from, failed, limited_exit, project_rung_or_exit, projects_on
from loop_claude.effects import Effects
from failure import Failure
from rung_replay.search import first_reply, replayed
from claude_code.transcript import first_timestamped_line, read_transcript
from claude_code.entries import parse_entry, structured_output
from model_routing.ladder import Rung
TOOL = "stability-replay"
LATEST = 10
REVIEWER = "stability-reviewer"
MATCH_SCHEMA = {"type": "object", "properties": {"found": {"type": "array", "items": {"type": "integer"}}}, "required": ["found"], "additionalProperties": False}
@dataclasses.dataclass(frozen=True)
class Code:
"""Where a round's reviewer read the code, the commit it read, the default branch, and where that branch stood as the chunk began."""
path: str
reviewed: str
default: str
base: str
def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, effects: Effects) -> int:
anchor = project_rung_or_exit(TOOL, argv, __doc__)
if isinstance(anchor, int):
return anchor
rounds = archived_rounds(anchor.model, archive_root(home, env), home)
if isinstance(rounds, Failure):
return failed(TOOL, rounds.reason)
codes = {}
for found in rounds:
if isinstance(code := code_of(found, effects), Failure):
print(f"{found.root} {found.at} is left out: {code.reason}", file=sys.stderr, flush=True)
else:
codes[found] = code
if not codes:
return failed(TOOL, f"no archived stability round of projects on {anchor.model} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find")
chosen = sample(list(codes), LATEST)
# The project's rung is the reference, so the candidates start below it.
replay = replayed(chosen, anchor, down_from(anchor)[1:], lambda found, rung: findings_found(found, codes[found], rung, anchor, effects), lambda _, found: sorted(found), "stability rounds")
if isinstance(replay, Limited):
return limited_exit(TOOL, replay)
print(report(anchor.model, anchor, chosen, replay))
return 0
def archived_rounds(model: str, root: pathlib.Path, home: pathlib.Path) -> list[Round] | Failure:
projects = projects_on(model, home)
if isinstance(projects, Failure):
return projects
found = []
try:
for key in projects:
for chunk in sorted(path for path in (root / key).glob("*") if path.is_dir()):
found += chunk_rounds(checkout_of(key, home), chunk)
except (OSError, UnicodeDecodeError, ValueError) as failure:
return Failure(f"could not read the archive {root}: {failure}")
return found
def chunk_rounds(checkout: str, chunk: pathlib.Path) -> list[Round]:
copies = sorted(chunk.glob("*.jsonl"))
began = parse_entry(first_timestamped_line(copies[0].read_bytes())) if copies else None
if not began or not isinstance(began.get("timestamp"), str):
return []
found = []
for meta in sorted(chunk.glob("session/*/subagents/*.meta.json")):
described = json.loads(meta.read_text(encoding="utf-8"))
if not isinstance(described, dict) or described.get("agentType") != REVIEWER:
continue
entries = read_transcript(str(meta.with_name(meta.name.removesuffix(".meta.json") + ".jsonl")))
if not isinstance(entries, Failure) and (one := round_from(checkout, stamp_seconds(began["timestamp"]), entries)):
found.append(one)
return found
def code_of(found: Round, effects: Effects) -> Code | Failure:
"""Where, and at which commits, found's reviewer read the code, as the archive lets it be found again: the prompt's removed worktree, and the commits authored by then.
Recording the reviewed commit at review time would need no such reading, but then no round archived before that change could be replayed. A wrong guess costs no rung: the reference reviewer, replayed on that code, finds fewer findings on record, and no rung is held to those.
"""
if not os.path.isdir(found.root):
return Failure("the project has no checkout")
path = reviewed_path(found.prompt, found.root, os.path.exists)
if path is None:
return Failure("the prompt names neither one removed worktree nor the checkout")
default = git(effects, found.root, "symbolic-ref", "--short", "refs/remotes/origin/HEAD")
if isinstance(default, Failure):
return default
default = default.strip().removeprefix("origin/")
log = git(effects, found.root, "log", "--format=%H %at", f"refs/remotes/origin/{default}")
main_line = git(effects, found.root, "log", "--first-parent", "--format=%H %at", f"refs/remotes/origin/{default}")
if isinstance(log, Failure):
return log
if isinstance(main_line, Failure):
return main_line
reviewed, base = authored_by(commit_log(log), found.seconds), authored_by(commit_log(main_line), found.began)
if reviewed is None or base is None:
return Failure("no commit was authored by then")
return Code(path, reviewed, default, base)
def commit_log(listed: str) -> list[tuple[str, float]]:
return [(commit, float(authored)) for commit, authored in (line.split() for line in listed.splitlines())]
def findings_found(found: Round, code: Code, rung: Rung, anchor: Rung, effects: Effects) -> set[int] | Limited:
"""The indexes of the round's findings that its replay on rung reports, as the reviewer on anchor matches them."""
reviewed: str | Failure | Limited = first_reply(lambda: review(found, code, rung, effects))
if isinstance(reviewed, Limited):
return reviewed
matched: set[int] | Failure | Limited = reviewed if isinstance(reviewed, Failure) else first_reply(lambda: match(found.findings, reviewed, found.root, anchor, effects))
if isinstance(matched, Failure):
print(f" {found.root} {found.at} on {rung} found nothing: {matched.reason}", file=sys.stderr, flush=True)
return set()
return matched
# The shell makes and removes the clone itself, and is tested on a temporary home as judge-replay's is; rounds.py's core takes the filesystem as an argument.
def review(found: Round, code: Code, rung: Rung, effects: Effects) -> str | Failure | Limited:
clone = tempfile.mkdtemp(prefix="stability-replay-")
try:
for command in (
("clone", "--quiet", "--shared", "--no-checkout", found.root, clone),
("-C", clone, "update-ref", f"refs/remotes/origin/{code.default}", code.base),
("-C", clone, "update-ref", f"refs/heads/{code.default}", code.base),
("-C", clone, "checkout", "--quiet", *(("-B", found.branch) if found.branch and found.branch != code.default else ("--detach",)), code.reviewed),
):
if isinstance(ran := git(effects, found.root, *command), Failure):
return ran
args = ["--agent", REVIEWER, *rung.flags()]
reply = claude_reply(effects.claude, args, clone, found.prompt.replace(code.path, clone), "claude")
if isinstance(reply, Failure | Limited):
return reply
return reply["result"] if isinstance(reply.get("result"), str) else Failure("the reviewer replied with no report")
finally:
shutil.rmtree(clone, ignore_errors=True)
def match(findings: tuple[str, ...], replay_report: str, root: str, rung: Rung, effects: Effects) -> set[int] | Failure | Limited:
numbered = "\n\n".join(f"{n}. {finding}" for n, finding in enumerate(findings, 1))
prompt = (
"A code reviewer reported the numbered findings below. Another reviewer then reviewed the same change; its report follows them. "
"List the number of each finding that the other report also raises as 🔴 or 🟡: the same defect, at the same place or from the same cause, however it is worded. "
"A finding it raises only as 🟢 or ❓, or not at all, is not listed.\n\n"
f"The findings:\n{numbered}\n\nThe other report:\n{replay_report}"
)
args = ["--json-schema", json.dumps(MATCH_SCHEMA), *rung.flags(), "--tools", ""]
reply = claude_reply(effects.claude, args, root, prompt, "claude")
if isinstance(reply, Failure | Limited):
return reply
output = structured_output(reply)
numbers = output.get("found") if isinstance(output, dict) else None
if not isinstance(numbers, list) or not all(isinstance(n, int) for n in numbers):
return Failure(f"the match replied: {json.dumps(reply)}")
return {n - 1 for n in numbers if 1 <= n <= len(findings)}
def git(effects: Effects, cwd: str, *args: str) -> str | Failure:
try:
ran = effects.git(list(args), cwd)
except OSError as failure:
return Failure(f"git {' '.join(args)} could not start in {cwd}: {failure.strerror}")
return ran.stdout if ran.returncode == 0 else Failure(f"git {' '.join(args)}: {ran.stderr.strip()}")
+110
View File
@@ -0,0 +1,110 @@
"""Which archived stability rounds to replay, the code each reviewed, and the report: docs/decisions.md → `routing: the stability reviewer stays on the project's rung`."""
import dataclasses
import datetime
import pathlib
import re
import typing
from claude_code.entries import assistant_parts, content, part_text, parts
from rung_replay.search import Replay
from model_routing.ladder import Rung
# A line that opens with a 🔴 or 🟡 mark, after any list, table, heading or bold markup, as review-findings.md has reviewers report them.
FINDING = re.compile(r"^[\s#>*+|-]*(?:\d+[.)]\s*)?[*_]*\s*(?:🔴|🟡)")
# An absolute path in a prompt, up to the quote, bracket or punctuation that ends it.
PATH = re.compile(r"/[^\s`'\"()<>\[\],;]+")
LINE_OR_STOP = re.compile(r"(?::\d+(?:-\d+)?)?[.:]?$")
@dataclasses.dataclass(frozen=True)
class Round:
"""One archived stability review of the checkout at root, at the UTC stamp at, in a chunk that began at began: its prompt, and the 🔴 and 🟡 findings its report holds, each with its lines up to the next blank line or finding."""
root: str
branch: str
at: str
began: float
prompt: str
findings: tuple[str, ...]
@property
def seconds(self) -> float:
return stamp_seconds(self.at)
def round_from(root: str, began: float, entries: list[dict]) -> Round | None:
"""The round a stability reviewer's transcript holds: its first entry's prompt, and its last text. None without a 🔴 or 🟡 finding to hold a rung to."""
reports = [text for found in entries for part in assistant_parts(found) if (text := part_text(part))]
first = entries[0] if entries else {}
if not stamped(first) or not reports or not (findings := recorded_findings(reports[-1])):
return None
prompt = content(first)
prompt = prompt if isinstance(prompt, str) else "".join(part_text(part) or "" for part in parts(first))
return Round(root, first.get("gitBranch", ""), first["timestamp"], began, prompt, tuple(findings))
def stamped(entry: dict) -> bool:
try:
return isinstance(entry.get("timestamp"), str) and bool(stamp_seconds(entry["timestamp"]))
except ValueError:
return False
def stamp_seconds(stamp: str) -> float:
return datetime.datetime.fromisoformat(stamp.replace("Z", "+00:00")).timestamp()
def recorded_findings(report_text: str) -> list[str]:
findings: list[list[str]] = []
open_finding = False
for line in report_text.splitlines():
if FINDING.match(line):
findings.append([line])
open_finding = True
elif open_finding and line.strip():
findings[-1].append(line)
else:
open_finding = False
return ["\n".join(lines) for lines in findings]
def reviewed_path(prompt: str, root: str, exists: typing.Callable[[str], bool]) -> str | None:
"""Where the reviewer read the code: the one gone directory the prompt names beside the checkout or inside it, as a removed worktree, else the checkout when the prompt names it; None when it names neither, or two gone ones."""
checkout = pathlib.PurePath(root)
# A path that ends a sentence or names a line, such as /code/p/a.py:3, loses that ending.
named = [LINE_OR_STOP.sub("", path) for path in PATH.findall(prompt)]
gone = set()
for path in map(pathlib.PurePath, named):
# The path's directories from the top down, without the filesystem root; the first one gone is what a removed worktree took with it.
down_the_path = [*reversed(path.parents), path][1:]
first_gone = next((prefix for prefix in down_the_path if not exists(str(prefix))), None)
if first_gone and (first_gone.parent == checkout.parent or checkout in first_gone.parents):
gone.add(str(first_gone))
if len(gone) == 1:
return gone.pop()
return root if not gone and root in named else None
def authored_by(log: list[tuple[str, float]], seconds: float) -> str | None:
"""The newest commit in log, which lists newest first, authored at or before seconds. A rebase keeps author times, so a merged branch's commits still say what it held."""
return next((commit for commit, authored in log if authored <= seconds), None)
def sample(rounds: list[Round], latest: int) -> list[Round]:
"""The latest rounds, in the order they ran."""
return sorted(rounds, key=lambda found: (found.seconds, found.root))[-latest:]
def report(model: str, reference: Rung, rounds: list[Round], replay: Replay[Round, int, set[int]]) -> str:
total = sum(len(found_round.findings) for found_round in rounds)
lines = [
f"Replayed {len(rounds)} stability rounds of {model} projects, holding {total} 🔴 and 🟡 findings.",
f"The reference reviewer, {reference}, found {sum(len(found) for found in replay.on_reference)} of them.",
]
for tried in replay.tried:
lines += [f"{tried.rung} missed {len(tried.missed)}{':' if tried.missed else '.'}", *(f" {miss.case.root} {miss.case.at}: {miss.case.findings[miss.item].splitlines()[0]}" for miss in tried.missed)]
if not replay.tried:
lines.append("No rung tried: the reference reviewer found no finding to hold one to.")
else:
lowest = replay.lowest or "none below the project's rung"
lines.append(f"Lowest rung that finds every finding the reference reviewer finds: {lowest}")
return "\n".join(lines)
@@ -0,0 +1,321 @@
import contextlib
import dataclasses
import io
import json
import os
import pathlib
import re
import shutil
import subprocess
import sys
import tempfile
import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from stability_replay.rounds import authored_by, recorded_findings, report, reviewed_path, Round, round_from, sample
from loop_claude.effects import Effects, system_effects
from failure import Failure
from stability_replay import main
from rung_replay.search import Miss, Replay, Tried
from model_routing.ladder import Rung
TOOL = pathlib.Path(__file__).resolve().parents[1] / "stability-replay"
REFERENCE = Rung("claude-opus-5-5", "medium")
MIDDLE = Rung("claude-sonnet-5-5", "high")
CHEAP = Rung("claude-haiku-5-5", "max")
def entry(kind, text, **fields):
return {"type": kind, "message": {"content": [{"type": "text", "text": text}]}, **fields}
def a_round(at, findings=("🔴 a",)):
return Round("/code/p", "x", f"2026-10-14T10:00:0{at}Z", 0, "Review it.", tuple(findings))
class RecordedFindings(unittest.TestCase):
def test_takes_each_red_and_yellow_finding_with_the_lines_up_to_the_next_blank_line_or_finding(self):
report_text = "BLOCK\n\n1 🔴, 2 🟡 in all.\n\n1. 🔴 **Loop** at a.py:3\nnever ends.\n- 🟡 b.py:1 slow\n| 🟡 | c.py | odd |\n\n🟢 nit\n### 🔴 Heading\n**🟡 Bold**"
self.assertEqual(recorded_findings(report_text), ["1. 🔴 **Loop** at a.py:3\nnever ends.", "- 🟡 b.py:1 slow", "| 🟡 | c.py | odd |", "### 🔴 Heading", "**🟡 Bold**"])
def test_skips_a_count_or_verdict_line_that_names_the_marks(self):
self.assertEqual(recorded_findings("APPROVE\n\nNo 🔴 or 🟡 findings.\n0 🔴, 0 🟡"), [])
class RoundFrom(unittest.TestCase):
def test_reads_the_prompt_the_last_report_and_when_the_reviewer_started(self):
entries = [
entry("user", "Review branch x in /code/p-x.", timestamp="2026-10-14T10:00:00.500Z", cwd="/code/p-x", gitBranch="x"),
entry("assistant", "Reading."),
entry("assistant", "BLOCK\n\n🔴 a.py:1 breaks"),
]
self.assertEqual(round_from("/code/p", 5.0, entries), Round("/code/p", "x", "2026-10-14T10:00:00.500Z", 5.0, "Review branch x in /code/p-x.", ("🔴 a.py:1 breaks",)))
self.assertEqual(round_from("/code/p", 5.0, entries).seconds, 1791972000.5)
def test_reads_a_prompt_given_as_a_string(self):
start = {"type": "user", "message": {"content": "Review branch x in /code/p-x."}, "timestamp": "2026-10-14T10:00:00Z", "gitBranch": "x"}
self.assertEqual(round_from("/code/p", 5.0, [start, entry("assistant", "🔴 a")]).prompt, "Review branch x in /code/p-x.")
def test_skips_a_round_with_no_red_or_yellow_finding_or_no_report(self):
start = entry("user", "Review.", timestamp="2026-10-14T10:00:00Z", cwd="/code/p", gitBranch="x")
self.assertIsNone(round_from("/code/p", 5.0, [start, entry("assistant", "APPROVE\n\n🟢 nit")]))
self.assertIsNone(round_from("/code/p", 5.0, [start]))
self.assertIsNone(round_from("/code/p", 5.0, []))
self.assertIsNone(round_from("/code/p", 5.0, [{**start, "timestamp": "yesterday"}, entry("assistant", "🔴 a")]))
class ReviewedPath(unittest.TestCase):
def test_names_the_worktree_the_prompt_names_that_is_gone(self):
exists = {"/code", "/code/p", "/code/p/worktrees", "/tmp"}.__contains__
self.assertEqual(reviewed_path("Review /code/p-x/src/a.py:3-5; the diff is at /tmp/s/x.diff; /code/p is older.", "/code/p", exists), "/code/p-x")
self.assertEqual(reviewed_path("Review `/code/p/worktrees/x`.", "/code/p", exists), "/code/p/worktrees/x")
def test_names_the_checkout_when_the_prompt_names_no_gone_worktree_and_none_when_it_names_neither_or_two(self):
exists = {"/code", "/code/p"}.__contains__
self.assertEqual(reviewed_path("Review /code/p.", "/code/p", exists), "/code/p")
self.assertIsNone(reviewed_path("Review the branch.", "/code/p", exists))
self.assertIsNone(reviewed_path("Review /code/p-x and /code/p-43.", "/code/p", exists))
class AuthoredBy(unittest.TestCase):
def test_takes_the_newest_commit_authored_by_then(self):
log = [("c3", 30), ("c2", 20), ("c1", 10)]
self.assertEqual(authored_by(log, 25), "c2")
self.assertEqual(authored_by(log, 30), "c3")
self.assertIsNone(authored_by(log, 5))
class Sample(unittest.TestCase):
def test_keeps_the_latest_rounds_in_the_order_they_ran(self):
rounds = [a_round(2), a_round(1), a_round(3)]
self.assertEqual(sample(rounds, 2), [rounds[0], rounds[2]])
class Report(unittest.TestCase):
def test_counts_the_findings_and_names_each_miss_and_the_lowest_rung_that_missed_none(self):
rounds = [a_round(1, ("🔴 a\nmore", "🟡 b")), a_round(2)]
lines = report("claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, set()], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(rounds[0], 0, {1})])], MIDDLE)).splitlines()
self.assertEqual(lines, [
"Replayed 2 stability rounds of claude-opus-5-5 projects, holding 3 🔴 and 🟡 findings.",
"The reference reviewer, claude-opus-5-5 at medium effort, found 2 of them.",
"claude-sonnet-5-5 at high effort missed 0.",
"claude-haiku-5-5 at max effort missed 1:",
" /code/p 2026-10-14T10:00:01Z: 🔴 a",
"Lowest rung that finds every finding the reference reviewer finds: claude-sonnet-5-5 at high effort",
])
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, {0}], [Tried(MIDDLE, [Miss(rounds[1], 0, set())])], None)).splitlines()[-1], "Lowest rung that finds every finding the reference reviewer finds: none below the project's rung")
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay([set(), set()], [], None)).splitlines()[-1], "No rung tried: the reference reviewer found no finding to hold one to.")
class Match(unittest.TestCase):
def run_match(self, stdout):
self.asked = []
def claude(args, cwd, prompt):
self.asked.append((args, cwd, prompt))
return subprocess.CompletedProcess(args, 0, stdout, "")
return main.match(("🔴 a", "🟡 b"), "the other report", "/code/p", REFERENCE, dataclasses.replace(system_effects(), claude=claude))
def test_asks_for_the_numbered_findings_the_other_report_raises_as_json_with_no_tools(self):
self.run_match("{}")
[(args, cwd, prompt)] = self.asked
self.assertEqual(cwd, "/code/p")
self.assertIn("1. 🔴 a\n\n2. 🟡 b\n\nThe other report:\nthe other report", prompt)
self.assertEqual(json.loads(args[args.index("--json-schema") + 1]), main.MATCH_SCHEMA)
self.assertEqual(args[args.index("--tools") + 1], "")
self.assertEqual(args[args.index("--model"):args.index("--model") + 4], REFERENCE.flags())
def test_takes_the_numbers_in_range_and_fails_on_a_reply_of_another_shape(self):
self.assertEqual(self.run_match(json.dumps({"type": "result", "structured_output": {"found": [2, 0, 3, 1]}})), {0, 1})
for output in ({"found": ["1"]}, {"found": 1}, [1]):
with self.subTest(output):
self.assertIsInstance(self.run_match(json.dumps({"type": "result", "structured_output": output})), Failure)
class Git(unittest.TestCase):
def test_says_why_git_cannot_start_in_a_directory_that_is_gone(self):
def gone(args, cwd):
raise FileNotFoundError(2, "No such file or directory")
ran = main.git(dataclasses.replace(system_effects(), git=gone), "/code/gone", "log")
self.assertEqual(ran, Failure("git log could not start in /code/gone: No such file or directory"))
def unused(*_):
raise AssertionError("not called")
def run_git(cwd, *args, at=None):
"""at, when given, is the commit's author and committer time, so git log lists the commits in a fixed order."""
env = {**os.environ, "GIT_AUTHOR_DATE": at, "GIT_COMMITTER_DATE": at} if at else None
subprocess.run(["git", "-c", "user.name=t", "-c", "user.email=t@t", *args], cwd=cwd, check=True, capture_output=True, env=env)
class Main(unittest.TestCase):
"""The tool on a temporary home: a chunk that began after the checkout's first commit, and whose stability reviewer reviewed branch x at its second, from a worktree since removed; a prose reviewer's files sort before the stability reviewer's."""
def setUp(self):
self.home = pathlib.Path(self.enterContext(tempfile.TemporaryDirectory()))
self.archive = self.home / ".claude" / "loop-claude-transcripts"
(self.home / ".claude").mkdir()
(self.home / ".claude" / "loop-claude-models.json").write_text(json.dumps({"proj": "claude-opus-5-5", "other": "claude-sonnet-5-5"}))
origin, self.checkout = self.home / "origin.git", self.home / "code" / "proj"
run_git(self.home, "init", "-q", "--bare", "-b", "main", str(origin))
run_git(self.home, "clone", "-q", str(origin), str(self.checkout))
for n, stamp in enumerate(("2026-10-14T09:00:00Z", "2026-10-14T09:30:00Z", "2026-10-14T11:00:00Z")):
(self.checkout / "a.txt").write_text(f"{n}\n")
run_git(self.checkout, "add", "a.txt")
run_git(self.checkout, "commit", "-qm", f"c{n}", at=stamp)
run_git(self.checkout, "push", "-q", "origin", "main")
run_git(self.checkout, "remote", "set-head", "origin", "main")
for key in ("proj", "other"):
self.write_chunk(key, "c1", "2026-10-14T10:00:00Z")
self.reviews = []
def write_chunk(self, key, chunk, at, began="2026-10-14T09:15:00Z"):
"""A chunk that began at began, whose stability round began at at."""
subagents = self.archive / key / chunk / "session" / "s" / "subagents"
subagents.mkdir(parents=True, exist_ok=True)
(subagents.parents[2] / "t0.jsonl").write_text(json.dumps({"type": "summary"}) + "\n" + json.dumps(entry("user", "Do the next chunk.", timestamp=began)) + "\n")
(subagents / "agent-a.meta.json").write_text(json.dumps({"agentType": "prose-reviewer"}))
(subagents / "agent-b.meta.json").write_text(json.dumps({"agentType": "stability-reviewer"}))
gone = self.home / "code" / "proj-x"
lines = [entry("user", f"Review branch x in {gone}.", timestamp=at, cwd=str(gone), gitBranch="x"), entry("assistant", "BLOCK\n\n🔴 a.txt:1 wrong\n\n🟡 slow")]
for name in ("agent-a.jsonl", "agent-b.jsonl"):
(subagents / name).write_text("".join(json.dumps(line) + "\n" for line in lines))
def rewrite_round(self, change):
path = self.archive / "proj" / "c1" / "session" / "s" / "subagents" / "agent-b.jsonl"
lines = [json.loads(line) for line in path.read_text().splitlines()]
change(lines[0])
path.write_text("".join(json.dumps(line) + "\n" for line in lines))
def set_branch(self, branch):
self.rewrite_round(lambda first: first.update(gitBranch=branch))
def set_prompt(self, prompt):
self.rewrite_round(lambda first: first["message"]["content"][0].update(text=prompt))
def run_main(self, argv, found, review=lambda rung: {"type": "result", "result": f"review on {rung.model} {rung.effort}"}):
"""found(rung) is the indexes of the recorded findings the reviewer on rung finds; review(rung), that reviewer's reply."""
def claude(args, cwd, prompt):
rung = Rung(args[args.index("--model") + 1], args[args.index("--effort") + 1])
if "--agent" in args:
shown = [subprocess.run(["git", *command], cwd=cwd, capture_output=True, text=True).stdout for command in (["show", "HEAD:a.txt"], ["branch", "--show-current"], ["show", "origin/main:a.txt"])]
self.reviews.append((rung, *shown, prompt, cwd))
result = review(rung)
else:
reviewed = Rung(*re.search(r"review on (\S+) (\S+)", prompt).groups())
result = found(reviewed)
result = result if "api_error_status" in result else {"type": "result", "structured_output": {"found": [n + 1 for n in result["found"]]}}
return subprocess.CompletedProcess(args, 0, json.dumps(result), "")
effects = Effects(claude=claude, claude_on_terminal=unused, execv=unused, git=system_effects().git, input=unused, local_time=unused, now=unused, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused)
out, err = io.StringIO(), io.StringIO()
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
code = main.main(argv, {}, self.home, effects)
return code, out.getvalue(), err.getvalue()
def test_replays_the_models_rounds_on_the_commit_reviewed_from_the_project_rung_down(self):
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0] if rung.model == "claude-haiku-5-5" else [0, 1]})
self.assertEqual(code, 0, err)
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE, MIDDLE, CHEAP])
_, reviewed, branch, base, prompt, cwd = self.reviews[0]
# The default branch stands where it stood when the chunk began, so the prompt's diff against it shows the branch's change.
self.assertEqual((reviewed, branch, base), ("1\n", "x\n", "0\n"))
self.assertEqual(prompt, f"Review branch x in {cwd}.")
self.assertFalse(pathlib.Path(cwd).exists())
self.assertIn("The reference reviewer, claude-opus-5-5 at medium effort, found 2 of them.", out)
self.assertIn(f" {self.checkout} 2026-10-14T10:00:00Z: 🟡 slow", out)
self.assertTrue(out.endswith("finds: claude-sonnet-5-5 at high effort\n"))
def test_replays_the_rounds_of_every_chunk(self):
self.write_chunk("proj", "c2", "2026-10-14T10:30:00Z")
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0, 1]})
self.assertEqual(code, 0, err)
self.assertTrue(out.startswith("Replayed 2 stability rounds"), out)
def test_checks_out_the_commit_detached_when_the_reviewer_started_on_the_default_branch(self):
self.set_branch("main")
self.assertEqual(self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0]})[0], 0)
self.assertEqual(self.reviews[0][1:3], ("1\n", ""))
def test_sets_the_default_branch_to_its_own_line_and_not_to_a_branch_merged_after_the_chunk_began(self):
# Branch side's commit was authored before the chunk began, but the default branch held it only once merged, after.
run_git(self.checkout, "reset", "-q", "--hard", "HEAD~2")
run_git(self.checkout, "checkout", "-qb", "side")
(self.checkout / "a.txt").write_text("s\n")
run_git(self.checkout, "commit", "-qam", "s", at="2026-10-14T09:10:00Z")
run_git(self.checkout, "checkout", "-q", "main")
run_git(self.checkout, "merge", "-q", "--no-ff", "-m", "m", "side", at="2026-10-14T09:40:00Z")
run_git(self.checkout, "push", "-qf", "origin", "main")
self.assertEqual(self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0]})[0], 0)
self.assertEqual(self.reviews[0][1:4], ("s\n", "x\n", "0\n"))
def test_finds_nothing_in_a_round_whose_review_has_no_report(self):
code, out, err = self.run_main(["claude-opus-5-5"], unused, review=lambda rung: {"type": "result"})
self.assertEqual(code, 0, err)
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE] * 3)
self.assertIn(f" {self.checkout} 2026-10-14T10:00:00Z on {REFERENCE} found nothing: the reviewer replied with no report", err)
self.assertTrue(out.endswith("No rung tried: the reference reviewer found no finding to hold one to.\n"))
def assert_no_round(self):
code, _, err = self.run_main(["claude-opus-5-5"], unused)
self.assertEqual(code, 1)
self.assertEqual(err.splitlines(), [f"stability-replay: no archived stability round of projects on claude-opus-5-5 in {self.archive} holds a 🔴 or 🟡 on code it can find"])
def test_skips_a_chunk_with_no_transcript_copy(self):
(self.archive / "proj" / "c1" / "t0.jsonl").unlink()
self.assert_no_round()
def test_skips_a_chunk_whose_transcript_copy_has_no_timestamp(self):
(self.archive / "proj" / "c1" / "t0.jsonl").write_text(json.dumps({"type": "summary"}) + "\n")
self.assert_no_round()
def assert_left_out(self, reason, root=None):
code, _, err = self.run_main(["claude-opus-5-5"], unused)
self.assertEqual(code, 1)
self.assertIn(f"{root or self.checkout} 2026-10-14T10:00:00Z is left out: {reason}", err)
def test_leaves_out_a_round_whose_prompt_names_no_worktree(self):
self.set_prompt("Review the branch.")
self.assert_left_out("the prompt names neither one removed worktree nor the checkout")
def test_leaves_out_a_round_whose_checkout_names_no_default_branch(self):
run_git(self.checkout, "remote", "set-head", "origin", "-d")
self.assert_left_out("git symbolic-ref --short refs/remotes/origin/HEAD: ")
def test_leaves_out_a_round_whose_checkout_is_gone(self):
shutil.rmtree(self.checkout)
# With no folder under ~/code, the key names /proj.
self.assert_left_out("the project has no checkout", "/proj")
def test_leaves_out_a_round_whose_chunk_began_before_the_checkouts_first_commit(self):
self.write_chunk("proj", "c1", "2026-10-14T10:00:00Z", began="2026-10-14T08:00:00Z")
self.assert_left_out("no commit was authored by then")
def test_exits_1_on_a_usage_limit_bad_arguments_or_no_round_to_replay(self):
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung: {"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
self.assertEqual(code, 1)
self.assertIn("usage limit: You've hit your limit", err)
for argv in ([], ["off-ladder"], ["claude-opus-5-5", "x"]):
self.assertEqual(self.run_main(argv, unused)[0], 1)
code, _, err = self.run_main(["claude-haiku-5-5"], unused)
self.assertEqual(code, 1)
self.assertIn("no archived stability round of projects on claude-haiku-5-5", err)
def test_prints_the_help(self):
code, out, _ = self.run_main(["-h"], unused)
self.assertEqual((code, out.splitlines()[2]), (0, "usage: stability-replay <model>"))
def test_the_launcher_prints_the_help(self):
ran = subprocess.run([sys.executable, str(TOOL), "--help"], capture_output=True, text=True)
self.assertEqual(ran.returncode, 0)
self.assertIn("usage: stability-replay <model>", ran.stdout)
if __name__ == "__main__":
unittest.main()
+37
View File
@@ -1770,3 +1770,40 @@ Junior A, decided by:
- Self-sufficiency: `bin/routing_figures/measure.py:23` `ESCALATION`: trigger numbers mean something only in `CLAUDE.md:60`'s list, and renumbering it would silently change old figures
- Self-sufficiency, Shape: `CLAUDE.md:60` "Working with the codebase": "trigger" now names an escalation condition as well as the review-pass condition Vocabulary defines
- Self-sufficiency, Locality: `bin/loop_claude/cycle/prompts.py:40` `escalation`: the past-the-cap branch is missing from the docstring, and the off-ladder guard differs from `escalation_rungs`'s
## 2026-10-15T19:39:18Z, PR #276 at f643a14, against eb23048
Ruling: same
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|---|---|---|---|---|---|
| Mid A | same | worse | worse | same | worse |
| Inherited architect | same | same | worse | better | same |
| Maintainability senior | same | same | worse | same | same |
| Junior A | same | same | worse | same | same |
Mid A, decided by:
- Shape, Locality: `bin/rung_replay/search.py:44` `replayed`: generic over `Case`, `Item` and `Reply` with a `reproduced` callback returning a list of items, worked through `held`, `aim` and `got`; for the judge the list is only `[kind]` or `[]`, so the reader decodes an abstraction built for two consumers before following one
- Locality: `bin/judge_replay/replay.py:70` `report`: rebuilds the unreproduced verdicts by calling `reproduced` again on `replay.on_reference`, so it is right only while `main.py` passes that same function to `replayed`
- Shape: `bin/stability_replay/main.py:83` `main`: `found` names a `Round`, a `set[int]` and a list in one file, and one call packs the reference, the candidates, a closure over `codes` and an inline lambda
- Shape: `bin/rung_replay/command.py:14` `project_rung_or_exit`: returns `Rung | int`, the int an exit code, so each caller tests `isinstance(anchor, int)`
Inherited architect, decided by:
- Shape: `bin/rung_replay/search.py:44` `replayed`: generic over its type variables, so a reader works out what `reproduced` returns before the search reads; it was a concrete comparison of kinds
- Shape: `bin/judge_replay/replay.py:65` `reproduced`: returns a list of zero or one kinds where a yes or no would do, to fit the generic shape
- Shape: `bin/stability_replay/main.py:83` `main`: `found` is a `Round` in one lambda and a `set[int]` in the next, and an accumulator list elsewhere in the file
- Self-sufficiency: `bin/rung_replay/search.py:52` `replayed`: the docstring says why the search goes down from the highest rung and stops at the first miss, which the cheapest-first search assumed unsaid
- Self-sufficiency: `bin/stability_replay/main.py:120` `code_of`: says what it guesses and what a wrong guess costs, so no record is needed beside it
Maintainability senior, decided by:
- Shape: `bin/rung_replay/search.py:44` `replayed`: generic as `Replay[Case, Item, Reply]` with a `reproduced` callback returning a list, which judge-replay fits only with a list of one or none and stability-replay with an inline adapter; following one `report` line takes three files and the generics
- Shape: `bin/stability_replay/main.py:66` `main`: one file walks the archive, rebuilds the reviewed commits from two git logs, clones, reviews and matches, with long lines packing the `replayed` call, a nested conditional over `first_reply` and a starred conditional checkout command
Junior A, decided by:
- Shape: `bin/rung_replay/search.py:37` `Replay`: the search is generic over three type parameters with two callbacks and a label, so a reader works out what `Item` and `Reply` are for each caller before the loop makes sense
- Shape: `bin/judge_replay/replay.py:65` `reproduced`: returns `[kind]` or `[]` only to fit the generic shape, and `report` rebuilds the unreproduced list by calling it again
- Shape: `bin/rung_replay/command.py:1` `command`: the name says little, and the module mixes argument parsing, a models-file lookup, rung arithmetic and stderr exits
+10
View File
@@ -229,3 +229,13 @@ On 2026-10-15, `bin/judge-replay claude-opus-5-5` replayed the latest 15 archive
Only Opus 5.5 was replayed, so the judge of a project on another model keeps the effort its settings give until `judge-replay` covers that model (goal 1.1). Every routed chunk on record had an item of R + C 9, so the replay could not show that the project's rung suffices for harder items, and the judge climbs with the routed item as the worker does. For R + C 11 to 14 it still runs at or below xhigh (goal 1.4); from 15, `routing: one ladder` records that lilleman wants the highest quality at any price. An item the worker takes in later lifts the worker, not the judge, until `todo.md` item 136 routes a chunk on the item its worker takes.
Valid while the archived verdicts show what a judge should return, and until a new model joins the ladder: replay then.
## routing: the stability reviewer stays on the project's rung while the worker runs below it, until `stability-replay` shows a lower rung finding every 🔴 and 🟡 the project's rung finds
2026-10-15, lilleman proposed running the stability reviewer on the worker's rung; the replay that gates it is the agent's proposal. Serves goals 1.1 and 1.4: the stability pass is the only one that mutation-tests the changed lines, so a defect it misses may merge, and a cheaper reviewer saves money only where it misses nothing.
`bin/stability-replay <model>` replays archived stability rounds on the code each reviewed, first on the project's rung, then on each rung below it, highest first, and stops at the first that misses a finding the project's rung found again. A rung qualifies only where it finds every such 🔴 and 🟡.
On 2026-10-15, `bin/stability-replay claude-opus-5-5` replayed the latest 10 rounds holding a 🔴 or 🟡, 40 findings in all, 8 of the rounds from this repo. Opus 5.5 at medium, the project's rung, found 29 of them again. Sonnet 5.5 at high missed 13 of those 29, 2 of them 🔴, so no rung below the project's rung qualifies, and Haiku 5.5, below Sonnet on the ladder, was not tried. So the stability reviewer runs where `routing: one ladder` puts every reviewer: on the project's rung while the worker runs below it.
Valid until a new model joins the ladder, or the archive holds rounds a new reviewer prompt produced: replay then.
+17
View File
@@ -384,3 +384,20 @@ Read most (opens and searches):
- `AGENTS.md` 1
- `bin/loop_claude/routing.py` 1
- `bin/model_routing/ladder.py` 1
## Reading cost, chunk d1481c54d520, finished 2026-10-15T17:52:35Z
90662 new input tokens before the first commit.
Read most (opens and searches):
- `todo.md` 6
- `docs/decisions.md` 5
- `skills/larv-review/SKILL.md` 3
- `CLAUDE.md` 2
- `README.md` 2
- `bin/loop_claude/cycle/prompts.py` 2
- `bin/loop_claude/loop/main.py` 2
- `bin/loop_claude/test_loop_claude.py` 2
- `skills/comprehension-panel/SKILL.md` 2
- `skills/comprehension-panel/seat-prompts.md` 2
+3 -12
View File
@@ -26,7 +26,6 @@ Release 0.3.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |
|---|---|---|---|---|---|---|---|---|---|---|
| 145 | 0.1.0 | | **Run the stability reviewer on the worker's rung while the worker runs below the project's rung, once a replay of past stability rounds on that rung finds the 🔴 and 🟡 findings on record.** | 4 | 5 | 6 | 6 | 7 | 1.4 | 20.2 |
| 143 | 0.1.0 | | **Resume a worker one rung up in the next cycle when its run ends asking for a higher rung and the judge finds the escalation trigger it cites in the transcript.** | 5 | 5 | 7 | 6 | 7 | 1.1, 1.4 | 19.6 |
| 136 | 0.1.0 | decision | **Route a chunk on the item its worker says it takes, in place of the next release's first row.** | 2 | 3 | 4 | 4 | 5 | 1.4 | 14.8 |
| 105 | 0.2.0 | decision | **Compare review rounds per pass and reading cost for the five chunks before and after the `worker:` decision moved the principles out of the worker's context, and file an item if either rose.** | 1 | 2 | 3 | 5 | 6 | 1.4 | 20.0 |
@@ -49,7 +48,7 @@ Release 0.3.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
| 133 | 0.3.0 | | **Show, beside `bin/rule_hook/override_log.py`'s `this_chunks`, one example per case it decides: an amended or rebased commit, a renamed branch, and a commit on the default branch.** | 1 | 2 | 2 | 3 | 4 | 1.1 | 12.4 |
| 134 | 0.3.0 | | **Name in `bin/shell_command.py`'s `segments` the shell forms it does not parse, so a caller knows what it misses.** | 1 | 2 | 2 | 3 | 4 | 1.1 | 12.4 |
| 140 | 0.3.0 | | **Split `docs/decisions.md` → `routing: one ladder` into one entry per decision, so a reader finds the rules for the ladder, the worker's rung, the routing figures and delegation by their titles.** | 1 | 2 | 1 | 3 | 4 | 1.1 | 12.4 |
| 114 | 0.3.0 | decision | **Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.** | 3 | 4 | 4 | 3 | 5 | 1.1 | 11.9 |
| 114 | 0.3.0 | decision | **Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost`, `stability_replay` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.** | 3 | 4 | 4 | 3 | 5 | 1.1 | 11.9 |
| 117 | 0.3.0 | | **Give `todo_table.items` one way to say a plan has no items table, so no caller tells it from an empty plan by truthiness.** | 2 | 2 | 3 | 3 | 4 | 1.1 | 11.4 |
| 124 | 0.3.0 | decision | **Group the shared modules at `bin/`'s root into packages named by purpose, so the root holds only launchers and stand-alone tools.** | 3 | 6 | 3 | 3 | 5 | 1.1 | 11.4 |
| 130 | 0.3.0 | | **Move the vote counting out of `bin/larv_review/ruling.py`'s `votes`, so reading a votes file decides nothing.** | 2 | 3 | 3 | 3 | 4 | 1.1 | 11.2 |
@@ -86,14 +85,6 @@ Release 0.3.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
## Details
### 145. Run the stability reviewer on the worker's rung while the worker runs below the project's rung, once a replay of past stability rounds on that rung finds the 🔴 and 🟡 findings on record.
On 2026-10-15 the human proposed running the stability reviewer on the worker's rung; the replay gate is the agent's proposal. Today a worker below the project's rung launches every reviewer on the project's rung. Under this item, the stability reviewer runs on the worker's rung instead. Every other reviewer, the comprehension seats and the security pass's subagent stay on the project's rung, so a change that fires their triggers is still reviewed on that rung.
The stability pass is the only one that mutation-tests the changed lines, so a defect it misses may merge. Before the switch, replay archived stability rounds on each rung below the project's, as `bin/judge-replay` does for the judge. Route the reviewer onto a rung only where it finds every 🔴 and 🟡 on record that a replay on the project's rung also finds. A rung that misses one keeps the reviewer on the project's rung. In the judge's replay on 2026-10-15, Haiku 5.5 at xhigh missed 7 of 31 verdicts, Sonnet 5.5 at high missed 3 interrupts, and Opus 5.5 at medium missed none.
This revises `docs/decisions.md` → `routing: one ladder`.
### 143. Resume a worker one rung up in the next cycle when its run ends asking for a higher rung and the judge finds the escalation trigger it cites in the transcript.
The human asked for this on 2026-10-15. A worker may end its run asking for a higher rung, citing an escalation trigger of `CLAUDE.md` → "Working with the codebase". The judge looks for that trigger in the transcript. Where it finds it, the next cycle resumes the session one rung up; otherwise the rung holds. The worker never steps down within a chunk, and goes at most two rungs above the rung it started the chunk on unless the human agrees.
@@ -190,11 +181,11 @@ Junior A and Mid A of PR #257's scoring run named `bin/shell_command.py:31` `seg
The entry holds 7 paragraphs under one title of about 70 words chaining three decisions: the ladder and bands, the worker below the project's rung with its reviewers, the routing figures and calls, delegation, and when the worker changes rung. A reader after one rule reads the whole entry. The prose pass of the chunk that split out the judge's rung asked for the rest. Once split, the delegation entry's title names the floor that `todo.md: only a session or subagent on the project's rung or above scores or reshapes the plan…` puts under a subagent that files or scores items.
### 114. Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.
### 114. Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost`, `stability_replay` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.
Shape is the lowest dimension, so this item goes before 0.3.0's other items, per `docs/decisions.md` → `larv-review: a project lifting itself to the comprehension floor starts at the lowest dimension`.
The comprehension seats on PR #254 found five git runners with different contracts: `git_repo.git`, `rule_hook/git.py`'s guarded one, `metric_bench/main.py`'s, `reading_cost/main.py`'s `located` and `loop_claude/effects.py`'s. The inherited architect of PR #257's scoring run named `bin/git_repo.py:8` `git` again. Make `git_repo.git` easier to read.
The comprehension seats on PR #254 found five git runners with different contracts: `git_repo.git`, `rule_hook/git.py`'s guarded one, `metric_bench/main.py`'s, `reading_cost/main.py`'s `located` and `loop_claude/effects.py`'s. PR #276 added a sixth, `stability_replay/main.py`'s `git`. The inherited architect of PR #257's scoring run named `bin/git_repo.py:8` `git` again. Make `git_repo.git` easier to read.
### 117. Give `todo_table.items` one way to say a plan has no items table, so no caller tells it from an empty plan by truthiness.