Answer architecture round 1: 1 🔴, 2 🟡; share one rung search between both replays, searching down from the project's rung, name the sixth git runner in item 114, decline recording the reviewed commit

This commit is contained in:
2026-10-15 20:52:50 +02:00
parent ae7161d2f4
commit 4f722d94f4
21 changed files with 398 additions and 321 deletions
+2
View File
@@ -53,6 +53,8 @@
!bin/routing_figures/*.py
!bin/rule_hook/
!bin/rule_hook/*.py
!bin/rung_replay/
!bin/rung_replay/*.py
!bin/score-outcomes
!bin/score_outcomes/
!bin/score_outcomes/*.py
+2 -2
View File
@@ -2,11 +2,11 @@
## Unreleased
- The new `bin/stability-replay <model>` replays archived stability reviews of the projects run on the model, each on the code it reviewed, first on the project's rung and then on each rung below, and prints the lowest rung that finds every 🔴 and 🟡 finding the project's rung finds. Its first run, on Opus 5.5 projects, found none: Sonnet 5.5 at high missed 13 of 29. So the stability reviewer stays on the project's rung while the worker runs below it.
- The new `bin/stability-replay <model>` replays archived stability reviews of the projects run on the model, each on the code it reviewed, first on the project's rung and then on each rung below, and prints the lowest rung that, like each rung above it, finds every 🔴 and 🟡 finding the project's rung finds. It shares its search with `judge-replay`, which now also tries rungs from the project's rung down. Its first run, on Opus 5.5 projects, found none: Sonnet 5.5 at high missed 13 of 29. So the stability reviewer stays on the project's rung while the worker runs below it.
- A `loop-claude` worker hands a task that proves too hard for its rung to a subagent one rung up, edits included, when a trigger fires: a re-score of the item on the project's rung or above picks a higher rung, the same failing test or finding survives two fix attempts, or a review pass reports a 🔴 in round 2 or later. Each further trigger climbs one more rung, to at most two rungs above the rung the chunk's `todo.md` item picks, unless you agree. `CLAUDE.md` states the rule, and `loop-claude`'s prompt names the rungs a trigger opens. The worker opens the subagent's prompt with `Escalation trigger <n>:`, and `routing-figures.md` counts the escalations by trigger. Expect a chunk that gets stuck to spend more on the stuck part, and fewer unfinished verdicts.
- A reviewer that rules a chunk's work below standard, with systems-architect's BLOCK or a finding against larv-review's `standards/quality.md`, may ask to rerun one rung up, as may a comprehension seat whose vote in a paired ruling `record-ruling` counts as worse. The worker relaunches it once, on the rung `loop-claude`'s prompt names. A relaunched reviewer gets the report that asked and confirms or dismisses each of its findings. The relaunch's report stands in place of the one that asked, in the same round, so it adds no round. The ruling that stands is a fourth escalation trigger in `CLAUDE.md`. Expect a chunk whose work falls below standard to spend more on its review and its fix.
- A session or subagent below the project's rung, the cheapest rung of the project's model, no longer scores or reshapes `todo.md`: it drafts each new item and each change to a filed item, and before its next commit one subagent on the project's rung or above applies and scores the drafts and sets their release. It may still remove finished items. `rules/todo-items.md` states the rule. `CLAUDE.md` has a session launch a subagent that files or scores items on the rung the new `bin/subagent-rung --plan <model> <R+C>` prints, never below the project's rung, and `loop-claude`'s prompt to a worker below that rung names the command. Expect one extra subagent in a chunk that files items while its worker runs below the project's rung.
- `loop-claude` runs the judge of an Opus 5.5 project on the rung the chunk's `todo.md` item picks, never below Opus 5.5 at medium effort, the project's rung. Before, the judge ran at the effort `settings.json` sets, xhigh. A replay of 39 archived verdicts found every cheaper rung missing verdicts. The judge keeps its rung when the worker steps up or takes in another item, and runs on the project's rung when no item routed the chunk. A project on another model keeps its judge at its settings' effort. Each verdict in the archive now names the judge's rung on a `Judge:` line. The new `bin/judge-replay <model>` replays the judge on archived verdicts and prints the cheapest rung that returns them. Expect the judge to cost less on items of R + C 12 or less, and more on items of R + C 15 or more.
- `loop-claude` runs the judge of an Opus 5.5 project on the rung the chunk's `todo.md` item picks, never below Opus 5.5 at medium effort, the project's rung. Before, the judge ran at the effort `settings.json` sets, xhigh. A replay of 39 archived verdicts found every cheaper rung missing verdicts. The judge keeps its rung when the worker steps up or takes in another item, and runs on the project's rung when no item routed the chunk. A project on another model keeps its judge at its settings' effort. Each verdict in the archive now names the judge's rung on a `Judge:` line. The new `bin/judge-replay <model>` replays the judge on archived verdicts and prints the lowest rung that, like each rung above it, returns them. Expect the judge to cost less on items of R + C 12 or less, and more on items of R + C 15 or more.
- Every 20 cycles across all projects, `loop-claude` measures every finished chunk in its archive, with no model run, and writes the figures to `routing-figures.md` at the archive's root: tokens by project, subagent type, rung and tool, and, for each chunk a `todo.md` item routed, grouped by the rung it started on: its review rounds past each pass's first, its unfinished verdicts and its subagents' rungs. It interrupts you, with `claude` on the project's model on the terminal in `~/.claude`, when chunks routed onto a rung took more review rounds or unfinished verdicts than chunks routed onto the next rung up, each rung with 5 chunks or more, or when 5 or more subagents of one type wrote reports longer than the tool results they read. Each call is raised once, then listed in `routing-calls-raised.md` beside the figures; the next chunk of this repo files a `todo.md` item for it. Off a terminal, the calls are said on stderr and the loop goes on. They are raised at the next measurement on a terminal. After a routing change, a `- <UTC time> Reset` line in that file makes the figures count only chunks finished after it, and lets its calls be raised again. Expect a pause of about ten seconds at each measurement.
- A session delegates a task to a subagent only when it needs just the task's conclusion, such as a search across many files, and keeps edits in its own thread. It scores the task's R and C as for a `todo.md` item, and launches the subagent on the model and effort that the new `bin/subagent-rung <model> <R+C>` picks from the project's model, such as `model sonnet, effort high`. `CLAUDE.md` states the rule. Expect searches to run on cheaper models, and less of each session's context to fill with search output.
- `loop-claude` routes each chunk's worker on a ladder of model and effort rungs, from Haiku 5.5 at low effort to Fable 5.1 at max, when the project's model is on it. As a chunk starts, the R + C of `todo.md`'s top item in the next release picks the rung: 11–12 picks one rung above the project's rung, so Opus 5.5 at high for an Opus 5.5 project, and so on up to four rungs. An item of R + C 10 or less runs the worker at or below the project's model's cheapest rung, which for Opus 5.5 is medium effort, below the xhigh `settings.json` sets. While the worker runs below the project's model's cheapest rung, it launches every reviewer, comprehension seat and security pass, the full scan included, in a subagent on that rung. While it runs above that rung, it launches each comprehension seat and the security pass on that rung. It steps up one rung after every second unfinished verdict on a chunk, and to a higher rung when it takes in an item of higher R + C. Each cycle's header, and each finished or unfinished verdict, stop and step-up in the archive, name the worker's rung; a rerun resumes an open chunk on the latest rung archived for it. `docs/decisions.md` → `routing:` gives the ladder, its source and the bands. Expect a chunk on an easy item to cost much less than under xhigh, and perhaps to take more review rounds, and one on an item of R + C 15 or more to cost more.
+2 -2
View File
@@ -67,7 +67,7 @@ Rename `## Unreleased` in `CHANGELOG.md` to the version, commit, tag that commit
Install puts only `loop-claude` on `PATH`; call the others by their path, such as `~/.claude/bin/metric-bench`, from the project's checkout.
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints the cheapest rung that returns them; `judge-replay --help` states the replay.
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints the lowest rung that, like every rung between it and the project's, returns them; `judge-replay --help` states the replay.
- `larv-review <command>` holds larv-review's mechanical rules: the paired ruling's base and vote count, each pass's round count, the scoring run's trigger, and the comprehension history's format and latest places; `larv-review --help` states them.
- `loop-claude <task>` runs a project's next chunk cycle after cycle; `loop-claude --help` states its contract.
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
@@ -94,7 +94,7 @@ pytest_add_cli_args=-k
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and `stability_replay` imports `loop_claude`'s retry and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and both it and `stability_replay` import `rung_replay/`: their shared rung search and command steps, which import `loop_claude`'s retry count and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
## Tracking a new file
+32
View File
@@ -0,0 +1,32 @@
"""One `claude -p --output-format json` run, as loop-claude's judge and the replays start it."""
import dataclasses
import subprocess
import typing
from failure import Failure
from claude_code.entries import hit_limit, limit_said, parse_entry
# Effects.claude: claude's arguments, its working directory and the prompt it reads on stdin.
Claude = typing.Callable[[list[str], str, str], subprocess.CompletedProcess[str]]
@dataclasses.dataclass(frozen=True)
class Limited:
"""claude hit a usage limit; said is claude's message about it, and resets_at is when the limit resets, in seconds since the epoch, or None when unknown."""
resets_at: int | None
said: str
def claude_reply(claude: Claude, args: list[str], cwd: str, prompt: str, who: str) -> dict | Failure | Limited:
"""claude's json reply, or why there is none; who names the run in the message for a non-zero exit, such as the judge."""
try:
proc = claude(args, cwd, prompt)
except OSError as failure:
return Failure(f"could not start claude: {failure.strerror}")
reply = parse_entry(proc.stdout) or {}
# The json output names the reset only in prose, such as "resets 7:40pm (Europe/Stockholm)", so its limit carries no reset time.
if hit_limit(reply):
return Limited(None, limit_said(reply))
if proc.returncode != 0:
return Failure(proc.stderr.strip() or f"{who} exited {proc.returncode}")
return reply
+22 -1
View File
@@ -1,11 +1,13 @@
import json
import pathlib
import subprocess
import sys
import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from claude_code import entries, transcript, trust
from claude_code import entries, reply, transcript, trust
from failure import Failure
def assistant(*parts):
@@ -80,6 +82,25 @@ class Parse(unittest.TestCase):
self.assertIsNone(entries.parse_entry("[1, 2]"))
class ClaudeReply(unittest.TestCase):
@staticmethod
def claude(*outcomes):
def run(args, cwd, prompt):
outcome = outcomes[0]
if isinstance(outcome, OSError):
raise outcome
return subprocess.CompletedProcess(args, *outcome)
return run
def test_returns_the_json_reply_a_usage_limit_whatever_the_exit_code_or_why_there_is_none(self):
self.assertEqual(reply.claude_reply(self.claude((0, '{"result": "ok"}', "")), [], "/r", "p", "the judge"), {"result": "ok"})
limit = json.dumps({"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
self.assertEqual(reply.claude_reply(self.claude((1, limit, "")), [], "/r", "p", "the judge"), reply.Limited(None, "You've hit your limit"))
self.assertEqual(reply.claude_reply(self.claude((2, "", " ")), [], "/r", "p", "the judge"), Failure("the judge exited 2"))
self.assertEqual(reply.claude_reply(self.claude((2, "", "boom\n")), [], "/r", "p", "the judge"), Failure("boom"))
self.assertEqual(reply.claude_reply(self.claude(FileNotFoundError(2, "No such file or directory")), [], "/r", "p", "claude"), Failure("could not start claude: No such file or directory"))
class MarkTrusted(unittest.TestCase):
def test_returns_the_updated_config_and_leaves_the_one_it_was_handed(self):
config = {"other": 1, "projects": {"/a": {"hasTrustDialogAccepted": True}, "/b": {"kept": True}}}
+27 -53
View File
@@ -1,4 +1,4 @@
"""Replay loop-claude's judge on archived verdicts, to find the cheapest rung that returns them: docs/decisions.md → `routing: on a model judge-replay covered`.
"""Replay loop-claude's judge on archived verdicts, to find the lowest rung that returns them: docs/decisions.md → `routing: on a model judge-replay covered`.
usage: judge-replay <model>
@@ -11,104 +11,78 @@ with the task, premises and records each project holds now.
It first replays every verdict on the reference judge: the model at xhigh effort, the rung the
judge ran on before loop-claude routed it. A verdict the reference judge returns another kind on
may rest on records or judge rules that have changed since; either way no rung is held to it. It
then replays the rest on each candidate rung, cheapest first: from the rung the lowest R + C picks
up to the model's cheapest rung. It stops at the first rung that returns all of them. It prints
each verdict the reference judge did not reproduce, each verdict a rung missed, and the cheapest
rung that missed none. A rung it names is the evidence to add the model to JUDGE_REPLAYED in
bin/loop_claude/routing.py, which routes that model's judge, and to record the run in
docs/decisions.md → `routing: on a model judge-replay covered`.
then replays the rest on each candidate rung, highest first: from the model's cheapest rung down
to the rung the lowest R + C picks. It stops at the first rung that misses one. It prints each
verdict the reference judge did not reproduce, each verdict a rung missed, and the lowest rung
that, like each rung above it, missed none. A rung it names is the evidence to add the model to
JUDGE_REPLAYED in bin/loop_claude/routing.py, which routes that model's judge, and to record the
run in docs/decisions.md → `routing: on a model judge-replay covered`.
Like the loop, it tries a judge three times running before it counts the reply as failed. A
verdict whose project or transcript cannot be read replays as failed too. Exit 1, saying why, on
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
or a usage limit.
"""
import concurrent.futures
import pathlib
import sys
import typing
from transcript_archive.location import archive_root, transcript_of
from transcript_archive.read import archived_outcomes
from judge_replay.replay import Case, case_from, Kind, replayed, report, sample, verdict_kind
from judge_replay.replay import Case, case_from, Kind, report, reproduced, sample, verdict_kind
from project_key import checkout_of
from rung_replay.command import down_from, failed, limited_exit, project_rung_or_exit, projects_on
from loop_claude.effects import Effects
from failure import Failure
from loop_claude.cycle.retry import first_reply, Limited
from rung_replay.search import first_reply, replayed
from loop_claude.cycle.judge import judge_once, Verdict
from model_routing.ladder import LADDER, picked, position, project_rung, Rung
from loop_claude.project.model_choice import read_models_file
from claude_code.reply import Limited
from loop_claude.project.premises import read_premises
from loop_claude.project.records import read_records
from claude_code.transcript import read_transcript, render
from model_routing.ladder import Rung
from loop_claude.stop import Stop
TOOL = "judge-replay"
PER_KIND = 15
# The effort settings.json gave every judge before loop-claude routed it; the archived verdicts came from it.
REFERENCE_EFFORT = "xhigh"
# Judges replayed at once.
AT_ONCE = 4
LOWEST_R_PLUS_C = 2
USAGE = "usage: judge-replay <model>, a model on the routing ladder"
def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, effects: Effects) -> int:
if argv[:1] in (["-h"], ["--help"]):
print(__doc__.strip())
return 0
anchor = project_rung(argv[0]) if len(argv) == 1 else None
if anchor is None:
print(f"judge-replay: cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; {USAGE}", file=sys.stderr)
return 1
cases = archived_cases(argv[0], archive_root(home, env), home)
anchor = project_rung_or_exit(TOOL, argv, __doc__)
if isinstance(anchor, int):
return anchor
cases = archived_cases(anchor.model, archive_root(home, env), home)
if isinstance(cases, Failure):
print(f"judge-replay: {cases.reason}", file=sys.stderr)
return 1
return failed(TOOL, cases.reason)
if not cases:
print(f"judge-replay: no archived verdicts of projects on {argv[0]} in {archive_root(home, env)}; run loop-claude on a project on that model first", file=sys.stderr)
return 1
return failed(TOOL, f"no archived verdicts of projects on {anchor.model} in {archive_root(home, env)}; run loop-claude on a project on that model first")
cases = sample(cases, PER_KIND)
reference = Rung(argv[0], REFERENCE_EFFORT)
candidates = list(LADDER[position(picked(anchor, LOWEST_R_PLUS_C)):position(anchor) + 1])
replay = replayed(cases, reference, candidates, lambda on, rung: rule_all(on, rung, home / ".claude", effects))
reference = Rung(anchor.model, REFERENCE_EFFORT)
replay = replayed(cases, reference, down_from(anchor), lambda found, rung: rule(found, rung, home / ".claude", effects), reproduced, "verdicts")
if isinstance(replay, Limited):
print(f"judge-replay: claude hit a usage limit: {replay.said}; replay again once it resets", file=sys.stderr)
return 1
print(report(argv[0], reference, cases, replay))
return limited_exit(TOOL, replay)
print(report(anchor.model, reference, cases, replay))
return 0
def archived_cases(model: str, root: pathlib.Path, home: pathlib.Path) -> list[Case] | Failure:
models = read_models_file(home / ".claude" / "loop-claude-models.json")
if isinstance(models, Stop):
return Failure(models.reason)
projects = projects_on(model, home)
if isinstance(projects, Failure):
return projects
listed = archived_outcomes(root)
if isinstance(listed, Failure):
return listed
found = []
try:
for key, outcomes in listed:
case = case_from(checkout_of(key, home), outcomes.read_text(encoding="utf-8"), transcript_of(outcomes)) if models.get(key) == model else None
case = case_from(checkout_of(key, home), outcomes.read_text(encoding="utf-8"), transcript_of(outcomes)) if key in projects else None
found += [case] if case else []
except (OSError, UnicodeDecodeError) as failure:
return Failure(f"could not read the archive {root}: {failure}")
return found
def rule_all(cases: list[Case], rung: Rung, claude_dir: pathlib.Path, effects: Effects) -> list[Kind] | Limited:
print(f"replaying {len(cases)} verdicts on {rung}", file=sys.stderr, flush=True)
with concurrent.futures.ThreadPoolExecutor(AT_ONCE) as pool:
running = [pool.submit(rule, found, rung, claude_dir, effects) for found in cases]
kinds = []
for future in running:
kind = future.result()
if isinstance(kind, Limited):
pool.shutdown(cancel_futures=True)
return kind
kinds.append(kind)
return kinds
def rule(case: Case, rung: Rung, claude_dir: pathlib.Path, effects: Effects) -> Kind | Limited:
premises, records, entries = read_premises(case.root), read_records(case.root, claude_dir), read_transcript(str(case.transcript))
if isinstance(premises, Stop):
+12 -40
View File
@@ -1,4 +1,4 @@
"""Which archived verdicts to replay, the cheapest rung that returns them, and the report: docs/decisions.md → `routing: on a model judge-replay covered`."""
"""Which archived verdicts to replay, what a replay reproduces, and the report: docs/decisions.md → `routing: on a model judge-replay covered`."""
import collections
import dataclasses
import pathlib
@@ -7,7 +7,7 @@ import typing
from transcript_archive.outcomes import ANSWERED, FINISHED, first_verdict, INTERRUPTED, task_of, UNFINISHED
from failure import Failure
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict
from loop_claude.cycle.retry import Limited
from rung_replay.search import Replay
from model_routing.ladder import Rung
# claude's reply as the replay compares it: a verdict as the archive words it, such as FINISHED, or a failure with its reason.
@@ -24,21 +24,6 @@ class Case:
on_record: Kind
@dataclasses.dataclass(frozen=True)
class Tried:
"""A rung, and each case it returned another kind on, with that kind."""
rung: Rung
missed: list[tuple[Case, Kind]]
@dataclasses.dataclass(frozen=True)
class Replay:
"""unreproduced holds each case the reference judge returned another kind on; tried, each rung tried, cheapest first; cheapest, the first rung that missed none, or None."""
unreproduced: list[tuple[Case, Kind]]
tried: list[Tried]
cheapest: Rung | None
def case_from(root: str, outcomes: str, transcript: pathlib.Path) -> Case | None:
"""The first verdict an outcomes file holds: the judge ruled on the transcript copy beside it as the copy stands. None when it holds none.
@@ -77,39 +62,26 @@ def verdict_kind(verdict: Verdict | Failure) -> Kind:
typing.assert_never(verdict)
def replayed(cases: list[Case], reference: Rung, candidates: list[Rung], rule_all: typing.Callable[[list[Case], Rung], list[Kind] | Limited]) -> Replay | Limited:
"""Replays the cases on the reference rung, then only those it reproduces on each candidate, cheapest first, until one returns every verdict."""
ruled = rule_all(cases, reference)
if isinstance(ruled, Limited):
return ruled
reproduced = [found for found, kind in zip(cases, ruled) if kind == found.on_record]
unreproduced = [(found, kind) for found, kind in zip(cases, ruled) if kind != found.on_record]
tried: list[Tried] = []
# With no verdict to hold a rung to, every rung would miss none.
for rung in candidates if reproduced else []:
ruled = rule_all(reproduced, rung)
if isinstance(ruled, Limited):
return ruled
tried.append(Tried(rung, [(found, kind) for found, kind in zip(reproduced, ruled) if kind != found.on_record]))
if not tried[-1].missed:
return Replay(unreproduced, tried, rung)
return Replay(unreproduced, tried, None)
def reproduced(case: Case, kind: Kind) -> list[Kind]:
"""The kind on record, where the reply returns it: a rung is held only to a verdict the reference judge reproduces."""
return [kind] if kind == case.on_record else []
def report(model: str, reference: Rung, cases: list[Case], replay: Replay) -> str:
def report(model: str, reference: Rung, cases: list[Case], replay: Replay[Case, Kind, Kind]) -> str:
kinds = collections.Counter(found.on_record for found in cases)
unreproduced = [(found, kind) for found, kind in zip(cases, replay.on_reference) if not reproduced(found, kind)]
lines = [
f"Replayed {len(cases)} verdicts of {model} projects: {', '.join(f'{count} {kind}' for kind, count in sorted(kinds.items()))}.",
f"The reference judge, {reference}, reproduced {len(cases) - len(replay.unreproduced)}{'. It returned another kind on:' if replay.unreproduced else '.'}",
*missed_lines(replay.unreproduced),
f"The reference judge, {reference}, reproduced {len(cases) - len(unreproduced)}{'. It returned another kind on:' if unreproduced else '.'}",
*missed_lines(unreproduced),
]
for tried in replay.tried:
lines += [f"{tried.rung} missed {len(tried.missed)}{':' if tried.missed else '.'}", *missed_lines(tried.missed)]
lines += [f"{tried.rung} missed {len(tried.missed)}{':' if tried.missed else '.'}", *missed_lines([(miss.case, miss.reply) for miss in tried.missed])]
if not replay.tried:
lines.append("No rung tried: the reference judge reproduced no verdict to hold one to.")
else:
cheapest = replay.cheapest or "none up to the project's rung"
lines.append(f"Cheapest rung that returns every verdict the reference judge reproduces: {cheapest}")
lowest = replay.lowest or "none, the project's rung misses one"
lines.append(f"Lowest rung that, like each rung above it, returns every verdict the reference judge reproduces: {lowest}")
return "\n".join(lines)
+22 -43
View File
@@ -10,12 +10,12 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINISHED
from judge_replay.replay import Case, case_from, Replay, replayed, report, sample, Tried, verdict_kind
from judge_replay.replay import Case, case_from, report, reproduced, sample, verdict_kind
from loop_claude.effects import Effects
from failure import Failure
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished
from loop_claude.cycle.retry import Limited
from judge_replay import main
from rung_replay.search import Miss, Replay, Tried
from model_routing.ladder import Rung
TOOL = pathlib.Path(__file__).resolve().parents[1] / "judge-replay"
@@ -59,52 +59,29 @@ class VerdictKind(unittest.TestCase):
self.assertEqual(verdict_kind(Failure("the judge quoted x")), "failed: the judge quoted x")
class Replayed(unittest.TestCase):
def test_tries_rungs_cheapest_first_on_the_verdicts_the_reference_reproduces_and_stops_at_the_first_that_returns_all(self):
cases = [case("2026-10-14T10:00:00Z", FINISHED), case("2026-10-14T10:00:01Z", INTERRUPTED), case("2026-10-14T10:00:02Z", FINISHED)]
replies = {REFERENCE: [FINISHED, INTERRUPTED, UNFINISHED], CHEAP: [FINISHED, FINISHED], MIDDLE: [FINISHED, INTERRUPTED]}
asked = []
def rule_all(on, rung):
asked.append((len(on), rung))
return replies[rung]
replay = replayed(cases, REFERENCE, [CHEAP, MIDDLE, Rung("claude-opus-5-5", "medium")], rule_all)
self.assertEqual(replay, Replay([(cases[2], UNFINISHED)], [Tried(CHEAP, [(cases[1], FINISHED)]), Tried(MIDDLE, [])], MIDDLE))
self.assertEqual(asked, [(3, REFERENCE), (2, CHEAP), (2, MIDDLE)])
def test_names_no_rung_when_every_candidate_misses_a_verdict(self):
cases = [case("2026-10-14T10:00:00Z", FINISHED)]
replay = replayed(cases, REFERENCE, [CHEAP], lambda on, rung: [FINISHED] if rung == REFERENCE else [INTERRUPTED])
self.assertEqual(replay, Replay([], [Tried(CHEAP, [(cases[0], INTERRUPTED)])], None))
def test_tries_no_rung_when_the_reference_reproduces_no_verdict(self):
cases = [case("2026-10-14T10:00:00Z", FINISHED)]
self.assertEqual(replayed(cases, REFERENCE, [CHEAP], lambda on, rung: [INTERRUPTED] * len(on)), Replay([(cases[0], INTERRUPTED)], [], None))
def test_stops_at_a_usage_limit(self):
limit = Limited(None, "limit")
self.assertEqual(replayed([case("2026-10-14T10:00:00Z", FINISHED)], REFERENCE, [CHEAP], lambda on, rung: limit), limit)
self.assertEqual(replayed([case("2026-10-14T10:00:00Z", FINISHED)], REFERENCE, [CHEAP], lambda on, rung: [FINISHED] if rung == REFERENCE else limit), limit)
class Reproduced(unittest.TestCase):
def test_holds_a_rung_to_a_verdict_only_where_the_reference_returned_the_kind_on_record(self):
self.assertEqual(reproduced(case("2026-10-14T10:00:00Z", FINISHED), FINISHED), [FINISHED])
self.assertEqual(reproduced(case("2026-10-14T10:00:00Z", FINISHED), INTERRUPTED), [])
class Report(unittest.TestCase):
def test_counts_the_verdicts_and_names_each_miss_and_the_cheapest_rung(self):
def test_counts_the_verdicts_and_names_each_miss_and_the_lowest_rung(self):
cases = [case("2026-10-14T10:00:00Z", FINISHED), case("2026-10-14T10:00:01Z", INTERRUPTED)]
replay = Replay([(cases[0], UNFINISHED)], [Tried(CHEAP, [(cases[1], FINISHED)]), Tried(MIDDLE, [])], MIDDLE)
replay = Replay([UNFINISHED, INTERRUPTED], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(cases[1], INTERRUPTED, FINISHED)])], MIDDLE)
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, replay).splitlines(), [
"Replayed 2 verdicts of claude-opus-5-5 projects: 1 Verdict: finished, 1 Verdict: interrupt.",
"The reference judge, claude-opus-5-5 at xhigh effort, reproduced 1. It returned another kind on:",
" /code/p 2026-10-14T10:00:00Z: on record Verdict: finished, replayed Verdict: unfinished",
"claude-sonnet-5-5 at high effort missed 0.",
"claude-haiku-5-5 at xhigh effort missed 1:",
" /code/p 2026-10-14T10:00:01Z: on record Verdict: interrupt, replayed Verdict: finished",
"claude-sonnet-5-5 at high effort missed 0.",
"Cheapest rung that returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort",
"Lowest rung that, like each rung above it, returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort",
])
missed_all = report("claude-opus-5-5", REFERENCE, cases, Replay([], [Tried(CHEAP, [(cases[1], FINISHED)])], None)).splitlines()
missed_all = report("claude-opus-5-5", REFERENCE, cases, Replay([FINISHED, INTERRUPTED], [Tried(MIDDLE, [Miss(cases[1], INTERRUPTED, FINISHED)])], None)).splitlines()
self.assertEqual(missed_all[1], "The reference judge, claude-opus-5-5 at xhigh effort, reproduced 2.")
self.assertEqual(missed_all[-1], "Cheapest rung that returns every verdict the reference judge reproduces: none up to the project's rung")
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, Replay([(cases[0], UNFINISHED), (cases[1], FINISHED)], [], None)).splitlines()[-1], "No rung tried: the reference judge reproduced no verdict to hold one to.")
self.assertEqual(missed_all[-1], "Lowest rung that, like each rung above it, returns every verdict the reference judge reproduces: none, the project's rung misses one")
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, Replay([UNFINISHED, FINISHED], [], None)).splitlines()[-1], "No rung tried: the reference judge reproduced no verdict to hold one to.")
def unused(*_):
@@ -148,25 +125,25 @@ class Main(unittest.TestCase):
output = {"answer": "", "bundled": 0, "evidence": evidence, "finished": not interrupt, "interrupt": interrupt, "question": "Which?" if interrupt else ""}
return {"type": "result", "structured_output": output}
def test_replays_the_models_projects_on_the_reference_then_cheapest_first_and_prints_the_rung_that_returns_all(self):
# Haiku misses the interrupt; Sonnet returns both.
def test_replays_the_models_projects_on_the_reference_then_from_the_projects_rung_down_and_prints_the_lowest_that_returns_all(self):
# Haiku misses the interrupt; Sonnet and Opus return both.
def reply(rung, interrupted):
return self.verdict(interrupted and rung.model != "claude-haiku-5-5", f"said {INTERRUPTED if interrupted else FINISHED}")
code, out, err = self.run_main(["claude-opus-5-5"], reply)
self.assertEqual(code, 0, err)
self.assertEqual({cwd for _, _, cwd in self.calls}, {str(self.home / "code" / "proj")})
self.assertEqual([(model, effort) for model, effort, _ in self.calls[::2]], [("claude-opus-5-5", "xhigh"), ("claude-haiku-5-5", "xhigh"), ("claude-haiku-5-5", "max"), ("claude-sonnet-5-5", "high")])
self.assertEqual([(model, effort) for model, effort, _ in self.calls[::2]], [("claude-opus-5-5", "xhigh"), ("claude-opus-5-5", "medium"), ("claude-sonnet-5-5", "high"), ("claude-haiku-5-5", "max")])
self.assertIn("The reference judge, claude-opus-5-5 at xhigh effort, reproduced 2.", out)
self.assertTrue(out.endswith("returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort\n"))
for said in ("The task an agent was given:\nDo it.", "1. Ship.", "From ~/.claude/CLAUDE.md:\nBe kind."):
self.assertIn(said, self.prompts[0])
def test_tries_rungs_up_to_the_models_cheapest_and_names_none_when_all_miss(self):
def test_stops_at_the_projects_rung_and_names_none_when_it_misses(self):
code, out, _ = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted and rung == REFERENCE, f"said {INTERRUPTED if interrupted else FINISHED}"))
self.assertEqual(code, 0)
self.assertEqual(self.calls[-1][:2], ("claude-opus-5-5", "medium"))
self.assertTrue(out.endswith(": none up to the project's rung\n"))
self.assertEqual({(model, effort) for model, effort, _ in self.calls[2:]}, {("claude-opus-5-5", "medium")})
self.assertTrue(out.endswith(": none, the project's rung misses one\n"))
def test_replays_a_verdict_whose_transcript_is_gone_as_failed(self):
(self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1" / "t0.jsonl").unlink()
@@ -174,14 +151,16 @@ class Main(unittest.TestCase):
self.assertEqual(code, 0)
self.assertIn("replayed failed: could not read the transcript", out)
def test_tries_up_to_the_models_cheapest_rung_and_retries_a_failed_reply_as_the_loop_does(self):
def test_tries_the_projects_rung_first_and_retries_a_failed_reply_as_the_loop_does(self):
def reply(rung, interrupted):
return self.verdict(interrupted, "not in the transcript" if rung.model != "claude-opus-5-5" else f"said {INTERRUPTED if interrupted else FINISHED}")
code, out, _ = self.run_main(["claude-opus-5-5"], reply)
self.assertEqual(code, 0)
self.assertEqual(sum(model == "claude-sonnet-5-5" for model, _, _ in self.calls), 6)
self.assertNotIn("claude-haiku-5-5", {model for model, _, _ in self.calls})
self.assertIn("claude-opus-5-5 at medium effort missed 0.", out)
self.assertTrue(out.endswith(": claude-opus-5-5 at medium effort\n"))
def test_exits_1_on_a_usage_limit_bad_arguments_or_an_unreadable_models_file(self):
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: {"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
+7 -13
View File
@@ -4,13 +4,14 @@ import re
import typing
from loop_claude.cycle.prompts import ANSWER_DESTINATIONS, ANSWERS_RECORDED, either
from claude_code.reply import claude_reply, Limited
from loop_claude.effects import Effects
from failure import Failure
from claude_code.entries import hit_limit, limit_said, parse_entry, structured_output
from loop_claude.project.records import HUMAN_RULE_NAMES
from loop_claude.cycle.retry import Limited, retried
from loop_claude.cycle.retry import retried
from model_routing.ladder import Rung
from loop_claude.stop import Stop
from claude_code.entries import structured_output
from todo_table import TOP_SCORE
# At 3 to 4 characters a token, 100,000 to 133,000 tokens, which leaves a 200,000-token context room for the reply; the largest session rendered here was 364,000 characters on 2026-10-03.
@@ -145,19 +146,12 @@ def judge_args(rung: Rung) -> list[str]:
def run_judge(root: str, args: list[str], prompt: str, shown: str, quotable: str, effects: Effects) -> Verdict | Failure | Limited:
try:
proc = effects.claude(args, root, prompt)
except OSError as failure:
return Failure(f"could not start claude: {failure.strerror}")
reply = parse_entry(proc.stdout) or {}
# The json output names the reset only in prose, such as "resets 7:40pm (Europe/Stockholm)", so the judge's limit carries no reset time.
if hit_limit(reply):
return Limited(None, limit_said(reply))
if proc.returncode != 0:
return Failure(proc.stderr.strip() or f"the judge exited {proc.returncode}")
reply = claude_reply(effects.claude, args, root, prompt, "the judge")
if isinstance(reply, Failure | Limited):
return reply
verdict = verdict_from(structured_output(reply))
if isinstance(verdict, Failure):
return Failure(f"{verdict.reason}; it replied: {proc.stdout.strip() or 'nothing'}")
return Failure(f"{verdict.reason}; it replied: {json.dumps(reply, ensure_ascii=False) if reply else 'nothing'}")
if not grounded(verdict.evidence, shown):
return Failure(f"the judge quoted {verdict.evidence!r}, which is not in the transcript" + (f", for the question: {verdict.question}" if isinstance(verdict, Interrupt | QuotedAnswer) else ""))
# Not a Failure: a misquote would otherwise exit an unattended loop. Never elided: stitched fragments can say what the records do not.
+1 -18
View File
@@ -1,9 +1,9 @@
import dataclasses
import sys
import typing
from loop_claude.effects import Effects
from failure import Failure
from claude_code.reply import Limited
from loop_claude.stop import Stop
T = typing.TypeVar("T")
@@ -14,13 +14,6 @@ RETRY_DELAY = 60
UNKNOWN_RESET_POLL = 600
@dataclasses.dataclass(frozen=True)
class Limited:
"""claude hit a usage limit; said is claude's message about it, and resets_at is when the limit resets, in seconds since the epoch, or None when unknown."""
resets_at: int | None
said: str
def retried(who: str, attempt: typing.Callable[[], T | Failure | Limited], effects: Effects) -> T | Stop:
failures = 0
while True:
@@ -48,13 +41,3 @@ def limit_wait(limited: Limited, now: float) -> float:
return limited.resets_at + PAST_RESET - now
return UNKNOWN_RESET_POLL
def first_reply(attempt: typing.Callable[[], T | Failure | Limited]) -> T | Failure | Limited:
"""The first reply that is no Failure, or the last of FAILURES_RUNNING failures: a replay's retry.
It counts failures as retried does, but neither waits between attempts nor waits out a usage limit, which ends a replay.
"""
for _ in range(FAILURES_RUNNING - 1):
if not isinstance(reply := attempt(), Failure):
return reply
return attempt()
+1 -1
View File
@@ -1,7 +1,7 @@
from claude_code.entries import assistant_parts, ends_stream, hit_limit, limit_said, parse_entry, rejected_until
from loop_claude.effects import Effects
from failure import Failure
from loop_claude.cycle.retry import Limited
from claude_code.reply import Limited
from claude_code.transcript import part_line
from model_routing.ladder import Rung
from claude_code.session import Session
+9 -7
View File
@@ -19,12 +19,12 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from routing_figures import archive
from loop_claude.project import checkout, model_choice, records
from loop_claude.effects import Effects
from claude_code import entries, session
from claude_code import entries, reply, session
from failure import Failure
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished, verdict_from
from loop_claude.cycle import interrupt, judge, prompts, retry, run, states, worker
from model_routing.ladder import LADDER, Rung
from loop_claude.cycle.retry import Limited
from claude_code.reply import Limited
from loop_claude.loop import main, reload, routing_check
from transcript_archive import outcomes, write
from loop_claude import paths, routing
@@ -886,20 +886,22 @@ class ExitContract(unittest.TestCase):
class JudgeFailures(unittest.TestCase):
def test_every_failed_judge_run_has_its_clause_in_help(self):
started, field, unasked = "claude cannot start or exits non-zero", "its reply misses or mistypes a field", "interrupts or quotes an answer without a question"
# Every Failure the judge's run returns, in source order: path, function, start of reason, its --help clause.
# Every Failure the judge's run returns, in source order of judge.py, then of the claude run it calls: function, start of reason, its --help clause.
clauses = [
("verdict_from", "'the judge returned no reply'", field),
("verdict_from", 'f"the judge\'s reply has no {name}', field),
("verdict_from", "'the judge interrupted without a question'", unasked),
("verdict_from", "'the judge quoted an answer without a question'", unasked),
("verdict_from", 'f"the judge\'s bundled R + C', "gives an unfinished verdict an R + C outside 0 to 20"),
("run_judge", "f'could not start claude:", started),
("run_judge", "proc.stderr.strip() or f'the judge exited", started),
("run_judge", 'f"{verdict.reason}; it replied:', field),
("run_judge", "f'the judge quoted {verdict.evidence!r}, which is not in the transcript", "quotes a passage the transcript does not hold"),
("claude_reply", "f'could not start claude:", started),
("claude_reply", "proc.stderr.strip() or f'{who} exited", started),
]
functions = [node for node in ast.walk(ast.parse(pathlib.Path(judge.__file__).read_text())) if isinstance(node, ast.FunctionDef)]
found = sorted((call.lineno, function.name, ast.unparse(call.args[0])) for function in functions for call in ast.walk(function) if isinstance(call, ast.Call) and getattr(call.func, "id", None) == "Failure")
found = []
for module in (judge, reply):
functions = [node for node in ast.walk(ast.parse(pathlib.Path(module.__file__).read_text())) if isinstance(node, ast.FunctionDef)]
found += sorted((call.lineno, function.name, ast.unparse(call.args[0])) for function in functions for call in ast.walk(function) if isinstance(call, ast.Call) and getattr(call.func, "id", None) == "Failure")
self.assertEqual(len(found), len(clauses), found)
for (_, function, reason), (want_function, reason_start, _) in zip(found, clauses):
self.assertEqual(function, want_function)
+1 -1
View File
@@ -15,5 +15,5 @@ class Types(unittest.TestCase):
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
if shutil.which("uv") is None:
self.fail("uv on PATH fetches mypy; see README.md's Test line")
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "rule_hook", "-p", "score_outcomes", "-p", "stability_replay", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "rule_hook", "-p", "rung_replay", "-p", "score_outcomes", "-p", "stability_replay", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
View File
+44
View File
@@ -0,0 +1,44 @@
"""What judge-replay's and stability-replay's commands share: the argument, the projects on its model, the candidate rungs and the exits."""
import pathlib
import sys
from failure import Failure
from model_routing.ladder import LADDER, picked, position, project_rung, Rung
from claude_code.reply import Limited
from loop_claude.project.model_choice import read_models_file
from loop_claude.stop import Stop
LOWEST_R_PLUS_C = 2
def project_rung_or_exit(tool: str, argv: list[str], help_text: str) -> Rung | int:
"""The project's rung of the model argv names, or the exit code: 0 after printing help_text for -h or --help, 1 after saying why argv cannot be read."""
if argv[:1] in (["-h"], ["--help"]):
print(help_text.strip())
return 0
anchor = project_rung(argv[0]) if len(argv) == 1 else None
if anchor is None:
return failed(tool, f"cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; usage: {tool} <model>, a model on the routing ladder")
return anchor
def projects_on(model: str, home: pathlib.Path) -> list[str] | Failure:
"""The keys of the projects ~/.claude/loop-claude-models.json runs on model, sorted."""
models = read_models_file(home / ".claude" / "loop-claude-models.json")
if isinstance(models, Stop):
return Failure(models.reason)
return sorted(key for key, on in models.items() if on == model)
def down_from(top: Rung) -> list[Rung]:
"""top and each rung below it, down to the rung the lowest R + C picks from top, highest first."""
return list(LADDER[position(picked(top, LOWEST_R_PLUS_C)):position(top) + 1])[::-1]
def limited_exit(tool: str, limited: Limited) -> int:
return failed(tool, f"claude hit a usage limit: {limited.said}; replay again once it resets")
def failed(tool: str, reason: str) -> int:
print(f"{tool}: {reason}", file=sys.stderr)
return 1
+101
View File
@@ -0,0 +1,101 @@
"""The rung search judge-replay and stability-replay share: replay archived cases on a reference rung, then hold each candidate rung to what the reference reproduced."""
import concurrent.futures
import dataclasses
import sys
import typing
from failure import Failure
from loop_claude.cycle.retry import FAILURES_RUNNING
from claude_code.reply import Limited
from model_routing.ladder import Rung
# Cases replayed at once.
AT_ONCE = 4
Case = typing.TypeVar("Case")
# What of a case's record a reply reproduces, such as a verdict's kind or a finding's index.
Item = typing.TypeVar("Item")
Reply = typing.TypeVar("Reply")
T = typing.TypeVar("T")
@dataclasses.dataclass(frozen=True)
class Miss(typing.Generic[Case, Item, Reply]):
"""An item the reference reproduced in case that a rung's reply did not."""
case: Case
item: Item
reply: Reply
@dataclasses.dataclass(frozen=True)
class Tried(typing.Generic[Case, Item, Reply]):
rung: Rung
missed: list[Miss[Case, Item, Reply]]
@dataclasses.dataclass(frozen=True)
class Replay(typing.Generic[Case, Item, Reply]):
"""on_reference holds the reference's reply on each case; tried, each candidate tried, highest first; lowest, the lowest rung that, like each candidate above it, missed nothing, or None."""
on_reference: list[Reply]
tried: list[Tried[Case, Item, Reply]]
lowest: Rung | None
def replayed(
cases: list[Case],
reference: Rung,
candidates: list[Rung],
replay_one: typing.Callable[[Case, Rung], Reply | Limited],
reproduced: typing.Callable[[Case, Reply], list[Item]],
what: str,
) -> Replay[Case, Item, Reply] | Limited:
"""Replays the cases on the reference rung, then each candidate, highest first, on the cases where the reference reproduced anything, until one misses an item it reproduced.
Going down, every rung at or above the one named was tried and missed nothing, so the result needs no assumption that a higher rung does at least as well; and a miss high up spares the cheaper rungs their replays.
"""
on_reference = replayed_on(cases, reference, replay_one, what)
if isinstance(on_reference, Limited):
return on_reference
held = [(case, aim) for case, reply in zip(cases, on_reference) if (aim := reproduced(case, reply))]
tried: list[Tried[Case, Item, Reply]] = []
lowest = None
# With nothing to hold a rung to, every rung would miss nothing.
for rung in candidates if held else []:
replies = replayed_on([case for case, _ in held], rung, replay_one, what)
if isinstance(replies, Limited):
return replies
misses: list[Miss[Case, Item, Reply]] = []
for (case, aim), reply in zip(held, replies):
got = reproduced(case, reply)
misses += [Miss(case, item, reply) for item in aim if item not in got]
tried.append(Tried(rung, misses))
if misses:
break
lowest = rung
return Replay(on_reference, tried, lowest)
def replayed_on(cases: list[Case], rung: Rung, replay_one: typing.Callable[[Case, Rung], Reply | Limited], what: str) -> list[Reply] | Limited:
"""Each case's reply on rung, AT_ONCE at a time; a usage limit cancels the rest."""
print(f"replaying {len(cases)} {what} on {rung}", file=sys.stderr, flush=True)
with concurrent.futures.ThreadPoolExecutor(AT_ONCE) as pool:
running = [pool.submit(replay_one, case, rung) for case in cases]
replies: list[Reply] = []
for future in running:
reply = future.result()
if isinstance(reply, Limited):
pool.shutdown(cancel_futures=True)
return reply
replies.append(reply)
return replies
def first_reply(attempt: typing.Callable[[], T | Failure | Limited]) -> T | Failure | Limited:
"""The first reply that is no Failure, or the last of FAILURES_RUNNING failures.
It counts failures as loop-claude's retried does, but neither waits between attempts nor waits out a usage limit, which ends a replay.
"""
for _ in range(FAILURES_RUNNING - 1):
if not isinstance(reply := attempt(), Failure):
return reply
return attempt()
+67
View File
@@ -0,0 +1,67 @@
import contextlib
import io
import pathlib
import sys
import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from rung_replay.command import down_from
from failure import Failure
from rung_replay.search import first_reply, Miss, Replay, replayed, Tried
from claude_code.reply import Limited
from model_routing.ladder import Rung
REFERENCE = Rung("claude-opus-5-5", "medium")
MIDDLE = Rung("claude-sonnet-5-5", "high")
CHEAP = Rung("claude-haiku-5-5", "max")
LOWER = Rung("claude-haiku-5-5", "xhigh")
def search(cases, candidates, replies):
"""replayed on cases named by strings, where replies[rung][case] is the set of items a rung reproduces in the case."""
asked = []
def replay_one(case, rung):
asked.append((case, rung))
reply = replies[rung]
return reply if isinstance(reply, Limited) else reply[case]
with contextlib.redirect_stderr(io.StringIO()):
return replayed(cases, REFERENCE, candidates, replay_one, lambda _, reply: sorted(reply), "cases"), asked
class Replayed(unittest.TestCase):
def test_holds_each_candidate_highest_first_to_what_the_reference_reproduced_and_stops_at_the_first_that_misses(self):
replies = {REFERENCE: {"a": {0, 1}, "b": set()}, MIDDLE: {"a": {0, 1}}, CHEAP: {"a": {1}}}
replay, asked = search(["a", "b"], [MIDDLE, CHEAP, LOWER], replies)
self.assertEqual(replay, Replay([{0, 1}, set()], [Tried(MIDDLE, []), Tried(CHEAP, [Miss("a", 0, {1})])], MIDDLE))
self.assertEqual(sorted(asked, key=lambda pair: (pair[1] != REFERENCE, pair[0])), [("a", REFERENCE), ("b", REFERENCE), ("a", MIDDLE), ("a", CHEAP)])
def test_names_no_rung_when_the_first_misses_and_tries_none_when_the_reference_reproduced_nothing(self):
self.assertEqual(search(["a"], [MIDDLE, CHEAP], {REFERENCE: {"a": {0}}, MIDDLE: {"a": set()}})[0], Replay([{0}], [Tried(MIDDLE, [Miss("a", 0, set())])], None))
self.assertEqual(search(["a"], [MIDDLE], {REFERENCE: {"a": set()}}), (Replay([set()], [], None), [("a", REFERENCE)]))
def test_stops_at_a_usage_limit(self):
limit = Limited(None, "limit")
self.assertEqual(search(["a"], [MIDDLE], {REFERENCE: limit})[0], limit)
self.assertEqual(search(["a"], [MIDDLE, CHEAP], {REFERENCE: {"a": {0}}, MIDDLE: limit})[0], limit)
class FirstReply(unittest.TestCase):
def test_tries_three_times_and_returns_the_first_reply_that_is_no_failure(self):
replies = [Failure("one"), Failure("two"), "done"]
self.assertEqual(first_reply(lambda: replies.pop(0)), "done")
replies = [Failure("one"), Limited(None, "limit")]
self.assertEqual(first_reply(lambda: replies.pop(0)), Limited(None, "limit"))
replies = [Failure("one"), Failure("two"), Failure("three"), "unasked"]
self.assertEqual(first_reply(lambda: replies.pop(0)), Failure("three"))
class DownFrom(unittest.TestCase):
def test_runs_from_the_projects_rung_down_to_the_rung_the_lowest_r_plus_c_picks(self):
self.assertEqual(down_from(REFERENCE), [REFERENCE, MIDDLE, CHEAP, LOWER])
if __name__ == "__main__":
unittest.main()
+29 -61
View File
@@ -24,7 +24,6 @@ Like the loop, it tries claude three times running before it counts a reply as f
review or match finds nothing. Exit 1, saying why, on bad arguments, an unreadable archive or
models file, no archived round to replay, or a usage limit.
"""
import concurrent.futures
import dataclasses
import json
import os
@@ -35,23 +34,20 @@ import tempfile
import typing
from transcript_archive.location import archive_root
from stability_replay.rounds import authored_by, replayed, report, reviewed_path, Round, round_from, sample, stamp_seconds
from stability_replay.rounds import authored_by, report, reviewed_path, Round, round_from, sample, stamp_seconds
from project_key import checkout_of
from claude_code.reply import claude_reply, Limited
from rung_replay.command import down_from, failed, limited_exit, project_rung_or_exit, projects_on
from loop_claude.effects import Effects
from failure import Failure
from loop_claude.cycle.retry import first_reply, Limited
from rung_replay.search import first_reply, replayed
from claude_code.transcript import first_timestamped_line, read_transcript
from claude_code.entries import hit_limit, limit_said, parse_entry, structured_output
from model_routing.ladder import LADDER, picked, position, project_rung, Rung
from loop_claude.project.model_choice import read_models_file
from loop_claude.stop import Stop
from claude_code.entries import parse_entry, structured_output
from model_routing.ladder import Rung
TOOL = "stability-replay"
LATEST = 10
# Reviews replayed at once.
AT_ONCE = 4
LOWEST_R_PLUS_C = 2
REVIEWER = "stability-reviewer"
USAGE = "usage: stability-replay <model>, a model on the routing ladder"
MATCH_SCHEMA = {"type": "object", "properties": {"found": {"type": "array", "items": {"type": "integer"}}}, "required": ["found"], "additionalProperties": False}
@@ -65,38 +61,33 @@ class Code:
def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, effects: Effects) -> int:
if argv[:1] in (["-h"], ["--help"]):
print(__doc__.strip())
return 0
anchor = project_rung(argv[0]) if len(argv) == 1 else None
if anchor is None:
print(f"stability-replay: cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; {USAGE}", file=sys.stderr)
return 1
rounds = archived_rounds(argv[0], archive_root(home, env), home)
anchor = project_rung_or_exit(TOOL, argv, __doc__)
if isinstance(anchor, int):
return anchor
rounds = archived_rounds(anchor.model, archive_root(home, env), home)
if isinstance(rounds, Failure):
print(f"stability-replay: {rounds.reason}", file=sys.stderr)
return 1
return failed(TOOL, rounds.reason)
codes = {found: code for found in rounds if not isinstance(code := code_of(found, effects), Failure)}
if len(codes) < len(rounds):
print(f"{len(rounds) - len(codes)} of {len(rounds)} rounds holding a 🔴 or 🟡 are left out: their code cannot be found", file=sys.stderr, flush=True)
if not codes:
print(f"stability-replay: no archived stability round of projects on {argv[0]} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find", file=sys.stderr)
return 1
return failed(TOOL, f"no archived stability round of projects on {anchor.model} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find")
chosen = sample(list(codes), LATEST)
candidates = list(LADDER[position(picked(anchor, LOWEST_R_PLUS_C)):position(anchor)])[::-1]
replay = replayed(chosen, anchor, candidates, lambda on, rung: review_all([(found, codes[found]) for found in on], rung, anchor, effects))
# The project's rung is the reference, so the candidates start below it.
replay = replayed(chosen, anchor, down_from(anchor)[1:], lambda found, rung: findings_found(found, codes[found], rung, anchor, effects), lambda _, found: sorted(found), "stability rounds")
if isinstance(replay, Limited):
print(f"stability-replay: claude hit a usage limit: {replay.said}; replay again once it resets", file=sys.stderr)
return 1
print(report(argv[0], anchor, chosen, replay))
return limited_exit(TOOL, replay)
print(report(anchor.model, anchor, chosen, replay))
return 0
def archived_rounds(model: str, root: pathlib.Path, home: pathlib.Path) -> list[Round] | Failure:
models = read_models_file(home / ".claude" / "loop-claude-models.json")
if isinstance(models, Stop):
return Failure(models.reason)
projects = projects_on(model, home)
if isinstance(projects, Failure):
return projects
found = []
try:
for key in sorted(key for key, on in models.items() if on == model):
for key in projects:
for chunk in sorted(path for path in (root / key).glob("*") if path.is_dir()):
found += chunk_rounds(checkout_of(key, home), chunk)
except (OSError, UnicodeDecodeError, ValueError) as failure:
@@ -120,6 +111,10 @@ def chunk_rounds(checkout: str, chunk: pathlib.Path) -> list[Round]:
def code_of(found: Round, effects: Effects) -> Code | Failure:
"""The code found's reviewer read, as the archive lets it be found again: the prompt's removed worktree, and the commits authored by then.
Recording the reviewed commit at review time would need no such reading, but then no round archived before that change could be replayed. A wrong guess costs no rung: the reference reviewer, replayed on that code, finds fewer findings on record, and no rung is held to those.
"""
path = reviewed_path(found.prompt, found.root, os.path.exists)
if path is None:
return Failure("the prompt names no one worktree")
@@ -143,20 +138,6 @@ def commit_log(listed: str) -> list[tuple[str, float]]:
return [(commit, float(authored)) for commit, authored in (line.split() for line in listed.splitlines())]
def review_all(on: list[tuple[Round, Code]], rung: Rung, anchor: Rung, effects: Effects) -> list[set[int]] | Limited:
print(f"replaying {len(on)} stability rounds on {rung}", file=sys.stderr, flush=True)
with concurrent.futures.ThreadPoolExecutor(AT_ONCE) as pool:
running = [pool.submit(findings_found, found, code, rung, anchor, effects) for found, code in on]
found_sets = []
for future in running:
found = future.result()
if isinstance(found, Limited):
pool.shutdown(cancel_futures=True)
return found
found_sets.append(found)
return found_sets
def findings_found(found: Round, code: Code, rung: Rung, anchor: Rung, effects: Effects) -> set[int] | Limited:
"""The indexes of the round's findings that its replay on rung reports, as the reviewer on anchor matches them."""
reviewed: str | Failure | Limited = first_reply(lambda: review(found, code, rung, effects))
@@ -181,7 +162,7 @@ def review(found: Round, code: Code, rung: Rung, effects: Effects) -> str | Fail
if isinstance(ran := git(effects, found.root, *command), Failure):
return ran
args = ["-p", "--agent", REVIEWER, *rung.flags(), "--no-session-persistence", "--output-format", "json", "--permission-mode", "bypassPermissions"]
reply = claude_reply(effects, args, clone, found.prompt.replace(code.path, clone))
reply = claude_reply(effects.claude, args, clone, found.prompt.replace(code.path, clone), "claude")
if isinstance(reply, Failure | Limited):
return reply
return reply["result"] if isinstance(reply.get("result"), str) else Failure("the reviewer replied with no report")
@@ -198,7 +179,7 @@ def match(findings: tuple[str, ...], replay_report: str, root: str, rung: Rung,
f"The findings:\n{numbered}\n\nThe other report:\n{replay_report}"
)
args = ["-p", "--json-schema", json.dumps(MATCH_SCHEMA), *rung.flags(), "--no-session-persistence", "--output-format", "json", "--permission-mode", "bypassPermissions", "--tools", ""]
reply = claude_reply(effects, args, root, prompt)
reply = claude_reply(effects.claude, args, root, prompt, "claude")
if isinstance(reply, Failure | Limited):
return reply
output = structured_output(reply)
@@ -208,19 +189,6 @@ def match(findings: tuple[str, ...], replay_report: str, root: str, rung: Rung,
return {n - 1 for n in numbers if 1 <= n <= len(findings)}
def claude_reply(effects: Effects, args: list[str], cwd: str, prompt: str) -> dict | Failure | Limited:
try:
proc = effects.claude(args, cwd, prompt)
except OSError as failure:
return Failure(f"could not start claude: {failure.strerror}")
reply = parse_entry(proc.stdout) or {}
if hit_limit(reply):
return Limited(None, limit_said(reply))
if proc.returncode != 0:
return Failure(proc.stderr.strip() or f"claude exited {proc.returncode}")
return reply
def git(effects: Effects, cwd: str, *args: str) -> str | Failure:
ran = effects.git(list(args), cwd)
return ran.stdout if ran.returncode == 0 else Failure(f"git {' '.join(args)}: {ran.stderr.strip()}")
+8 -45
View File
@@ -1,4 +1,4 @@
"""Which archived stability rounds to replay, the lowest rung that finds their findings, and the report: docs/decisions.md → `routing: the stability reviewer stays on the project's rung`."""
"""Which archived stability rounds to replay, the code each reviewed, and the report: docs/decisions.md → `routing: the stability reviewer stays on the project's rung`."""
import dataclasses
import datetime
import pathlib
@@ -6,7 +6,7 @@ import re
import typing
from claude_code.entries import assistant_parts, content, part_text, parts
from loop_claude.cycle.retry import Limited
from rung_replay.search import Replay
from model_routing.ladder import Rung
# A line that opens with a 🔴 or 🟡 mark, after any list, table, heading or bold markup, as review-findings.md has reviewers report them.
@@ -28,21 +28,6 @@ class Round:
findings: tuple[str, ...]
@dataclasses.dataclass(frozen=True)
class Tried:
"""A rung, and each finding it missed that the reference reviewer found, with its round."""
rung: Rung
missed: list[tuple[Round, str]]
@dataclasses.dataclass(frozen=True)
class Replay:
"""unfound counts the findings on record the reference reviewer did not find; tried holds each rung tried, highest first; lowest, the lowest rung that missed none of the rest, or None."""
unfound: int
tried: list[Tried]
lowest: Rung | None
def round_from(root: str, began: float, entries: list[dict]) -> Round | None:
"""The round a stability reviewer's transcript holds: its first entry's prompt, and its last text. None without a 🔴 or 🟡 finding to hold a rung to."""
reports = [text for found in entries for part in assistant_parts(found) if (text := part_text(part))]
@@ -79,7 +64,9 @@ def reviewed_path(prompt: str, root: str, exists: typing.Callable[[str], bool])
named = [LINE_OR_STOP.sub("", path) for path in PATH.findall(prompt)]
gone = set()
for path in map(pathlib.PurePath, named):
first_gone = next((prefix for prefix in [*reversed(path.parents), path][1:] if not exists(str(prefix))), None)
# The path's directories from the top down, without the filesystem root; the first one gone is what a removed worktree took with it.
down_the_path = [*reversed(path.parents), path][1:]
first_gone = next((prefix for prefix in down_the_path if not exists(str(prefix))), None)
if first_gone and (first_gone.parent == checkout.parent or checkout in first_gone.parents):
gone.add(str(first_gone))
if len(gone) == 1:
@@ -97,38 +84,14 @@ def sample(rounds: list[Round], latest: int) -> list[Round]:
return sorted(rounds, key=lambda found: (found.seconds, found.root))[-latest:]
def replayed(rounds: list[Round], reference: Rung, candidates: list[Rung], review_all: typing.Callable[[list[Round], Rung], list[set[int]] | Limited]) -> Replay | Limited:
"""Replays the rounds on the reference rung, then each candidate, highest first, on the rounds where the reference found a finding, until one misses a finding it found.
Going down, every rung at or above the lowest that missed none was tried.
"""
found = review_all(rounds, reference)
if isinstance(found, Limited):
return found
held = [(found_round, kept) for found_round, kept in zip(rounds, found) if kept]
unfound = sum(len(found_round.findings) for found_round in rounds) - sum(len(kept) for _, kept in held)
tried: list[Tried] = []
lowest = None
# With no finding to hold a rung to, every rung would miss none.
for rung in candidates if held else []:
found = review_all([found_round for found_round, _ in held], rung)
if isinstance(found, Limited):
return found
tried.append(Tried(rung, [(found_round, found_round.findings[n]) for (found_round, kept), on_rung in zip(held, found) for n in sorted(kept - on_rung)]))
if tried[-1].missed:
break
lowest = rung
return Replay(unfound, tried, lowest)
def report(model: str, reference: Rung, rounds: list[Round], replay: Replay) -> str:
def report(model: str, reference: Rung, rounds: list[Round], replay: Replay[Round, int, set[int]]) -> str:
total = sum(len(found_round.findings) for found_round in rounds)
lines = [
f"Replayed {len(rounds)} stability rounds of {model} projects, holding {total} 🔴 and 🟡 findings.",
f"The reference reviewer, {reference}, found {total - replay.unfound} of them.",
f"The reference reviewer, {reference}, found {sum(len(found) for found in replay.on_reference)} of them.",
]
for tried in replay.tried:
lines += [f"{tried.rung} missed {len(tried.missed)}{':' if tried.missed else '.'}", *(f" {found_round.root} {found_round.at}: {finding.splitlines()[0]}" for found_round, finding in tried.missed)]
lines += [f"{tried.rung} missed {len(tried.missed)}{':' if tried.missed else '.'}", *(f" {miss.case.root} {miss.case.at}: {miss.case.findings[miss.item].splitlines()[0]}" for miss in tried.missed)]
if not replay.tried:
lines.append("No rung tried: the reference reviewer found no finding to hold one to.")
else:
+6 -31
View File
@@ -10,10 +10,10 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from stability_replay.rounds import authored_by, recorded_findings, Replay, replayed, report, reviewed_path, Round, round_from, sample, Tried
from stability_replay.rounds import authored_by, recorded_findings, report, reviewed_path, Round, round_from, sample
from loop_claude.effects import Effects, system_effects
from loop_claude.cycle.retry import Limited
from stability_replay import main
from rung_replay.search import Miss, Replay, Tried
from model_routing.ladder import Rung
TOOL = pathlib.Path(__file__).resolve().parents[1] / "stability-replay"
@@ -82,35 +82,10 @@ class Sample(unittest.TestCase):
self.assertEqual(sample(rounds, 2), [rounds[0], rounds[2]])
class Replayed(unittest.TestCase):
def test_holds_each_rung_highest_first_to_what_the_reference_found_and_stops_at_the_first_that_misses(self):
rounds = [a_round(1, ("🔴 a", "🟡 b")), a_round(2, ("🔴 c",))]
found = {REFERENCE: [{0, 1}, set()], MIDDLE: [{0, 1}], CHEAP: [{1}]}
asked = []
def review_all(on, rung):
asked.append((len(on), rung))
return found[rung]
replay = replayed(rounds, REFERENCE, [MIDDLE, CHEAP, Rung("claude-haiku-5-5", "xhigh")], review_all)
self.assertEqual(replay, Replay(1, [Tried(MIDDLE, []), Tried(CHEAP, [(rounds[0], "🔴 a")])], MIDDLE))
self.assertEqual(asked, [(2, REFERENCE), (1, MIDDLE), (1, CHEAP)])
def test_names_no_rung_when_the_first_misses_or_the_reference_found_nothing(self):
rounds = [a_round(1)]
self.assertEqual(replayed(rounds, REFERENCE, [MIDDLE], lambda on, rung: [{0}] if rung == REFERENCE else [set()]), Replay(0, [Tried(MIDDLE, [(rounds[0], "🔴 a")])], None))
self.assertEqual(replayed(rounds, REFERENCE, [MIDDLE], lambda on, rung: [set()]), Replay(1, [], None))
def test_stops_at_a_usage_limit(self):
limit = Limited(None, "limit")
self.assertEqual(replayed([a_round(1)], REFERENCE, [MIDDLE], lambda on, rung: limit), limit)
self.assertEqual(replayed([a_round(1)], REFERENCE, [MIDDLE], lambda on, rung: [{0}] if rung == REFERENCE else limit), limit)
class Report(unittest.TestCase):
def test_counts_the_findings_and_names_each_miss_and_the_lowest_rung_that_missed_none(self):
rounds = [a_round(1, ("🔴 a", "🟡 b")), a_round(2)]
lines = report("claude-opus-5-5", REFERENCE, rounds, Replay(1, [Tried(MIDDLE, []), Tried(CHEAP, [(rounds[0], "🔴 a\nmore")])], MIDDLE)).splitlines()
rounds = [a_round(1, ("🔴 a\nmore", "🟡 b")), a_round(2)]
lines = report("claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, set()], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(rounds[0], 0, {1})])], MIDDLE)).splitlines()
self.assertEqual(lines, [
"Replayed 2 stability rounds of claude-opus-5-5 projects, holding 3 🔴 and 🟡 findings.",
"The reference reviewer, claude-opus-5-5 at medium effort, found 2 of them.",
@@ -119,8 +94,8 @@ class Report(unittest.TestCase):
" /code/p 2026-10-14T10:00:01Z: 🔴 a",
"Lowest rung that finds every finding the reference reviewer finds: claude-sonnet-5-5 at high effort",
])
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay(0, [Tried(MIDDLE, [(rounds[1], "🔴 a")])], None)).splitlines()[-1], "Lowest rung that finds every finding the reference reviewer finds: none below the project's rung")
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay(3, [], None)).splitlines()[-1], "No rung tried: the reference reviewer found no finding to hold one to.")
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, {0}], [Tried(MIDDLE, [Miss(rounds[1], 0, set())])], None)).splitlines()[-1], "Lowest rung that finds every finding the reference reviewer finds: none below the project's rung")
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay([set(), set()], [], None)).splitlines()[-1], "No rung tried: the reference reviewer found no finding to hold one to.")
def unused(*_):
+3 -3
View File
@@ -48,7 +48,7 @@ Release 0.3.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
| 133 | 0.3.0 | | **Show, beside `bin/rule_hook/override_log.py`'s `this_chunks`, one example per case it decides: an amended or rebased commit, a renamed branch, and a commit on the default branch.** | 1 | 2 | 2 | 3 | 4 | 1.1 | 12.4 |
| 134 | 0.3.0 | | **Name in `bin/shell_command.py`'s `segments` the shell forms it does not parse, so a caller knows what it misses.** | 1 | 2 | 2 | 3 | 4 | 1.1 | 12.4 |
| 140 | 0.3.0 | | **Split `docs/decisions.md` → `routing: one ladder` into one entry per decision, so a reader finds the rules for the ladder, the worker's rung, the routing figures and delegation by their titles.** | 1 | 2 | 1 | 3 | 4 | 1.1 | 12.4 |
| 114 | 0.3.0 | decision | **Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.** | 3 | 4 | 4 | 3 | 5 | 1.1 | 11.9 |
| 114 | 0.3.0 | decision | **Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost`, `stability_replay` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.** | 3 | 4 | 4 | 3 | 5 | 1.1 | 11.9 |
| 117 | 0.3.0 | | **Give `todo_table.items` one way to say a plan has no items table, so no caller tells it from an empty plan by truthiness.** | 2 | 2 | 3 | 3 | 4 | 1.1 | 11.4 |
| 124 | 0.3.0 | decision | **Group the shared modules at `bin/`'s root into packages named by purpose, so the root holds only launchers and stand-alone tools.** | 3 | 6 | 3 | 3 | 5 | 1.1 | 11.4 |
| 130 | 0.3.0 | | **Move the vote counting out of `bin/larv_review/ruling.py`'s `votes`, so reading a votes file decides nothing.** | 2 | 3 | 3 | 3 | 4 | 1.1 | 11.2 |
@@ -181,11 +181,11 @@ Junior A and Mid A of PR #257's scoring run named `bin/shell_command.py:31` `seg
The entry holds 7 paragraphs under one title of about 70 words chaining three decisions: the ladder and bands, the worker below the project's rung with its reviewers, the routing figures and calls, delegation, and when the worker changes rung. A reader after one rule reads the whole entry. The prose pass of the chunk that split out the judge's rung asked for the rest. Once split, the delegation entry's title names the floor that `todo.md: only a session or subagent on the project's rung or above scores or reshapes the plan…` puts under a subagent that files or scores items.
### 114. Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.
### 114. Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost`, `stability_replay` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.
Shape is the lowest dimension, so this item goes before 0.3.0's other items, per `docs/decisions.md` → `larv-review: a project lifting itself to the comprehension floor starts at the lowest dimension`.
The comprehension seats on PR #254 found five git runners with different contracts: `git_repo.git`, `rule_hook/git.py`'s guarded one, `metric_bench/main.py`'s, `reading_cost/main.py`'s `located` and `loop_claude/effects.py`'s. The inherited architect of PR #257's scoring run named `bin/git_repo.py:8` `git` again. Make `git_repo.git` easier to read.
The comprehension seats on PR #254 found five git runners with different contracts: `git_repo.git`, `rule_hook/git.py`'s guarded one, `metric_bench/main.py`'s, `reading_cost/main.py`'s `located` and `loop_claude/effects.py`'s. PR #276 added a sixth, `stability_replay/main.py`'s `git`. The inherited architect of PR #257's scoring run named `bin/git_repo.py:8` `git` again. Make `git_repo.git` easier to read.
### 117. Give `todo_table.items` one way to say a plan has no items table, so no caller tells it from an empty plan by truthiness.