Answer architecture round 1: 1 🔴, 2 🟡; share one rung search between both replays, searching down from the project's rung, name the sixth git runner in item 114, decline recording the reviewed commit
This commit is contained in:
@@ -53,6 +53,8 @@
|
||||
!bin/routing_figures/*.py
|
||||
!bin/rule_hook/
|
||||
!bin/rule_hook/*.py
|
||||
!bin/rung_replay/
|
||||
!bin/rung_replay/*.py
|
||||
!bin/score-outcomes
|
||||
!bin/score_outcomes/
|
||||
!bin/score_outcomes/*.py
|
||||
|
||||
+2
-2
@@ -2,11 +2,11 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- The new `bin/stability-replay <model>` replays archived stability reviews of the projects run on the model, each on the code it reviewed, first on the project's rung and then on each rung below, and prints the lowest rung that finds every 🔴 and 🟡 finding the project's rung finds. Its first run, on Opus 5.5 projects, found none: Sonnet 5.5 at high missed 13 of 29. So the stability reviewer stays on the project's rung while the worker runs below it.
|
||||
- The new `bin/stability-replay <model>` replays archived stability reviews of the projects run on the model, each on the code it reviewed, first on the project's rung and then on each rung below, and prints the lowest rung that, like each rung above it, finds every 🔴 and 🟡 finding the project's rung finds. It shares its search with `judge-replay`, which now also tries rungs from the project's rung down. Its first run, on Opus 5.5 projects, found none: Sonnet 5.5 at high missed 13 of 29. So the stability reviewer stays on the project's rung while the worker runs below it.
|
||||
- A `loop-claude` worker hands a task that proves too hard for its rung to a subagent one rung up, edits included, when a trigger fires: a re-score of the item on the project's rung or above picks a higher rung, the same failing test or finding survives two fix attempts, or a review pass reports a 🔴 in round 2 or later. Each further trigger climbs one more rung, to at most two rungs above the rung the chunk's `todo.md` item picks, unless you agree. `CLAUDE.md` states the rule, and `loop-claude`'s prompt names the rungs a trigger opens. The worker opens the subagent's prompt with `Escalation trigger <n>:`, and `routing-figures.md` counts the escalations by trigger. Expect a chunk that gets stuck to spend more on the stuck part, and fewer unfinished verdicts.
|
||||
- A reviewer that rules a chunk's work below standard, with systems-architect's BLOCK or a finding against larv-review's `standards/quality.md`, may ask to rerun one rung up, as may a comprehension seat whose vote in a paired ruling `record-ruling` counts as worse. The worker relaunches it once, on the rung `loop-claude`'s prompt names. A relaunched reviewer gets the report that asked and confirms or dismisses each of its findings. The relaunch's report stands in place of the one that asked, in the same round, so it adds no round. The ruling that stands is a fourth escalation trigger in `CLAUDE.md`. Expect a chunk whose work falls below standard to spend more on its review and its fix.
|
||||
- A session or subagent below the project's rung, the cheapest rung of the project's model, no longer scores or reshapes `todo.md`: it drafts each new item and each change to a filed item, and before its next commit one subagent on the project's rung or above applies and scores the drafts and sets their release. It may still remove finished items. `rules/todo-items.md` states the rule. `CLAUDE.md` has a session launch a subagent that files or scores items on the rung the new `bin/subagent-rung --plan <model> <R+C>` prints, never below the project's rung, and `loop-claude`'s prompt to a worker below that rung names the command. Expect one extra subagent in a chunk that files items while its worker runs below the project's rung.
|
||||
- `loop-claude` runs the judge of an Opus 5.5 project on the rung the chunk's `todo.md` item picks, never below Opus 5.5 at medium effort, the project's rung. Before, the judge ran at the effort `settings.json` sets, xhigh. A replay of 39 archived verdicts found every cheaper rung missing verdicts. The judge keeps its rung when the worker steps up or takes in another item, and runs on the project's rung when no item routed the chunk. A project on another model keeps its judge at its settings' effort. Each verdict in the archive now names the judge's rung on a `Judge:` line. The new `bin/judge-replay <model>` replays the judge on archived verdicts and prints the cheapest rung that returns them. Expect the judge to cost less on items of R + C 12 or less, and more on items of R + C 15 or more.
|
||||
- `loop-claude` runs the judge of an Opus 5.5 project on the rung the chunk's `todo.md` item picks, never below Opus 5.5 at medium effort, the project's rung. Before, the judge ran at the effort `settings.json` sets, xhigh. A replay of 39 archived verdicts found every cheaper rung missing verdicts. The judge keeps its rung when the worker steps up or takes in another item, and runs on the project's rung when no item routed the chunk. A project on another model keeps its judge at its settings' effort. Each verdict in the archive now names the judge's rung on a `Judge:` line. The new `bin/judge-replay <model>` replays the judge on archived verdicts and prints the lowest rung that, like each rung above it, returns them. Expect the judge to cost less on items of R + C 12 or less, and more on items of R + C 15 or more.
|
||||
- Every 20 cycles across all projects, `loop-claude` measures every finished chunk in its archive, with no model run, and writes the figures to `routing-figures.md` at the archive's root: tokens by project, subagent type, rung and tool, and, for each chunk a `todo.md` item routed, grouped by the rung it started on: its review rounds past each pass's first, its unfinished verdicts and its subagents' rungs. It interrupts you, with `claude` on the project's model on the terminal in `~/.claude`, when chunks routed onto a rung took more review rounds or unfinished verdicts than chunks routed onto the next rung up, each rung with 5 chunks or more, or when 5 or more subagents of one type wrote reports longer than the tool results they read. Each call is raised once, then listed in `routing-calls-raised.md` beside the figures; the next chunk of this repo files a `todo.md` item for it. Off a terminal, the calls are said on stderr and the loop goes on. They are raised at the next measurement on a terminal. After a routing change, a `- <UTC time> Reset` line in that file makes the figures count only chunks finished after it, and lets its calls be raised again. Expect a pause of about ten seconds at each measurement.
|
||||
- A session delegates a task to a subagent only when it needs just the task's conclusion, such as a search across many files, and keeps edits in its own thread. It scores the task's R and C as for a `todo.md` item, and launches the subagent on the model and effort that the new `bin/subagent-rung <model> <R+C>` picks from the project's model, such as `model sonnet, effort high`. `CLAUDE.md` states the rule. Expect searches to run on cheaper models, and less of each session's context to fill with search output.
|
||||
- `loop-claude` routes each chunk's worker on a ladder of model and effort rungs, from Haiku 5.5 at low effort to Fable 5.1 at max, when the project's model is on it. As a chunk starts, the R + C of `todo.md`'s top item in the next release picks the rung: 11–12 picks one rung above the project's rung, so Opus 5.5 at high for an Opus 5.5 project, and so on up to four rungs. An item of R + C 10 or less runs the worker at or below the project's model's cheapest rung, which for Opus 5.5 is medium effort, below the xhigh `settings.json` sets. While the worker runs below the project's model's cheapest rung, it launches every reviewer, comprehension seat and security pass, the full scan included, in a subagent on that rung. While it runs above that rung, it launches each comprehension seat and the security pass on that rung. It steps up one rung after every second unfinished verdict on a chunk, and to a higher rung when it takes in an item of higher R + C. Each cycle's header, and each finished or unfinished verdict, stop and step-up in the archive, name the worker's rung; a rerun resumes an open chunk on the latest rung archived for it. `docs/decisions.md` → `routing:` gives the ladder, its source and the bands. Expect a chunk on an easy item to cost much less than under xhigh, and perhaps to take more review rounds, and one on an item of R + C 15 or more to cost more.
|
||||
|
||||
@@ -67,7 +67,7 @@ Rename `## Unreleased` in `CHANGELOG.md` to the version, commit, tag that commit
|
||||
|
||||
Install puts only `loop-claude` on `PATH`; call the others by their path, such as `~/.claude/bin/metric-bench`, from the project's checkout.
|
||||
|
||||
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints the cheapest rung that returns them; `judge-replay --help` states the replay.
|
||||
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints the lowest rung that, like every rung between it and the project's, returns them; `judge-replay --help` states the replay.
|
||||
- `larv-review <command>` holds larv-review's mechanical rules: the paired ruling's base and vote count, each pass's round count, the scoring run's trigger, and the comprehension history's format and latest places; `larv-review --help` states them.
|
||||
- `loop-claude <task>` runs a project's next chunk cycle after cycle; `loop-claude --help` states its contract.
|
||||
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
|
||||
@@ -94,7 +94,7 @@ pytest_add_cli_args=-k
|
||||
|
||||
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
|
||||
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and `stability_replay` imports `loop_claude`'s retry and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and both it and `stability_replay` import `rung_replay/`: their shared rung search and command steps, which import `loop_claude`'s retry count and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
|
||||
## Tracking a new file
|
||||
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
"""One `claude -p --output-format json` run, as loop-claude's judge and the replays start it."""
|
||||
import dataclasses
|
||||
import subprocess
|
||||
import typing
|
||||
|
||||
from failure import Failure
|
||||
from claude_code.entries import hit_limit, limit_said, parse_entry
|
||||
|
||||
# Effects.claude: claude's arguments, its working directory and the prompt it reads on stdin.
|
||||
Claude = typing.Callable[[list[str], str, str], subprocess.CompletedProcess[str]]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Limited:
|
||||
"""claude hit a usage limit; said is claude's message about it, and resets_at is when the limit resets, in seconds since the epoch, or None when unknown."""
|
||||
resets_at: int | None
|
||||
said: str
|
||||
|
||||
|
||||
def claude_reply(claude: Claude, args: list[str], cwd: str, prompt: str, who: str) -> dict | Failure | Limited:
|
||||
"""claude's json reply, or why there is none; who names the run in the message for a non-zero exit, such as the judge."""
|
||||
try:
|
||||
proc = claude(args, cwd, prompt)
|
||||
except OSError as failure:
|
||||
return Failure(f"could not start claude: {failure.strerror}")
|
||||
reply = parse_entry(proc.stdout) or {}
|
||||
# The json output names the reset only in prose, such as "resets 7:40pm (Europe/Stockholm)", so its limit carries no reset time.
|
||||
if hit_limit(reply):
|
||||
return Limited(None, limit_said(reply))
|
||||
if proc.returncode != 0:
|
||||
return Failure(proc.stderr.strip() or f"{who} exited {proc.returncode}")
|
||||
return reply
|
||||
@@ -1,11 +1,13 @@
|
||||
import json
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from claude_code import entries, transcript, trust
|
||||
from claude_code import entries, reply, transcript, trust
|
||||
from failure import Failure
|
||||
|
||||
|
||||
def assistant(*parts):
|
||||
@@ -80,6 +82,25 @@ class Parse(unittest.TestCase):
|
||||
self.assertIsNone(entries.parse_entry("[1, 2]"))
|
||||
|
||||
|
||||
class ClaudeReply(unittest.TestCase):
|
||||
@staticmethod
|
||||
def claude(*outcomes):
|
||||
def run(args, cwd, prompt):
|
||||
outcome = outcomes[0]
|
||||
if isinstance(outcome, OSError):
|
||||
raise outcome
|
||||
return subprocess.CompletedProcess(args, *outcome)
|
||||
return run
|
||||
|
||||
def test_returns_the_json_reply_a_usage_limit_whatever_the_exit_code_or_why_there_is_none(self):
|
||||
self.assertEqual(reply.claude_reply(self.claude((0, '{"result": "ok"}', "")), [], "/r", "p", "the judge"), {"result": "ok"})
|
||||
limit = json.dumps({"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
|
||||
self.assertEqual(reply.claude_reply(self.claude((1, limit, "")), [], "/r", "p", "the judge"), reply.Limited(None, "You've hit your limit"))
|
||||
self.assertEqual(reply.claude_reply(self.claude((2, "", " ")), [], "/r", "p", "the judge"), Failure("the judge exited 2"))
|
||||
self.assertEqual(reply.claude_reply(self.claude((2, "", "boom\n")), [], "/r", "p", "the judge"), Failure("boom"))
|
||||
self.assertEqual(reply.claude_reply(self.claude(FileNotFoundError(2, "No such file or directory")), [], "/r", "p", "claude"), Failure("could not start claude: No such file or directory"))
|
||||
|
||||
|
||||
class MarkTrusted(unittest.TestCase):
|
||||
def test_returns_the_updated_config_and_leaves_the_one_it_was_handed(self):
|
||||
config = {"other": 1, "projects": {"/a": {"hasTrustDialogAccepted": True}, "/b": {"kept": True}}}
|
||||
|
||||
+27
-53
@@ -1,4 +1,4 @@
|
||||
"""Replay loop-claude's judge on archived verdicts, to find the cheapest rung that returns them: docs/decisions.md → `routing: on a model judge-replay covered`.
|
||||
"""Replay loop-claude's judge on archived verdicts, to find the lowest rung that returns them: docs/decisions.md → `routing: on a model judge-replay covered`.
|
||||
|
||||
usage: judge-replay <model>
|
||||
|
||||
@@ -11,104 +11,78 @@ with the task, premises and records each project holds now.
|
||||
It first replays every verdict on the reference judge: the model at xhigh effort, the rung the
|
||||
judge ran on before loop-claude routed it. A verdict the reference judge returns another kind on
|
||||
may rest on records or judge rules that have changed since; either way no rung is held to it. It
|
||||
then replays the rest on each candidate rung, cheapest first: from the rung the lowest R + C picks
|
||||
up to the model's cheapest rung. It stops at the first rung that returns all of them. It prints
|
||||
each verdict the reference judge did not reproduce, each verdict a rung missed, and the cheapest
|
||||
rung that missed none. A rung it names is the evidence to add the model to JUDGE_REPLAYED in
|
||||
bin/loop_claude/routing.py, which routes that model's judge, and to record the run in
|
||||
docs/decisions.md → `routing: on a model judge-replay covered`.
|
||||
then replays the rest on each candidate rung, highest first: from the model's cheapest rung down
|
||||
to the rung the lowest R + C picks. It stops at the first rung that misses one. It prints each
|
||||
verdict the reference judge did not reproduce, each verdict a rung missed, and the lowest rung
|
||||
that, like each rung above it, missed none. A rung it names is the evidence to add the model to
|
||||
JUDGE_REPLAYED in bin/loop_claude/routing.py, which routes that model's judge, and to record the
|
||||
run in docs/decisions.md → `routing: on a model judge-replay covered`.
|
||||
|
||||
Like the loop, it tries a judge three times running before it counts the reply as failed. A
|
||||
verdict whose project or transcript cannot be read replays as failed too. Exit 1, saying why, on
|
||||
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
|
||||
or a usage limit.
|
||||
"""
|
||||
import concurrent.futures
|
||||
import pathlib
|
||||
import sys
|
||||
import typing
|
||||
|
||||
from transcript_archive.location import archive_root, transcript_of
|
||||
from transcript_archive.read import archived_outcomes
|
||||
from judge_replay.replay import Case, case_from, Kind, replayed, report, sample, verdict_kind
|
||||
from judge_replay.replay import Case, case_from, Kind, report, reproduced, sample, verdict_kind
|
||||
from project_key import checkout_of
|
||||
from rung_replay.command import down_from, failed, limited_exit, project_rung_or_exit, projects_on
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.retry import first_reply, Limited
|
||||
from rung_replay.search import first_reply, replayed
|
||||
from loop_claude.cycle.judge import judge_once, Verdict
|
||||
from model_routing.ladder import LADDER, picked, position, project_rung, Rung
|
||||
from loop_claude.project.model_choice import read_models_file
|
||||
from claude_code.reply import Limited
|
||||
from loop_claude.project.premises import read_premises
|
||||
from loop_claude.project.records import read_records
|
||||
from claude_code.transcript import read_transcript, render
|
||||
from model_routing.ladder import Rung
|
||||
from loop_claude.stop import Stop
|
||||
|
||||
TOOL = "judge-replay"
|
||||
PER_KIND = 15
|
||||
# The effort settings.json gave every judge before loop-claude routed it; the archived verdicts came from it.
|
||||
REFERENCE_EFFORT = "xhigh"
|
||||
# Judges replayed at once.
|
||||
AT_ONCE = 4
|
||||
LOWEST_R_PLUS_C = 2
|
||||
USAGE = "usage: judge-replay <model>, a model on the routing ladder"
|
||||
|
||||
|
||||
def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, effects: Effects) -> int:
|
||||
if argv[:1] in (["-h"], ["--help"]):
|
||||
print(__doc__.strip())
|
||||
return 0
|
||||
anchor = project_rung(argv[0]) if len(argv) == 1 else None
|
||||
if anchor is None:
|
||||
print(f"judge-replay: cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; {USAGE}", file=sys.stderr)
|
||||
return 1
|
||||
cases = archived_cases(argv[0], archive_root(home, env), home)
|
||||
anchor = project_rung_or_exit(TOOL, argv, __doc__)
|
||||
if isinstance(anchor, int):
|
||||
return anchor
|
||||
cases = archived_cases(anchor.model, archive_root(home, env), home)
|
||||
if isinstance(cases, Failure):
|
||||
print(f"judge-replay: {cases.reason}", file=sys.stderr)
|
||||
return 1
|
||||
return failed(TOOL, cases.reason)
|
||||
if not cases:
|
||||
print(f"judge-replay: no archived verdicts of projects on {argv[0]} in {archive_root(home, env)}; run loop-claude on a project on that model first", file=sys.stderr)
|
||||
return 1
|
||||
return failed(TOOL, f"no archived verdicts of projects on {anchor.model} in {archive_root(home, env)}; run loop-claude on a project on that model first")
|
||||
cases = sample(cases, PER_KIND)
|
||||
reference = Rung(argv[0], REFERENCE_EFFORT)
|
||||
candidates = list(LADDER[position(picked(anchor, LOWEST_R_PLUS_C)):position(anchor) + 1])
|
||||
replay = replayed(cases, reference, candidates, lambda on, rung: rule_all(on, rung, home / ".claude", effects))
|
||||
reference = Rung(anchor.model, REFERENCE_EFFORT)
|
||||
replay = replayed(cases, reference, down_from(anchor), lambda found, rung: rule(found, rung, home / ".claude", effects), reproduced, "verdicts")
|
||||
if isinstance(replay, Limited):
|
||||
print(f"judge-replay: claude hit a usage limit: {replay.said}; replay again once it resets", file=sys.stderr)
|
||||
return 1
|
||||
print(report(argv[0], reference, cases, replay))
|
||||
return limited_exit(TOOL, replay)
|
||||
print(report(anchor.model, reference, cases, replay))
|
||||
return 0
|
||||
|
||||
|
||||
def archived_cases(model: str, root: pathlib.Path, home: pathlib.Path) -> list[Case] | Failure:
|
||||
models = read_models_file(home / ".claude" / "loop-claude-models.json")
|
||||
if isinstance(models, Stop):
|
||||
return Failure(models.reason)
|
||||
projects = projects_on(model, home)
|
||||
if isinstance(projects, Failure):
|
||||
return projects
|
||||
listed = archived_outcomes(root)
|
||||
if isinstance(listed, Failure):
|
||||
return listed
|
||||
found = []
|
||||
try:
|
||||
for key, outcomes in listed:
|
||||
case = case_from(checkout_of(key, home), outcomes.read_text(encoding="utf-8"), transcript_of(outcomes)) if models.get(key) == model else None
|
||||
case = case_from(checkout_of(key, home), outcomes.read_text(encoding="utf-8"), transcript_of(outcomes)) if key in projects else None
|
||||
found += [case] if case else []
|
||||
except (OSError, UnicodeDecodeError) as failure:
|
||||
return Failure(f"could not read the archive {root}: {failure}")
|
||||
return found
|
||||
|
||||
|
||||
def rule_all(cases: list[Case], rung: Rung, claude_dir: pathlib.Path, effects: Effects) -> list[Kind] | Limited:
|
||||
print(f"replaying {len(cases)} verdicts on {rung}", file=sys.stderr, flush=True)
|
||||
with concurrent.futures.ThreadPoolExecutor(AT_ONCE) as pool:
|
||||
running = [pool.submit(rule, found, rung, claude_dir, effects) for found in cases]
|
||||
kinds = []
|
||||
for future in running:
|
||||
kind = future.result()
|
||||
if isinstance(kind, Limited):
|
||||
pool.shutdown(cancel_futures=True)
|
||||
return kind
|
||||
kinds.append(kind)
|
||||
return kinds
|
||||
|
||||
|
||||
def rule(case: Case, rung: Rung, claude_dir: pathlib.Path, effects: Effects) -> Kind | Limited:
|
||||
premises, records, entries = read_premises(case.root), read_records(case.root, claude_dir), read_transcript(str(case.transcript))
|
||||
if isinstance(premises, Stop):
|
||||
|
||||
+12
-40
@@ -1,4 +1,4 @@
|
||||
"""Which archived verdicts to replay, the cheapest rung that returns them, and the report: docs/decisions.md → `routing: on a model judge-replay covered`."""
|
||||
"""Which archived verdicts to replay, what a replay reproduces, and the report: docs/decisions.md → `routing: on a model judge-replay covered`."""
|
||||
import collections
|
||||
import dataclasses
|
||||
import pathlib
|
||||
@@ -7,7 +7,7 @@ import typing
|
||||
from transcript_archive.outcomes import ANSWERED, FINISHED, first_verdict, INTERRUPTED, task_of, UNFINISHED
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict
|
||||
from loop_claude.cycle.retry import Limited
|
||||
from rung_replay.search import Replay
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
# claude's reply as the replay compares it: a verdict as the archive words it, such as FINISHED, or a failure with its reason.
|
||||
@@ -24,21 +24,6 @@ class Case:
|
||||
on_record: Kind
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Tried:
|
||||
"""A rung, and each case it returned another kind on, with that kind."""
|
||||
rung: Rung
|
||||
missed: list[tuple[Case, Kind]]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Replay:
|
||||
"""unreproduced holds each case the reference judge returned another kind on; tried, each rung tried, cheapest first; cheapest, the first rung that missed none, or None."""
|
||||
unreproduced: list[tuple[Case, Kind]]
|
||||
tried: list[Tried]
|
||||
cheapest: Rung | None
|
||||
|
||||
|
||||
def case_from(root: str, outcomes: str, transcript: pathlib.Path) -> Case | None:
|
||||
"""The first verdict an outcomes file holds: the judge ruled on the transcript copy beside it as the copy stands. None when it holds none.
|
||||
|
||||
@@ -77,39 +62,26 @@ def verdict_kind(verdict: Verdict | Failure) -> Kind:
|
||||
typing.assert_never(verdict)
|
||||
|
||||
|
||||
def replayed(cases: list[Case], reference: Rung, candidates: list[Rung], rule_all: typing.Callable[[list[Case], Rung], list[Kind] | Limited]) -> Replay | Limited:
|
||||
"""Replays the cases on the reference rung, then only those it reproduces on each candidate, cheapest first, until one returns every verdict."""
|
||||
ruled = rule_all(cases, reference)
|
||||
if isinstance(ruled, Limited):
|
||||
return ruled
|
||||
reproduced = [found for found, kind in zip(cases, ruled) if kind == found.on_record]
|
||||
unreproduced = [(found, kind) for found, kind in zip(cases, ruled) if kind != found.on_record]
|
||||
tried: list[Tried] = []
|
||||
# With no verdict to hold a rung to, every rung would miss none.
|
||||
for rung in candidates if reproduced else []:
|
||||
ruled = rule_all(reproduced, rung)
|
||||
if isinstance(ruled, Limited):
|
||||
return ruled
|
||||
tried.append(Tried(rung, [(found, kind) for found, kind in zip(reproduced, ruled) if kind != found.on_record]))
|
||||
if not tried[-1].missed:
|
||||
return Replay(unreproduced, tried, rung)
|
||||
return Replay(unreproduced, tried, None)
|
||||
def reproduced(case: Case, kind: Kind) -> list[Kind]:
|
||||
"""The kind on record, where the reply returns it: a rung is held only to a verdict the reference judge reproduces."""
|
||||
return [kind] if kind == case.on_record else []
|
||||
|
||||
|
||||
def report(model: str, reference: Rung, cases: list[Case], replay: Replay) -> str:
|
||||
def report(model: str, reference: Rung, cases: list[Case], replay: Replay[Case, Kind, Kind]) -> str:
|
||||
kinds = collections.Counter(found.on_record for found in cases)
|
||||
unreproduced = [(found, kind) for found, kind in zip(cases, replay.on_reference) if not reproduced(found, kind)]
|
||||
lines = [
|
||||
f"Replayed {len(cases)} verdicts of {model} projects: {', '.join(f'{count} {kind}' for kind, count in sorted(kinds.items()))}.",
|
||||
f"The reference judge, {reference}, reproduced {len(cases) - len(replay.unreproduced)}{'. It returned another kind on:' if replay.unreproduced else '.'}",
|
||||
*missed_lines(replay.unreproduced),
|
||||
f"The reference judge, {reference}, reproduced {len(cases) - len(unreproduced)}{'. It returned another kind on:' if unreproduced else '.'}",
|
||||
*missed_lines(unreproduced),
|
||||
]
|
||||
for tried in replay.tried:
|
||||
lines += [f"{tried.rung} missed {len(tried.missed)}{':' if tried.missed else '.'}", *missed_lines(tried.missed)]
|
||||
lines += [f"{tried.rung} missed {len(tried.missed)}{':' if tried.missed else '.'}", *missed_lines([(miss.case, miss.reply) for miss in tried.missed])]
|
||||
if not replay.tried:
|
||||
lines.append("No rung tried: the reference judge reproduced no verdict to hold one to.")
|
||||
else:
|
||||
cheapest = replay.cheapest or "none up to the project's rung"
|
||||
lines.append(f"Cheapest rung that returns every verdict the reference judge reproduces: {cheapest}")
|
||||
lowest = replay.lowest or "none, the project's rung misses one"
|
||||
lines.append(f"Lowest rung that, like each rung above it, returns every verdict the reference judge reproduces: {lowest}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
|
||||
@@ -10,12 +10,12 @@ import unittest
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINISHED
|
||||
from judge_replay.replay import Case, case_from, Replay, replayed, report, sample, Tried, verdict_kind
|
||||
from judge_replay.replay import Case, case_from, report, reproduced, sample, verdict_kind
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished
|
||||
from loop_claude.cycle.retry import Limited
|
||||
from judge_replay import main
|
||||
from rung_replay.search import Miss, Replay, Tried
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
TOOL = pathlib.Path(__file__).resolve().parents[1] / "judge-replay"
|
||||
@@ -59,52 +59,29 @@ class VerdictKind(unittest.TestCase):
|
||||
self.assertEqual(verdict_kind(Failure("the judge quoted x")), "failed: the judge quoted x")
|
||||
|
||||
|
||||
class Replayed(unittest.TestCase):
|
||||
def test_tries_rungs_cheapest_first_on_the_verdicts_the_reference_reproduces_and_stops_at_the_first_that_returns_all(self):
|
||||
cases = [case("2026-10-14T10:00:00Z", FINISHED), case("2026-10-14T10:00:01Z", INTERRUPTED), case("2026-10-14T10:00:02Z", FINISHED)]
|
||||
replies = {REFERENCE: [FINISHED, INTERRUPTED, UNFINISHED], CHEAP: [FINISHED, FINISHED], MIDDLE: [FINISHED, INTERRUPTED]}
|
||||
asked = []
|
||||
|
||||
def rule_all(on, rung):
|
||||
asked.append((len(on), rung))
|
||||
return replies[rung]
|
||||
|
||||
replay = replayed(cases, REFERENCE, [CHEAP, MIDDLE, Rung("claude-opus-5-5", "medium")], rule_all)
|
||||
self.assertEqual(replay, Replay([(cases[2], UNFINISHED)], [Tried(CHEAP, [(cases[1], FINISHED)]), Tried(MIDDLE, [])], MIDDLE))
|
||||
self.assertEqual(asked, [(3, REFERENCE), (2, CHEAP), (2, MIDDLE)])
|
||||
|
||||
def test_names_no_rung_when_every_candidate_misses_a_verdict(self):
|
||||
cases = [case("2026-10-14T10:00:00Z", FINISHED)]
|
||||
replay = replayed(cases, REFERENCE, [CHEAP], lambda on, rung: [FINISHED] if rung == REFERENCE else [INTERRUPTED])
|
||||
self.assertEqual(replay, Replay([], [Tried(CHEAP, [(cases[0], INTERRUPTED)])], None))
|
||||
|
||||
def test_tries_no_rung_when_the_reference_reproduces_no_verdict(self):
|
||||
cases = [case("2026-10-14T10:00:00Z", FINISHED)]
|
||||
self.assertEqual(replayed(cases, REFERENCE, [CHEAP], lambda on, rung: [INTERRUPTED] * len(on)), Replay([(cases[0], INTERRUPTED)], [], None))
|
||||
|
||||
def test_stops_at_a_usage_limit(self):
|
||||
limit = Limited(None, "limit")
|
||||
self.assertEqual(replayed([case("2026-10-14T10:00:00Z", FINISHED)], REFERENCE, [CHEAP], lambda on, rung: limit), limit)
|
||||
self.assertEqual(replayed([case("2026-10-14T10:00:00Z", FINISHED)], REFERENCE, [CHEAP], lambda on, rung: [FINISHED] if rung == REFERENCE else limit), limit)
|
||||
class Reproduced(unittest.TestCase):
|
||||
def test_holds_a_rung_to_a_verdict_only_where_the_reference_returned_the_kind_on_record(self):
|
||||
self.assertEqual(reproduced(case("2026-10-14T10:00:00Z", FINISHED), FINISHED), [FINISHED])
|
||||
self.assertEqual(reproduced(case("2026-10-14T10:00:00Z", FINISHED), INTERRUPTED), [])
|
||||
|
||||
|
||||
class Report(unittest.TestCase):
|
||||
def test_counts_the_verdicts_and_names_each_miss_and_the_cheapest_rung(self):
|
||||
def test_counts_the_verdicts_and_names_each_miss_and_the_lowest_rung(self):
|
||||
cases = [case("2026-10-14T10:00:00Z", FINISHED), case("2026-10-14T10:00:01Z", INTERRUPTED)]
|
||||
replay = Replay([(cases[0], UNFINISHED)], [Tried(CHEAP, [(cases[1], FINISHED)]), Tried(MIDDLE, [])], MIDDLE)
|
||||
replay = Replay([UNFINISHED, INTERRUPTED], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(cases[1], INTERRUPTED, FINISHED)])], MIDDLE)
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, replay).splitlines(), [
|
||||
"Replayed 2 verdicts of claude-opus-5-5 projects: 1 Verdict: finished, 1 Verdict: interrupt.",
|
||||
"The reference judge, claude-opus-5-5 at xhigh effort, reproduced 1. It returned another kind on:",
|
||||
" /code/p 2026-10-14T10:00:00Z: on record Verdict: finished, replayed Verdict: unfinished",
|
||||
"claude-sonnet-5-5 at high effort missed 0.",
|
||||
"claude-haiku-5-5 at xhigh effort missed 1:",
|
||||
" /code/p 2026-10-14T10:00:01Z: on record Verdict: interrupt, replayed Verdict: finished",
|
||||
"claude-sonnet-5-5 at high effort missed 0.",
|
||||
"Cheapest rung that returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort",
|
||||
"Lowest rung that, like each rung above it, returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort",
|
||||
])
|
||||
missed_all = report("claude-opus-5-5", REFERENCE, cases, Replay([], [Tried(CHEAP, [(cases[1], FINISHED)])], None)).splitlines()
|
||||
missed_all = report("claude-opus-5-5", REFERENCE, cases, Replay([FINISHED, INTERRUPTED], [Tried(MIDDLE, [Miss(cases[1], INTERRUPTED, FINISHED)])], None)).splitlines()
|
||||
self.assertEqual(missed_all[1], "The reference judge, claude-opus-5-5 at xhigh effort, reproduced 2.")
|
||||
self.assertEqual(missed_all[-1], "Cheapest rung that returns every verdict the reference judge reproduces: none up to the project's rung")
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, Replay([(cases[0], UNFINISHED), (cases[1], FINISHED)], [], None)).splitlines()[-1], "No rung tried: the reference judge reproduced no verdict to hold one to.")
|
||||
self.assertEqual(missed_all[-1], "Lowest rung that, like each rung above it, returns every verdict the reference judge reproduces: none, the project's rung misses one")
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, Replay([UNFINISHED, FINISHED], [], None)).splitlines()[-1], "No rung tried: the reference judge reproduced no verdict to hold one to.")
|
||||
|
||||
|
||||
def unused(*_):
|
||||
@@ -148,25 +125,25 @@ class Main(unittest.TestCase):
|
||||
output = {"answer": "", "bundled": 0, "evidence": evidence, "finished": not interrupt, "interrupt": interrupt, "question": "Which?" if interrupt else ""}
|
||||
return {"type": "result", "structured_output": output}
|
||||
|
||||
def test_replays_the_models_projects_on_the_reference_then_cheapest_first_and_prints_the_rung_that_returns_all(self):
|
||||
# Haiku misses the interrupt; Sonnet returns both.
|
||||
def test_replays_the_models_projects_on_the_reference_then_from_the_projects_rung_down_and_prints_the_lowest_that_returns_all(self):
|
||||
# Haiku misses the interrupt; Sonnet and Opus return both.
|
||||
def reply(rung, interrupted):
|
||||
return self.verdict(interrupted and rung.model != "claude-haiku-5-5", f"said {INTERRUPTED if interrupted else FINISHED}")
|
||||
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], reply)
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual({cwd for _, _, cwd in self.calls}, {str(self.home / "code" / "proj")})
|
||||
self.assertEqual([(model, effort) for model, effort, _ in self.calls[::2]], [("claude-opus-5-5", "xhigh"), ("claude-haiku-5-5", "xhigh"), ("claude-haiku-5-5", "max"), ("claude-sonnet-5-5", "high")])
|
||||
self.assertEqual([(model, effort) for model, effort, _ in self.calls[::2]], [("claude-opus-5-5", "xhigh"), ("claude-opus-5-5", "medium"), ("claude-sonnet-5-5", "high"), ("claude-haiku-5-5", "max")])
|
||||
self.assertIn("The reference judge, claude-opus-5-5 at xhigh effort, reproduced 2.", out)
|
||||
self.assertTrue(out.endswith("returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort\n"))
|
||||
for said in ("The task an agent was given:\nDo it.", "1. Ship.", "From ~/.claude/CLAUDE.md:\nBe kind."):
|
||||
self.assertIn(said, self.prompts[0])
|
||||
|
||||
def test_tries_rungs_up_to_the_models_cheapest_and_names_none_when_all_miss(self):
|
||||
def test_stops_at_the_projects_rung_and_names_none_when_it_misses(self):
|
||||
code, out, _ = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted and rung == REFERENCE, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
self.assertEqual(code, 0)
|
||||
self.assertEqual(self.calls[-1][:2], ("claude-opus-5-5", "medium"))
|
||||
self.assertTrue(out.endswith(": none up to the project's rung\n"))
|
||||
self.assertEqual({(model, effort) for model, effort, _ in self.calls[2:]}, {("claude-opus-5-5", "medium")})
|
||||
self.assertTrue(out.endswith(": none, the project's rung misses one\n"))
|
||||
|
||||
def test_replays_a_verdict_whose_transcript_is_gone_as_failed(self):
|
||||
(self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1" / "t0.jsonl").unlink()
|
||||
@@ -174,14 +151,16 @@ class Main(unittest.TestCase):
|
||||
self.assertEqual(code, 0)
|
||||
self.assertIn("replayed failed: could not read the transcript", out)
|
||||
|
||||
def test_tries_up_to_the_models_cheapest_rung_and_retries_a_failed_reply_as_the_loop_does(self):
|
||||
def test_tries_the_projects_rung_first_and_retries_a_failed_reply_as_the_loop_does(self):
|
||||
def reply(rung, interrupted):
|
||||
return self.verdict(interrupted, "not in the transcript" if rung.model != "claude-opus-5-5" else f"said {INTERRUPTED if interrupted else FINISHED}")
|
||||
|
||||
code, out, _ = self.run_main(["claude-opus-5-5"], reply)
|
||||
self.assertEqual(code, 0)
|
||||
self.assertEqual(sum(model == "claude-sonnet-5-5" for model, _, _ in self.calls), 6)
|
||||
self.assertNotIn("claude-haiku-5-5", {model for model, _, _ in self.calls})
|
||||
self.assertIn("claude-opus-5-5 at medium effort missed 0.", out)
|
||||
self.assertTrue(out.endswith(": claude-opus-5-5 at medium effort\n"))
|
||||
|
||||
def test_exits_1_on_a_usage_limit_bad_arguments_or_an_unreadable_models_file(self):
|
||||
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: {"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
|
||||
|
||||
@@ -4,13 +4,14 @@ import re
|
||||
import typing
|
||||
|
||||
from loop_claude.cycle.prompts import ANSWER_DESTINATIONS, ANSWERS_RECORDED, either
|
||||
from claude_code.reply import claude_reply, Limited
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from claude_code.entries import hit_limit, limit_said, parse_entry, structured_output
|
||||
from loop_claude.project.records import HUMAN_RULE_NAMES
|
||||
from loop_claude.cycle.retry import Limited, retried
|
||||
from loop_claude.cycle.retry import retried
|
||||
from model_routing.ladder import Rung
|
||||
from loop_claude.stop import Stop
|
||||
from claude_code.entries import structured_output
|
||||
from todo_table import TOP_SCORE
|
||||
|
||||
# At 3 to 4 characters a token, 100,000 to 133,000 tokens, which leaves a 200,000-token context room for the reply; the largest session rendered here was 364,000 characters on 2026-10-03.
|
||||
@@ -145,19 +146,12 @@ def judge_args(rung: Rung) -> list[str]:
|
||||
|
||||
|
||||
def run_judge(root: str, args: list[str], prompt: str, shown: str, quotable: str, effects: Effects) -> Verdict | Failure | Limited:
|
||||
try:
|
||||
proc = effects.claude(args, root, prompt)
|
||||
except OSError as failure:
|
||||
return Failure(f"could not start claude: {failure.strerror}")
|
||||
reply = parse_entry(proc.stdout) or {}
|
||||
# The json output names the reset only in prose, such as "resets 7:40pm (Europe/Stockholm)", so the judge's limit carries no reset time.
|
||||
if hit_limit(reply):
|
||||
return Limited(None, limit_said(reply))
|
||||
if proc.returncode != 0:
|
||||
return Failure(proc.stderr.strip() or f"the judge exited {proc.returncode}")
|
||||
reply = claude_reply(effects.claude, args, root, prompt, "the judge")
|
||||
if isinstance(reply, Failure | Limited):
|
||||
return reply
|
||||
verdict = verdict_from(structured_output(reply))
|
||||
if isinstance(verdict, Failure):
|
||||
return Failure(f"{verdict.reason}; it replied: {proc.stdout.strip() or 'nothing'}")
|
||||
return Failure(f"{verdict.reason}; it replied: {json.dumps(reply, ensure_ascii=False) if reply else 'nothing'}")
|
||||
if not grounded(verdict.evidence, shown):
|
||||
return Failure(f"the judge quoted {verdict.evidence!r}, which is not in the transcript" + (f", for the question: {verdict.question}" if isinstance(verdict, Interrupt | QuotedAnswer) else ""))
|
||||
# Not a Failure: a misquote would otherwise exit an unattended loop. Never elided: stitched fragments can say what the records do not.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
import dataclasses
|
||||
import sys
|
||||
import typing
|
||||
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from claude_code.reply import Limited
|
||||
from loop_claude.stop import Stop
|
||||
|
||||
T = typing.TypeVar("T")
|
||||
@@ -14,13 +14,6 @@ RETRY_DELAY = 60
|
||||
UNKNOWN_RESET_POLL = 600
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Limited:
|
||||
"""claude hit a usage limit; said is claude's message about it, and resets_at is when the limit resets, in seconds since the epoch, or None when unknown."""
|
||||
resets_at: int | None
|
||||
said: str
|
||||
|
||||
|
||||
def retried(who: str, attempt: typing.Callable[[], T | Failure | Limited], effects: Effects) -> T | Stop:
|
||||
failures = 0
|
||||
while True:
|
||||
@@ -48,13 +41,3 @@ def limit_wait(limited: Limited, now: float) -> float:
|
||||
return limited.resets_at + PAST_RESET - now
|
||||
return UNKNOWN_RESET_POLL
|
||||
|
||||
|
||||
def first_reply(attempt: typing.Callable[[], T | Failure | Limited]) -> T | Failure | Limited:
|
||||
"""The first reply that is no Failure, or the last of FAILURES_RUNNING failures: a replay's retry.
|
||||
|
||||
It counts failures as retried does, but neither waits between attempts nor waits out a usage limit, which ends a replay.
|
||||
"""
|
||||
for _ in range(FAILURES_RUNNING - 1):
|
||||
if not isinstance(reply := attempt(), Failure):
|
||||
return reply
|
||||
return attempt()
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
from claude_code.entries import assistant_parts, ends_stream, hit_limit, limit_said, parse_entry, rejected_until
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.retry import Limited
|
||||
from claude_code.reply import Limited
|
||||
from claude_code.transcript import part_line
|
||||
from model_routing.ladder import Rung
|
||||
from claude_code.session import Session
|
||||
|
||||
@@ -19,12 +19,12 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
from routing_figures import archive
|
||||
from loop_claude.project import checkout, model_choice, records
|
||||
from loop_claude.effects import Effects
|
||||
from claude_code import entries, session
|
||||
from claude_code import entries, reply, session
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished, verdict_from
|
||||
from loop_claude.cycle import interrupt, judge, prompts, retry, run, states, worker
|
||||
from model_routing.ladder import LADDER, Rung
|
||||
from loop_claude.cycle.retry import Limited
|
||||
from claude_code.reply import Limited
|
||||
from loop_claude.loop import main, reload, routing_check
|
||||
from transcript_archive import outcomes, write
|
||||
from loop_claude import paths, routing
|
||||
@@ -886,20 +886,22 @@ class ExitContract(unittest.TestCase):
|
||||
class JudgeFailures(unittest.TestCase):
|
||||
def test_every_failed_judge_run_has_its_clause_in_help(self):
|
||||
started, field, unasked = "claude cannot start or exits non-zero", "its reply misses or mistypes a field", "interrupts or quotes an answer without a question"
|
||||
# Every Failure the judge's run returns, in source order: path, function, start of reason, its --help clause.
|
||||
# Every Failure the judge's run returns, in source order of judge.py, then of the claude run it calls: function, start of reason, its --help clause.
|
||||
clauses = [
|
||||
("verdict_from", "'the judge returned no reply'", field),
|
||||
("verdict_from", 'f"the judge\'s reply has no {name}', field),
|
||||
("verdict_from", "'the judge interrupted without a question'", unasked),
|
||||
("verdict_from", "'the judge quoted an answer without a question'", unasked),
|
||||
("verdict_from", 'f"the judge\'s bundled R + C', "gives an unfinished verdict an R + C outside 0 to 20"),
|
||||
("run_judge", "f'could not start claude:", started),
|
||||
("run_judge", "proc.stderr.strip() or f'the judge exited", started),
|
||||
("run_judge", 'f"{verdict.reason}; it replied:', field),
|
||||
("run_judge", "f'the judge quoted {verdict.evidence!r}, which is not in the transcript", "quotes a passage the transcript does not hold"),
|
||||
("claude_reply", "f'could not start claude:", started),
|
||||
("claude_reply", "proc.stderr.strip() or f'{who} exited", started),
|
||||
]
|
||||
functions = [node for node in ast.walk(ast.parse(pathlib.Path(judge.__file__).read_text())) if isinstance(node, ast.FunctionDef)]
|
||||
found = sorted((call.lineno, function.name, ast.unparse(call.args[0])) for function in functions for call in ast.walk(function) if isinstance(call, ast.Call) and getattr(call.func, "id", None) == "Failure")
|
||||
found = []
|
||||
for module in (judge, reply):
|
||||
functions = [node for node in ast.walk(ast.parse(pathlib.Path(module.__file__).read_text())) if isinstance(node, ast.FunctionDef)]
|
||||
found += sorted((call.lineno, function.name, ast.unparse(call.args[0])) for function in functions for call in ast.walk(function) if isinstance(call, ast.Call) and getattr(call.func, "id", None) == "Failure")
|
||||
self.assertEqual(len(found), len(clauses), found)
|
||||
for (_, function, reason), (want_function, reason_start, _) in zip(found, clauses):
|
||||
self.assertEqual(function, want_function)
|
||||
|
||||
@@ -15,5 +15,5 @@ class Types(unittest.TestCase):
|
||||
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
|
||||
if shutil.which("uv") is None:
|
||||
self.fail("uv on PATH fetches mypy; see README.md's Test line")
|
||||
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "rule_hook", "-p", "score_outcomes", "-p", "stability_replay", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
|
||||
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "rule_hook", "-p", "rung_replay", "-p", "score_outcomes", "-p", "stability_replay", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
|
||||
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
"""What judge-replay's and stability-replay's commands share: the argument, the projects on its model, the candidate rungs and the exits."""
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
from failure import Failure
|
||||
from model_routing.ladder import LADDER, picked, position, project_rung, Rung
|
||||
from claude_code.reply import Limited
|
||||
from loop_claude.project.model_choice import read_models_file
|
||||
from loop_claude.stop import Stop
|
||||
|
||||
LOWEST_R_PLUS_C = 2
|
||||
|
||||
|
||||
def project_rung_or_exit(tool: str, argv: list[str], help_text: str) -> Rung | int:
|
||||
"""The project's rung of the model argv names, or the exit code: 0 after printing help_text for -h or --help, 1 after saying why argv cannot be read."""
|
||||
if argv[:1] in (["-h"], ["--help"]):
|
||||
print(help_text.strip())
|
||||
return 0
|
||||
anchor = project_rung(argv[0]) if len(argv) == 1 else None
|
||||
if anchor is None:
|
||||
return failed(tool, f"cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; usage: {tool} <model>, a model on the routing ladder")
|
||||
return anchor
|
||||
|
||||
|
||||
def projects_on(model: str, home: pathlib.Path) -> list[str] | Failure:
|
||||
"""The keys of the projects ~/.claude/loop-claude-models.json runs on model, sorted."""
|
||||
models = read_models_file(home / ".claude" / "loop-claude-models.json")
|
||||
if isinstance(models, Stop):
|
||||
return Failure(models.reason)
|
||||
return sorted(key for key, on in models.items() if on == model)
|
||||
|
||||
|
||||
def down_from(top: Rung) -> list[Rung]:
|
||||
"""top and each rung below it, down to the rung the lowest R + C picks from top, highest first."""
|
||||
return list(LADDER[position(picked(top, LOWEST_R_PLUS_C)):position(top) + 1])[::-1]
|
||||
|
||||
|
||||
def limited_exit(tool: str, limited: Limited) -> int:
|
||||
return failed(tool, f"claude hit a usage limit: {limited.said}; replay again once it resets")
|
||||
|
||||
|
||||
def failed(tool: str, reason: str) -> int:
|
||||
print(f"{tool}: {reason}", file=sys.stderr)
|
||||
return 1
|
||||
@@ -0,0 +1,101 @@
|
||||
"""The rung search judge-replay and stability-replay share: replay archived cases on a reference rung, then hold each candidate rung to what the reference reproduced."""
|
||||
import concurrent.futures
|
||||
import dataclasses
|
||||
import sys
|
||||
import typing
|
||||
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.retry import FAILURES_RUNNING
|
||||
from claude_code.reply import Limited
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
# Cases replayed at once.
|
||||
AT_ONCE = 4
|
||||
|
||||
Case = typing.TypeVar("Case")
|
||||
# What of a case's record a reply reproduces, such as a verdict's kind or a finding's index.
|
||||
Item = typing.TypeVar("Item")
|
||||
Reply = typing.TypeVar("Reply")
|
||||
T = typing.TypeVar("T")
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Miss(typing.Generic[Case, Item, Reply]):
|
||||
"""An item the reference reproduced in case that a rung's reply did not."""
|
||||
case: Case
|
||||
item: Item
|
||||
reply: Reply
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Tried(typing.Generic[Case, Item, Reply]):
|
||||
rung: Rung
|
||||
missed: list[Miss[Case, Item, Reply]]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Replay(typing.Generic[Case, Item, Reply]):
|
||||
"""on_reference holds the reference's reply on each case; tried, each candidate tried, highest first; lowest, the lowest rung that, like each candidate above it, missed nothing, or None."""
|
||||
on_reference: list[Reply]
|
||||
tried: list[Tried[Case, Item, Reply]]
|
||||
lowest: Rung | None
|
||||
|
||||
|
||||
def replayed(
|
||||
cases: list[Case],
|
||||
reference: Rung,
|
||||
candidates: list[Rung],
|
||||
replay_one: typing.Callable[[Case, Rung], Reply | Limited],
|
||||
reproduced: typing.Callable[[Case, Reply], list[Item]],
|
||||
what: str,
|
||||
) -> Replay[Case, Item, Reply] | Limited:
|
||||
"""Replays the cases on the reference rung, then each candidate, highest first, on the cases where the reference reproduced anything, until one misses an item it reproduced.
|
||||
|
||||
Going down, every rung at or above the one named was tried and missed nothing, so the result needs no assumption that a higher rung does at least as well; and a miss high up spares the cheaper rungs their replays.
|
||||
"""
|
||||
on_reference = replayed_on(cases, reference, replay_one, what)
|
||||
if isinstance(on_reference, Limited):
|
||||
return on_reference
|
||||
held = [(case, aim) for case, reply in zip(cases, on_reference) if (aim := reproduced(case, reply))]
|
||||
tried: list[Tried[Case, Item, Reply]] = []
|
||||
lowest = None
|
||||
# With nothing to hold a rung to, every rung would miss nothing.
|
||||
for rung in candidates if held else []:
|
||||
replies = replayed_on([case for case, _ in held], rung, replay_one, what)
|
||||
if isinstance(replies, Limited):
|
||||
return replies
|
||||
misses: list[Miss[Case, Item, Reply]] = []
|
||||
for (case, aim), reply in zip(held, replies):
|
||||
got = reproduced(case, reply)
|
||||
misses += [Miss(case, item, reply) for item in aim if item not in got]
|
||||
tried.append(Tried(rung, misses))
|
||||
if misses:
|
||||
break
|
||||
lowest = rung
|
||||
return Replay(on_reference, tried, lowest)
|
||||
|
||||
|
||||
def replayed_on(cases: list[Case], rung: Rung, replay_one: typing.Callable[[Case, Rung], Reply | Limited], what: str) -> list[Reply] | Limited:
|
||||
"""Each case's reply on rung, AT_ONCE at a time; a usage limit cancels the rest."""
|
||||
print(f"replaying {len(cases)} {what} on {rung}", file=sys.stderr, flush=True)
|
||||
with concurrent.futures.ThreadPoolExecutor(AT_ONCE) as pool:
|
||||
running = [pool.submit(replay_one, case, rung) for case in cases]
|
||||
replies: list[Reply] = []
|
||||
for future in running:
|
||||
reply = future.result()
|
||||
if isinstance(reply, Limited):
|
||||
pool.shutdown(cancel_futures=True)
|
||||
return reply
|
||||
replies.append(reply)
|
||||
return replies
|
||||
|
||||
|
||||
def first_reply(attempt: typing.Callable[[], T | Failure | Limited]) -> T | Failure | Limited:
|
||||
"""The first reply that is no Failure, or the last of FAILURES_RUNNING failures.
|
||||
|
||||
It counts failures as loop-claude's retried does, but neither waits between attempts nor waits out a usage limit, which ends a replay.
|
||||
"""
|
||||
for _ in range(FAILURES_RUNNING - 1):
|
||||
if not isinstance(reply := attempt(), Failure):
|
||||
return reply
|
||||
return attempt()
|
||||
@@ -0,0 +1,67 @@
|
||||
import contextlib
|
||||
import io
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from rung_replay.command import down_from
|
||||
from failure import Failure
|
||||
from rung_replay.search import first_reply, Miss, Replay, replayed, Tried
|
||||
from claude_code.reply import Limited
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
REFERENCE = Rung("claude-opus-5-5", "medium")
|
||||
MIDDLE = Rung("claude-sonnet-5-5", "high")
|
||||
CHEAP = Rung("claude-haiku-5-5", "max")
|
||||
LOWER = Rung("claude-haiku-5-5", "xhigh")
|
||||
|
||||
|
||||
def search(cases, candidates, replies):
|
||||
"""replayed on cases named by strings, where replies[rung][case] is the set of items a rung reproduces in the case."""
|
||||
asked = []
|
||||
|
||||
def replay_one(case, rung):
|
||||
asked.append((case, rung))
|
||||
reply = replies[rung]
|
||||
return reply if isinstance(reply, Limited) else reply[case]
|
||||
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
return replayed(cases, REFERENCE, candidates, replay_one, lambda _, reply: sorted(reply), "cases"), asked
|
||||
|
||||
|
||||
class Replayed(unittest.TestCase):
|
||||
def test_holds_each_candidate_highest_first_to_what_the_reference_reproduced_and_stops_at_the_first_that_misses(self):
|
||||
replies = {REFERENCE: {"a": {0, 1}, "b": set()}, MIDDLE: {"a": {0, 1}}, CHEAP: {"a": {1}}}
|
||||
replay, asked = search(["a", "b"], [MIDDLE, CHEAP, LOWER], replies)
|
||||
self.assertEqual(replay, Replay([{0, 1}, set()], [Tried(MIDDLE, []), Tried(CHEAP, [Miss("a", 0, {1})])], MIDDLE))
|
||||
self.assertEqual(sorted(asked, key=lambda pair: (pair[1] != REFERENCE, pair[0])), [("a", REFERENCE), ("b", REFERENCE), ("a", MIDDLE), ("a", CHEAP)])
|
||||
|
||||
def test_names_no_rung_when_the_first_misses_and_tries_none_when_the_reference_reproduced_nothing(self):
|
||||
self.assertEqual(search(["a"], [MIDDLE, CHEAP], {REFERENCE: {"a": {0}}, MIDDLE: {"a": set()}})[0], Replay([{0}], [Tried(MIDDLE, [Miss("a", 0, set())])], None))
|
||||
self.assertEqual(search(["a"], [MIDDLE], {REFERENCE: {"a": set()}}), (Replay([set()], [], None), [("a", REFERENCE)]))
|
||||
|
||||
def test_stops_at_a_usage_limit(self):
|
||||
limit = Limited(None, "limit")
|
||||
self.assertEqual(search(["a"], [MIDDLE], {REFERENCE: limit})[0], limit)
|
||||
self.assertEqual(search(["a"], [MIDDLE, CHEAP], {REFERENCE: {"a": {0}}, MIDDLE: limit})[0], limit)
|
||||
|
||||
|
||||
class FirstReply(unittest.TestCase):
|
||||
def test_tries_three_times_and_returns_the_first_reply_that_is_no_failure(self):
|
||||
replies = [Failure("one"), Failure("two"), "done"]
|
||||
self.assertEqual(first_reply(lambda: replies.pop(0)), "done")
|
||||
replies = [Failure("one"), Limited(None, "limit")]
|
||||
self.assertEqual(first_reply(lambda: replies.pop(0)), Limited(None, "limit"))
|
||||
replies = [Failure("one"), Failure("two"), Failure("three"), "unasked"]
|
||||
self.assertEqual(first_reply(lambda: replies.pop(0)), Failure("three"))
|
||||
|
||||
|
||||
class DownFrom(unittest.TestCase):
|
||||
def test_runs_from_the_projects_rung_down_to_the_rung_the_lowest_r_plus_c_picks(self):
|
||||
self.assertEqual(down_from(REFERENCE), [REFERENCE, MIDDLE, CHEAP, LOWER])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -24,7 +24,6 @@ Like the loop, it tries claude three times running before it counts a reply as f
|
||||
review or match finds nothing. Exit 1, saying why, on bad arguments, an unreadable archive or
|
||||
models file, no archived round to replay, or a usage limit.
|
||||
"""
|
||||
import concurrent.futures
|
||||
import dataclasses
|
||||
import json
|
||||
import os
|
||||
@@ -35,23 +34,20 @@ import tempfile
|
||||
import typing
|
||||
|
||||
from transcript_archive.location import archive_root
|
||||
from stability_replay.rounds import authored_by, replayed, report, reviewed_path, Round, round_from, sample, stamp_seconds
|
||||
from stability_replay.rounds import authored_by, report, reviewed_path, Round, round_from, sample, stamp_seconds
|
||||
from project_key import checkout_of
|
||||
from claude_code.reply import claude_reply, Limited
|
||||
from rung_replay.command import down_from, failed, limited_exit, project_rung_or_exit, projects_on
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.retry import first_reply, Limited
|
||||
from rung_replay.search import first_reply, replayed
|
||||
from claude_code.transcript import first_timestamped_line, read_transcript
|
||||
from claude_code.entries import hit_limit, limit_said, parse_entry, structured_output
|
||||
from model_routing.ladder import LADDER, picked, position, project_rung, Rung
|
||||
from loop_claude.project.model_choice import read_models_file
|
||||
from loop_claude.stop import Stop
|
||||
from claude_code.entries import parse_entry, structured_output
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
TOOL = "stability-replay"
|
||||
LATEST = 10
|
||||
# Reviews replayed at once.
|
||||
AT_ONCE = 4
|
||||
LOWEST_R_PLUS_C = 2
|
||||
REVIEWER = "stability-reviewer"
|
||||
USAGE = "usage: stability-replay <model>, a model on the routing ladder"
|
||||
MATCH_SCHEMA = {"type": "object", "properties": {"found": {"type": "array", "items": {"type": "integer"}}}, "required": ["found"], "additionalProperties": False}
|
||||
|
||||
|
||||
@@ -65,38 +61,33 @@ class Code:
|
||||
|
||||
|
||||
def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, effects: Effects) -> int:
|
||||
if argv[:1] in (["-h"], ["--help"]):
|
||||
print(__doc__.strip())
|
||||
return 0
|
||||
anchor = project_rung(argv[0]) if len(argv) == 1 else None
|
||||
if anchor is None:
|
||||
print(f"stability-replay: cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; {USAGE}", file=sys.stderr)
|
||||
return 1
|
||||
rounds = archived_rounds(argv[0], archive_root(home, env), home)
|
||||
anchor = project_rung_or_exit(TOOL, argv, __doc__)
|
||||
if isinstance(anchor, int):
|
||||
return anchor
|
||||
rounds = archived_rounds(anchor.model, archive_root(home, env), home)
|
||||
if isinstance(rounds, Failure):
|
||||
print(f"stability-replay: {rounds.reason}", file=sys.stderr)
|
||||
return 1
|
||||
return failed(TOOL, rounds.reason)
|
||||
codes = {found: code for found in rounds if not isinstance(code := code_of(found, effects), Failure)}
|
||||
if len(codes) < len(rounds):
|
||||
print(f"{len(rounds) - len(codes)} of {len(rounds)} rounds holding a 🔴 or 🟡 are left out: their code cannot be found", file=sys.stderr, flush=True)
|
||||
if not codes:
|
||||
print(f"stability-replay: no archived stability round of projects on {argv[0]} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find", file=sys.stderr)
|
||||
return 1
|
||||
return failed(TOOL, f"no archived stability round of projects on {anchor.model} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find")
|
||||
chosen = sample(list(codes), LATEST)
|
||||
candidates = list(LADDER[position(picked(anchor, LOWEST_R_PLUS_C)):position(anchor)])[::-1]
|
||||
replay = replayed(chosen, anchor, candidates, lambda on, rung: review_all([(found, codes[found]) for found in on], rung, anchor, effects))
|
||||
# The project's rung is the reference, so the candidates start below it.
|
||||
replay = replayed(chosen, anchor, down_from(anchor)[1:], lambda found, rung: findings_found(found, codes[found], rung, anchor, effects), lambda _, found: sorted(found), "stability rounds")
|
||||
if isinstance(replay, Limited):
|
||||
print(f"stability-replay: claude hit a usage limit: {replay.said}; replay again once it resets", file=sys.stderr)
|
||||
return 1
|
||||
print(report(argv[0], anchor, chosen, replay))
|
||||
return limited_exit(TOOL, replay)
|
||||
print(report(anchor.model, anchor, chosen, replay))
|
||||
return 0
|
||||
|
||||
|
||||
def archived_rounds(model: str, root: pathlib.Path, home: pathlib.Path) -> list[Round] | Failure:
|
||||
models = read_models_file(home / ".claude" / "loop-claude-models.json")
|
||||
if isinstance(models, Stop):
|
||||
return Failure(models.reason)
|
||||
projects = projects_on(model, home)
|
||||
if isinstance(projects, Failure):
|
||||
return projects
|
||||
found = []
|
||||
try:
|
||||
for key in sorted(key for key, on in models.items() if on == model):
|
||||
for key in projects:
|
||||
for chunk in sorted(path for path in (root / key).glob("*") if path.is_dir()):
|
||||
found += chunk_rounds(checkout_of(key, home), chunk)
|
||||
except (OSError, UnicodeDecodeError, ValueError) as failure:
|
||||
@@ -120,6 +111,10 @@ def chunk_rounds(checkout: str, chunk: pathlib.Path) -> list[Round]:
|
||||
|
||||
|
||||
def code_of(found: Round, effects: Effects) -> Code | Failure:
|
||||
"""The code found's reviewer read, as the archive lets it be found again: the prompt's removed worktree, and the commits authored by then.
|
||||
|
||||
Recording the reviewed commit at review time would need no such reading, but then no round archived before that change could be replayed. A wrong guess costs no rung: the reference reviewer, replayed on that code, finds fewer findings on record, and no rung is held to those.
|
||||
"""
|
||||
path = reviewed_path(found.prompt, found.root, os.path.exists)
|
||||
if path is None:
|
||||
return Failure("the prompt names no one worktree")
|
||||
@@ -143,20 +138,6 @@ def commit_log(listed: str) -> list[tuple[str, float]]:
|
||||
return [(commit, float(authored)) for commit, authored in (line.split() for line in listed.splitlines())]
|
||||
|
||||
|
||||
def review_all(on: list[tuple[Round, Code]], rung: Rung, anchor: Rung, effects: Effects) -> list[set[int]] | Limited:
|
||||
print(f"replaying {len(on)} stability rounds on {rung}", file=sys.stderr, flush=True)
|
||||
with concurrent.futures.ThreadPoolExecutor(AT_ONCE) as pool:
|
||||
running = [pool.submit(findings_found, found, code, rung, anchor, effects) for found, code in on]
|
||||
found_sets = []
|
||||
for future in running:
|
||||
found = future.result()
|
||||
if isinstance(found, Limited):
|
||||
pool.shutdown(cancel_futures=True)
|
||||
return found
|
||||
found_sets.append(found)
|
||||
return found_sets
|
||||
|
||||
|
||||
def findings_found(found: Round, code: Code, rung: Rung, anchor: Rung, effects: Effects) -> set[int] | Limited:
|
||||
"""The indexes of the round's findings that its replay on rung reports, as the reviewer on anchor matches them."""
|
||||
reviewed: str | Failure | Limited = first_reply(lambda: review(found, code, rung, effects))
|
||||
@@ -181,7 +162,7 @@ def review(found: Round, code: Code, rung: Rung, effects: Effects) -> str | Fail
|
||||
if isinstance(ran := git(effects, found.root, *command), Failure):
|
||||
return ran
|
||||
args = ["-p", "--agent", REVIEWER, *rung.flags(), "--no-session-persistence", "--output-format", "json", "--permission-mode", "bypassPermissions"]
|
||||
reply = claude_reply(effects, args, clone, found.prompt.replace(code.path, clone))
|
||||
reply = claude_reply(effects.claude, args, clone, found.prompt.replace(code.path, clone), "claude")
|
||||
if isinstance(reply, Failure | Limited):
|
||||
return reply
|
||||
return reply["result"] if isinstance(reply.get("result"), str) else Failure("the reviewer replied with no report")
|
||||
@@ -198,7 +179,7 @@ def match(findings: tuple[str, ...], replay_report: str, root: str, rung: Rung,
|
||||
f"The findings:\n{numbered}\n\nThe other report:\n{replay_report}"
|
||||
)
|
||||
args = ["-p", "--json-schema", json.dumps(MATCH_SCHEMA), *rung.flags(), "--no-session-persistence", "--output-format", "json", "--permission-mode", "bypassPermissions", "--tools", ""]
|
||||
reply = claude_reply(effects, args, root, prompt)
|
||||
reply = claude_reply(effects.claude, args, root, prompt, "claude")
|
||||
if isinstance(reply, Failure | Limited):
|
||||
return reply
|
||||
output = structured_output(reply)
|
||||
@@ -208,19 +189,6 @@ def match(findings: tuple[str, ...], replay_report: str, root: str, rung: Rung,
|
||||
return {n - 1 for n in numbers if 1 <= n <= len(findings)}
|
||||
|
||||
|
||||
def claude_reply(effects: Effects, args: list[str], cwd: str, prompt: str) -> dict | Failure | Limited:
|
||||
try:
|
||||
proc = effects.claude(args, cwd, prompt)
|
||||
except OSError as failure:
|
||||
return Failure(f"could not start claude: {failure.strerror}")
|
||||
reply = parse_entry(proc.stdout) or {}
|
||||
if hit_limit(reply):
|
||||
return Limited(None, limit_said(reply))
|
||||
if proc.returncode != 0:
|
||||
return Failure(proc.stderr.strip() or f"claude exited {proc.returncode}")
|
||||
return reply
|
||||
|
||||
|
||||
def git(effects: Effects, cwd: str, *args: str) -> str | Failure:
|
||||
ran = effects.git(list(args), cwd)
|
||||
return ran.stdout if ran.returncode == 0 else Failure(f"git {' '.join(args)}: {ran.stderr.strip()}")
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Which archived stability rounds to replay, the lowest rung that finds their findings, and the report: docs/decisions.md → `routing: the stability reviewer stays on the project's rung`."""
|
||||
"""Which archived stability rounds to replay, the code each reviewed, and the report: docs/decisions.md → `routing: the stability reviewer stays on the project's rung`."""
|
||||
import dataclasses
|
||||
import datetime
|
||||
import pathlib
|
||||
@@ -6,7 +6,7 @@ import re
|
||||
import typing
|
||||
|
||||
from claude_code.entries import assistant_parts, content, part_text, parts
|
||||
from loop_claude.cycle.retry import Limited
|
||||
from rung_replay.search import Replay
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
# A line that opens with a 🔴 or 🟡 mark, after any list, table, heading or bold markup, as review-findings.md has reviewers report them.
|
||||
@@ -28,21 +28,6 @@ class Round:
|
||||
findings: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Tried:
|
||||
"""A rung, and each finding it missed that the reference reviewer found, with its round."""
|
||||
rung: Rung
|
||||
missed: list[tuple[Round, str]]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Replay:
|
||||
"""unfound counts the findings on record the reference reviewer did not find; tried holds each rung tried, highest first; lowest, the lowest rung that missed none of the rest, or None."""
|
||||
unfound: int
|
||||
tried: list[Tried]
|
||||
lowest: Rung | None
|
||||
|
||||
|
||||
def round_from(root: str, began: float, entries: list[dict]) -> Round | None:
|
||||
"""The round a stability reviewer's transcript holds: its first entry's prompt, and its last text. None without a 🔴 or 🟡 finding to hold a rung to."""
|
||||
reports = [text for found in entries for part in assistant_parts(found) if (text := part_text(part))]
|
||||
@@ -79,7 +64,9 @@ def reviewed_path(prompt: str, root: str, exists: typing.Callable[[str], bool])
|
||||
named = [LINE_OR_STOP.sub("", path) for path in PATH.findall(prompt)]
|
||||
gone = set()
|
||||
for path in map(pathlib.PurePath, named):
|
||||
first_gone = next((prefix for prefix in [*reversed(path.parents), path][1:] if not exists(str(prefix))), None)
|
||||
# The path's directories from the top down, without the filesystem root; the first one gone is what a removed worktree took with it.
|
||||
down_the_path = [*reversed(path.parents), path][1:]
|
||||
first_gone = next((prefix for prefix in down_the_path if not exists(str(prefix))), None)
|
||||
if first_gone and (first_gone.parent == checkout.parent or checkout in first_gone.parents):
|
||||
gone.add(str(first_gone))
|
||||
if len(gone) == 1:
|
||||
@@ -97,38 +84,14 @@ def sample(rounds: list[Round], latest: int) -> list[Round]:
|
||||
return sorted(rounds, key=lambda found: (found.seconds, found.root))[-latest:]
|
||||
|
||||
|
||||
def replayed(rounds: list[Round], reference: Rung, candidates: list[Rung], review_all: typing.Callable[[list[Round], Rung], list[set[int]] | Limited]) -> Replay | Limited:
|
||||
"""Replays the rounds on the reference rung, then each candidate, highest first, on the rounds where the reference found a finding, until one misses a finding it found.
|
||||
|
||||
Going down, every rung at or above the lowest that missed none was tried.
|
||||
"""
|
||||
found = review_all(rounds, reference)
|
||||
if isinstance(found, Limited):
|
||||
return found
|
||||
held = [(found_round, kept) for found_round, kept in zip(rounds, found) if kept]
|
||||
unfound = sum(len(found_round.findings) for found_round in rounds) - sum(len(kept) for _, kept in held)
|
||||
tried: list[Tried] = []
|
||||
lowest = None
|
||||
# With no finding to hold a rung to, every rung would miss none.
|
||||
for rung in candidates if held else []:
|
||||
found = review_all([found_round for found_round, _ in held], rung)
|
||||
if isinstance(found, Limited):
|
||||
return found
|
||||
tried.append(Tried(rung, [(found_round, found_round.findings[n]) for (found_round, kept), on_rung in zip(held, found) for n in sorted(kept - on_rung)]))
|
||||
if tried[-1].missed:
|
||||
break
|
||||
lowest = rung
|
||||
return Replay(unfound, tried, lowest)
|
||||
|
||||
|
||||
def report(model: str, reference: Rung, rounds: list[Round], replay: Replay) -> str:
|
||||
def report(model: str, reference: Rung, rounds: list[Round], replay: Replay[Round, int, set[int]]) -> str:
|
||||
total = sum(len(found_round.findings) for found_round in rounds)
|
||||
lines = [
|
||||
f"Replayed {len(rounds)} stability rounds of {model} projects, holding {total} 🔴 and 🟡 findings.",
|
||||
f"The reference reviewer, {reference}, found {total - replay.unfound} of them.",
|
||||
f"The reference reviewer, {reference}, found {sum(len(found) for found in replay.on_reference)} of them.",
|
||||
]
|
||||
for tried in replay.tried:
|
||||
lines += [f"{tried.rung} missed {len(tried.missed)}{':' if tried.missed else '.'}", *(f" {found_round.root} {found_round.at}: {finding.splitlines()[0]}" for found_round, finding in tried.missed)]
|
||||
lines += [f"{tried.rung} missed {len(tried.missed)}{':' if tried.missed else '.'}", *(f" {miss.case.root} {miss.case.at}: {miss.case.findings[miss.item].splitlines()[0]}" for miss in tried.missed)]
|
||||
if not replay.tried:
|
||||
lines.append("No rung tried: the reference reviewer found no finding to hold one to.")
|
||||
else:
|
||||
|
||||
@@ -10,10 +10,10 @@ import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from stability_replay.rounds import authored_by, recorded_findings, Replay, replayed, report, reviewed_path, Round, round_from, sample, Tried
|
||||
from stability_replay.rounds import authored_by, recorded_findings, report, reviewed_path, Round, round_from, sample
|
||||
from loop_claude.effects import Effects, system_effects
|
||||
from loop_claude.cycle.retry import Limited
|
||||
from stability_replay import main
|
||||
from rung_replay.search import Miss, Replay, Tried
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
TOOL = pathlib.Path(__file__).resolve().parents[1] / "stability-replay"
|
||||
@@ -82,35 +82,10 @@ class Sample(unittest.TestCase):
|
||||
self.assertEqual(sample(rounds, 2), [rounds[0], rounds[2]])
|
||||
|
||||
|
||||
class Replayed(unittest.TestCase):
|
||||
def test_holds_each_rung_highest_first_to_what_the_reference_found_and_stops_at_the_first_that_misses(self):
|
||||
rounds = [a_round(1, ("🔴 a", "🟡 b")), a_round(2, ("🔴 c",))]
|
||||
found = {REFERENCE: [{0, 1}, set()], MIDDLE: [{0, 1}], CHEAP: [{1}]}
|
||||
asked = []
|
||||
|
||||
def review_all(on, rung):
|
||||
asked.append((len(on), rung))
|
||||
return found[rung]
|
||||
|
||||
replay = replayed(rounds, REFERENCE, [MIDDLE, CHEAP, Rung("claude-haiku-5-5", "xhigh")], review_all)
|
||||
self.assertEqual(replay, Replay(1, [Tried(MIDDLE, []), Tried(CHEAP, [(rounds[0], "🔴 a")])], MIDDLE))
|
||||
self.assertEqual(asked, [(2, REFERENCE), (1, MIDDLE), (1, CHEAP)])
|
||||
|
||||
def test_names_no_rung_when_the_first_misses_or_the_reference_found_nothing(self):
|
||||
rounds = [a_round(1)]
|
||||
self.assertEqual(replayed(rounds, REFERENCE, [MIDDLE], lambda on, rung: [{0}] if rung == REFERENCE else [set()]), Replay(0, [Tried(MIDDLE, [(rounds[0], "🔴 a")])], None))
|
||||
self.assertEqual(replayed(rounds, REFERENCE, [MIDDLE], lambda on, rung: [set()]), Replay(1, [], None))
|
||||
|
||||
def test_stops_at_a_usage_limit(self):
|
||||
limit = Limited(None, "limit")
|
||||
self.assertEqual(replayed([a_round(1)], REFERENCE, [MIDDLE], lambda on, rung: limit), limit)
|
||||
self.assertEqual(replayed([a_round(1)], REFERENCE, [MIDDLE], lambda on, rung: [{0}] if rung == REFERENCE else limit), limit)
|
||||
|
||||
|
||||
class Report(unittest.TestCase):
|
||||
def test_counts_the_findings_and_names_each_miss_and_the_lowest_rung_that_missed_none(self):
|
||||
rounds = [a_round(1, ("🔴 a", "🟡 b")), a_round(2)]
|
||||
lines = report("claude-opus-5-5", REFERENCE, rounds, Replay(1, [Tried(MIDDLE, []), Tried(CHEAP, [(rounds[0], "🔴 a\nmore")])], MIDDLE)).splitlines()
|
||||
rounds = [a_round(1, ("🔴 a\nmore", "🟡 b")), a_round(2)]
|
||||
lines = report("claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, set()], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(rounds[0], 0, {1})])], MIDDLE)).splitlines()
|
||||
self.assertEqual(lines, [
|
||||
"Replayed 2 stability rounds of claude-opus-5-5 projects, holding 3 🔴 and 🟡 findings.",
|
||||
"The reference reviewer, claude-opus-5-5 at medium effort, found 2 of them.",
|
||||
@@ -119,8 +94,8 @@ class Report(unittest.TestCase):
|
||||
" /code/p 2026-10-14T10:00:01Z: 🔴 a",
|
||||
"Lowest rung that finds every finding the reference reviewer finds: claude-sonnet-5-5 at high effort",
|
||||
])
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay(0, [Tried(MIDDLE, [(rounds[1], "🔴 a")])], None)).splitlines()[-1], "Lowest rung that finds every finding the reference reviewer finds: none below the project's rung")
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay(3, [], None)).splitlines()[-1], "No rung tried: the reference reviewer found no finding to hold one to.")
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay([{0, 1}, {0}], [Tried(MIDDLE, [Miss(rounds[1], 0, set())])], None)).splitlines()[-1], "Lowest rung that finds every finding the reference reviewer finds: none below the project's rung")
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, rounds, Replay([set(), set()], [], None)).splitlines()[-1], "No rung tried: the reference reviewer found no finding to hold one to.")
|
||||
|
||||
|
||||
def unused(*_):
|
||||
|
||||
@@ -48,7 +48,7 @@ Release 0.3.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
|
||||
| 133 | 0.3.0 | | **Show, beside `bin/rule_hook/override_log.py`'s `this_chunks`, one example per case it decides: an amended or rebased commit, a renamed branch, and a commit on the default branch.** | 1 | 2 | 2 | 3 | 4 | 1.1 | 12.4 |
|
||||
| 134 | 0.3.0 | | **Name in `bin/shell_command.py`'s `segments` the shell forms it does not parse, so a caller knows what it misses.** | 1 | 2 | 2 | 3 | 4 | 1.1 | 12.4 |
|
||||
| 140 | 0.3.0 | | **Split `docs/decisions.md` → `routing: one ladder` into one entry per decision, so a reader finds the rules for the ladder, the worker's rung, the routing figures and delegation by their titles.** | 1 | 2 | 1 | 3 | 4 | 1.1 | 12.4 |
|
||||
| 114 | 0.3.0 | decision | **Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.** | 3 | 4 | 4 | 3 | 5 | 1.1 | 11.9 |
|
||||
| 114 | 0.3.0 | decision | **Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost`, `stability_replay` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.** | 3 | 4 | 4 | 3 | 5 | 1.1 | 11.9 |
|
||||
| 117 | 0.3.0 | | **Give `todo_table.items` one way to say a plan has no items table, so no caller tells it from an empty plan by truthiness.** | 2 | 2 | 3 | 3 | 4 | 1.1 | 11.4 |
|
||||
| 124 | 0.3.0 | decision | **Group the shared modules at `bin/`'s root into packages named by purpose, so the root holds only launchers and stand-alone tools.** | 3 | 6 | 3 | 3 | 5 | 1.1 | 11.4 |
|
||||
| 130 | 0.3.0 | | **Move the vote counting out of `bin/larv_review/ruling.py`'s `votes`, so reading a votes file decides nothing.** | 2 | 3 | 3 | 3 | 4 | 1.1 | 11.2 |
|
||||
@@ -181,11 +181,11 @@ Junior A and Mid A of PR #257's scoring run named `bin/shell_command.py:31` `seg
|
||||
|
||||
The entry holds 7 paragraphs under one title of about 70 words chaining three decisions: the ladder and bands, the worker below the project's rung with its reviewers, the routing figures and calls, delegation, and when the worker changes rung. A reader after one rule reads the whole entry. The prose pass of the chunk that split out the judge's rung asked for the rest. Once split, the delegation entry's title names the floor that `todo.md: only a session or subagent on the project's rung or above scores or reshapes the plan…` puts under a subagent that files or scores items.
|
||||
|
||||
### 114. Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.
|
||||
### 114. Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost`, `stability_replay` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.
|
||||
|
||||
Shape is the lowest dimension, so this item goes before 0.3.0's other items, per `docs/decisions.md` → `larv-review: a project lifting itself to the comprehension floor starts at the lowest dimension`.
|
||||
|
||||
The comprehension seats on PR #254 found five git runners with different contracts: `git_repo.git`, `rule_hook/git.py`'s guarded one, `metric_bench/main.py`'s, `reading_cost/main.py`'s `located` and `loop_claude/effects.py`'s. The inherited architect of PR #257's scoring run named `bin/git_repo.py:8` `git` again. Make `git_repo.git` easier to read.
|
||||
The comprehension seats on PR #254 found five git runners with different contracts: `git_repo.git`, `rule_hook/git.py`'s guarded one, `metric_bench/main.py`'s, `reading_cost/main.py`'s `located` and `loop_claude/effects.py`'s. PR #276 added a sixth, `stability_replay/main.py`'s `git`. The inherited architect of PR #257's scoring run named `bin/git_repo.py:8` `git` again. Make `git_repo.git` easier to read.
|
||||
|
||||
### 117. Give `todo_table.items` one way to say a plan has no items table, so no caller tells it from an empty plan by truthiness.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user