Files
claude-config/bin/review_replay/main.py
T

143 lines
9.8 KiB
Python

"""Replay a review pass's archived rounds on the rungs below the project's, to find which rungs may run its reviewer: docs/decisions.md → `routing: a review pass runs below the project's rung only on a rung whose replay of its rounds finds a share of their 🔴 and 🟡 findings no more than a margin below the project's rung's own replay`.
usage: review-replay <pass> <model> [--omit-claude-md | --bash-ceiling | --read-limit]
<pass> is one of architecture, principles, overrides, stability, product-owner or prose.
<model> is a project model on the routing ladder this setup ships, such as claude-opus-5-5. The tool replays the
latest 10 archived rounds of the pass whose report holds a 🔴 or 🟡 finding, from the projects
that ~/.claude/loop-claude-models.json runs on the model. It replays each round on the code it
reviewed: in a throwaway clone of the project's checkout, at the newest commit of origin/HEAD
authored by the time the round began, on the branch the reviewer started on. The clone's default
branch stands at the newest commit authored before the round's chunk began, so a diff against it
shows the chunk's change. The prompt names the clone in place of the worktree it named, which the
chunk has since removed. A round whose code it cannot find in this way is not replayed; stderr
names it and why. The prompt also names a copy of the chunk's review folder in place of that
folder, holding only the reports written after the chunk began and before the round did. A replay
fails where its prompt still names the review folder, or names a file there, other than a round's
report, that the folder did not hold from the chunk's start until the round's.
It replays every round twice on the project's rung. Even that rung finds only part of its own
archived findings again, so the bar for a rung below is the mean share of those two replays, less
a margin of 10 percentage points. Rounds that either replay failed on are left out, and where
fewer than 8 remain, or the bar is zero or below, no rung is tried. It
then replays the rounds on each rung below, highest first: the project's model at each lower
effort, which the ladder leaves out, then each ladder rung down to the ladder's lowest. It stops at
the first rung that doesn't qualify. A rung qualifies where it finds as many findings as reach the
bar and misses no 🔴 that both replays on the project's rung found; a replay that failed on it
finds none. For
each replay, claude on the project's rung matches the report to the archived findings, and counts
a finding only where the report raises it as 🔴 or 🟡. The tool prints each rung's count, each 🔴
it missed, and the lowest rung that, like each rung above it, qualifies. That rung is the evidence
for running the pass's reviewer on it; the decision above records each run.
With --omit-claude-md it tests instead whether the pass's reviewer may start without CLAUDE.md:
docs/decisions.md → `agents/: a reviewer starts without CLAUDE.md only where a replay of its pass
missed…`. It replays every round four times on the project's rung: three as archived, then once
without the instruction files. The unchanged reviewer does not find the same findings twice, so
each unchanged replay is a control: the tool prints how many findings it missed that both of the
other two found. The replay without CLAUDE.md is held to the findings both of the first two found,
and passes where it missed no more of them than the most any unchanged replay missed. It prints
how many findings each replay found, how many the replay without CLAUDE.md is held to, and each of
those it missed. A round that any of the four replays failed on is left out, and where the first
two share no finding the last line says there is no evidence.
Claude Code ignores an agent's `omitClaudeMd` where the agent runs the session, as it does in a
replay, and runs no SubagentStart hook there. So the replay without CLAUDE.md turns the
instruction files off in its settings and opens the prompt with what the hook hands a reviewer, and
closes it with the header's line on the project principles.
--bash-ceiling and --read-limit run the same comparison for two limits on what one tool result adds to a
claude's context, which a subagent inherits (bin/claude_code/output_limits.py). loop-claude hands the worker's
claude the Bash ceiling, and doesn't hand it the Read limit yet. The replay with --bash-ceiling starts under
BASH_MAX_OUTPUT_LENGTH=6000, which saves a Bash result of more than 6,000 characters to a file and hands the
reviewer its path and the first 2,000 characters. The replay with --read-limit starts under
CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS=5000, which fails a Read of more than 5,000 tokens. The replay with
the limit passes where it missed no more of the findings both of the first two found than the most an
unchanged replay missed. The tool doesn't count what the replays read: each run's stream, with every tool
result, is in the archive. It refuses to run where its own environment already holds the variable, since the
replays as archived would inherit it.
It keeps each run, a reviewer's tool calls included, and the report it prints in loop-claude's archive, in
replays/review-replay-<pass>-<model>[-<flag without its leading dashes>]-<UTC stamp>/ under ~/.claude/loop-claude-transcripts
or $LOOP_CLAUDE_TRANSCRIPTS. A run's .jsonl file starts with a line holding claude's arguments, working
directory, prompt, exit code and stderr, then holds claude's stream-json output; result.md holds the report.
Like the loop, it tries claude three times running before it counts a reply as failed. A failed
review or match leaves its round out, except on a rung below the project's, where it finds
none. Exit 1, saying why, on bad arguments, an unreadable archive or models file, no archived
round to replay, or a usage limit.
"""
import pathlib
import typing
from transcript_archive.location import archive_root
from review_replay.rounds import archived_rounds, Round, sample
from rung_replay.archive import archiving, replay_folder
from rung_replay.command import arguments_or_exit, failed, limited_exit, print_report, say
from rung_replay.search import AT_ONCE
from review_replay.changes import Change, CHANGES
from review_replay.replay import codes_found, findings_replayed
from review_replay.change_comparison import compared
from loop_effects.effects import Effects
from review_replay.recall import ENOUGH_ROUNDS, recall_report, recalled
from failure import Failure
from model_routing.ladder import Ladder, Rung
from claude_code.reply import Limited
from model_routing.ladder_file import read_ladder, SHIPPED
from review_rules.review_passes import ReviewedPass, REVIEWERS
TOOL = "review-replay"
USAGE = f"{TOOL} <pass> <model> [{' | '.join(change.flag for change in CHANGES)}], <pass> one of {', '.join(REVIEWERS)} and <model> a model on the routing ladder"
LATEST = 10
# A test sets the three limits: the fewest rounds a rung is tried on, the rounds replayed at once, and the folder each clone is made in, the system's temporary folder where None.
def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, effects: Effects, *, enough_rounds: int = ENOUGH_ROUNDS, at_once: int = AT_ONCE, scratch: pathlib.Path | None = None) -> int:
# The shipped ladder, not a project's: the replay spans every project on the model.
ladder = read_ladder(SHIPPED)
if isinstance(ladder, Failure):
return failed(TOOL, ladder.reason)
read = arguments_or_exit(TOOL, argv, __doc__, USAGE, lambda argv: pass_and_model(ladder, argv))
if isinstance(read, int):
return read
reviewed, project, change = read
# The replays as archived start under this environment too, so a change it already holds would reach all four.
if change and (inherited := [name for name in change.env if name in env]):
return failed(TOOL, f"the replays as archived would inherit {', '.join(inherited)} from this environment, and the comparison needs them without it: unset it and run again")
rounds = archived_rounds(REVIEWERS[reviewed], project.model, archive_root(home, env), home)
if isinstance(rounds, Failure):
return failed(TOOL, rounds.reason)
codes = codes_found(rounds, effects, say)
if not codes:
return failed(TOOL, f"no archived {reviewed} round of projects on {project.model} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find")
chosen = sample(list(codes), LATEST)
folder = replay_folder(archive_root(home, env), TOOL, argv, effects.now())
effects = archiving(effects, folder, say)
if change:
report = compared(reviewed, project, chosen, change, lambda found, applied: findings_replayed(found, codes[found], project, project, home, effects, applied, scratch), say, at_once)
if isinstance(report, Limited):
return limited_exit(TOOL, report)
print_report(folder, report)
return 0
def replay_one(found: Round, rung: Rung) -> set[int] | Failure | Limited:
matched = findings_replayed(found, codes[found], rung, project, home, effects, None, scratch)
if isinstance(matched, Failure):
say(f" {found.root} {found.at} on {rung} failed: {matched.reason}")
return matched
recall = recalled(chosen, ladder, project, replay_one, f"{reviewed} rounds", say, at_once, enough_rounds)
if isinstance(recall, Limited):
return limited_exit(TOOL, recall)
print_report(folder, recall_report(reviewed, project, recall, enough_rounds))
return 0
def pass_and_model(ladder: Ladder, argv: list[str]) -> tuple[ReviewedPass, Rung, Change | None] | None:
"""The pass, the project's rung on ladder of the model, and the change a flag names, as argv names them in `<pass> <model> [<flag>]`; None for any other argv."""
change = next((each for each in CHANGES if argv[2:] == [each.flag]), None)
if argv[2:] and change is None or len(argv) < 2:
return None
reviewed = next((kind for kind in REVIEWERS if kind == argv[0]), None)
project = ladder.project_rung(argv[1])
return (reviewed, project.rung, change) if reviewed and project else None