143 lines
9.8 KiB
Python
143 lines
9.8 KiB
Python
"""Replay a review pass's archived rounds on the rungs below the project's, to find which rungs may run its reviewer: docs/decisions.md → `routing: a review pass runs below the project's rung only on a rung whose replay of its rounds finds a share of their 🔴 and 🟡 findings no more than a margin below the project's rung's own replay`.
|
|
|
|
usage: review-replay <pass> <model> [--omit-claude-md | --bash-ceiling | --read-limit]
|
|
|
|
<pass> is one of architecture, principles, overrides, stability, product-owner or prose.
|
|
<model> is a project model on the routing ladder this setup ships, such as claude-opus-5-5. The tool replays the
|
|
latest 10 archived rounds of the pass whose report holds a 🔴 or 🟡 finding, from the projects
|
|
that ~/.claude/loop-claude-models.json runs on the model. It replays each round on the code it
|
|
reviewed: in a throwaway clone of the project's checkout, at the newest commit of origin/HEAD
|
|
authored by the time the round began, on the branch the reviewer started on. The clone's default
|
|
branch stands at the newest commit authored before the round's chunk began, so a diff against it
|
|
shows the chunk's change. The prompt names the clone in place of the worktree it named, which the
|
|
chunk has since removed. A round whose code it cannot find in this way is not replayed; stderr
|
|
names it and why. The prompt also names a copy of the chunk's review folder in place of that
|
|
folder, holding only the reports written after the chunk began and before the round did. A replay
|
|
fails where its prompt still names the review folder, or names a file there, other than a round's
|
|
report, that the folder did not hold from the chunk's start until the round's.
|
|
|
|
It replays every round twice on the project's rung. Even that rung finds only part of its own
|
|
archived findings again, so the bar for a rung below is the mean share of those two replays, less
|
|
a margin of 10 percentage points. Rounds that either replay failed on are left out, and where
|
|
fewer than 8 remain, or the bar is zero or below, no rung is tried. It
|
|
then replays the rounds on each rung below, highest first: the project's model at each lower
|
|
effort, which the ladder leaves out, then each ladder rung down to the ladder's lowest. It stops at
|
|
the first rung that doesn't qualify. A rung qualifies where it finds as many findings as reach the
|
|
bar and misses no 🔴 that both replays on the project's rung found; a replay that failed on it
|
|
finds none. For
|
|
each replay, claude on the project's rung matches the report to the archived findings, and counts
|
|
a finding only where the report raises it as 🔴 or 🟡. The tool prints each rung's count, each 🔴
|
|
it missed, and the lowest rung that, like each rung above it, qualifies. That rung is the evidence
|
|
for running the pass's reviewer on it; the decision above records each run.
|
|
|
|
With --omit-claude-md it tests instead whether the pass's reviewer may start without CLAUDE.md:
|
|
docs/decisions.md → `agents/: a reviewer starts without CLAUDE.md only where a replay of its pass
|
|
missed…`. It replays every round four times on the project's rung: three as archived, then once
|
|
without the instruction files. The unchanged reviewer does not find the same findings twice, so
|
|
each unchanged replay is a control: the tool prints how many findings it missed that both of the
|
|
other two found. The replay without CLAUDE.md is held to the findings both of the first two found,
|
|
and passes where it missed no more of them than the most any unchanged replay missed. It prints
|
|
how many findings each replay found, how many the replay without CLAUDE.md is held to, and each of
|
|
those it missed. A round that any of the four replays failed on is left out, and where the first
|
|
two share no finding the last line says there is no evidence.
|
|
Claude Code ignores an agent's `omitClaudeMd` where the agent runs the session, as it does in a
|
|
replay, and runs no SubagentStart hook there. So the replay without CLAUDE.md turns the
|
|
instruction files off in its settings and opens the prompt with what the hook hands a reviewer, and
|
|
closes it with the header's line on the project principles.
|
|
|
|
--bash-ceiling and --read-limit run the same comparison for two limits on what one tool result adds to a
|
|
claude's context, which a subagent inherits (bin/claude_code/output_limits.py). loop-claude hands the worker's
|
|
claude the Bash ceiling, and doesn't hand it the Read limit yet. The replay with --bash-ceiling starts under
|
|
BASH_MAX_OUTPUT_LENGTH=6000, which saves a Bash result of more than 6,000 characters to a file and hands the
|
|
reviewer its path and the first 2,000 characters. The replay with --read-limit starts under
|
|
CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS=5000, which fails a Read of more than 5,000 tokens. The replay with
|
|
the limit passes where it missed no more of the findings both of the first two found than the most an
|
|
unchanged replay missed. The tool doesn't count what the replays read: each run's stream, with every tool
|
|
result, is in the archive. It refuses to run where its own environment already holds the variable, since the
|
|
replays as archived would inherit it.
|
|
|
|
It keeps each run, a reviewer's tool calls included, and the report it prints in loop-claude's archive, in
|
|
replays/review-replay-<pass>-<model>[-<flag without its leading dashes>]-<UTC stamp>/ under ~/.claude/loop-claude-transcripts
|
|
or $LOOP_CLAUDE_TRANSCRIPTS. A run's .jsonl file starts with a line holding claude's arguments, working
|
|
directory, prompt, exit code and stderr, then holds claude's stream-json output; result.md holds the report.
|
|
|
|
Like the loop, it tries claude three times running before it counts a reply as failed. A failed
|
|
review or match leaves its round out, except on a rung below the project's, where it finds
|
|
none. Exit 1, saying why, on bad arguments, an unreadable archive or models file, no archived
|
|
round to replay, or a usage limit.
|
|
"""
|
|
import pathlib
|
|
import typing
|
|
|
|
from transcript_archive.location import archive_root
|
|
from review_replay.rounds import archived_rounds, Round, sample
|
|
from rung_replay.archive import archiving, replay_folder
|
|
from rung_replay.command import arguments_or_exit, failed, limited_exit, print_report, say
|
|
from rung_replay.search import AT_ONCE
|
|
from review_replay.changes import Change, CHANGES
|
|
from review_replay.replay import codes_found, findings_replayed
|
|
from review_replay.change_comparison import compared
|
|
from loop_effects.effects import Effects
|
|
from review_replay.recall import ENOUGH_ROUNDS, recall_report, recalled
|
|
from failure import Failure
|
|
from model_routing.ladder import Ladder, Rung
|
|
from claude_code.reply import Limited
|
|
from model_routing.ladder_file import read_ladder, SHIPPED
|
|
from review_rules.review_passes import ReviewedPass, REVIEWERS
|
|
|
|
TOOL = "review-replay"
|
|
USAGE = f"{TOOL} <pass> <model> [{' | '.join(change.flag for change in CHANGES)}], <pass> one of {', '.join(REVIEWERS)} and <model> a model on the routing ladder"
|
|
LATEST = 10
|
|
|
|
|
|
# A test sets the three limits: the fewest rounds a rung is tried on, the rounds replayed at once, and the folder each clone is made in, the system's temporary folder where None.
|
|
def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, effects: Effects, *, enough_rounds: int = ENOUGH_ROUNDS, at_once: int = AT_ONCE, scratch: pathlib.Path | None = None) -> int:
|
|
# The shipped ladder, not a project's: the replay spans every project on the model.
|
|
ladder = read_ladder(SHIPPED)
|
|
if isinstance(ladder, Failure):
|
|
return failed(TOOL, ladder.reason)
|
|
read = arguments_or_exit(TOOL, argv, __doc__, USAGE, lambda argv: pass_and_model(ladder, argv))
|
|
if isinstance(read, int):
|
|
return read
|
|
reviewed, project, change = read
|
|
# The replays as archived start under this environment too, so a change it already holds would reach all four.
|
|
if change and (inherited := [name for name in change.env if name in env]):
|
|
return failed(TOOL, f"the replays as archived would inherit {', '.join(inherited)} from this environment, and the comparison needs them without it: unset it and run again")
|
|
rounds = archived_rounds(REVIEWERS[reviewed], project.model, archive_root(home, env), home)
|
|
if isinstance(rounds, Failure):
|
|
return failed(TOOL, rounds.reason)
|
|
codes = codes_found(rounds, effects, say)
|
|
if not codes:
|
|
return failed(TOOL, f"no archived {reviewed} round of projects on {project.model} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find")
|
|
chosen = sample(list(codes), LATEST)
|
|
folder = replay_folder(archive_root(home, env), TOOL, argv, effects.now())
|
|
effects = archiving(effects, folder, say)
|
|
if change:
|
|
report = compared(reviewed, project, chosen, change, lambda found, applied: findings_replayed(found, codes[found], project, project, home, effects, applied, scratch), say, at_once)
|
|
if isinstance(report, Limited):
|
|
return limited_exit(TOOL, report)
|
|
print_report(folder, report)
|
|
return 0
|
|
|
|
def replay_one(found: Round, rung: Rung) -> set[int] | Failure | Limited:
|
|
matched = findings_replayed(found, codes[found], rung, project, home, effects, None, scratch)
|
|
if isinstance(matched, Failure):
|
|
say(f" {found.root} {found.at} on {rung} failed: {matched.reason}")
|
|
return matched
|
|
|
|
recall = recalled(chosen, ladder, project, replay_one, f"{reviewed} rounds", say, at_once, enough_rounds)
|
|
if isinstance(recall, Limited):
|
|
return limited_exit(TOOL, recall)
|
|
print_report(folder, recall_report(reviewed, project, recall, enough_rounds))
|
|
return 0
|
|
|
|
|
|
def pass_and_model(ladder: Ladder, argv: list[str]) -> tuple[ReviewedPass, Rung, Change | None] | None:
|
|
"""The pass, the project's rung on ladder of the model, and the change a flag names, as argv names them in `<pass> <model> [<flag>]`; None for any other argv."""
|
|
change = next((each for each in CHANGES if argv[2:] == [each.flag]), None)
|
|
if argv[2:] and change is None or len(argv) < 2:
|
|
return None
|
|
reviewed = next((kind for kind in REVIEWERS if kind == argv[0]), None)
|
|
project = ladder.project_rung(argv[1])
|
|
return (reviewed, project.rung, change) if reviewed and project else None
|