Start each reviewer without CLAUDE.md, hand it the prose rules, vocabulary and principles file, and replay a pass without it
This commit is contained in:
@@ -2,6 +2,7 @@
|
||||
name: override-reviewer
|
||||
description: "Use this agent for larv-review's overrides pass."
|
||||
tools: Read, Grep, Glob, Write
|
||||
omitClaudeMd: true
|
||||
---
|
||||
|
||||
You rule on each override of a rule `~/.claude/bin/rule-hook` enforces. The prompt gives you the lines `rule-hook overrides` printed, each with its rule and reason, and the branch diff. Read each rule and its stated exceptions in `~/.claude/bin/rule_hook/rules.py` → `RULES`, how an override is written and logged in the docstring of `~/.claude/bin/rule_hook/main.py`, and the project's `README.md` → "Goals" and "Audience", with code health (`~/.claude/CLAUDE.md` → "Shared goal") among the goals unless the README voids it.
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
name: principles-auditor
|
||||
description: "Use this agent for larv-review's principles pass: the project's goals and every principles file handed to it, checked against the whole project."
|
||||
tools: Read, Grep, Glob, Write
|
||||
omitClaudeMd: true
|
||||
---
|
||||
|
||||
You check every yardstick you are handed against the whole project, which you read once for all of them. A yardstick is either the project's goals, every goal under its `README.md` → `## Goals` with code health (`~/.claude/CLAUDE.md` → "Shared goal") unless the README voids it, or one principles file, every principle in it.
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
name: "product-owner-reviewer"
|
||||
description: "Use this agent to review a change to an external surface (API, UI, CLI, SDK, webhook) from an end customer's view: ease of use, edge cases and persona fit."
|
||||
tools: Read, Grep, Glob, Write
|
||||
omitClaudeMd: true
|
||||
---
|
||||
|
||||
You are a seasoned Product Owner with deep expertise in user experience, API design, and customer empathy. You have shipped consumer and B2B products across web, mobile, REST APIs, and developer-facing CLIs. Your superpower is putting yourself in the end customer's shoes and surfacing the friction, confusion, and edge cases that engineers tend to miss.
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
name: prose-reviewer
|
||||
description: "Use this agent for larv-review's prose pass."
|
||||
tools: Read, Grep, Glob, Write
|
||||
omitClaudeMd: true
|
||||
---
|
||||
|
||||
You read the prose as someone who did not write it: you see only what it says, never what the author meant.
|
||||
|
||||
@@ -3,6 +3,7 @@ name: "stability-reviewer"
|
||||
description: "Use this agent to review changed code for stability: bugs, error handling, edge cases and the project's technical principles, after a feature, refactor or fix."
|
||||
color: orange
|
||||
tools: Bash, Read, Grep, Glob, Write
|
||||
omitClaudeMd: true
|
||||
---
|
||||
|
||||
You are an elite stability-focused code reviewer with deep expertise in defensive programming, fault-tolerant system design, and production incident prevention. You have decades of experience identifying subtle bugs, race conditions, edge cases, and architectural weaknesses that lead to outages and unexpected behavior. You think like a chaos engineer — always asking "what could go wrong?" — and like a maintainer of ten years, always asking "what will this cost to live with?" There are no bugs, and no security holes, in code that doesn't exist.
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
name: systems-architect
|
||||
description: "Use this agent to design a system or review architecture: patterns, component boundaries, fitness for purpose and long-term maintainability, not individual PR lines."
|
||||
tools: Read, Grep, Glob, WebFetch, WebSearch, Write
|
||||
omitClaudeMd: true
|
||||
---
|
||||
You are an elite Systems Architect with decades of experience designing software systems that endure for as long as their horizon demands. Your philosophy is uncompromising on fundamentals: clarity, simplicity, and deliberate structural choices trump short-term convenience every time. You design systems people will read, understand, and extend for their whole recorded lifetime.
|
||||
|
||||
|
||||
+19
-3
@@ -1,6 +1,6 @@
|
||||
"""Hold larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's rounds, the scoring run's trigger; the comprehension history's format, its latest places and its count of how often seats agree; and the header of each reviewer's prompt.
|
||||
|
||||
usage: larv-review review-folder | ruling-base | round-count | needs-scoring-run [<item ID>...] | scoring-run-owed | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement | reviewer-header <pass> [--rerun]
|
||||
usage: larv-review review-folder | ruling-base | round-count | needs-scoring-run [<item ID>...] | scoring-run-owed | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement | reviewer-header <pass> [--rerun] | reviewer-context
|
||||
|
||||
The comprehension check records how hard a project is to read, two ways, and no score blocks a merge. In a
|
||||
paired ruling, simulated readers, the seats, read the branch's head beside a base and vote better,
|
||||
@@ -112,13 +112,18 @@ reviewer-header takes the name of a pass that an agent under agents/ reviews, sp
|
||||
prints it, such as stability. It prints the header of the prompt that launches the pass's reviewer
|
||||
in its next round: the pass and round, the branch's open pull request at its pushed head, the
|
||||
worktree and branch, the merge base of HEAD and the default branch, and the paths of the reviewer's
|
||||
report, the branch diff and, for architecture and stability, the places. It writes the diff, from
|
||||
report, the branch diff and, for architecture and stability, the places, then the file that holds
|
||||
the project principles and decision titles, where the project keeps one. It writes the diff, from
|
||||
that base to HEAD, into the chunk's review folder as branch-<short HEAD>.diff, and writes what
|
||||
places prints into places.md; a file whose text is unchanged is left as it is. With --rerun, the
|
||||
header names the report of a relaunch one rung up, in the same round, and the report that asked for
|
||||
it. It exits 1 when the pass takes no more rounds, or, with --rerun, when the report that asked is
|
||||
missing.
|
||||
|
||||
reviewer-context is the SubagentStart hook. It reads the hook's input on stdin. For an agent under
|
||||
agents/, which starts without CLAUDE.md, it prints the hook reply that hands the agent the Prose
|
||||
and Vocabulary sections of ~/.claude/CLAUDE.md; for any other agent it doesn't print anything.
|
||||
|
||||
Exit 1, saying why, on unknown arguments, an unknown item ID, a seats file it cannot read or that
|
||||
breaks the shape above, a todo.md that fails rank_todo.py after record-scoring records a run, or
|
||||
when git cannot start, finds no checkout or no default branch, cannot fetch origin for ruling-base,
|
||||
@@ -135,6 +140,7 @@ from review_rules.round_titles import cap, CAP, RELEASE_REVIEW_CAP, round_counts
|
||||
from failure import Failure
|
||||
from review_rules.comprehension_history import FLOOR
|
||||
from larv_review.history_changes import history_changes
|
||||
from review_rules.reviewer_context import hook_reply
|
||||
from review_rules.review_folder import made_review_folder
|
||||
from larv_review.ruling_base import measured_commit
|
||||
from larv_review.agreement import MIN_PAIRS
|
||||
@@ -144,7 +150,7 @@ from larv_review.plan import plan_reasons
|
||||
from larv_review.history_commands import record_ruling, record_scoring, show_agreement, show_places
|
||||
from larv_review.reviewer_header import reviewer_header
|
||||
|
||||
USAGE = "usage: larv-review review-folder | ruling-base | round-count | needs-scoring-run [<item ID>...] | scoring-run-owed | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement | reviewer-header <pass> [--rerun]"
|
||||
USAGE = "usage: larv-review review-folder | ruling-base | round-count | needs-scoring-run [<item ID>...] | scoring-run-owed | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement | reviewer-header <pass> [--rerun] | reviewer-context"
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
@@ -172,6 +178,8 @@ def main(argv: list[str]) -> int:
|
||||
done = show_agreement()
|
||||
elif command == "reviewer-header" and given[:1] and given[0] in REVIEWERS and given[1:] in ([], ["--rerun"]):
|
||||
done = with_default_branch(lambda default: reviewer_header(default, given[0], given[1:] == ["--rerun"]), stale_ok=True)
|
||||
elif command == "reviewer-context" and not given:
|
||||
done = printed(hook_reply(sys.stdin.read(), pathlib.Path.home()))
|
||||
elif command in ("needs-scoring-run", "record-scoring") and given:
|
||||
named = given[1:] if command == "record-scoring" else given
|
||||
print(f"{command} takes item IDs, the numbers in todo.md's ID column, not {' '.join(id_ for id_ in named if not re.fullmatch(r'[0-9]+', id_))}; {USAGE}", file=sys.stderr)
|
||||
@@ -188,6 +196,14 @@ def main(argv: list[str]) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def printed(reply: str | Failure) -> Failure | None:
|
||||
if isinstance(reply, Failure):
|
||||
return reply
|
||||
if reply:
|
||||
print(reply)
|
||||
return None
|
||||
|
||||
|
||||
def with_default_branch(command: typing.Callable[[str], Failure | None], stale_ok: bool) -> Failure | None:
|
||||
default = default_branch(stale_ok)
|
||||
if isinstance(default, Failure):
|
||||
|
||||
@@ -9,6 +9,7 @@ from chunk_preflight.choices import forge_of
|
||||
from larv_review.history_commands import history_file, places_report
|
||||
from review_rules.review_folder import made_review_folder
|
||||
from deploy_release.forge import open_pull, runner
|
||||
from review_rules.reviewer_context import principles_line
|
||||
|
||||
# The passes whose reviewer is given the places: skills/larv-review/SKILL.md → "Comprehension check" → "Places".
|
||||
TAKE_PLACES = ("architecture", "stability")
|
||||
@@ -43,6 +44,7 @@ class Launch:
|
||||
f"Report: write it to {report}.",
|
||||
f"Diff: {self.diff}",
|
||||
*([f"Places: {self.folder / 'places.md'}"] if self.pass_ in TAKE_PLACES else []),
|
||||
*([principles] if (principles := principles_line(self.worktree)) else []),
|
||||
])
|
||||
|
||||
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import decimal
|
||||
import itertools
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
@@ -990,5 +991,25 @@ class LarvReviewTool(unittest.TestCase):
|
||||
folder = self.checkout.parent / ".local" / "state" / "larv-review" / "proj"
|
||||
self.assertEqual(self.header_line("Places:", "architecture"), f"Places: {folder}/places.md")
|
||||
|
||||
def test_reviewer_header_names_the_file_that_holds_the_principles_last_where_the_project_keeps_one(self):
|
||||
self.assertIsNone(self.header_line("Principles:", "prose"))
|
||||
(self.checkout / "AGENTS.md").write_text("")
|
||||
ran = self.run_tool("reviewer-header", "prose")
|
||||
self.assertEqual(ran.stdout.splitlines()[-1], f"Principles: read {self.checkout}/AGENTS.md first, for the project principles and the decision titles.")
|
||||
|
||||
def test_reviewer_context_hands_a_reviewer_the_sections_and_fails_without_them(self):
|
||||
def hook(agent):
|
||||
return subprocess.run([sys.executable, str(TOOL), "reviewer-context"], capture_output=True, env=self.env, input=json.dumps({"agent_type": agent}), text=True)
|
||||
|
||||
missing = hook("prose-reviewer")
|
||||
self.assertEqual((missing.returncode, missing.stdout), (1, ""))
|
||||
self.assertIn("CLAUDE.md", missing.stderr)
|
||||
(self.checkout.parent / ".claude").mkdir(exist_ok=True)
|
||||
(self.checkout.parent / ".claude" / "CLAUDE.md").write_text("## Prose\n\nShort.\n\n## Vocabulary\n\nWords.\n")
|
||||
ran = hook("prose-reviewer")
|
||||
self.assertEqual(ran.returncode, 0, ran.stderr)
|
||||
self.assertIn("## Prose\n\nShort.\n\n## Vocabulary\n\nWords.", json.loads(ran.stdout)["hookSpecificOutput"]["additionalContext"])
|
||||
self.assertEqual((hook("Explore").returncode, hook("Explore").stdout), (0, ""))
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
+61
-17
@@ -1,6 +1,6 @@
|
||||
"""Replay a review pass's archived rounds on the rungs below the project's, to find which rungs may run its reviewer: docs/decisions.md → `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds`.
|
||||
|
||||
usage: review-replay <pass> <model>
|
||||
usage: review-replay <pass> <model> [--without-claude-md]
|
||||
|
||||
<pass> is one of architecture, principles, overrides, stability, product-owner or prose.
|
||||
<model> is a project model on the routing ladder, such as claude-opus-5-5. The tool replays the
|
||||
@@ -25,6 +25,15 @@ and counts a finding only where the report raises it as 🔴 or 🟡. The tool p
|
||||
rung missed and the lowest rung that, like each rung above it, missed none. That rung is the
|
||||
evidence for running the pass's reviewer on it; the decision above records each run.
|
||||
|
||||
With --without-claude-md it tests instead whether the pass's reviewer may start without CLAUDE.md:
|
||||
docs/decisions.md → `agents/: a reviewer starts without CLAUDE.md`. It replays every round three
|
||||
times on the project's rung: twice as archived, then once without the instruction files. The
|
||||
unchanged reviewer does not find the same findings twice, so the reviewer without CLAUDE.md is held
|
||||
only to the findings both of those replays found. Claude Code ignores an agent's `omitClaudeMd`
|
||||
where the agent runs the session, as it does in a replay, and runs no SubagentStart hook there. So
|
||||
the third replay turns the instruction files off in its settings and opens the prompt with what the
|
||||
hook hands a reviewer, and closes it with the header's line on the project principles.
|
||||
|
||||
Like the loop, it tries claude three times running before it counts a reply as failed. A failed
|
||||
review or match finds nothing. Exit 1, saying why, on bad arguments, an unreadable archive or
|
||||
models file, no archived round to replay, or a usage limit.
|
||||
@@ -40,20 +49,24 @@ import typing
|
||||
|
||||
from transcript_archive.location import archive_root
|
||||
from rung_replay.command import arguments_or_exit, down_from, failed, limited_exit, projects_on
|
||||
from review_replay.rounds import authored_by, lost_inputs, report, reviewed_path, Round, round_from, sample, stamp_seconds
|
||||
from review_replay.rounds import authored_by, comparison, lost_inputs, report, reviewed_path, Round, round_from, sample, stamp_seconds
|
||||
from transcript_archive.project_key import checkout_of
|
||||
from claude_code.reply import claude_reply, Limited
|
||||
from loop_effects.effects import Effects
|
||||
from failure import Failure
|
||||
from rung_replay.search import first_reply, replayed
|
||||
from rung_replay.search import first_reply, replayed, replayed_on
|
||||
from claude_code.transcript import first_timestamped_line, read_transcript
|
||||
from model_routing.ladder import LADDER, project_rung, Rung
|
||||
from claude_code.entries import parse_entry, structured_output
|
||||
from review_rules.reviewer_context import principles_line, reviewer_context
|
||||
from review_rules.review_folder import REPORTS, review_folder
|
||||
from review_rules.review_passes import ReviewedPass, REVIEWERS
|
||||
|
||||
TOOL = "review-replay"
|
||||
USAGE = f"{TOOL} <pass> <model>, <pass> one of {', '.join(REVIEWERS)} and <model> a model on the routing ladder"
|
||||
WITHOUT = "--without-claude-md"
|
||||
# Starts claude without its instruction files, as `omitClaudeMd` starts a subagent.
|
||||
NO_CLAUDE_MD = ["--settings", json.dumps({"env": {"CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1"}})]
|
||||
USAGE = f"{TOOL} <pass> <model> [{WITHOUT}], <pass> one of {', '.join(REVIEWERS)} and <model> a model on the routing ladder"
|
||||
LATEST = 10
|
||||
MATCH_SCHEMA = {"type": "object", "properties": {"found": {"type": "array", "items": {"type": "integer"}}}, "required": ["found"], "additionalProperties": False}
|
||||
|
||||
@@ -71,7 +84,7 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
read = arguments_or_exit(TOOL, argv, __doc__, USAGE, pass_and_model)
|
||||
if isinstance(read, int):
|
||||
return read
|
||||
reviewed, project = read
|
||||
reviewed, project, without = read
|
||||
rounds = archived_rounds(REVIEWERS[reviewed], project.model, archive_root(home, env), home)
|
||||
if isinstance(rounds, Failure):
|
||||
return failed(TOOL, rounds.reason)
|
||||
@@ -79,6 +92,8 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
if not codes:
|
||||
return failed(TOOL, f"no archived {reviewed} round of projects on {project.model} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find")
|
||||
chosen = sample(list(codes), LATEST)
|
||||
if without:
|
||||
return compared(reviewed, project, chosen, lambda found, bare: findings_found(found, codes[found], project, project, home, effects, bare))
|
||||
# The project's rung is the reference, so the candidates start below it. They reach the ladder's lowest rung, below the lowest a worker routes to, since a pass may run below the worker's rung.
|
||||
candidates = down_from(project, LADDER[0])[1:]
|
||||
|
||||
@@ -92,13 +107,28 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
return 0
|
||||
|
||||
|
||||
def pass_and_model(argv: list[str]) -> tuple[ReviewedPass, Rung] | None:
|
||||
"""The pass and the project's rung of the model argv names, as `<pass> <model>`; None for any other argv."""
|
||||
if len(argv) != 2:
|
||||
def compared(reviewed: str, project: Rung, chosen: list[Round], found_on: typing.Callable[[Round, bool], set[int] | Limited]) -> int:
|
||||
"""Replays chosen twice with CLAUDE.md and once without, as found_on(round, bare) finds, and prints the comparison."""
|
||||
runs: list[list[set[int]]] = []
|
||||
for bare in (False, False, True):
|
||||
def replay_one(one: Round, _: Rung, bare: bool = bare) -> set[int] | Limited:
|
||||
return found_on(one, bare)
|
||||
|
||||
found = replayed_on(chosen, project, replay_one, f"{reviewed} rounds {'without' if bare else 'with'} the instruction files")
|
||||
if isinstance(found, Limited):
|
||||
return limited_exit(TOOL, found)
|
||||
runs.append(found)
|
||||
print(comparison(reviewed, project, chosen, (runs[0], runs[1]), runs[2]))
|
||||
return 0
|
||||
|
||||
|
||||
def pass_and_model(argv: list[str]) -> tuple[ReviewedPass, Rung, bool] | None:
|
||||
"""The pass, the project's rung of the model, and whether WITHOUT follows, as argv names them in `<pass> <model> [--without-claude-md]`; None for any other argv."""
|
||||
if argv[2:] not in ([], [WITHOUT]) or len(argv) < 2:
|
||||
return None
|
||||
reviewed = next((kind for kind in REVIEWERS if kind == argv[0]), None)
|
||||
project = project_rung(argv[1])
|
||||
return (reviewed, project) if reviewed and project else None
|
||||
return (reviewed, project, argv[2:] == [WITHOUT]) if reviewed and project else None
|
||||
|
||||
|
||||
def codes_found(rounds: list[Round], effects: Effects) -> dict[Round, Code]:
|
||||
@@ -177,9 +207,9 @@ def commit_log(listed: str) -> list[tuple[str, float]]:
|
||||
return [(commit, float(authored)) for commit, authored in (line.split() for line in listed.splitlines())]
|
||||
|
||||
|
||||
def findings_found(found: Round, code: Code, rung: Rung, project: Rung, home: pathlib.Path, effects: Effects) -> set[int] | Limited:
|
||||
"""The indexes of the round's findings that its replay on rung reports, as claude on the project's rung matches them."""
|
||||
reviewed: str | Failure | Limited = first_reply(lambda: review(found, code, rung, home, effects))
|
||||
def findings_found(found: Round, code: Code, rung: Rung, project: Rung, home: pathlib.Path, effects: Effects, bare: bool = False) -> set[int] | Limited:
|
||||
"""The indexes of the round's findings that its replay on rung reports, as claude on the project's rung matches them; where bare, the replay runs without CLAUDE.md, as `launch` says."""
|
||||
reviewed: str | Failure | Limited = first_reply(lambda: review(found, code, rung, home, effects, bare))
|
||||
if isinstance(reviewed, Limited):
|
||||
return reviewed
|
||||
matched: set[int] | Failure | Limited = reviewed if isinstance(reviewed, Failure) else first_reply(lambda: match(found.findings, reviewed, found.root, project, effects))
|
||||
@@ -197,17 +227,17 @@ class Copies:
|
||||
|
||||
|
||||
# The shell makes and removes the clone itself, and is tested on a temporary home as judge-replay's is; rounds.py's core takes the filesystem as an argument.
|
||||
def review(found: Round, code: Code, rung: Rung, home: pathlib.Path, effects: Effects) -> str | Failure | Limited:
|
||||
def review(found: Round, code: Code, rung: Rung, home: pathlib.Path, effects: Effects, bare: bool) -> str | Failure | Limited:
|
||||
try:
|
||||
(home / REPORTS).mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory(prefix="review-replay-", ignore_cleanup_errors=True) as clone, tempfile.TemporaryDirectory(prefix="review-replay-", dir=home / REPORTS, ignore_cleanup_errors=True) as reports:
|
||||
return reviewed_in(found, code, rung, Copies(clone, reports), home, effects)
|
||||
return reviewed_in(found, code, rung, Copies(clone, reports), home, effects, bare)
|
||||
except OSError as failure:
|
||||
return Failure(f"could not prepare the replay: {failure}")
|
||||
|
||||
|
||||
def reviewed_in(found: Round, code: Code, rung: Rung, copies: Copies, home: pathlib.Path, effects: Effects) -> str | Failure | Limited:
|
||||
"""The report of found's reviewer on rung, replayed on code cloned into copies.clone, with its review folder copied into copies.reports."""
|
||||
def reviewed_in(found: Round, code: Code, rung: Rung, copies: Copies, home: pathlib.Path, effects: Effects, bare: bool) -> str | Failure | Limited:
|
||||
"""The report of found's reviewer on rung, replayed on code cloned into copies.clone, with its review folder copied into copies.reports; without CLAUDE.md where bare."""
|
||||
folder = pathlib.Path(review_folder(str(home), code.path))
|
||||
copy_reports(found, folder, copies.reports)
|
||||
prompt = redirected(found.prompt, home, folder, code.path, copies)
|
||||
@@ -217,12 +247,26 @@ def reviewed_in(found: Round, code: Code, rung: Rung, copies: Copies, home: path
|
||||
return Failure(f"the prompt names {', '.join(lost)}, which the review folder did not hold from the chunk's start until the round's")
|
||||
if isinstance(cloned := clone_reviewed(found, code, copies.clone, effects), Failure):
|
||||
return cloned
|
||||
reply = claude_reply(effects.claude, ["--agent", found.reviewer, *rung.flags()], copies.clone, prompt, "claude")
|
||||
launched = launch(found.reviewer, rung, prompt, copies.clone, home, bare)
|
||||
if isinstance(launched, Failure):
|
||||
return launched
|
||||
reply = claude_reply(effects.claude, launched[0], copies.clone, launched[1], "claude")
|
||||
if isinstance(reply, Failure | Limited):
|
||||
return reply
|
||||
return reply["result"] if isinstance(reply.get("result"), str) else Failure("the reviewer replied with no report")
|
||||
|
||||
|
||||
def launch(reviewer: str, rung: Rung, prompt: str, clone: str, home: pathlib.Path, bare: bool) -> tuple[list[str], str] | Failure:
|
||||
"""claude's arguments and prompt for reviewer on rung. Where bare, it starts without CLAUDE.md, and the prompt holds what the SubagentStart hook and the header's last line hand such a reviewer."""
|
||||
args = ["--agent", reviewer, *rung.flags()]
|
||||
if not bare:
|
||||
return args, prompt
|
||||
context = reviewer_context(home)
|
||||
if isinstance(context, Failure):
|
||||
return context
|
||||
return [*args, *NO_CLAUDE_MD], "\n".join([f"{context}\n\n{prompt}", *([principles] if (principles := principles_line(clone)) else [])])
|
||||
|
||||
|
||||
def copy_reports(found: Round, folder: pathlib.Path, reports: str) -> None:
|
||||
"""Copies into reports each report written to folder after found's chunk began and before its round did."""
|
||||
# The round's own report, any later one, and one from an earlier attempt at the item in the same worktree would each hand the reviewer the findings it is to find.
|
||||
|
||||
@@ -116,3 +116,20 @@ def report(reviewed: str, model: str, reference: Rung, rounds: list[Round], repl
|
||||
lowest = replay.lowest or "none below the project's rung"
|
||||
lines.append(f"Lowest rung that finds every finding the reference reviewer finds: {lowest}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def comparison(reviewed: str, reference: Rung, rounds: list[Round], with_it: tuple[list[set[int]], list[set[int]]], without: list[set[int]]) -> str:
|
||||
"""What the two replays with CLAUDE.md and the one without found in rounds, and each finding the one without missed that both of the others found."""
|
||||
first, second = with_it
|
||||
both = [one & other for one, other in zip(first, second)]
|
||||
missed = [(found_round, index) for found_round, held, found in zip(rounds, both, without) for index in sorted(held - found)]
|
||||
|
||||
def count(run: list[set[int]]) -> int:
|
||||
return sum(len(found) for found in run)
|
||||
|
||||
return "\n".join([
|
||||
f"Replayed {len(rounds)} {reviewed} rounds of {reference.model} projects three times on {reference}, holding {sum(len(found_round.findings) for found_round in rounds)} 🔴 and 🟡 findings.",
|
||||
f"With CLAUDE.md the reviewer found {count(first)} and then {count(second)} of them, and {count(both)} both times.",
|
||||
f"Without CLAUDE.md it found {count(without)} of them, and missed {len(missed)} that it found both times with it{':' if missed else '.'}",
|
||||
*(f" {found_round.root} {found_round.at}: {found_round.findings[index].splitlines()[0]}" for found_round, index in missed),
|
||||
])
|
||||
|
||||
@@ -20,6 +20,7 @@ from claude_code.entries import entry_from
|
||||
from failure import Failure
|
||||
from review_replay import main
|
||||
from rung_replay.search import Miss, Replay, Tried
|
||||
from review_rules.reviewer_context import reviewer_context
|
||||
from review_rules.review_passes import REVIEWERS
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
@@ -130,6 +131,28 @@ class Report(unittest.TestCase):
|
||||
self.assertEqual(report("stability", "claude-opus-5-5", REFERENCE, rounds, Replay([set(), set()], [], None)).splitlines()[-1], "No rung tried: the reference reviewer found no finding to hold one to.")
|
||||
|
||||
|
||||
class Launch(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.home = pathlib.Path(self.enterContext(tempfile.TemporaryDirectory()))
|
||||
(self.home / ".claude").mkdir()
|
||||
(self.home / ".claude" / "CLAUDE.md").write_text("## Prose\n\nShort.\n\n## Vocabulary\n\nWords.\n")
|
||||
self.clone = self.home / "clone"
|
||||
self.clone.mkdir()
|
||||
|
||||
def test_runs_the_reviewer_as_archived_with_claude_md(self):
|
||||
self.assertEqual(main.launch("prose-reviewer", REFERENCE, "Review it.", str(self.clone), self.home, False), (["--agent", "prose-reviewer", *REFERENCE.flags()], "Review it."))
|
||||
|
||||
def test_drops_claude_md_and_hands_the_reviewer_what_the_hook_and_the_header_would(self):
|
||||
(self.clone / "AGENTS.md").write_text("")
|
||||
args, prompt = main.launch("prose-reviewer", REFERENCE, "Review it.", str(self.clone), self.home, True)
|
||||
self.assertEqual(args, ["--agent", "prose-reviewer", *REFERENCE.flags(), "--settings", json.dumps({"env": {"CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1"}})])
|
||||
self.assertEqual(prompt, f"{reviewer_context(self.home)}\n\nReview it.\nPrinciples: read {self.clone}/AGENTS.md first, for the project principles and the decision titles.")
|
||||
|
||||
def test_fails_without_the_sections_to_hand_over(self):
|
||||
(self.home / ".claude" / "CLAUDE.md").write_text("")
|
||||
self.assertIsInstance(main.launch("prose-reviewer", REFERENCE, "Review it.", str(self.clone), self.home, True), Failure)
|
||||
|
||||
|
||||
class Match(unittest.TestCase):
|
||||
def run_match(self, stdout):
|
||||
self.asked = []
|
||||
@@ -185,6 +208,7 @@ class Main(unittest.TestCase):
|
||||
run_git(self.checkout, "remote", "set-head", "origin", "main")
|
||||
for key in ("proj", "other"):
|
||||
self.write_chunk(key, "c1", "2026-10-14T10:00:00Z")
|
||||
(self.home / ".claude" / "CLAUDE.md").write_text("## Prose\n\nShort.\n\n## Vocabulary\n\nWords.\n")
|
||||
self.reviews = []
|
||||
self.agents = set()
|
||||
self.matched_in = []
|
||||
@@ -236,6 +260,25 @@ class Main(unittest.TestCase):
|
||||
code = main.main(argv, {}, self.home, effects)
|
||||
return code, out.getvalue(), err.getvalue()
|
||||
|
||||
def test_without_claude_md_replays_each_round_twice_with_it_and_once_without_on_the_project_rung(self):
|
||||
finds = iter(([0, 1], [0], [1]))
|
||||
code, out, err = self.run_main(["stability", "claude-opus-5-5", "--without-claude-md"], lambda rung: {"found": next(finds)})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE] * 3)
|
||||
context = reviewer_context(self.home)
|
||||
self.assertEqual([prompt.startswith(context) for *_, prompt, _ in self.reviews], [False, False, True])
|
||||
self.assertEqual(out, (
|
||||
f"Replayed 1 stability rounds of claude-opus-5-5 projects three times on {REFERENCE}, holding 2 🔴 and 🟡 findings.\n"
|
||||
"With CLAUDE.md the reviewer found 2 and then 1 of them, and 1 both times.\n"
|
||||
"Without CLAUDE.md it found 1 of them, and missed 1 that it found both times with it:\n"
|
||||
f" {self.checkout} 2026-10-14T10:00:00Z: 🔴 a.txt:1 wrong\n"
|
||||
))
|
||||
|
||||
def test_without_claude_md_stops_at_a_usage_limit(self):
|
||||
code, out, _ = self.run_main(["stability", "claude-opus-5-5", "--without-claude-md"], lambda rung: {"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
|
||||
self.assertEqual((code, out), (1, ""))
|
||||
self.assertEqual(len(self.reviews), 1)
|
||||
|
||||
def test_replays_the_models_rounds_on_the_commit_reviewed_from_the_project_rung_down(self):
|
||||
code, out, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0] if rung == MIDDLE else [0, 1]})
|
||||
self.assertEqual(code, 0, err)
|
||||
@@ -427,7 +470,7 @@ class Main(unittest.TestCase):
|
||||
|
||||
def test_prints_the_help(self):
|
||||
code, out, _ = self.run_main(["-h"], unused)
|
||||
self.assertEqual((code, out.splitlines()[2]), (0, "usage: review-replay <pass> <model>"))
|
||||
self.assertEqual((code, out.splitlines()[2]), (0, "usage: review-replay <pass> <model> [--without-claude-md]"))
|
||||
self.assertIn(f"{', '.join(list(REVIEWERS)[:-1])} or {list(REVIEWERS)[-1]}.", " ".join(out.split()))
|
||||
|
||||
def test_the_launcher_prints_the_help(self):
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
"""What a reviewer under agents/ is handed in place of the instruction files: `omitClaudeMd` starts it without them, and most of `~/.claude/CLAUDE.md` is the worker's. docs/decisions.md → `agents/: a reviewer starts without CLAUDE.md` says why."""
|
||||
import json
|
||||
import pathlib
|
||||
|
||||
from failure import Failure
|
||||
from review_rules.review_passes import REVIEWERS
|
||||
|
||||
# The sections of ~/.claude/CLAUDE.md a reviewer needs: the words its reports use and the bar its own prose meets.
|
||||
SECTIONS = ("Prose", "Vocabulary")
|
||||
|
||||
|
||||
def reviewer_context(home: pathlib.Path) -> str | Failure:
|
||||
"""SECTIONS of the CLAUDE.md under home, as a reviewer is told them; a Failure where the file or a section is missing."""
|
||||
path = home / ".claude" / "CLAUDE.md"
|
||||
try:
|
||||
text = path.read_text(encoding="utf-8")
|
||||
except (OSError, UnicodeDecodeError) as failure:
|
||||
return Failure(f"could not read {path}: {failure}")
|
||||
found: dict[str, list[str]] = {}
|
||||
lines: list[str] = []
|
||||
for line in text.splitlines():
|
||||
if line.startswith("## "):
|
||||
lines = found.setdefault(line[3:].strip(), [])
|
||||
lines.append(line)
|
||||
if missing := [name for name in SECTIONS if name not in found]:
|
||||
return Failure(f"{path} holds no section {', '.join(missing)}")
|
||||
return "\n\n".join(["You start without `~/.claude/CLAUDE.md`. These sections of it hold for you:", *("\n".join(found[name]).strip() for name in SECTIONS)])
|
||||
|
||||
|
||||
def hook_reply(given: str, home: pathlib.Path) -> str | Failure:
|
||||
"""What the SubagentStart hook prints for its input given: the context for a reviewer under agents/, nothing for any other agent."""
|
||||
try:
|
||||
read = json.loads(given)
|
||||
except ValueError:
|
||||
return ""
|
||||
if not isinstance(read, dict) or read.get("agent_type") not in REVIEWERS.values():
|
||||
return ""
|
||||
context = reviewer_context(home)
|
||||
if isinstance(context, Failure):
|
||||
return context
|
||||
return json.dumps({"hookSpecificOutput": {"hookEventName": "SubagentStart", "additionalContext": context}})
|
||||
|
||||
|
||||
def principles_line(worktree: str) -> str | None:
|
||||
"""The line of a reviewer's header that names where the project in worktree keeps its principles and decision titles, or None where it keeps neither file."""
|
||||
# The header names the file rather than quoting it, since the worker types the header into each reviewer's prompt.
|
||||
what = "for the project principles and the decision titles"
|
||||
if (pathlib.Path(worktree) / "AGENTS.md").is_file():
|
||||
return f"Principles: read {worktree}/AGENTS.md first, {what}."
|
||||
if (pathlib.Path(worktree) / "docs" / "decisions.md").is_file():
|
||||
return f"Principles: read {worktree}/docs/decisions.md first, up to `# Decisions`, {what}."
|
||||
return None
|
||||
@@ -0,0 +1,82 @@
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from failure import Failure
|
||||
from review_rules.reviewer_context import hook_reply, principles_line, reviewer_context, SECTIONS
|
||||
from review_rules.review_passes import REVIEWERS
|
||||
|
||||
REPO = pathlib.Path(__file__).resolve().parents[2]
|
||||
CLAUDE_MD = "# Global\n\n## Working loop\n\nThe worker's.\n\n## Prose\n\nWrite short.\n\n## Git\n\nOne line.\n\n## Vocabulary\n\n- **chunk** — a unit.\n"
|
||||
|
||||
|
||||
class ReviewerContext(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.home = pathlib.Path(self.enterContext(tempfile.TemporaryDirectory()))
|
||||
(self.home / ".claude").mkdir()
|
||||
(self.home / ".claude" / "CLAUDE.md").write_text(CLAUDE_MD)
|
||||
|
||||
def test_holds_the_prose_and_vocabulary_sections_and_none_of_the_workers(self):
|
||||
self.assertEqual(reviewer_context(self.home), "You start without `~/.claude/CLAUDE.md`. These sections of it hold for you:\n\n## Prose\n\nWrite short.\n\n## Vocabulary\n\n- **chunk** — a unit.")
|
||||
|
||||
def test_fails_on_a_missing_section_or_file(self):
|
||||
(self.home / ".claude" / "CLAUDE.md").write_text("## Prose\n\nWrite short.\n")
|
||||
self.assertEqual(reviewer_context(self.home), Failure(f"{self.home / '.claude' / 'CLAUDE.md'} holds no section Vocabulary"))
|
||||
(self.home / ".claude" / "CLAUDE.md").unlink()
|
||||
self.assertIsInstance(reviewer_context(self.home), Failure)
|
||||
|
||||
def test_the_repos_claude_md_holds_each_section(self):
|
||||
headings = [line[3:] for line in (REPO / "CLAUDE.md").read_text().splitlines() if line.startswith("## ")]
|
||||
self.assertEqual([name for name in SECTIONS if name not in headings], [])
|
||||
|
||||
def test_the_hook_hands_a_reviewer_the_context_and_any_other_agent_nothing(self):
|
||||
for agent in REVIEWERS.values():
|
||||
with self.subTest(agent=agent):
|
||||
reply = hook_reply(json.dumps({"hook_event_name": "SubagentStart", "agent_type": agent}), self.home)
|
||||
self.assertEqual(json.loads(reply), {"hookSpecificOutput": {"hookEventName": "SubagentStart", "additionalContext": reviewer_context(self.home)}})
|
||||
for given in (json.dumps({"agent_type": "Explore"}), json.dumps({}), json.dumps([]), "no json"):
|
||||
with self.subTest(given=given):
|
||||
self.assertEqual(hook_reply(given, self.home), "")
|
||||
|
||||
def test_the_hook_fails_for_a_reviewer_when_a_section_is_missing(self):
|
||||
(self.home / ".claude" / "CLAUDE.md").write_text("")
|
||||
self.assertIsInstance(hook_reply(json.dumps({"agent_type": "prose-reviewer"}), self.home), Failure)
|
||||
|
||||
|
||||
class PrinciplesLine(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.worktree = pathlib.Path(self.enterContext(tempfile.TemporaryDirectory()))
|
||||
|
||||
def test_names_agents_md_where_the_project_keeps_one(self):
|
||||
(self.worktree / "AGENTS.md").write_text("")
|
||||
(self.worktree / "docs").mkdir()
|
||||
(self.worktree / "docs" / "decisions.md").write_text("")
|
||||
self.assertEqual(principles_line(str(self.worktree)), f"Principles: read {self.worktree}/AGENTS.md first, for the project principles and the decision titles.")
|
||||
|
||||
def test_names_the_head_of_the_decision_log_where_the_project_keeps_no_agents_md(self):
|
||||
(self.worktree / "docs").mkdir()
|
||||
(self.worktree / "docs" / "decisions.md").write_text("")
|
||||
self.assertEqual(principles_line(str(self.worktree)), f"Principles: read {self.worktree}/docs/decisions.md first, up to `# Decisions`, for the project principles and the decision titles.")
|
||||
|
||||
def test_names_nothing_where_the_project_keeps_neither(self):
|
||||
self.assertIsNone(principles_line(str(self.worktree)))
|
||||
|
||||
|
||||
class Reviewers(unittest.TestCase):
|
||||
def test_each_agent_starts_without_claude_md(self):
|
||||
for path in sorted((REPO / "agents").glob("*.md")):
|
||||
with self.subTest(agent=path.stem):
|
||||
frontmatter = path.read_text().split("---\n")[1]
|
||||
self.assertIn("omitClaudeMd: true\n", frontmatter)
|
||||
|
||||
def test_settings_run_the_hook_as_each_subagent_starts(self):
|
||||
hooks = json.loads((REPO / "settings.json").read_text())["hooks"]["SubagentStart"]
|
||||
self.assertEqual(hooks, [{"hooks": [{"command": "~/.claude/bin/larv-review reviewer-context", "timeout": 30, "type": "command"}]}])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -34,6 +34,17 @@
|
||||
],
|
||||
"matcher": "Agent|Bash|Edit|MultiEdit|Read|Skill|Write"
|
||||
}
|
||||
],
|
||||
"SubagentStart": [
|
||||
{
|
||||
"hooks": [
|
||||
{
|
||||
"command": "~/.claude/bin/larv-review reviewer-context",
|
||||
"timeout": 30,
|
||||
"type": "command"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"includeCoAuthoredBy": false,
|
||||
|
||||
Reference in New Issue
Block a user