Launch comprehension seats as an agent type of their own, without CLAUDE.md
CI / tests (pull_request) Successful in 4m45s
CI / tests (pull_request) Successful in 4m45s
This commit is contained in:
@@ -0,0 +1,8 @@
|
||||
---
|
||||
name: comprehension-seat
|
||||
description: "Use this agent for a seat of larv-review's comprehension check, with the prompt `larv-review prepare-scoring` writes or the paired ruling's brief in skills/comprehension-panel/seat-prompts.md."
|
||||
tools: Read, Grep, Glob, Bash
|
||||
omitClaudeMd: true
|
||||
---
|
||||
|
||||
You are a simulated reader of a project, and the prompt you are launched with says who: your background, the checkout you read, what you may open and the report to write. Read the project as that reader meets it, knowing only what its files tell you. Nothing here adds to the prompt; follow it exactly.
|
||||
@@ -93,9 +93,7 @@ skills/comprehension-panel/seat-prompts.md builds them. It prints the worktree's
|
||||
prompt's. It takes {{language}}, {{kind}}, {{domain}}, {{domain docs}} and {{3am question}} from the
|
||||
checkout's AGENTS.md → "Scoring run". claude-config keeps no AGENTS.md, so there it takes them from
|
||||
seat-prompts.md → "claude-config's values". It fills the other placeholders from the worktree and
|
||||
the panel's SKILL.md. --preloaded replaces {{preloaded}}, which by default names each CLAUDE.md the main checkout's claudeMdExcludes doesn't exclude, its
|
||||
imports and each rule file without `paths:` that Claude Code loads into a seat. Where any one of
|
||||
the five values is missing, it exits 1 and doesn't write anything. Where a setup that
|
||||
the panel's SKILL.md. Where any one of the five values is missing, it exits 1 and doesn't write anything. Where a setup that
|
||||
record-scoring never removed is still there, it removes it first and says so.
|
||||
|
||||
record-scoring takes the scores file of a run needs-scoring-run calls, whose first line names no
|
||||
@@ -213,7 +211,7 @@ def run_needs_scoring_run(given: list[str]) -> Failure | None:
|
||||
|
||||
|
||||
def fits_prepare_scoring(given: list[str]) -> bool:
|
||||
return not given or len(given) == 2 and given[0] == "--preloaded"
|
||||
return not given
|
||||
|
||||
|
||||
def run_record_scoring(given: list[str]) -> Failure | None:
|
||||
@@ -236,7 +234,7 @@ COMMANDS = {
|
||||
"needs-scoring-run": Command("needs-scoring-run [<item ID>...]", are_ids, run_needs_scoring_run, ids_from=0),
|
||||
"scoring-run-owed": Command("scoring-run-owed", no_arguments, lambda given: with_default_branch(lambda default: owed(default, pathlib.Path.home()), stale_ok=False)),
|
||||
"record-ruling": Command("record-ruling <votes file>", lambda given: len(given) == 1, lambda given: printed(record_ruling(pathlib.Path(given[0]), pathlib.Path.home(), time.time()))),
|
||||
"prepare-scoring": Command("prepare-scoring [--preloaded <text>]", fits_prepare_scoring, lambda given: prepared_scoring(given[1] if given else None)),
|
||||
"prepare-scoring": Command("prepare-scoring", fits_prepare_scoring, lambda given: prepared_scoring()),
|
||||
"record-scoring": Command("record-scoring <scores file> [<item ID>...]", lambda given: bool(given) and are_ids(given[1:]), run_record_scoring, ids_from=1),
|
||||
"places": Command("places", no_arguments, lambda given: printed(read_places(pathlib.Path.home()))),
|
||||
"agreement": Command("agreement", no_arguments, lambda given: printed(read_agreement(pathlib.Path.home()))),
|
||||
@@ -293,9 +291,9 @@ def with_default_branch(command: typing.Callable[[str], Failure | None], stale_o
|
||||
return command(branch)
|
||||
|
||||
|
||||
def prepared_scoring(preloaded: str | None) -> Failure | None:
|
||||
def prepared_scoring() -> Failure | None:
|
||||
worktree = top()
|
||||
return worktree if isinstance(worktree, Failure) else printed(prepare_scoring(worktree, preloaded, pathlib.Path.home()))
|
||||
return worktree if isinstance(worktree, Failure) else printed(prepare_scoring(worktree, pathlib.Path.home()))
|
||||
|
||||
|
||||
def make_review_folder() -> Failure | None:
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
"""Set up a scoring run: the worktree its seats read and each seat's prompt, filled from skills/comprehension-panel/seat-prompts.md."""
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
@@ -13,7 +12,6 @@ from review_rules.review_folder import made_review_folder, review_folder
|
||||
from project_config.settings import ProjectConfig
|
||||
from record_store.stores import store
|
||||
|
||||
CLAUDE_MD = "CLAUDE.md"
|
||||
PANEL = pathlib.Path(__file__).resolve().parents[2] / "skills" / "comprehension-panel"
|
||||
# The file name of each seat's prompt, its label in seat-prompts.md → "Backgrounds", and the heading of its report there.
|
||||
SEATS = (
|
||||
@@ -24,6 +22,8 @@ SEATS = (
|
||||
)
|
||||
# The dimensions a seat scores beside the overall, which comes last in DIMENSIONS.
|
||||
PANEL_DIMENSIONS = DIMENSIONS[:-1]
|
||||
# The agent under agents/ that runs each seat: it starts without CLAUDE.md, so a seat holds only what its prompt and its reads give it.
|
||||
SEAT_AGENT = "comprehension-seat"
|
||||
VALUE_NAMES = ("language", "kind", "domain", "domain docs", "3am question")
|
||||
# The files no seat may open, by the setting that places each, and what each is.
|
||||
FORBIDDEN = (("plan", "the plan"), ("decisions", "the decision log"), ("comprehension-history", "past panels' records"), ("reading-cost", "past chunks' reading cost"))
|
||||
@@ -103,45 +103,6 @@ def read(path: pathlib.Path) -> str | Failure:
|
||||
return Failure(f"cannot read {path}: {failure}")
|
||||
|
||||
|
||||
def preloaded_files(main: pathlib.Path, home: pathlib.Path) -> str | Failure:
|
||||
"""The files Claude Code loads into a seat started in main before its first tool call: each CLAUDE.md, the files it imports, and each rule file without `paths:`, leaving out those `claudeMdExcludes` names. A Failure where one of them cannot be read."""
|
||||
excludes = claude_md_excludes(main, home)
|
||||
found = []
|
||||
|
||||
def kept(path: pathlib.Path) -> bool:
|
||||
return not any(path.full_match(pattern) for pattern in excludes)
|
||||
|
||||
for shown, claude_md, rules in (("~/.claude/", home / ".claude" / CLAUDE_MD, home / ".claude" / "rules"), ("", main / CLAUDE_MD, main / ".claude" / "rules")):
|
||||
if claude_md.is_file() and kept(claude_md):
|
||||
text = read(claude_md)
|
||||
if isinstance(text, Failure):
|
||||
return text
|
||||
found.append(f"`{shown}{CLAUDE_MD}`")
|
||||
found += [f"`{shown}{name}`, which it imports" for name in re.findall(r"^@(\S+)", text, re.M)]
|
||||
for rule in filter(kept, sorted(rules.glob("*.md"))):
|
||||
text = read(rule)
|
||||
if isinstance(text, Failure):
|
||||
return text
|
||||
if not re.search(r"^paths:", text, re.M):
|
||||
found.append(f"`{shown}{rule.relative_to(claude_md.parent)}`")
|
||||
return listed(found, "and") if found else "none"
|
||||
|
||||
|
||||
def claude_md_excludes(main: pathlib.Path, home: pathlib.Path) -> list[str]:
|
||||
"""The `claudeMdExcludes` globs of the user's settings and main's project and local settings, merged as Claude Code merges them across layers.
|
||||
|
||||
Managed settings are left out, since they may come from the claude.ai console or an MDM policy, where no file shows them."""
|
||||
globs = []
|
||||
for path in (home / ".claude" / "settings.json", main / ".claude" / "settings.json", main / ".claude" / "settings.local.json"):
|
||||
try:
|
||||
settings = json.loads(path.read_text())
|
||||
except (OSError, ValueError):
|
||||
continue
|
||||
excludes = settings.get("claudeMdExcludes") if isinstance(settings, dict) else None
|
||||
globs += [glob for glob in excludes if isinstance(glob, str)] if isinstance(excludes, list) else []
|
||||
return globs
|
||||
|
||||
|
||||
def prompt_parts() -> dict[str, str] | Failure:
|
||||
"""seat-prompts.md's opening, shared brief and each report as their fenced text, and each seat's background, keyed by heading or label."""
|
||||
path = PANEL / "seat-prompts.md"
|
||||
@@ -167,7 +128,7 @@ def scoring_folder(worktree: str, home: pathlib.Path) -> pathlib.Path:
|
||||
return pathlib.Path(review_folder(str(home), worktree)) / "scoring"
|
||||
|
||||
|
||||
def prepare_scoring(worktree: str, preloaded: str | None, home: pathlib.Path) -> str | Failure:
|
||||
def prepare_scoring(worktree: str, home: pathlib.Path) -> str | Failure:
|
||||
"""Makes the scoring run's worktree at worktree's HEAD and writes each seat's prompt beside it, in the chunk's review folder; what it made, as the tool prints it."""
|
||||
folder = scoring_folder(worktree, home)
|
||||
# A setup record-scoring never removed, as where the seats failed, would otherwise block every later run.
|
||||
@@ -204,10 +165,6 @@ def prepare_scoring(worktree: str, preloaded: str | None, home: pathlib.Path) ->
|
||||
if isinstance(head, Failure):
|
||||
return head
|
||||
main = listing.splitlines()[0].removeprefix("worktree ")
|
||||
# Read before the scoring worktree is made, so a file that cannot be read leaves none behind.
|
||||
loaded = preloaded if preloaded is not None else preloaded_files(pathlib.Path(main), home)
|
||||
if isinstance(loaded, Failure):
|
||||
return loaded
|
||||
made = made_review_folder(worktree, home)
|
||||
if isinstance(made, Failure):
|
||||
return made
|
||||
@@ -227,7 +184,6 @@ def prepare_scoring(worktree: str, preloaded: str | None, home: pathlib.Path) ->
|
||||
"forbidden": forbidden_words,
|
||||
"git": f"`git -C {checkout} ls-files`",
|
||||
"files": file_globs(tracked.splitlines(), forbidden_files),
|
||||
"preloaded": loaded,
|
||||
"anchors": "\n".join(score[0]),
|
||||
"dimensions": "\n".join(score[1]),
|
||||
}
|
||||
@@ -242,7 +198,7 @@ def prepare_scoring(worktree: str, preloaded: str | None, home: pathlib.Path) ->
|
||||
return Failure(f"could not write the seat's prompt {path}: {failure}")
|
||||
written.append(str(path))
|
||||
before = f"Removed the scoring run set up before in {folder}.\n" if replaced else ""
|
||||
return before + f"Scoring worktree, at {head.strip()}: {checkout}\nPrompts:\n" + "\n".join(written)
|
||||
return before + f"Scoring worktree, at {head.strip()}: {checkout}\nPrompts, each for one {SEAT_AGENT} agent:\n" + "\n".join(written)
|
||||
|
||||
|
||||
def removed_setup(worktree: str, home: pathlib.Path) -> Failure | None:
|
||||
|
||||
@@ -941,16 +941,9 @@ class LarvReviewTool(unittest.TestCase):
|
||||
|
||||
VALUES = {"AGENTS.md": "## Scoring run\n\n- Language: Python\n- Kind: a CLI\n- Domain: parsing\n- Domain docs: the spec\n- 3am question: the parse hangs\n"}
|
||||
|
||||
def test_prepare_scoring_puts_the_preloaded_text_it_is_given_in_the_seats_prompts(self):
|
||||
def test_prepare_scoring_refuses_any_argument(self):
|
||||
self.commit("Values", self.VALUES)
|
||||
prepared = self.run_tool("prepare-scoring", "--preloaded", "the files the harness loaded")
|
||||
self.assertEqual(prepared.returncode, 0, prepared.stderr)
|
||||
prompts = [path.read_text() for path in (self.checkout.parent / ".local" / "state" / "larv-review").rglob("*.md") if "scoring" in path.parts]
|
||||
self.assertTrue(any("the files the harness loaded" in prompt for prompt in prompts), prepared.stdout)
|
||||
|
||||
def test_prepare_scoring_refuses_a_preloaded_flag_without_its_text_and_any_other_arguments(self):
|
||||
self.commit("Values", self.VALUES)
|
||||
for given in (["--preloaded"], ["text"], ["text", "--preloaded"], ["--preloaded", "text", "more"], ["--other", "text"]):
|
||||
for given in (["--preloaded"], ["--preloaded", "text"], ["text"]):
|
||||
with self.subTest(given=given):
|
||||
ran = self.run_tool("prepare-scoring", *given)
|
||||
self.assertEqual((ran.returncode, ran.stdout), (1, ""))
|
||||
|
||||
@@ -9,7 +9,7 @@ import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from larv_review.scoring_setup import bullets, CLAUDE_MD, file_globs, preloaded_files, prepare_scoring, project_values, removed_setup, score_bullets
|
||||
from larv_review.scoring_setup import bullets, file_globs, prepare_scoring, project_values, removed_setup, score_bullets
|
||||
from failure import Failure
|
||||
from test_env import isolated_env
|
||||
from project_config.test_project_config import write_config
|
||||
@@ -101,21 +101,21 @@ class PrepareScoring(unittest.TestCase):
|
||||
self.git("add", "-A")
|
||||
self.git("commit", "-q", "-m", "start")
|
||||
|
||||
def prepare(self, preloaded=None):
|
||||
def prepare(self):
|
||||
"""What prepare-scoring returns in the checkout, run in this process so mutation tests see the lines it runs."""
|
||||
with mock.patch.dict(os.environ, self.env, clear=True):
|
||||
return prepare_scoring(str(self.checkout), preloaded, pathlib.Path.home())
|
||||
return prepare_scoring(str(self.checkout), pathlib.Path.home())
|
||||
|
||||
def run_tool(self, *args):
|
||||
return subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, cwd=self.checkout, env=self.env, text=True)
|
||||
|
||||
def test_writes_each_seats_prompt_with_every_placeholder_filled_and_a_worktree_at_head(self):
|
||||
self.commit("# A\n\n## Scoring run\n\n- Language: Python\n- Kind: a CLI\n- Domain: parsing\n- Domain docs: the spec\n- 3am question: the parse hangs\n")
|
||||
ran = self.prepare("`README.md`")
|
||||
ran = self.prepare()
|
||||
folder = self.root / ".local" / "state" / "larv-review" / "proj" / "scoring"
|
||||
worktree = folder / "checkout"
|
||||
prompts = [folder / f"{seat}.md" for seat in ("junior-a", "mid-a", "maintainability-senior", "inherited-architect")]
|
||||
self.assertEqual(ran, f"Scoring worktree, at {self.git('rev-parse', '--short', 'HEAD').strip()}: {worktree}\nPrompts:\n" + "\n".join(str(prompt) for prompt in prompts))
|
||||
self.assertEqual(ran, f"Scoring worktree, at {self.git('rev-parse', '--short', 'HEAD').strip()}: {worktree}\nPrompts, each for one comprehension-seat agent:\n" + "\n".join(str(prompt) for prompt in prompts))
|
||||
self.assertEqual(subprocess.run(["git", "rev-parse", "HEAD"], cwd=worktree, env=self.env, capture_output=True, text=True).stdout, self.git("rev-parse", "HEAD"))
|
||||
junior, _, senior, architect = (prompt.read_text() for prompt in prompts)
|
||||
for prompt in (junior, architect):
|
||||
@@ -124,7 +124,6 @@ class PrepareScoring(unittest.TestCase):
|
||||
self.assertIn("\n\nYour background: ~2 years in Python, first codebase of this kind (a CLI); no exposure to the domain at all.\n\n", junior)
|
||||
self.assertIn(f"Target: the project checked out at {worktree} (branch `item-9`). Read ONLY there — never {self.checkout}, which is a different version.", junior)
|
||||
self.assertIn("every non-test tracked file: `bin/**`, `AGENTS.md` and `README.md`. Do NOT open todo.md, docs/decisions.md,", junior)
|
||||
self.assertIn("from a DIFFERENT version (main): `README.md`.", junior)
|
||||
self.assertIn("\n- 7 — Predictable. The layout answers", junior)
|
||||
self.assertIn("Report, in this shape:", senior)
|
||||
self.assertIn("~15 years taking over other people's systems", architect)
|
||||
@@ -153,7 +152,7 @@ class PrepareScoring(unittest.TestCase):
|
||||
(self.checkout / "plan").mkdir()
|
||||
(self.checkout / "plan" / "todo.md").write_text("# todo\n")
|
||||
self.commit("# The owners' own AGENTS.md, with no values\n")
|
||||
ran = self.prepare("`README.md`")
|
||||
ran = self.prepare()
|
||||
assert isinstance(ran, str), ran
|
||||
junior = pathlib.Path(ran.splitlines()[2]).read_text()
|
||||
self.assertIn("Your background: ~2 years in Go,", junior)
|
||||
@@ -163,7 +162,7 @@ class PrepareScoring(unittest.TestCase):
|
||||
self.config.write_text(self.config.read_text().replace('"checkout:', '"folder:').replace('"folder:README.md"', '"checkout:README.md"'))
|
||||
(self.config.parent / "AGENTS.md").write_text("# A\n\n## Scoring run\n\n- Language: Go\n- Kind: a CLI\n- Domain: parsing\n- Domain docs: the spec\n- 3am question: the parse hangs\n")
|
||||
self.commit()
|
||||
ran = self.prepare("`README.md`")
|
||||
ran = self.prepare()
|
||||
assert isinstance(ran, str), ran
|
||||
self.assertIn("every non-test tracked file: `bin/**` and `README.md`. Do NOT open any record of a past panel, nor any review output.", pathlib.Path(ran.splitlines()[2]).read_text())
|
||||
|
||||
@@ -175,74 +174,14 @@ class PrepareScoring(unittest.TestCase):
|
||||
self.assertFalse((self.root / ".local" / "state" / "larv-review" / "proj" / "scoring").exists())
|
||||
self.assertEqual(len(self.git("worktree", "list").splitlines()), 1)
|
||||
|
||||
def test_lists_the_harness_files_where_no_preloaded_is_given(self):
|
||||
(self.checkout / "CLAUDE.md").write_text("@AGENTS.md\n")
|
||||
self.commit("## Scoring run\n\n- Language: Python\n- Kind: a CLI\n- Domain: parsing\n- Domain docs: the spec\n- 3am question: the parse hangs\n")
|
||||
self.assertNotIsInstance(self.prepare(), Failure)
|
||||
junior = (self.root / ".local" / "state" / "larv-review" / "proj" / "scoring" / "junior-a.md").read_text()
|
||||
self.assertIn("from a DIFFERENT version (main): `CLAUDE.md` and `AGENTS.md`, which it imports.", junior)
|
||||
|
||||
def test_names_a_file_that_is_not_utf_8(self):
|
||||
(self.root / ".claude").mkdir()
|
||||
(self.root / ".claude" / "CLAUDE.md").write_bytes(b"\xff")
|
||||
unread = preloaded_files(self.checkout, self.root)
|
||||
assert isinstance(unread, Failure), unread
|
||||
self.assertTrue(unread.reason.startswith(f"cannot read {self.root / '.claude' / CLAUDE_MD}: "), unread.reason)
|
||||
|
||||
def test_fails_before_it_makes_the_worktree_where_a_file_it_reads_does_not_decode(self):
|
||||
values = "## Scoring run\n\n- Language: Python\n- Kind: a CLI\n- Domain: parsing\n- Domain docs: the spec\n- 3am question: the parse hangs\n"
|
||||
self.commit(values)
|
||||
(self.checkout / "CLAUDE.md").write_bytes(b"\xff")
|
||||
unread = self.prepare()
|
||||
assert isinstance(unread, Failure), unread
|
||||
self.assertTrue(unread.reason.startswith(f"cannot read {self.checkout / CLAUDE_MD}: "), unread.reason)
|
||||
self.assertEqual(len(self.git("worktree", "list").splitlines()), 1)
|
||||
(self.checkout / "CLAUDE.md").unlink()
|
||||
(self.checkout / "AGENTS.md").write_bytes(b"\xff")
|
||||
unread = self.prepare()
|
||||
assert isinstance(unread, Failure), unread
|
||||
self.assertTrue(unread.reason.startswith(f"cannot read {self.checkout}/AGENTS.md: "), unread.reason)
|
||||
|
||||
def test_leaves_out_a_claude_md_the_project_settings_exclude(self):
|
||||
(self.root / ".claude").mkdir()
|
||||
(self.root / ".claude" / "CLAUDE.md").write_text("# Home\n")
|
||||
(self.root / ".claude" / "rules").mkdir()
|
||||
rule = "always"
|
||||
(self.root / ".claude" / "rules" / f"{rule}.md").write_text("# Always\n")
|
||||
(self.checkout / ".claude").mkdir()
|
||||
(self.checkout / ".claude" / "settings.json").write_text('{"claudeMdExcludes": ["**/.claude/CLAUDE.md"]}\n')
|
||||
(self.checkout / ".claude" / "settings.local.json").write_text("not json")
|
||||
self.assertEqual(preloaded_files(self.checkout, self.root), f"`~/.claude/rules/{rule}.md`")
|
||||
(self.checkout / ".claude" / "settings.json").write_text('{"claudeMdExcludes": [3]}\n')
|
||||
self.assertEqual(preloaded_files(self.checkout, self.root), f"`~/.claude/CLAUDE.md` and `~/.claude/rules/{rule}.md`")
|
||||
|
||||
def test_names_each_import_and_each_rule_without_paths_leaving_out_what_the_user_settings_exclude(self):
|
||||
(self.root / ".claude" / "rules").mkdir(parents=True)
|
||||
(self.root / ".claude" / "settings.json").write_text('{"claudeMdExcludes": ["**/rules/skipped.md"]}')
|
||||
(self.root / ".claude" / "rules" / "skipped.md").write_text("# Skipped\n")
|
||||
(self.checkout / "CLAUDE.md").write_text("# Project\n\n@AGENTS.md\n@docs/x.md\n")
|
||||
(self.checkout / ".claude" / "rules").mkdir(parents=True)
|
||||
(self.checkout / ".claude" / "rules" / "always.md").write_text("# Always\n")
|
||||
(self.checkout / ".claude" / "rules" / "scoped.md").write_text("---\npaths: ['*.py']\n---\n")
|
||||
imports, rule = "`AGENTS.md`, which it imports, `docs/x.md`, which it imports", "always"
|
||||
self.assertEqual(preloaded_files(self.checkout, self.root), f"`{CLAUDE_MD}`, {imports} and `.claude/rules/{rule}.md`")
|
||||
|
||||
def test_reads_the_excludes_of_both_settings_files_and_names_none_where_nothing_loads(self):
|
||||
(self.root / ".claude").mkdir()
|
||||
(self.root / ".claude" / "CLAUDE.md").write_text("# Home\n")
|
||||
(self.checkout / "CLAUDE.md").write_text("# Project\n")
|
||||
(self.checkout / ".claude").mkdir()
|
||||
settings, local = self.checkout / ".claude" / "settings.json", self.checkout / ".claude" / "settings.local.json"
|
||||
for first, second in (("not json", '{"claudeMdExcludes": ["**/.claude/*.md", "**/proj/*.md"]}'), ('{"claudeMdExcludes": ["**/.claude/*.md"]}', '{"claudeMdExcludes": ["**/proj/*.md"]}')):
|
||||
with self.subTest(first=first):
|
||||
settings.write_text(first)
|
||||
local.write_text(second)
|
||||
self.assertEqual(preloaded_files(self.checkout, self.root), "none")
|
||||
for odd in ("[1]", '{"claudeMdExcludes": 3}'):
|
||||
with self.subTest(odd=odd):
|
||||
settings.write_text(odd)
|
||||
local.write_text(odd)
|
||||
self.assertEqual(preloaded_files(self.checkout, self.root), f"`~/.claude/{CLAUDE_MD}` and `{CLAUDE_MD}`")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -7,9 +7,8 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from review_rules.review_passes import PASS_AGENTS, PASSES
|
||||
from repo_checks import ROOT
|
||||
from rule_hook.reviewer_rung import SEAT, SECURITY_OPENER
|
||||
from rule_hook.reviewer_rung import SEAT_AGENT, SECURITY_OPENER
|
||||
|
||||
OPENING = re.compile(r"^## Opening\n+```\n([^\n]+?\.)(?: |\n)", re.MULTILINE)
|
||||
# An agent's name, as opposed to the skill or plugin that reviews a pass.
|
||||
AGENT_NAME = re.compile(r"[a-z]+(?:-[a-z]+)*")
|
||||
# A row of the pass table: its pass and its reviewer cell.
|
||||
@@ -25,10 +24,9 @@ class ReviewerLaunches(unittest.TestCase):
|
||||
rows = [(name.lower(), reviewer if AGENT_NAME.fullmatch(reviewer) else None) for name, reviewer in PASS_ROW.findall(PASS_TABLE)]
|
||||
self.assertEqual(rows, list(PASS_AGENTS.items()))
|
||||
|
||||
def test_the_seat_opening_is_the_sentence_that_opens_every_seat_prompt(self):
|
||||
opening = OPENING.search((ROOT / "skills" / "comprehension-panel" / "seat-prompts.md").read_text(encoding="utf-8"))
|
||||
self.assertIsNotNone(opening, "seat-prompts.md has a `## Opening` section whose code block opens with a sentence")
|
||||
self.assertEqual(SEAT, opening.group(1) if opening else "")
|
||||
def test_the_seat_agent_is_the_agent_under_agents_that_larv_review_has_every_seat_run_as(self):
|
||||
self.assertTrue((ROOT / "agents" / f"{SEAT_AGENT}.md").is_file())
|
||||
self.assertIn(f"`{SEAT_AGENT}`", LARV_REVIEW.split("### Comprehension check", 1)[1].split("### Security", 1)[0])
|
||||
|
||||
def test_the_security_opening_is_the_sentence_larv_review_has_the_security_pass_open_with(self):
|
||||
security = LARV_REVIEW.split("### Security", 1)[1].split("### Overrides", 1)[0]
|
||||
|
||||
@@ -13,9 +13,9 @@ class ReviewPasses(unittest.TestCase):
|
||||
self.assertEqual(list(REVIEWERS), list(typing.get_args(ReviewedPass)))
|
||||
self.assertEqual(list(REVIEWERS), [name for name in PASSES if PASS_AGENTS[name]])
|
||||
|
||||
def test_names_each_agent_under_agents(self):
|
||||
def test_names_each_agent_under_agents_beside_the_comprehension_seat_that_reviews_no_pass(self):
|
||||
agents = pathlib.Path(__file__).resolve().parents[2] / "agents"
|
||||
self.assertEqual(sorted(REVIEWERS.values()), sorted(path.stem for path in agents.glob("*.md")))
|
||||
self.assertEqual(sorted([*REVIEWERS.values(), "comprehension-seat"]), sorted(path.stem for path in agents.glob("*.md")))
|
||||
|
||||
def test_counts_markdown_an_agent_follows_as_config_and_the_rest_as_prose(self):
|
||||
for path in ("README.md", "docs/decisions.md", "todo.md", "workflow-bench/README.md"):
|
||||
|
||||
@@ -90,10 +90,10 @@ class PrinciplesLine(unittest.TestCase):
|
||||
|
||||
|
||||
class Reviewers(unittest.TestCase):
|
||||
def test_only_the_reviewers_whose_pass_met_the_replay_gate_start_without_claude_md(self):
|
||||
def test_only_the_reviewers_whose_pass_met_the_replay_gate_and_the_comprehension_seat_start_without_claude_md(self):
|
||||
# docs/decisions.md → `agents/: a reviewer starts without CLAUDE.md only where a replay of its pass missed no more findings than the most an unchanged replay missed` holds each pass's result; override-reviewer has no archived round to replay.
|
||||
bare = [path.stem for path in sorted((REPO / "agents").glob("*.md")) if "omitClaudeMd: true" in path.read_text().split("---\n")[1].splitlines()]
|
||||
self.assertEqual(bare, ["principles-auditor", "product-owner-reviewer", "prose-reviewer", "stability-reviewer", "systems-architect"])
|
||||
self.assertEqual(bare, ["comprehension-seat", "principles-auditor", "product-owner-reviewer", "prose-reviewer", "stability-reviewer", "systems-architect"])
|
||||
|
||||
def test_settings_run_the_hook_as_each_subagent_starts(self):
|
||||
hooks = json.loads((REPO / "settings.json").read_text())["hooks"]["SubagentStart"]
|
||||
|
||||
@@ -16,8 +16,8 @@ from claude_code.entries import UserText
|
||||
|
||||
# The pass each agent under agents/ reviews.
|
||||
PASS_OF = {agent: kind for kind, agent in REVIEWERS.items()}
|
||||
# The opening of every comprehension seat's prompt: skills/comprehension-panel/seat-prompts.md → "Opening".
|
||||
SEAT = "You are one seat of a comprehension panel."
|
||||
# The agent under agents/ that every comprehension seat runs as: skills/comprehension-panel/SKILL.md → "The scoring run".
|
||||
SEAT_AGENT = "comprehension-seat"
|
||||
# The opening of the security pass's prompt: skills/larv-review/SKILL.md → "Security".
|
||||
SECURITY_OPENER = "You are the security pass for this chunk."
|
||||
SUBJECTS: dict[ReviewKind, str] = {"seat": "a comprehension seat", "security": "the security pass's subagent"}
|
||||
@@ -79,10 +79,9 @@ def review_kind(given: dict[str, typing.Any]) -> ReviewKind | None:
|
||||
"""What a launch reviews, a reviewer's pass, a comprehension seat or the security pass; None for any other subagent."""
|
||||
if pass_reviewed := PASS_OF.get(text(given, "subagent_type")):
|
||||
return pass_reviewed
|
||||
opening = text(given, "prompt").lstrip()
|
||||
if opening.startswith(SEAT):
|
||||
if text(given, "subagent_type") == SEAT_AGENT:
|
||||
return "seat"
|
||||
if opening.startswith(SECURITY_OPENER):
|
||||
if text(given, "prompt").lstrip().startswith(SECURITY_OPENER):
|
||||
return "security"
|
||||
return None
|
||||
|
||||
|
||||
@@ -309,26 +309,27 @@ class ReviewerRung(unittest.TestCase):
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "fable", "effort": "max"}, self.ABOVE), [])
|
||||
lifted = self.ENVIRON | {"LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5"}
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}, lifted), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"prompt": "You are one seat of a comprehension panel.", "model": "opus", "effort": "medium"}, lifted), [])
|
||||
self.assertEqual(self.refused({"subagent_type": "comprehension-seat", "prompt": "Score the project.", "model": "opus", "effort": "medium"}, lifted), [])
|
||||
|
||||
def test_holds_a_reviewer_to_the_higher_of_the_workers_and_the_projects_rung_where_loop_claude_handed_no_e_plus_i(self):
|
||||
without = {name: rung for name, rung in self.ABOVE.items() if name != "LOOP_CLAUDE_ESCAPE_AND_IMPACT"}
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}, without), ["reviewer-rung"])
|
||||
below = {name: rung for name, rung in self.ENVIRON.items() if name != "LOOP_CLAUDE_ESCAPE_AND_IMPACT"}
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "sonnet", "effort": "high"}, below), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"prompt": "You are one seat of a comprehension panel.", "model": "sonnet", "effort": "high"}, without), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"subagent_type": "comprehension-seat", "prompt": "Score the project.", "model": "sonnet", "effort": "high"}, without), ["reviewer-rung"])
|
||||
|
||||
def test_a_seat_or_a_security_launch_is_held_to_the_project_rung_alone(self):
|
||||
seat = {"prompt": "You are one seat of a comprehension panel. Tools: Read, Grep, Glob.", "model": "opus", "effort": "medium"}
|
||||
seat = {"subagent_type": "comprehension-seat", "prompt": "Score the project.", "model": "opus", "effort": "medium"}
|
||||
self.assertEqual(self.refused(seat, self.ABOVE), [])
|
||||
self.assertEqual(self.refused({"prompt": "You are the security pass for this chunk. Invoke /security-review over the branch diff.", "model": "sonnet", "effort": "high"}), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"prompt": "You are the security pass for this chunk. Run /claude-security scan this branch's changes at high.", "model": "opus", "effort": "medium"}), [])
|
||||
|
||||
def test_a_seat_named_below_the_project_rung_is_refused(self):
|
||||
self.assertEqual(self.refused({"prompt": "You are one seat of a comprehension panel.", "model": "sonnet", "effort": "high"}), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"subagent_type": "comprehension-seat", "prompt": "Score the project.", "model": "sonnet", "effort": "high"}), ["reviewer-rung"])
|
||||
|
||||
def test_any_other_subagent_keeps_its_own_routing(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "Explore", "model": "haiku", "effort": "low", "prompt": "Find the callers of run_worker."}), [])
|
||||
self.assertEqual(self.refused({"prompt": "You are one seat of a comprehension panel.", "model": "haiku", "effort": "low"}), [], "the seat's opening sentence without its agent is no launch")
|
||||
self.assertEqual(self.refused({"prompt": "Note that security-review runs in CI.", "model": "haiku", "effort": "low"}), [])
|
||||
self.assertEqual(self.refused({"prompt": "Find where /security-review is named.", "model": "haiku", "effort": "low"}), [], "a mention without the opening sentence is no launch")
|
||||
self.assertEqual(self.refused({"prompt": "Quote the rule: 'You are the security pass for this chunk.' Then stop.", "model": "haiku", "effort": "low"}), [], "a quote midway is no launch")
|
||||
|
||||
@@ -2525,3 +2525,31 @@ Read most (opens and searches):
|
||||
- `bin/loop_claude/cycle/run.py` 4
|
||||
- `bin/loop_claude/loop/close.py` 3
|
||||
- `bin/model_routing/test_review_choice.py` 3
|
||||
|
||||
## Reading cost, chunk cfb3b6b8c710, finished 2026-11-17T08:13:19Z
|
||||
|
||||
182657 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `docs/decisions.md` 13
|
||||
- `README.md` 5
|
||||
- `skills/larv-review/SKILL.md` 4
|
||||
- `CLAUDE.md` 3
|
||||
- `CHANGELOG.md` 1
|
||||
- `bin/loop_claude/cycle/prompts.py` 1
|
||||
- `bin/todo-item` 1
|
||||
- `rules/decisions.md` 1
|
||||
- `todo.md` 1
|
||||
|
||||
## Reading cost, chunk 8174e3b60f07, finished 2026-11-17T08:26:23Z
|
||||
|
||||
92410 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `bin/loop_claude/cycle/prompts.py` 3
|
||||
- `rules/todo-items.md` 3
|
||||
- `workflow-bench/bench.py` 2
|
||||
- `bin/loop_claude/cycle/run.py` 1
|
||||
- `bin/loop_claude/cycle/worker.py` 1
|
||||
|
||||
@@ -23,13 +23,13 @@ Hold all of these or the comparison doesn't prove anything.
|
||||
one. Seats that read different amounts produce numbers nothing can compare.
|
||||
- **Blind.** Deny the project's own conclusions — the plan, a decision log, prior review
|
||||
output — and never show a panelist the metric under test. An anchored reader is a wasted one.
|
||||
- **Check what the harness pre-loads, before launching.** `CLAUDE.md` may hold only
|
||||
`@AGENTS.md`, and an `@file` import lands in a subagent's context ahead of its first tool call.
|
||||
That is harmless for a reader who was going to read the charter anyway, and fatal to the
|
||||
architects' README-and-tree-only step. Read the project's `CLAUDE.md` and resolve its imports
|
||||
first; name every pre-loaded file in the architects' brief and require them to mark each map
|
||||
entry that file could have fed, so the contaminated entries are discountable instead of
|
||||
invisible.
|
||||
- **Check what the harness pre-loads, before launching.** The scoring run and the paired ruling
|
||||
launch every seat as the `comprehension-seat` agent, which starts without `CLAUDE.md`, so nothing
|
||||
is pre-loaded. Any other launch: `CLAUDE.md` may hold only `@AGENTS.md`, and an `@file` import
|
||||
lands in a subagent's context ahead of its first tool call. That is harmless for a reader who was
|
||||
going to read the charter anyway, and fatal to the architects' README-and-tree-only step. Resolve
|
||||
the project's imports, name every pre-loaded file in the architects' brief, and require them to
|
||||
mark each map entry that file could have fed.
|
||||
- **Isolated and read-only.** Launch each with `isolation: "worktree"`, and forbid edits, test
|
||||
runs and builds. Entering a worktree yourself mid-run strands a non-isolated subagent.
|
||||
- **Parallel.** Launch them in one message. Nine agents reading a few thousand lines is minutes,
|
||||
@@ -97,7 +97,8 @@ A scoring run needs four seats — Junior A, Mid A, the maintainability senior a
|
||||
architect — plus the calibration seat ("The score") when more than one project is in play. `~/.claude/bin/larv-review prepare-scoring`
|
||||
builds the four seats' prompts from `seat-prompts.md`, filled in from the project's `AGENTS.md` →
|
||||
"Scoring run". Change neither between runs, or the scores are not comparable. A project without that
|
||||
section gets it at its first scoring run or paired ruling.
|
||||
section gets it at its first scoring run or paired ruling. Each seat runs as the `comprehension-seat`
|
||||
agent under `agents/`, which starts without `CLAUDE.md`.
|
||||
|
||||
## The paired ruling
|
||||
|
||||
|
||||
@@ -1,13 +1,12 @@
|
||||
# The comprehension seats' prompts
|
||||
|
||||
A scoring-run seat's prompt is the opening, the seat's background, the shared brief, and the report for its kind of seat, in that order. Copy the text verbatim and replace each `{placeholder}`:
|
||||
A scoring-run seat's prompt is the opening, the seat's background, the shared brief, and the report for its kind of seat, in that order. Launch each seat as the `comprehension-seat` agent, `agents/comprehension-seat.md`, which starts without `CLAUDE.md` and the rule files, so a seat holds only what its prompt and its reads give it. Copy the text verbatim and replace each `{placeholder}`:
|
||||
|
||||
- `{checkout}` and `{branch}`: the worktree the seats read, and its branch.
|
||||
- `{main}`: the project's main checkout.
|
||||
- `{forbidden}`: the plan, the decision log, past panels' records and past chunks' reading cost, each where the project's config puts it in the checkout. Where it puts all four at their usual paths: `todo.md, docs/decisions.md, docs/comprehension-history.md or docs/reading-cost.md (the plan, the decision log, past panels' records and past chunks' reading cost)`. A file the config puts outside the checkout is left out, since the seats' worktree doesn't hold it.
|
||||
- `{git}`: `git -C {checkout} ls-files`, and for a paired seat also `git -C {base checkout} ls-files`, each in backticks.
|
||||
- `{files}`: every tracked file in `git -C {checkout} ls-files`, except test files and those `{forbidden}` names. Give one recursive glob per top-level source directory, say which test files it leaves out, and list each file no glob matches. Example: `bin/**/*.py` except `test_*.py`, plus launchers such as `bin/loop-claude`.
|
||||
- `{preloaded}`: the files the harness pre-loads into a seat, found per `SKILL.md` → "What makes the result trustworthy".
|
||||
- `{anchors}` and `{dimensions}`: the six anchor bullets and the four dimension bullets in `SKILL.md` → "The score", verbatim, without bold and each on one line.
|
||||
- `{language}`, `{kind}`, `{domain}`, `{domain docs}` and `{3am question}`: the project's `AGENTS.md` → "Scoring run".
|
||||
|
||||
@@ -31,8 +30,6 @@ Target: the project checked out at {checkout} (branch `{branch}`). Read ONLY the
|
||||
|
||||
Depth 1, the whole project: read README.md, AGENTS.md and CLAUDE.md (the charter), then every non-test tracked file: {files}. Do NOT open {forbidden}, nor any review output. You may open a test file when the code does not answer a question; say when you did.
|
||||
|
||||
Your harness pre-loads these files into your context before your first tool call, from a DIFFERENT version (main): {preloaded}. Treat the worktree's own copies as the ones under review. A pre-loaded AGENTS.md may list decision titles; do not use them as evidence.
|
||||
|
||||
No running anything, no builds, no tests, no edits of any kind. Do not manufacture confusion.
|
||||
|
||||
Scale (verbatim anchors):
|
||||
@@ -60,7 +57,7 @@ Report, in this shape:
|
||||
```
|
||||
Standard: Code That Fits in Your Head — at each level a bounded number of areas, few enough to hold at once. Guidelines, not rules; a justified break is a finding too.
|
||||
Enforce this order:
|
||||
1. README.md and the file tree only (`git -C {checkout} ls-files`). No other file yet. Write down the top-level areas you believe exist and what each does, what you expect inside each source directory, anything whose purpose the name does not give, anything the README led you to expect that is not there. Mark each map entry the pre-loaded files could have fed. This map goes in your report verbatim.
|
||||
1. README.md and the file tree only (`git -C {checkout} ls-files`). No other file yet. Write down the top-level areas you believe exist and what each does, what you expect inside each source directory, anything whose purpose the name does not give, anything the README led you to expect that is not there. This map goes in your report verbatim.
|
||||
2. Descend breadth-first, forming an expectation before opening each directory; record how many distinct things each level demanded.
|
||||
3. Read enough implementation to check the map (depth 1: every non-test file is in scope, but your job is shape).
|
||||
Report: the verbatim map; where it was wrong and how costly each correction was; level-by-level fan-out with the worst level named; names that mislead, concepts with two names, one name over two concepts, areas you could not name; what you would restructure, ranked, with files and costs; what the structure gets right. The 3am question: {3am question} — walking in cold, how long to the right file and unit? Where does it rot first, and where would the next two features land? Close with Navigation, Locality, Shape, Self-sufficiency and overall, each with one sentence naming its anchor and the concrete thing holding it there.
|
||||
@@ -80,8 +77,6 @@ Depth 1, the whole project in both versions: read README.md, AGENTS.md and CLAUD
|
||||
{changed}
|
||||
Do NOT open {forbidden}, nor any review output. You may open a test file when the code does not answer a question; say when you did.
|
||||
|
||||
Your harness pre-loads these files into your context before your first tool call, from main, which may match either version or neither: {preloaded}. Treat the two worktrees' own copies as the ones under review. A pre-loaded AGENTS.md may list decision titles; do not use them as evidence.
|
||||
|
||||
No running anything, no builds, no tests, no edits of any kind.
|
||||
|
||||
Rule whether the project reads better, the same or worse at {head} than at {base} on each of these four dimensions, and as a whole:
|
||||
|
||||
@@ -155,7 +155,7 @@ The comprehension check records how hard the project is for its contributor to r
|
||||
|
||||
**Scoring run**, when `~/.claude/bin/larv-review needs-scoring-run <the IDs of the items the chunk ships, if any>` prints `Scoring run due:`. `chunk-pr merge` refuses until the branch's committed history records every run the plan calls, which `larv-review scoring-run-owed` counts at merge time from the merge base; an earlier due line is only a snapshot.
|
||||
|
||||
- **Seats.** Junior A, Mid A, the maintainability senior and the inherited architect each score the project at the head from 1 to 10 overall and on four dimensions: Navigation, Locality, Shape and Self-sufficiency. Run `~/.claude/bin/larv-review prepare-scoring`, adding `--preloaded <text>` only where its default list of pre-loaded files is wrong. It makes the worktree the seats read and prints each seat's prompt file; launch each seat with its file's text as its prompt. Write their lines into a scores file and run `~/.claude/bin/larv-review record-scoring <scores file> <the same item IDs>`, which appends the run to the history and prints the four seats' means.
|
||||
- **Seats.** Junior A, Mid A, the maintainability senior and the inherited architect each score the project at the head from 1 to 10 overall and on four dimensions: Navigation, Locality, Shape and Self-sufficiency. Run `~/.claude/bin/larv-review prepare-scoring`. It makes the worktree the seats read and prints each seat's prompt file; launch each seat as the `comprehension-seat` agent, with its file's text as its prompt. The paired ruling's seats are launched as that agent too. Write their lines into a scores file and run `~/.claude/bin/larv-review record-scoring <scores file> <the same item IDs>`, which appends the run to the history and prints the four seats' means.
|
||||
- **Floor item.** Where `record-scoring` prints `Floor:`, a `Next:` line follows: file one floor item, or refresh the open one. A floor item is a plan item meant to bring every mean to the project's readability floor, which `record-scoring`'s `Floor:` line names. File it as one ambitious refactor, not touch-ups. It restructures the area behind the places `larv-review places` prints for this run, chosen where they bear on the lowest dimension, sized to bring every mean to that floor, and quotes those places. Its tagline opens with `Reach the comprehension floor:`, so its own chunk runs a scoring run; while a mean stays under the floor, that run's `Next:` line asks for the next floor item. Mark it `owner`, since the human asked that such a project always hold one, so no score deletes it. Put it in the earliest release the plan names after the one the chunk ships into, or else in the next version after that one that `~/.claude/rules/todo-items.md` allows, so the floor doesn't hold up any release. While one is open, refresh its places and aim from the newest run instead of filing another; the item the chunk ships doesn't count as open. Commit it with the run, as "Record" says.
|
||||
- **Second run.** Where `needs-scoring-run` printed that the chunk `leaves release … with no plan item`, run `prepare-scoring` again and launch the same four seats afresh on the same head. Their scores file's first line is `<PR or branch> rerun at <commit>`. Record it with the same `record-scoring` command.
|
||||
- **Agreement.** After the second run, run `~/.claude/bin/larv-review agreement` and put its counts in the summary. On `Assumption fails:`, ask the user whether to keep the paired ruling or change the decision `review: the comprehension check records …`. Record the answer as a decision in the repo whose history was counted, with the counts that led to the ask. A later run asks again only when the rulings' rate falls, or the runs' rate rises, against the recorded counts.
|
||||
|
||||
Reference in New Issue
Block a user