Merge pull request 'Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries' (#310) from item-73 into main
This commit was merged in pull request #310.
This commit is contained in:
@@ -50,6 +50,7 @@
|
||||
!bin/reading_cost/*.py
|
||||
!bin/repo_checks/
|
||||
!bin/repo_checks/*.py
|
||||
!bin/review_folder.py
|
||||
!bin/round_titles.py
|
||||
!bin/rule-hook
|
||||
!bin/routing_figures/
|
||||
|
||||
@@ -37,6 +37,7 @@ Decisions live in `docs/decisions.md`:
|
||||
- rules/: `CLAUDE.md` holds the worker's rules that no file path can scope, and each topic's rules for the worker live in a `rules/` file that Claude Code loads by its `paths:` frontmatter
|
||||
- worker: a chunk's worker writes the simplest change that solves its item, test first wherever a test can check the change, and reads a technical principle only where the item, a finding or its own record names it; larv-review's reviewers hold the change to the principles and to the standards in `skills/larv-review/standards/`
|
||||
- tests: each test earns its keep as the technical principles' test bullet says, and the worker takes a test's expected values from its item, never from the code
|
||||
- review: each reviewer under `agents/` writes its full report into the chunk's review folder under `~/.local/state/larv-review/` and returns the worker its verdict and one line per finding; rule-hook refuses its Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`
|
||||
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
|
||||
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
|
||||
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Each reviewer under `agents/` now writes its full report to a file that larv-review names in the reviewer's prompt, in a folder per chunk under `~/.local/state/larv-review/`. It returns only its verdict and one line per finding, so a review takes less of the worker's context. The worker reads a finding's full entry before it fixes or answers a 🔴 or 🟡. Each reviewer now holds Write. `rule-hook` refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`; no override waives this rule. Each chunk's folder stays after the merge, so `stability-replay` can replay a round that reads an earlier round's report. It replays on a copy of the folder.
|
||||
- `loop-claude` asks an interrupt's question on its own terminal and resumes the worker headless with your one-line answer, so the loop no longer waits for you to leave a claude session. Type `/talk` instead to open the worker's session in claude, as every interrupt did before, to read the diff or discuss first. A release ready to cut opens that session at once. Pressing Enter on an empty answer stops the loop, exit 2. Keys typed before the question shows are dropped, and a pasted answer keeps only its first line; type `/talk` for a longer one. The judge counts that answer as yours, like one typed in claude: the chunk stays open until the answer is recorded, where a later chunk could need it.
|
||||
- `chunk-preflight` and `loop-claude` no longer count a renamed release as finished: one that `todo.md` named since the last tag, whose last items all moved to other releases in one change, none leaving the plan. An example is a planned 0.2.1 that becomes 0.3.0 because an item turned out breaking. Before, they reported it finished and ready to cut, and the loop stopped on every cycle while the release it became still held open items. larv-review's scoring-run trigger calls no run for such a release either. A release whose leftover items you defer to a later release reads the same way now, so cut it yourself when you want it cut.
|
||||
- When a worker's run ends with a question and a request for a higher rung, `loop-claude` now steps the worker one rung up as the judge interrupts, or as it answers the question with a quote from `README.md` or another file it may quote. The judge must quote a passage showing that the escalation trigger the run cites fired. Before, the step was lost for an answered question, and the worker carried on from the answer on its old rung. The step has the same limits as the step on an unfinished verdict, and no later ruling of that cycle carries a second escalation. The archive's interrupt and quoted-answer outcomes now name the worker's rung.
|
||||
|
||||
@@ -66,7 +66,7 @@ The rules only reviewers apply, for quality, comments, READMEs and prose, live i
|
||||
4. A review rules the work below standard: systems-architect's BLOCK, a finding against larv-review's `standards/quality.md`, or a vote `record-ruling` counts as worse, in the report that stands after any relaunch one rung up (`larv-review` §3 → "Rung up").
|
||||
|
||||
When one fires, hand the hard part, edits included, to a subagent on the rung loop-claude's prompt names, one above yours. Open the subagent's prompt with the marker loop-claude's prompt gives, then the trigger's evidence. Each later subagent on that task runs on that rung or higher, and each further escalation trigger moves the task one rung higher, up to the highest rung loop-claude's prompt names. Ask the human before going higher. Where loop-claude's prompt names no escalation, as on the ladder's top rung or off it, keep the task in your own thread.
|
||||
- Give a subagent only the access its job needs, and state the limits in its prompt. One that only reads, such as a prose review, gets no shell and no edits. One that probes a build tags the image as throwaway and removes it afterwards, writes only to the session scratchpad, and removes no image or file it did not create.
|
||||
- Give a subagent only the access its job needs, and state the limits in its prompt. One that only reads, such as a prose review, gets no shell and no edits. A reviewer under `~/.claude/agents/` writes only its report, under `~/.local/state/larv-review/`, and rule-hook refuses it every other Write, every Edit and every MultiEdit. One that probes a build tags the image as throwaway and removes it afterwards, writes only to the session scratchpad, and removes no image or file it did not create.
|
||||
- Check `README.md` first for build, test, install, and codegen commands, and fix one that misbehaves in the same change. Refresh `README.md` whenever you change what it describes, keeping it true, not longer.
|
||||
- Where `~/.claude/bin/rule-hook` does not check it, such as an image named in test code, a Kubernetes manifest, a `docker run` line or a file a Bash command writes, pin a container image to its full patch version and a GitHub Action to a semver tag.
|
||||
|
||||
|
||||
@@ -84,7 +84,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
|
||||
- `chunk-preflight` finds the mechanical facts a chunk starts from and makes its worktree; `loop-claude` runs it before it routes each chunk, and `chunk-preflight --help` lists the facts.
|
||||
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints the lowest rung that, like each rung above it, returns them; `judge-replay --help` states the replay.
|
||||
- `larv-review <command>` holds larv-review's mechanical rules: the paired ruling's base and vote count, each pass's round count, the scoring run's trigger, and the comprehension history's format and latest places, and its count of how often seats agree; `larv-review --help` states them.
|
||||
- `larv-review <command>` holds larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's round count, the scoring run's trigger, and the comprehension history's format and latest places, and its count of how often seats agree; `larv-review --help` states them.
|
||||
- `loop-claude <task>` runs a project's next chunk cycle after cycle; `loop-claude --help` states its contract.
|
||||
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
|
||||
- `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: override-reviewer
|
||||
description: "Use this agent for larv-review's overrides pass."
|
||||
tools: Read, Grep, Glob
|
||||
tools: Read, Grep, Glob, Write
|
||||
---
|
||||
|
||||
You rule on each override of a rule `~/.claude/bin/rule-hook` enforces. The prompt gives you the lines `rule-hook overrides` printed, each with its rule and reason, and the branch diff. Read each rule and its stated exceptions in `~/.claude/bin/rule_hook/rules.py` → `RULES`, how an override is written and logged in the docstring of `~/.claude/bin/rule_hook/main.py`, and the project's `README.md` → "Goals" and "Audience".
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: principles-auditor
|
||||
description: "Use this agent for larv-review's principles pass: one yardstick checked against the whole project."
|
||||
tools: Read, Grep, Glob
|
||||
tools: Read, Grep, Glob, Write
|
||||
---
|
||||
|
||||
You check one yardstick against the whole project: every principle in the file you are handed, or every goal under the project's `README.md` → `## Goals`.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: "product-owner-reviewer"
|
||||
description: "Use this agent when you need a product-focused review of changes that affect external-facing surfaces (REST APIs, web UIs, CLI commands, public SDKs, webhooks, etc.) from an end-customer perspective. This agent evaluates ease of use, edge cases, persona fit, and reasonableness of usage paths — not internal implementation details unless they leak into the external experience. Invoke after a feature, endpoint, UI flow, or CLI command has been drafted or modified, or before opening a PR that changes user-facing behavior.\\n\\n<example>\\nContext: The developer just added a new REST endpoint for cancelling a subscription.\\nuser: \"I've added a DELETE /v4/subscriptions/{id} endpoint that cancels a subscription immediately.\"\\nassistant: \"Let me launch the product-owner-reviewer agent to evaluate this from an end-customer perspective.\"\\n<commentary>\\nA new external API surface was introduced. Use the Agent tool to launch product-owner-reviewer to check personas, edge cases (already cancelled? pending invoices? refund expectations?), and usability of the contract.\\n</commentary>\\n</example>\\n\\n<example>\\nContext: The developer added a new CLI subcommand.\\nuser: \"Added `mytool deploy --env prod --force` to skip confirmation.\"\\nassistant: \"I'll use the Agent tool to launch the product-owner-reviewer agent to review the CLI ergonomics and edge cases.\"\\n<commentary>\\nCLI usage is an external surface. The product-owner-reviewer should evaluate persona fit (who uses --force? is it discoverable? what happens on partial failure?).\\n</commentary>\\n</example>\\n\\n<example>\\nContext: The developer modified a checkout flow in the web UI.\\nuser: \"I changed the checkout to require email verification before payment.\"\\nassistant: \"Let me launch the product-owner-reviewer agent to review this user-facing change.\"\\n<commentary>\\nWeb UI flow change directly affects end customers. Use product-owner-reviewer to assess friction, error states, recovery paths, and whether the change makes sense for the documented personas.\\n</commentary>\\n</example>"
|
||||
tools: Read, Grep, Glob
|
||||
tools: Read, Grep, Glob, Write
|
||||
---
|
||||
|
||||
You are a seasoned Product Owner with deep expertise in user experience, API design, and customer empathy. You have shipped consumer and B2B products across web, mobile, REST APIs, and developer-facing CLIs. Your superpower is putting yourself in the end customer's shoes and surfacing the friction, confusion, and edge cases that engineers tend to miss.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: prose-reviewer
|
||||
description: "Use this agent for larv-review's prose pass."
|
||||
tools: Read, Grep, Glob
|
||||
tools: Read, Grep, Glob, Write
|
||||
---
|
||||
|
||||
You read the prose as someone who did not write it: you see only what it says, never what the author meant.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
name: "stability-reviewer"
|
||||
description: "Use this agent when code has been written or modified and needs review for stability, bug prevention, and adherence to coding best practices. This includes after implementing new features, refactoring existing code, or fixing bugs. The agent focuses on defensive programming, error handling, edge cases, and compliance with the project's technical principles.\\n\\nExamples:\\n\\n- user: \"Please implement a retry mechanism for our API client\"\\n assistant: \"Here is the retry mechanism implementation: [code written]\"\\n Since significant code was written, use the Agent tool to launch the stability-reviewer agent to review the new code for stability and best practices.\\n assistant: \"Now let me use the stability-reviewer agent to review this implementation for stability concerns.\"\\n\\n- user: \"Refactor the payment processing module to use the new gateway\"\\n assistant: \"I've refactored the payment processing module. Let me now use the Agent tool to launch the stability-reviewer agent to check for any stability issues or violations of our technical principles.\"\\n\\n- user: \"Can you review what I just wrote?\"\\n assistant: \"I'll use the Agent tool to launch the stability-reviewer agent to thoroughly review your recent changes.\"\\n\\n- After any significant code change, proactively launch the stability-reviewer agent to catch issues before they reach production."
|
||||
color: orange
|
||||
tools: Bash, Read, Grep, Glob
|
||||
tools: Bash, Read, Grep, Glob, Write
|
||||
---
|
||||
|
||||
You are an elite stability-focused code reviewer with deep expertise in defensive programming, fault-tolerant system design, and production incident prevention. You have decades of experience identifying subtle bugs, race conditions, edge cases, and architectural weaknesses that lead to outages and unexpected behavior. You think like a chaos engineer — always asking "what could go wrong?" — and like a maintainer of ten years, always asking "what will this cost to live with?" There are no bugs, and no security holes, in code that doesn't exist.
|
||||
@@ -12,7 +12,7 @@ You are an elite stability-focused code reviewer with deep expertise in defensiv
|
||||
1. **Scope the change.** You review every PR; a fresh invocation has no prior context, so determine the diff yourself. Diff the PR branch against its base (`gh pr diff`, or `git merge-base <base> HEAD` then `git diff`), and review the changed lines **plus enough surrounding code and callers** to judge correctness — read the whole changed file when practical; a diff alone hides whether a caller breaks. Read the PR description and, when the project uses an issue tracker (project-specific — check its docs) and the change references a ticket, that ticket: the diff is judged against its stated intent, and a scope mismatch in either direction is a finding — claimed-but-unimplemented, and implemented-but-unasked (#2).
|
||||
2. **Read every file under `~/.claude/principles/`**, the shared principles, and `~/.claude/skills/larv-review/standards/quality.md`. These are mandatory org guidelines; all code MUST comply with each that applies. A file's `Applies to` line says what it covers; one without the line covers every change. Re-read each time — they change. Flag violations prominently.
|
||||
3. **Read README.md, AGENTS.md and `docs/decisions.md`** in the current project for project-specific conventions, build instructions, tooling, and the accepted trade-offs a reviewer should honor rather than re-flag. Note the documented test, lint, and static-analysis commands — you will run them.
|
||||
4. **Verify, don't just read.** Run the test suite and static analysis/linters through the project's container tooling, unless a decision puts them on the host; tear down test containers afterwards. A full-suite run catches broken callers that diff-reading misses. When a suspected bug is cheap to confirm, reproduce it — a scratch script, a targeted test run — before reporting it. Put scratch work in a directory you create for this run, and delete it when done; delete nothing else, since a directory the prompt names may hold other agents' files. Leave the working tree exactly as you found it. Anything you could not run goes in Coverage & Verification, never silently skipped.
|
||||
4. **Verify, don't just read.** Run the test suite and static analysis/linters through the project's container tooling, unless a decision puts them on the host; tear down test containers afterwards. A full-suite run catches broken callers that diff-reading misses. When a suspected bug is cheap to confirm, reproduce it — a scratch script, a targeted test run — before reporting it. Make scratch work with Bash, since Write is for your report alone. Put it in a directory you create for this run, and delete it when done; delete nothing else, since a directory the prompt names may hold other agents' files. Leave the working tree exactly as you found it. Anything you could not run goes in Coverage & Verification, never silently skipped.
|
||||
5. **Mutation-test the changed lines.** Run the project's documented mutation command, or else a widely deployed mutation tool for the language: cargo-mutants for Rust, Infection for PHP, mutmut for Python, Stryker for JavaScript and TypeScript. Limit it to the lines the change adds or modifies, through the tool's own option where it has one (cargo-mutants `--in-diff <diff file>`, Infection `--git-diff-lines`, Stryker `--mutate <file>:<lines>`); otherwise mutate only the changed files and keep only the mutants on changed lines. Run it through the project's tooling, as in step 4, on a scratch copy: the tools write files. Report its wall time and mutant count in Coverage & Verification, or why nothing was mutated. Ignore a surviving mutant that leaves behaviour unchanged, or changes only message or log text no caller or user depends on. Report one 🟡 per test to add or strengthen, naming the surviving mutants it would catch.
|
||||
6. **Two passes minimum, then hunt absences.**
|
||||
- *Breadth:* every hunk in every changed file, no skimming. Track which files you have covered; batch large diffs, never truncate the review.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: systems-architect
|
||||
description: "Use this agent when designing new systems, planning major features, reviewing architectural decisions in PRs, or evaluating the high-level structure of a codebase. It runs as a periodic, deliberate alignment review — not a per-PR reviewer; the stability-reviewer is the per-PR, code-level companion. This agent focuses on architectural patterns (functional core/imperative shell), the external contract with consumers, component boundaries, internal API design, testability seams and verification strategy (never individual tests), delivery strategy (merge gate with tests and static analysis, deployment path, dependency automation — as architecture, never pipeline configs or lint rules), naming, file system organization, the simplicity and reasonability of the core (questioning unjustified duplication and accidental complexity, demanding a stated reason for every repeated or complex structure), quantitative fits-in-your-head guardrails (per Code That Fits in Your Head), simpler near-equivalent solutions (trading a small, named behavior delta for a large complexity reduction), and long-term maintainability rather than implementation details. It also evaluates fitness for purpose — whether the chosen language, frameworks, and overall solution approach are the best fit for the project's goals and priorities — and treats missing or contradictory goals, priorities, audience, problem statement, scale and horizon, or domain vocabulary as the first thing to fix. It should be used proactively when starting new projects or significant new subsystems, and when reviewing PRs that introduce new architectural patterns or restructure existing ones.\\n\\n<example>\\nContext: The user is starting a new project and needs architectural guidance before writing code.\\nuser: \"I'm building a service that ingests events from multiple sources, deduplicates them, and forwards them to downstream systems.\"\\nassistant: \"Before we start implementing, let me use the Agent tool to launch the systems-architect agent to design the high-level architecture and data flows.\"\\n<commentary>\\nA new project is being started — use the systems-architect agent to deliberately design the architecture before any code is written.\\n</commentary>\\n</example>\\n\\n<example>\\nContext: A PR has been opened that introduces a new module or restructures existing components.\\nuser: \"Here's the PR for the new payment processing subsystem: #4521\"\\nassistant: \"I'll use the Agent tool to launch the systems-architect agent to review the high-level architecture and component boundaries in this PR.\"\\n<commentary>\\nA PR introduces a new subsystem — the systems-architect agent should review architectural choices, not line-level details.\\n</commentary>\\n</example>\\n\\n<example>\\nContext: The user is adding a significant new feature that crosses multiple components.\\nuser: \"We need to add real-time notifications across the web app, mobile app, and email.\"\\nassistant: \"This crosses several system boundaries. Let me use the Agent tool to launch the systems-architect agent to design how this should fit into the existing architecture.\"\\n<commentary>\\nA feature crossing multiple components needs deliberate architectural design — use the systems-architect agent proactively.\\n</commentary>\\n</example>\\n\\n<example>\\nContext: The user asks for an architecture review but the project has no clearly stated goals or priorities.\\nuser: \"Can you review the architecture of our new sync service?\"\\nassistant: \"I'll use the Agent tool to launch the systems-architect agent. Because fitness-for-purpose depends on goals, it will first pin down the project's goals and priorities, then judge whether the language, frameworks, and approach actually serve them.\"\\n<commentary>\\nJudging whether an implementation is the best one requires explicit goals and priorities — the systems-architect agent drafts the missing goals and priorities and asks the user to confirm them before evaluating fitness.\\n</commentary>\\n</example>"
|
||||
tools: Read, Grep, Glob, WebFetch, WebSearch
|
||||
tools: Read, Grep, Glob, WebFetch, WebSearch, Write
|
||||
---
|
||||
You are an elite Systems Architect with decades of experience designing software systems that endure for as long as their horizon demands. Your philosophy is uncompromising on fundamentals: clarity, simplicity, and deliberate structural choices trump short-term convenience every time. You design systems people will read, understand, and extend for their whole recorded lifetime.
|
||||
|
||||
|
||||
+35
-11
@@ -1,17 +1,21 @@
|
||||
"""Hold larv-review's mechanical rules: the paired ruling's base and vote count, each pass's rounds, the scoring run's trigger, and the comprehension history's format and latest places, and its count of how often seats agree.
|
||||
"""Hold larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's rounds, the scoring run's trigger, and the comprehension history's format and latest places, and its count of how often seats agree.
|
||||
|
||||
usage: larv-review ruling-base | round-count | needs-scoring-run [<item ID>...] | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement
|
||||
usage: larv-review review-folder | ruling-base | round-count | needs-scoring-run [<item ID>...] | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement
|
||||
|
||||
The comprehension check records how hard a project is to read, two ways, and blocks no merge. In a
|
||||
paired ruling, simulated readers, the seats, read the branch's head beside a base and vote better,
|
||||
same or worse, overall and on each dimension. In a scoring run, four seats score the project from 1
|
||||
to 10. Each seat names the places it found hard.
|
||||
|
||||
Run it inside a project checkout or one of its worktrees. Each command but record-ruling, places and agreement
|
||||
first fetches origin, when the checkout has that remote, and takes origin's default branch, or else the
|
||||
local main or master, as the default branch. When the fetch fails, round-count reads origin as
|
||||
last fetched and says so on stderr; the other three exit 1. The history is
|
||||
docs/comprehension-history.md in the checkout.
|
||||
Run it inside a project checkout or one of its worktrees. Each command but review-folder,
|
||||
record-ruling, places and agreement first fetches origin, when the checkout has that remote, and
|
||||
takes origin's default branch, or else the local main or master, as the default branch. When the
|
||||
fetch fails, round-count reads origin as last fetched and says so on stderr; the other three exit 1.
|
||||
The history is docs/comprehension-history.md in the checkout.
|
||||
|
||||
review-folder makes the chunk's review folder, where each reviewer writes its report, and prints its
|
||||
path: ~/.local/state/larv-review/ followed by the worktree's path relative to ~, or to / outside it.
|
||||
It leaves ~/.local/state/larv-review readable by you alone.
|
||||
|
||||
ruling-base prints the commit a paired ruling reads against, then which of these found it. The last
|
||||
measured commit is the default branch's newest merged change that adds a paired ruling or a scoring
|
||||
@@ -88,7 +92,8 @@ when the history already holds an entry on what that line reads.
|
||||
|
||||
Exit 1, saying why, on unknown arguments, an unknown item ID, a seats file it cannot read or that
|
||||
breaks the shape above, or when git cannot start, finds no checkout or no default branch, cannot
|
||||
fetch origin for ruling-base, needs-scoring-run or record-scoring, or cannot read or write the history or todo.md.
|
||||
fetch origin for ruling-base, needs-scoring-run or record-scoring, or cannot read or write the
|
||||
history, todo.md or the review folder.
|
||||
"""
|
||||
import pathlib
|
||||
import re
|
||||
@@ -96,7 +101,7 @@ import sys
|
||||
import typing
|
||||
|
||||
from round_titles import CAP, PASSES, round_counts
|
||||
from git_repo import default_branch, git
|
||||
from git_repo import default_branch, git, top
|
||||
from failure import Failure
|
||||
from larv_review.scoring import FLOOR
|
||||
from larv_review.git import history_changes
|
||||
@@ -104,8 +109,9 @@ from larv_review.ruling_base import measured_commit
|
||||
from larv_review.agreement import MIN_PAIRS
|
||||
from larv_review.plan import plan_reasons
|
||||
from larv_review.record import record_ruling, record_scoring, show_agreement, show_places
|
||||
from review_folder import REPORTS, review_folder
|
||||
|
||||
USAGE = "usage: larv-review ruling-base | round-count | needs-scoring-run [<item ID>...] | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement"
|
||||
USAGE = "usage: larv-review review-folder | ruling-base | round-count | needs-scoring-run [<item ID>...] | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement"
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
@@ -113,7 +119,9 @@ def main(argv: list[str]) -> int:
|
||||
print(__doc__.strip().format(cap=CAP, floor=FLOOR, pairs=MIN_PAIRS))
|
||||
return 0
|
||||
command, *given = argv or [""]
|
||||
if command == "ruling-base" and not given:
|
||||
if command == "review-folder" and not given:
|
||||
done = make_review_folder()
|
||||
elif command == "ruling-base" and not given:
|
||||
done = with_default_branch(show_ruling_base, stale_ok=False)
|
||||
elif command == "round-count" and not given:
|
||||
done = with_default_branch(round_count, stale_ok=True)
|
||||
@@ -153,6 +161,22 @@ def with_default_branch(command: typing.Callable[[str], Failure | None], stale_o
|
||||
return command(branch)
|
||||
|
||||
|
||||
def make_review_folder() -> Failure | None:
|
||||
worktree = top()
|
||||
if isinstance(worktree, Failure):
|
||||
return worktree
|
||||
home = pathlib.Path.home()
|
||||
folder = pathlib.Path(review_folder(str(home), worktree))
|
||||
try:
|
||||
(home / REPORTS).mkdir(parents=True, exist_ok=True)
|
||||
(home / REPORTS).chmod(0o700)
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as failure:
|
||||
return Failure(f"could not make the review folder {folder}: {failure}")
|
||||
print(folder)
|
||||
return None
|
||||
|
||||
|
||||
def show_ruling_base(default: str) -> Failure | None:
|
||||
changes = history_changes(default)
|
||||
if isinstance(changes, Failure):
|
||||
|
||||
@@ -452,6 +452,34 @@ class LarvReviewTool(unittest.TestCase):
|
||||
self.git("remote", "set-head", "origin", "--delete")
|
||||
self.assertEqual(self.run_tool("needs-scoring-run", "2").stdout, "No scoring run due.\n")
|
||||
|
||||
def test_makes_and_prints_the_worktrees_review_folder_from_anywhere_in_it(self):
|
||||
reports = self.checkout.parent / ".local" / "state" / "larv-review"
|
||||
(self.checkout / "sub").mkdir()
|
||||
ran = subprocess.run([sys.executable, str(TOOL), "review-folder"], capture_output=True, cwd=self.checkout / "sub", env=self.env, text=True)
|
||||
self.assertEqual((ran.returncode, ran.stdout), (0, f"{reports}/proj\n"), ran.stderr)
|
||||
self.assertTrue((reports / "proj").is_dir())
|
||||
|
||||
def test_leaves_the_reports_folder_readable_by_the_user_alone_where_it_was_not(self):
|
||||
reports = self.checkout.parent / ".local" / "state" / "larv-review"
|
||||
reports.mkdir(parents=True, mode=0o755)
|
||||
self.assertEqual(self.run_tool("review-folder").returncode, 0)
|
||||
self.assertEqual(reports.stat().st_mode & 0o777, 0o700)
|
||||
|
||||
def test_fails_where_it_cannot_make_the_review_folder(self):
|
||||
(self.checkout.parent / ".local" / "state").mkdir(parents=True)
|
||||
(self.checkout.parent / ".local" / "state" / "larv-review").write_text("")
|
||||
ran = self.run_tool("review-folder")
|
||||
self.assertEqual(ran.returncode, 1)
|
||||
self.assertIn("could not make the review folder", ran.stderr)
|
||||
|
||||
def test_review_folder_fails_outside_a_checkout(self):
|
||||
ran = subprocess.run([sys.executable, str(TOOL), "review-folder"], capture_output=True, cwd=self.checkout.parent, env=self.env, text=True)
|
||||
self.assertEqual(ran.returncode, 1)
|
||||
self.assertNotIn("Traceback", ran.stderr)
|
||||
|
||||
def test_review_folder_takes_no_arguments(self):
|
||||
self.assertIn("unknown arguments", self.run_tool("review-folder", "x").stderr)
|
||||
|
||||
def test_takes_a_local_main_without_origin_and_reads_origin_as_last_fetched_when_the_fetch_fails(self):
|
||||
self.commit("Answer prose round 1: a")
|
||||
self.git("remote", "set-url", "origin", str(self.checkout.parent / "gone.git"))
|
||||
|
||||
@@ -41,6 +41,10 @@ class Gitignore(unittest.TestCase):
|
||||
ignored = subprocess.run(["git", "ls-files", "-ci", "--exclude-standard"], capture_output=True, check=True, cwd=ROOT, text=True).stdout
|
||||
self.assertEqual(ignored, "")
|
||||
|
||||
def test_ignores_no_python_module_or_test_under_bin(self):
|
||||
ignored = subprocess.run(["git", "ls-files", "-oi", "--exclude-standard", "--", "bin/*.py", "bin/**/*.py"], capture_output=True, check=True, cwd=ROOT, text=True).stdout
|
||||
self.assertEqual(ignored, "")
|
||||
|
||||
def test_ignores_every_private_file(self):
|
||||
ignored = subprocess.run(["git", "check-ignore", "--no-index", "--stdin"], capture_output=True, cwd=ROOT, input="\n".join(PRIVATE), text=True).stdout
|
||||
self.assertEqual(ignored.splitlines(), PRIVATE)
|
||||
|
||||
@@ -28,3 +28,6 @@ class ReviewerLaunches(unittest.TestCase):
|
||||
def test_the_security_opening_is_the_sentence_larv_review_has_the_security_pass_open_with(self):
|
||||
security = LARV_REVIEW.split("### Security", 1)[1].split("### Overrides", 1)[0]
|
||||
self.assertIn(SECURITY_OPENER, security)
|
||||
|
||||
def test_larv_review_has_the_tool_make_the_reports_folder_rule_hook_lets_a_reviewer_write_into(self):
|
||||
self.assertIn("`~/.claude/bin/larv-review review-folder`", LARV_REVIEW)
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
"""The chunk's review folder, where each reviewer writes its report: skills/larv-review/SKILL.md → "Reports"."""
|
||||
import os
|
||||
|
||||
# Why here: docs/decisions.md → `review: each reviewer under agents/ writes its full report into the chunk's review folder`.
|
||||
REPORTS = ".local/state/larv-review"
|
||||
|
||||
|
||||
def review_folder(home: str, worktree: str) -> str:
|
||||
"""The review folder of the chunk whose worktree is at worktree: its path relative to home, or to / outside it, under REPORTS."""
|
||||
inside = worktree.startswith(home.rstrip("/") + "/")
|
||||
return os.path.join(home, REPORTS, os.path.relpath(worktree, home) if inside else worktree.lstrip("/"))
|
||||
@@ -11,6 +11,8 @@ class Call:
|
||||
tool: str
|
||||
given: dict[str, typing.Any]
|
||||
cwd: str
|
||||
# The subagent the call comes from, such as prose-reviewer; empty for the session's own agent.
|
||||
agent: str = ""
|
||||
|
||||
|
||||
def parsed_call(hook_input: object) -> Call | Failure:
|
||||
@@ -19,7 +21,8 @@ def parsed_call(hook_input: object) -> Call | Failure:
|
||||
tool, given, cwd = hook_input.get("tool_name"), hook_input.get("tool_input"), hook_input.get("cwd")
|
||||
if not isinstance(tool, str) or not isinstance(given, dict) or not isinstance(cwd, str):
|
||||
return Failure("the hook input has no string tool_name and cwd and object tool_input")
|
||||
return Call(tool, given, cwd)
|
||||
agent = hook_input.get("agent_type")
|
||||
return Call(tool, given, cwd, agent if isinstance(agent, str) else "")
|
||||
|
||||
|
||||
def text(given: dict[str, typing.Any], key: str) -> str:
|
||||
|
||||
@@ -8,7 +8,7 @@ subagent. The rules:
|
||||
|
||||
{rules}
|
||||
|
||||
A file a Bash command writes, such as through `cat > Dockerfile`, is not checked. The reviewer-rung rule takes no override.
|
||||
A file a Bash command writes, such as through `cat > Dockerfile`, is not checked.
|
||||
|
||||
A call overrides a rule with `# rule-override <rule>: <reason>`:
|
||||
|
||||
@@ -24,7 +24,7 @@ default branch lacks, and those logged under the branch's own name.
|
||||
|
||||
Run with no arguments in a terminal, it prints this help.
|
||||
|
||||
Exit 2 with the rule, why it holds and how to override it, when a rule blocks the call, or when an
|
||||
Exit 2 with the rule, why it holds and, where the rule takes an override, how to write one, when a rule blocks the call, or when an
|
||||
override cannot be logged; 0 otherwise. `rule-hook overrides` exits 1 when git cannot read the
|
||||
checkout or the branch's reflog. Exit 1, a non-blocking error to Claude Code, on input it
|
||||
cannot read.
|
||||
@@ -41,6 +41,7 @@ from rule_hook.rules import Breach, command_breaches, file_breaches, Outcome, ru
|
||||
from git_command import commits
|
||||
from failure import Failure
|
||||
from rule_hook.override_log import Log, log_path, logged, print_overrides, write_override
|
||||
from rule_hook.reviewer_report import report_breaches
|
||||
from rule_hook.reviewer_rung import reviewer_breaches
|
||||
from shell_command import simple_commands
|
||||
from rule_hook.todo_check import todo_breaches
|
||||
@@ -81,7 +82,7 @@ def breaches_of(call: Call, home: str, environ: typing.Mapping[str, str]) -> tup
|
||||
return None
|
||||
before = read(path)
|
||||
added, after = applied(call, before)
|
||||
return added, path, file_breaches(path, before, added, after)
|
||||
return added, path, report_breaches(call, path, home) + file_breaches(path, before, added, after)
|
||||
|
||||
|
||||
def answer(judged: Outcome, what: str, now: float) -> int:
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
"""Refuse a reviewer's file call other than a Write of its report into the reports folder: skills/larv-review/review-findings.md → "Report"."""
|
||||
import os
|
||||
|
||||
from rule_hook.rules import Breach
|
||||
from rule_hook.call import Call
|
||||
from review_folder import REPORTS
|
||||
from rule_hook.reviewer_rung import REVIEWERS
|
||||
|
||||
|
||||
def report_breaches(call: Call, path: str, home: str) -> list[Breach]:
|
||||
"""The breach of a file call from a reviewer that is not a Write under REPORTS in home; none from any other agent."""
|
||||
if call.agent not in REVIEWERS:
|
||||
return []
|
||||
written = os.path.realpath(os.path.join(call.cwd, path))
|
||||
reports = os.path.realpath(os.path.join(home, REPORTS))
|
||||
if call.tool == "Write" and written.startswith(reports + "/"):
|
||||
return []
|
||||
return [Breach("reviewer-report", f"{call.agent} would {call.tool.lower()} {written}; write your report with Write to the path your prompt names, under {reports}/", call.cwd)]
|
||||
@@ -5,9 +5,10 @@ import typing
|
||||
|
||||
from git_command import commits
|
||||
from round_titles import lacks_counts
|
||||
from review_folder import REPORTS
|
||||
from shell_command import Simple
|
||||
|
||||
RuleName = typing.Literal["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "reviewer-rung", "round-title-counts", "todo-rank-check"]
|
||||
RuleName = typing.Literal["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "reviewer-report", "reviewer-rung", "round-title-counts", "todo-rank-check"]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -39,6 +40,12 @@ RULES: dict[RuleName, Rule] = {
|
||||
"no commit, PR, issue comment or file carries a Claude or Anthropic attribution, such as a co-author trailer or a generated-with line",
|
||||
"a git, gh or tea command, and any file written",
|
||||
),
|
||||
# A reviewer only reports, so no marker waives it.
|
||||
"reviewer-report": Rule(
|
||||
f"a reviewer writes nothing but its report, with Write, into a file under ~/{REPORTS}/",
|
||||
"a Write, Edit or MultiEdit call from inside an agent under agents/",
|
||||
overridable=False,
|
||||
),
|
||||
# A rung is not a judgment call, and the refusal names the arguments to pass, so no marker waives it.
|
||||
"reviewer-rung": Rule(
|
||||
"a reviewer, a comprehension seat or the security pass's subagent runs on the rung loop-claude names for it: for a seat or the security pass the project's rung, and for a reviewer the highest of the project's rung, the worker's and the one the chunk's item's E + I picks, so a worker below the project's rung cannot hand a review to its own cheaper model",
|
||||
@@ -180,4 +187,4 @@ def blocks(written: str) -> list[list[str]]:
|
||||
|
||||
|
||||
def rules_help() -> str:
|
||||
return "\n".join(f" {name}\n The rule: {rule.holds}.\n Checked in: {rule.checked}." for name, rule in RULES.items())
|
||||
return "\n".join(f" {name}\n The rule: {rule.holds}.\n Checked in: {rule.checked}." + ("" if rule.overridable else "\n Takes no override.") for name, rule in RULES.items())
|
||||
|
||||
@@ -233,6 +233,41 @@ class ReviewerRung(unittest.TestCase):
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "haiku", "effort": "low"}, {}), [])
|
||||
|
||||
|
||||
class ReviewerReport(unittest.TestCase):
|
||||
def breached(self, tool: str, path: str, agent: str = "prose-reviewer", cwd: str = "/r") -> list[str]:
|
||||
given = {"file_path": path, "content": "BLOCK\n"} if tool == "Write" else {"file_path": path, "old_string": "a", "new_string": "b"}
|
||||
found = breaches_of(Call(tool, given, cwd, agent), "/h", {})
|
||||
return rules(found[2]) if found else []
|
||||
|
||||
def test_a_reviewer_writes_its_report_into_the_reports_folder(self):
|
||||
self.assertEqual(self.breached("Write", "/h/.local/state/larv-review/.claude-item-73/prose-round-1.md"), [])
|
||||
|
||||
def test_a_relative_path_resolves_against_the_calls_cwd(self):
|
||||
self.assertEqual(self.breached("Write", "prose-round-1.md", cwd="/h/.local/state/larv-review/x"), [])
|
||||
|
||||
def test_a_symlink_in_the_reports_folder_is_followed_to_where_it_points(self):
|
||||
home = pathlib.Path(self.enterContext(tempfile.TemporaryDirectory()))
|
||||
(home / ".local" / "state" / "larv-review").mkdir(parents=True)
|
||||
(home / ".local" / "state" / "larv-review" / "out").symlink_to(home)
|
||||
found = breaches_of(Call("Write", {"file_path": f"{home}/.local/state/larv-review/out/a.md", "content": "x"}, "/r", "prose-reviewer"), str(home), {})
|
||||
self.assertEqual(rules(found[2]), ["reviewer-report"])
|
||||
|
||||
def test_a_reviewer_writing_anywhere_else_is_refused(self):
|
||||
for path in ["/r/README.md", "/h/.local/state/larv-review", "/h/.local/state/larv-review-x/a.md", "/h/.local/state/larv-review/../a.md", "/tmp/larv-review/a.md"]:
|
||||
with self.subTest(path):
|
||||
self.assertEqual(self.breached("Write", path), ["reviewer-report"])
|
||||
|
||||
def test_a_reviewer_edits_no_file_even_in_the_reports_folder(self):
|
||||
self.assertEqual(self.breached("Edit", "/h/.local/state/larv-review/x/prose-round-1.md"), ["reviewer-report"])
|
||||
|
||||
def test_a_reviewers_multiedit_is_refused(self):
|
||||
self.assertEqual(self.breached("MultiEdit", "/r/a.py", "stability-reviewer"), ["reviewer-report"])
|
||||
|
||||
def test_the_worker_and_other_subagents_write_anywhere(self):
|
||||
self.assertEqual(self.breached("Write", "/r/README.md", ""), [])
|
||||
self.assertEqual(self.breached("Write", "/r/README.md", "general-purpose"), [])
|
||||
|
||||
|
||||
class Hook(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.repo = tempfile.mkdtemp()
|
||||
@@ -307,6 +342,14 @@ class Hook(unittest.TestCase):
|
||||
call = {"cwd": self.repo, "tool_input": {"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium", "prompt": "Review."}, "tool_name": "Agent"}
|
||||
self.assertEqual(hook(call, self.repo, ReviewerRung.ENVIRON).returncode, 0)
|
||||
|
||||
def test_a_reviewers_write_outside_the_reports_folder_is_refused_and_offers_no_override(self):
|
||||
call = {"agent_type": "prose-reviewer", "cwd": self.repo, "tool_input": {"file_path": f"{self.repo}/a.md", "content": "<!-- rule-override reviewer-report: trust me -->\n"}, "tool_name": "Write"}
|
||||
ran = hook(call, self.repo)
|
||||
self.assertEqual(ran.returncode, 2)
|
||||
self.assertIn("Blocked by rule reviewer-report: prose-reviewer would write ", ran.stderr)
|
||||
self.assertIn("/.local/state/larv-review/", ran.stderr)
|
||||
self.assertNotIn("rule-override", ran.stderr)
|
||||
|
||||
def test_a_commit_override_is_logged_in_the_commit_s_checkout(self):
|
||||
outside = tempfile.mkdtemp()
|
||||
self.addCleanup(subprocess.run, ["rm", "-rf", outside])
|
||||
@@ -370,5 +413,8 @@ class Hook(unittest.TestCase):
|
||||
|
||||
def test_help_lists_every_rule(self):
|
||||
helped = subprocess.run([str(HOOK), "--help"], capture_output=True, check=True, env=isolated_env(), text=True).stdout
|
||||
for rule in ["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "round-title-counts", "todo-rank-check"]:
|
||||
for rule in ["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "reviewer-report", "reviewer-rung", "round-title-counts", "todo-rank-check"]:
|
||||
self.assertIn(f" {rule}\n The rule: ", helped)
|
||||
self.assertIn("reviewer-report\n The rule: a reviewer writes nothing but its report", helped)
|
||||
self.assertRegex(helped, r" reviewer-report\n( .*\n){2} Takes no override\.\n")
|
||||
self.assertRegex(helped, r" image-pin\n( .*\n){2} \S", "a rule a marker waives says nothing of overrides")
|
||||
|
||||
@@ -10,7 +10,9 @@ authored by the time the round began, on the branch the reviewer started on. The
|
||||
branch stands at the newest commit authored before the round's chunk began, so a diff against it
|
||||
shows the chunk's change. The prompt names the clone in place of the worktree it named, which the
|
||||
chunk has since removed. A round whose code it cannot find in this way is not replayed; stderr
|
||||
names it and why.
|
||||
names it and why. The prompt also names a copy of the chunk's review folder in place of that
|
||||
folder, holding only the reports written after the chunk began and before the round did. A replay
|
||||
whose prompt still names the review folder finds nothing.
|
||||
|
||||
It first replays every round on the project's rung, the reference reviewer. Findings on record
|
||||
that the reference reviewer does not raise again may rest on code or rules that have changed
|
||||
@@ -46,6 +48,7 @@ from failure import Failure
|
||||
from rung_replay.search import first_reply, replayed
|
||||
from claude_code.transcript import first_timestamped_line, read_transcript
|
||||
from claude_code.entries import parse_entry, structured_output
|
||||
from review_folder import REPORTS, review_folder
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
TOOL = "stability-replay"
|
||||
@@ -80,7 +83,7 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
return failed(TOOL, f"no archived stability round of projects on {anchor.model} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find")
|
||||
chosen = sample(list(codes), LATEST)
|
||||
# The project's rung is the reference, so the candidates start below it.
|
||||
replay = replayed(chosen, anchor, down_from(anchor)[1:], lambda found, rung: findings_found(found, codes[found], rung, anchor, effects), lambda _, found: sorted(found), "stability rounds")
|
||||
replay = replayed(chosen, anchor, down_from(anchor)[1:], lambda found, rung: findings_found(found, codes[found], rung, anchor, home, effects), lambda _, found: sorted(found), "stability rounds")
|
||||
if isinstance(replay, Limited):
|
||||
return limited_exit(TOOL, replay)
|
||||
print(report(anchor.model, anchor, chosen, replay))
|
||||
@@ -147,9 +150,9 @@ def commit_log(listed: str) -> list[tuple[str, float]]:
|
||||
return [(commit, float(authored)) for commit, authored in (line.split() for line in listed.splitlines())]
|
||||
|
||||
|
||||
def findings_found(found: Round, code: Code, rung: Rung, anchor: Rung, effects: Effects) -> set[int] | Limited:
|
||||
def findings_found(found: Round, code: Code, rung: Rung, anchor: Rung, home: pathlib.Path, effects: Effects) -> set[int] | Limited:
|
||||
"""The indexes of the round's findings that its replay on rung reports, as the reviewer on anchor matches them."""
|
||||
reviewed: str | Failure | Limited = first_reply(lambda: review(found, code, rung, effects))
|
||||
reviewed: str | Failure | Limited = first_reply(lambda: review(found, code, rung, home, effects))
|
||||
if isinstance(reviewed, Limited):
|
||||
return reviewed
|
||||
matched: set[int] | Failure | Limited = reviewed if isinstance(reviewed, Failure) else first_reply(lambda: match(found.findings, reviewed, found.root, anchor, effects))
|
||||
@@ -160,24 +163,40 @@ def findings_found(found: Round, code: Code, rung: Rung, anchor: Rung, effects:
|
||||
|
||||
|
||||
# The shell makes and removes the clone itself, and is tested on a temporary home as judge-replay's is; rounds.py's core takes the filesystem as an argument.
|
||||
def review(found: Round, code: Code, rung: Rung, effects: Effects) -> str | Failure | Limited:
|
||||
clone = tempfile.mkdtemp(prefix="stability-replay-")
|
||||
def review(found: Round, code: Code, rung: Rung, home: pathlib.Path, effects: Effects) -> str | Failure | Limited:
|
||||
try:
|
||||
for command in (
|
||||
("clone", "--quiet", "--shared", "--no-checkout", found.root, clone),
|
||||
("-C", clone, "update-ref", f"refs/remotes/origin/{code.default}", code.base),
|
||||
("-C", clone, "update-ref", f"refs/heads/{code.default}", code.base),
|
||||
("-C", clone, "checkout", "--quiet", *(("-B", found.branch) if found.branch and found.branch != code.default else ("--detach",)), code.reviewed),
|
||||
):
|
||||
if isinstance(ran := git(effects, found.root, *command), Failure):
|
||||
return ran
|
||||
args = ["--agent", REVIEWER, *rung.flags()]
|
||||
reply = claude_reply(effects.claude, args, clone, found.prompt.replace(code.path, clone), "claude")
|
||||
if isinstance(reply, Failure | Limited):
|
||||
return reply
|
||||
return reply["result"] if isinstance(reply.get("result"), str) else Failure("the reviewer replied with no report")
|
||||
finally:
|
||||
shutil.rmtree(clone, ignore_errors=True)
|
||||
(home / REPORTS).mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory(prefix="stability-replay-", ignore_cleanup_errors=True) as clone, tempfile.TemporaryDirectory(prefix="stability-replay-", dir=home / REPORTS, ignore_cleanup_errors=True) as reports:
|
||||
return reviewed_in(found, code, rung, home, effects, clone, reports)
|
||||
except OSError as failure:
|
||||
return Failure(f"could not prepare the replay: {failure}")
|
||||
|
||||
|
||||
def reviewed_in(found: Round, code: Code, rung: Rung, home: pathlib.Path, effects: Effects, clone: str, reports: str) -> str | Failure | Limited:
|
||||
"""The report of found's reviewer on rung, replayed on code cloned into clone, with its review folder copied into reports."""
|
||||
# The round's own report, any later one, and one from an earlier attempt at the item in the same worktree would each hand the reviewer the findings it is to find.
|
||||
folder = pathlib.Path(review_folder(str(home), code.path))
|
||||
for written in folder.iterdir() if folder.is_dir() else ():
|
||||
if written.is_file() and found.began <= written.stat().st_mtime < found.seconds:
|
||||
shutil.copy2(written, reports)
|
||||
# Replace the folder's path first: outside home it contains the worktree's path, which replacing code.path would otherwise turn into the clone's.
|
||||
prompt = found.prompt.replace(str(folder), reports).replace(f"~/{folder.relative_to(home)}", reports).replace(code.path, clone)
|
||||
# A spelling the rewrite misses, such as $HOME/, would leave the reviewer the live folder and the findings in it.
|
||||
if REPORTS in prompt.replace(reports, ""):
|
||||
return Failure("the prompt names the review folder in a spelling the replay does not redirect")
|
||||
for command in (
|
||||
("clone", "--quiet", "--shared", "--no-checkout", found.root, clone),
|
||||
("-C", clone, "update-ref", f"refs/remotes/origin/{code.default}", code.base),
|
||||
("-C", clone, "update-ref", f"refs/heads/{code.default}", code.base),
|
||||
("-C", clone, "checkout", "--quiet", *(("-B", found.branch) if found.branch and found.branch != code.default else ("--detach",)), code.reviewed),
|
||||
):
|
||||
if isinstance(ran := git(effects, found.root, *command), Failure):
|
||||
return ran
|
||||
args = ["--agent", REVIEWER, *rung.flags()]
|
||||
reply = claude_reply(effects.claude, args, clone, prompt, "claude")
|
||||
if isinstance(reply, Failure | Limited):
|
||||
return reply
|
||||
return reply["result"] if isinstance(reply.get("result"), str) else Failure("the reviewer replied with no report")
|
||||
|
||||
|
||||
def match(findings: tuple[str, ...], replay_report: str, root: str, rung: Rung, effects: Effects) -> set[int] | Failure | Limited:
|
||||
|
||||
@@ -10,10 +10,11 @@ import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import unittest.mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from stability_replay.rounds import authored_by, recorded_findings, report, reviewed_path, Round, round_from, sample
|
||||
from stability_replay.rounds import authored_by, recorded_findings, report, reviewed_path, Round, round_from, sample, stamp_seconds
|
||||
from loop_claude.effects import Effects, system_effects
|
||||
from failure import Failure
|
||||
from stability_replay import main
|
||||
@@ -232,6 +233,73 @@ class Main(unittest.TestCase):
|
||||
self.assertIn(f" {self.checkout} 2026-10-14T10:00:00Z: 🟡 slow", out)
|
||||
self.assertTrue(out.endswith("finds: claude-sonnet-5-5 at high effort\n"))
|
||||
|
||||
def review_folder_with_round_1(self):
|
||||
"""The chunk's review folder, holding round 1's report from before the replayed round began at 10:00, that round's own and a later pass's reports, written after, and an earlier attempt's report, from before the chunk began at 09:15."""
|
||||
folder = self.home / ".local" / "state" / "larv-review" / "code" / "proj-x"
|
||||
folder.mkdir(parents=True)
|
||||
(folder / "stability-round-1.md").write_text("round 1\n")
|
||||
before = stamp_seconds("2026-10-14T09:50:00Z")
|
||||
os.utime(folder / "stability-round-1.md", (before, before))
|
||||
(folder / "stability-round-2.md").write_text("🔴 a.txt:1 wrong\n")
|
||||
(folder / "prose-round-1.md").write_text("later\n")
|
||||
(folder / "stability-round-3.md").write_text("🔴 a.txt:1 wrong\n")
|
||||
attempt = stamp_seconds("2026-10-14T09:00:00Z")
|
||||
os.utime(folder / "stability-round-3.md", (attempt, attempt))
|
||||
return folder
|
||||
|
||||
def test_points_the_prompts_review_folder_at_a_copy_of_the_reports_from_before_the_round_that_it_removes(self):
|
||||
folder = self.review_folder_with_round_1()
|
||||
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to ~/.local/state/larv-review/code/proj-x/stability-round-2.md; the last round's is ~/.local/state/larv-review/code/proj-x/stability-round-1.md.")
|
||||
seen = []
|
||||
|
||||
def review(rung):
|
||||
named = pathlib.Path(re.search(r"report to (\S+)/stability-round-2\.md", self.reviews[-1][4]).group(1))
|
||||
seen.append((named, sorted(path.name for path in named.iterdir()), (named / "stability-round-1.md").read_text()))
|
||||
(named / "stability-round-2.md").write_text("replayed\n")
|
||||
return {"type": "result", "result": f"review on {rung.model} {rung.effort}"}
|
||||
|
||||
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0, 1]}, review)
|
||||
self.assertEqual(code, 0, err)
|
||||
named, copied, earlier = seen[0]
|
||||
self.assertEqual(named.parent, folder.parents[1], "the copy sits in the reports folder, where rule-hook lets the reviewer write")
|
||||
self.assertNotEqual(named, folder)
|
||||
self.assertEqual(copied, ["stability-round-1.md"], "the round's own report, later ones and an earlier attempt's would hand the replayed reviewer its findings")
|
||||
self.assertEqual(earlier, "round 1\n")
|
||||
self.assertFalse(named.exists())
|
||||
self.assertEqual(sorted(path.name for path in folder.iterdir()), ["prose-round-1.md", "stability-round-1.md", "stability-round-2.md", "stability-round-3.md"])
|
||||
self.assertEqual((folder / "stability-round-2.md").read_text(), "🔴 a.txt:1 wrong\n")
|
||||
|
||||
def test_points_the_review_folders_absolute_path_at_the_copy(self):
|
||||
folder = self.review_folder_with_round_1()
|
||||
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to {folder}/stability-round-2.md.")
|
||||
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
self.assertEqual(code, 0, err)
|
||||
prompt = self.reviews[0][4]
|
||||
self.assertNotIn(str(folder), prompt)
|
||||
self.assertRegex(prompt, rf"report to {folder.parents[1]}/stability-replay-\w+/stability-round-2\.md\.$")
|
||||
|
||||
def test_replays_no_round_whose_prompt_still_names_the_review_folder_after_the_rewrite(self):
|
||||
self.review_folder_with_round_1()
|
||||
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to $HOME/.local/state/larv-review/code/proj-x/stability-round-2.md.")
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], unused)
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual(self.reviews, [])
|
||||
self.assertIn(f"on {REFERENCE} found nothing: the prompt names the review folder in a spelling the replay does not redirect", err)
|
||||
self.assertTrue(out.endswith("No rung tried: the reference reviewer found no finding to hold one to.\n"))
|
||||
|
||||
def test_finds_nothing_and_leaves_no_folder_behind_when_it_cannot_copy_the_reports(self):
|
||||
folder = self.review_folder_with_round_1()
|
||||
(folder / "stability-round-1.md").chmod(0)
|
||||
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to ~/.local/state/larv-review/code/proj-x/stability-round-2.md.")
|
||||
scratch = pathlib.Path(self.enterContext(tempfile.TemporaryDirectory()))
|
||||
self.enterContext(unittest.mock.patch.object(tempfile, "tempdir", str(scratch)))
|
||||
code, _, err = self.run_main(["claude-opus-5-5"], unused)
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual(self.reviews, [])
|
||||
self.assertIn(f"on {REFERENCE} found nothing: could not prepare the replay: ", err)
|
||||
self.assertEqual(list(scratch.iterdir()), [], "the clone is removed")
|
||||
self.assertEqual([path.name for path in folder.parent.parent.iterdir()], ["code"], "the copy is removed")
|
||||
|
||||
def test_replays_the_rounds_of_every_chunk(self):
|
||||
self.write_chunk("proj", "c2", "2026-10-14T10:30:00Z")
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
|
||||
from review_folder import review_folder
|
||||
|
||||
|
||||
class ReviewFolder(unittest.TestCase):
|
||||
def test_keys_a_worktree_under_home_by_its_path_relative_to_home(self):
|
||||
self.assertEqual(review_folder("/h", "/h/code/proj-item-3"), "/h/.local/state/larv-review/code/proj-item-3")
|
||||
|
||||
def test_keys_a_worktree_outside_home_by_its_path_relative_to_the_root(self):
|
||||
self.assertEqual(review_folder("/h", "/srv/proj-item-3"), "/h/.local/state/larv-review/srv/proj-item-3")
|
||||
self.assertEqual(review_folder("/h", "/hx/proj"), "/h/.local/state/larv-review/hx/proj", "a sibling of home that shares its prefix is outside it")
|
||||
@@ -2141,3 +2141,24 @@ Inherited architect, decided by:
|
||||
- Locality: `bin/loop_claude/cycle/prompts.py:138` `TYPED_HERE` / `typed_answer`: the resume prompt doubles as a data channel back to the judge, resting on `typed`'s layout, the blank-line partition and `put_question` returning one line, in three modules linked only by a comment
|
||||
- Shape: `bin/loop_claude/routing.py:80` `stepped_from`: returns the stepped route, while `Cycle.stepped_from` holds the route before the step; `escalated` is likewise a function, a bool parameter and a local string
|
||||
- Shape: `bin/loop_claude/cycle/judge.py:95` `Interrupt`: shares `escalation` and `bundled` with `Unfinished` in the opposite positional order, and both are built positionally
|
||||
|
||||
## 2026-10-18T07:20:58Z, PR #310 at 1d8dd33, against fdd48bb
|
||||
|
||||
Ruling: same
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Mid A | better | same | same | same | same |
|
||||
| Inherited architect | same | same | same | same | same |
|
||||
|
||||
Mid A, decided by:
|
||||
|
||||
- Navigation: `bin/review_folder.py:5` `REPORTS`: the review folder's path is one constant in a module named for it, imported by `make_review_folder`, `report_breaches` and `reviewed_in`, so where reports go and who enforces it is found from one name
|
||||
|
||||
Inherited architect, decided by:
|
||||
|
||||
- Locality: `bin/todo_table.py:216` `renamed`: one function now holds the renamed-never-finished rule that `named_releases` and `plan.py`'s `emptied` both call, where a comment kept two files in step by hand
|
||||
- Locality: `bin/todo_table.py:205` `named_releases`: now a fold whose answer depends on the texts' order, fixed far away by `--reverse` in `git_repo.py`'s `todos_since` and enforced by nothing
|
||||
- Locality, Self-sufficiency: `bin/stability_replay/main.py:183` `reviewed_in`: three chained replaces that must run in a fixed order, and the copy placed under REPORTS for a reason that lives only in `reviewer_report.py`
|
||||
- Shape: `bin/stability_replay/main.py:175` `reviewed_in`: seven positional parameters with `clone` and `reports` adjacent strings, and REPORTS meaning the root, its realpath and one round's copy in three files
|
||||
- Locality: `bin/review_folder.py:5` `REPORTS`: the path is spelled again in `settings.json`, SKILL.md, CLAUDE.md and larv-review's help, and only the Python reads the constant
|
||||
|
||||
@@ -222,6 +222,22 @@ The decision `worker: a chunk's worker writes the simplest change…` has the wo
|
||||
|
||||
Valid while the worker writes the tests that larv-review's reviewers hold to the principles.
|
||||
|
||||
## review: each reviewer under `agents/` writes its full report into the chunk's review folder under `~/.local/state/larv-review/` and returns the worker its verdict and one line per finding; rule-hook refuses its Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`
|
||||
|
||||
2026-10-18, lilleman through todo item 73. Serves goal 1.4: a report the worker does not need line by line no longer fills its context. Goal 1.1 holds because the worker reads a finding's full entry before it answers a 🔴 or 🟡.
|
||||
|
||||
Each reviewer holds Write for its report. Claude Code passes a subagent's type to a PreToolUse hook as `agent_type`. So `bin/rule_hook/reviewer_report.py` refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`; no marker overrides it. A reviewer's Bash, as stability-reviewer holds, is not confined.
|
||||
|
||||
The folder is fixed, since loop-claude's headless sessions have no scratchpad. It sits outside every checkout, so no project sees a report. It survives a reboot, unlike `/tmp`, so a resumed chunk still reads its earlier rounds. It is in the user's home, so no other local user plants a symlink that widens it. A report may quote a secret, so `larv-review review-folder`, which makes the chunk's folder, leaves `~/.local/state/larv-review` readable by the user alone.
|
||||
|
||||
A chunk keeps its folder after the merge. `stability-replay` hands a replayed reviewer a copy holding only the reports written after the chunk began and before the round did. So the replay reads the earlier rounds its prompt names, never the findings it is to find. The chunk's own reports stay as they were.
|
||||
|
||||
Claude Code's own subagent prompt tells a subagent to return findings rather than write report files. `review-findings.md` → "Report" overrides that, since the file is the worker's input; a prose-reviewer probed on 2026-10-18 wrote the file its prompt named. A general-purpose wrapper that launches the reviewer was not tried: Claude Code's documentation says a subagent cannot start another.
|
||||
|
||||
Before this change, reports took a median 19k tokens per chunk, 8% of the worker's peak context, and lines carrying a severity mark were 9–24% of a report (2026-10-08, over 68 archived transcripts). `stability-replay` still counts each 🔴 and 🟡 by its line in the reviewer's reply, now a one-line summary. todo.md item 187 is to measure what the worker reads of the reports instead.
|
||||
|
||||
Valid while Claude Code passes `agent_type` to hooks inside a subagent. Without it, the rule refuses nothing, and a reviewer may write anywhere.
|
||||
|
||||
## review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
|
||||
|
||||
2026-10-13, lilleman through todo item 97, implemented by claude. Serves goals 1.4 and 1.5. Anthropic's Claude Code best practices: "A reviewer prompted to find gaps will usually report some, even when the work is sound… Chasing every finding leads to over-engineering." 320 of this repo's 876 commits answered a review round. So stability and product-owner drop a finding that is none of these, and stability leaves the style of comments to prose, which reads every comment a chunk touches. An architect finding that names no later change it makes harder cannot be checked, so it is dropped. The architecture pass and the comprehension check keep their whole-project view, which guards goal 1.1 as a project grows. Valid while the findings these rules drop keep changing nothing a user would notice.
|
||||
|
||||
@@ -636,3 +636,12 @@ Read most (opens and searches):
|
||||
## Reading cost, chunk 055546870eaf, finished 2026-10-17T20:46:45Z
|
||||
|
||||
100998 new input tokens before the first commit.
|
||||
|
||||
## Reading cost, chunk 6ab072ca20a3, finished 2026-10-17T22:04:13Z
|
||||
|
||||
248782 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `todo.md` 4
|
||||
- `rules/todo-items.md` 1
|
||||
|
||||
@@ -9,6 +9,7 @@
|
||||
"permissions": {
|
||||
"defaultMode": "auto",
|
||||
"additionalDirectories": [
|
||||
"~/.local/state/larv-review",
|
||||
"~/code"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -45,7 +45,9 @@ A comment is prose, not code. Passes run in this order: line-level polish on cod
|
||||
|
||||
Give a reviewer without Bash the branch diff and `git ls-files`'s output in its prompt: some harnesses grant it Read alone.
|
||||
|
||||
`~/.claude/bin/larv-review` holds this section's mechanical rules: the round count, the paired ruling's base and vote count, the scoring run's trigger, and the history's format and latest places, and its count of how often seats agree. Run it in the chunk's worktree; `~/.claude/bin/larv-review --help` states each rule. On exit 1, repair what its message names and run it again; when you cannot, ask the user.
|
||||
`~/.claude/bin/larv-review` holds this section's mechanical rules: the chunk's review folder, the round count, the paired ruling's base and vote count, the scoring run's trigger, and the history's format and latest places, and its count of how often seats agree. Run it in the chunk's worktree; `~/.claude/bin/larv-review --help` states each rule. On exit 1, repair what its message names and run it again; when you cannot, ask the user.
|
||||
|
||||
**Reports.** Run `~/.claude/bin/larv-review review-folder` in the chunk's worktree: it makes the chunk's review folder and prints its path. In each prompt to an agent under `~/.claude/agents/`, name the file in that folder for its full report: `<pass>-round-<n>.md`. For a principles-auditor, use `principles-<yardstick>-round-<n>.md`. For a relaunch one rung up, add `-rerun` before `.md`. Before you answer a 🔴 or 🟡, read its entry: Grep the report for the finding's place, with the lines after it, rather than Read the whole report. rule-hook refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`. Keep the folder after the merge: `stability-replay` replays a round from the reports its prompt names.
|
||||
|
||||
**Precedent, then ask** — for the architecture pass and the full scan:
|
||||
|
||||
@@ -64,7 +66,7 @@ Give a reviewer without Bash the branch diff and `git ls-files`'s output in its
|
||||
- **Round.** A round is one review by a pass's reviewer: the first, each re-entry, and the rerun after the architect returns Stage 0 questions ("Architecture" below). A relaunch one rung up ("Rung up" below) stays in its round.
|
||||
- **Auto-reviewer.** The external auto-reviewer (§4) is a pass of its own, each review it posts a round.
|
||||
- **Count.** Close each round with one commit that holds the round's fixes. Title it `Answer <pass> round <n>: <k> 🔴, <m> 🟡; …`: `<pass>` is the pass's name in lower case, such as `full scan`, and `<k>` and `<m>` are the 🔴 and 🟡 findings all the round's reviewers reported. Example: `Answer stability round 1: 1 🔴, 2 🟡; flag a deleted test`. A round that changed nothing gets an empty commit (`git commit --allow-empty`). Push the commits when §4 runs next, where there is a remote. `~/.claude/bin/larv-review round-count` prints each pass's count and marks a pass at the cap. The auto-reviewer's count is the number of its reviews on the PR, less its reviews of a `Record the comprehension check:` commit.
|
||||
- **Later rounds.** From round 2, an agent under `~/.claude/agents/` reviews only the answers to its last round's findings, and each fix from another pass that re-entered its pass. Prompt it with its last round's findings verbatim, each with its answer: the fix's diff, or the reason it was dismissed. Give it each other pass's fix as `git diff <the commit before that fix> <the fix's commit>`, since the fixes may be unpushed, with the finding it answered. It reports only an answer that is plainly wrong or worse than the change it answered, so its pass converges in its second round. Other reviewers review the branch as their tool does.
|
||||
- **Later rounds.** From round 2, an agent under `~/.claude/agents/` reviews only the answers to its last round's findings, and each fix from another pass that re-entered its pass. Prompt it with the path of its last round's report and each finding's answer: the fix's diff, or the reason it was dismissed. Give it each other pass's fix as `git diff <the commit before that fix> <the fix's commit>`, since the fixes may be unpushed, with the finding it answered. It reports only an answer that is plainly wrong or worse than the change it answered, so its pass converges in its second round. Other reviewers review the branch as their tool does.
|
||||
- **Done.** A pass is done when its latest round leaves no 🔴 or 🟡 standing: none was reported, or each was shown factually wrong, declined, filed, or named under the bar per "Findings", or settled in the PR summary per "Comprehension check" → "Places", or, for the prose pass, fixed. Resolve that round's 🟢s and ❓s per "Findings" too.
|
||||
- **Cap.** A pass takes at most three rounds per chunk. Instead of a fourth, list what is open and ask the user. For the auto-reviewer, ask before the push that would summon its fourth, and hold the fixes unpushed until the answer.
|
||||
- **Answer.** A yes grants that pass one more round, after which the ask repeats; a no leaves the PR unmerged.
|
||||
@@ -78,7 +80,7 @@ Give a reviewer without Bash the branch diff and `git ls-files`'s output in its
|
||||
**Rung up.** A reviewer rules the work below standard when its report holds systems-architect's BLOCK or a finding against `standards/quality.md`. A comprehension seat does when `record-ruling` counts one of its votes as worse.
|
||||
|
||||
- **Ask.** Such a reviewer or seat may end its report asking for a rerun one rung up. Relaunch it only on the rung loop-claude's prompt names for it; where the prompt names none, the report stands.
|
||||
- **Relaunch.** Relaunch a reviewer on that rung with the same prompt and the report that asked; its report stands in place of that one, in the same round. Relaunch a seat with the same prompt alone; its line replaces the one that asked in the votes file, before `record-ruling` runs. A relaunch cannot ask again.
|
||||
- **Relaunch.** Relaunch a reviewer on that rung with the same prompt, except that it names the `-rerun` report file per "Reports", plus the path of the report that asked; its report stands in place of that one, in the same round. Relaunch a seat with the same prompt alone; its line replaces the one that asked in the votes file, before `record-ruling` runs. A relaunch cannot ask again.
|
||||
- **Escalate.** A below-standard ruling in the report that stands is escalation trigger 4 in `~/.claude/CLAUDE.md` → "Working with the codebase".
|
||||
|
||||
**Findings.** A reviewer rates each finding 🔴 must fix, 🟡 should fix, 🟢 nit, ❓ question, or 🔀 escalation naming the reviewer whose concern it is.
|
||||
@@ -110,7 +112,7 @@ The comprehension check records how hard the project is to read, and blocks no m
|
||||
|
||||
- **Base.** The last measured commit, which `~/.claude/bin/larv-review ruling-base` prints.
|
||||
- **Head.** The head holds `origin/<default branch>`, merged in at §4 step 1, so it differs from the default branch only by this branch's change.
|
||||
- **Seats.** Mid A and the inherited architect. Write what they read and their replies into a votes file in your scratchpad, in the shape `~/.claude/bin/larv-review --help` shows, and run `~/.claude/bin/larv-review record-ruling <votes file>`. On `Split:`, launch the maintainability senior and Junior A together, add their lines, and run it again.
|
||||
- **Seats.** Mid A and the inherited architect. Write what they read and their replies into a votes file in the chunk's review folder, in the shape `~/.claude/bin/larv-review --help` shows, and run `~/.claude/bin/larv-review record-ruling <votes file>`. On `Split:`, launch the maintainability senior and Junior A together, add their lines, and run it again.
|
||||
|
||||
**Scoring run**, when `~/.claude/bin/larv-review needs-scoring-run <the IDs of the items the chunk ships, if any>` prints `Scoring run due:`. Run `needs-scoring-run` before removing the chunk's items from `todo.md`.
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Words: `~/.claude/CLAUDE.md` → "Vocabulary".
|
||||
|
||||
A reviewer reads and reports; it edits nothing. Read `docs/decisions.md` and `todo.md` first: a finding declined or disproved in `docs/decisions.md` or in a comment at the code it concerns, or filed in `todo.md`, is not raised again, unless the decline works against a goal or a technical principle and names no `todo.md` item ending it.
|
||||
A reviewer reads and reports; it edits nothing. It writes no file but its report, except the scratch files that a reviewer holding Bash makes with Bash. Read `docs/decisions.md` and `todo.md` first: a finding declined or disproved in `docs/decisions.md` or in a comment at the code it concerns, or filed in `todo.md`, is not raised again, unless the decline works against a goal or a technical principle and names no `todo.md` item ending it.
|
||||
|
||||
## Severity
|
||||
|
||||
@@ -26,9 +26,21 @@ A review's first line, derived from the findings:
|
||||
|
||||
Where your report holds a finding against `~/.claude/skills/larv-review/standards/quality.md`, or you are systems-architect and return BLOCK, you may end it with the line `Rerun one rung up` when you want a more capable reviewer to check that ruling.
|
||||
|
||||
## Report
|
||||
|
||||
Write your full report with the Write tool to the file your prompt names. The file is the worker's input, which it searches for each finding's entry, so write it even where your harness says to return findings directly. It holds the verdict, then every finding with its place, its evidence and its fix, as your agent file shapes them. Then return only:
|
||||
|
||||
- the verdict line
|
||||
- one line per finding, in this order: its severity mark, its place, and what is wrong, in one sentence. Where the finding breaks `standards/quality.md` or a file under `~/.claude/principles/`, end the line with that file's name. For a 🔀, end it with the reviewer it names. Example: ``🟡 `bin/example.py:42` `parse`: an empty line raises IndexError (technical-principles.md)``
|
||||
- for stability-reviewer, whether the tests passed, and at which commit
|
||||
- `Rerun one rung up`, where your report ends with it
|
||||
- the report's path
|
||||
|
||||
A prompt that names no file, or a Write that fails, gets the full report as your reply.
|
||||
|
||||
## Rounds
|
||||
|
||||
- Handed neither earlier findings nor another pass's fix: surface everything.
|
||||
- Handed an earlier round's findings, each with its fix or the reason it was dismissed: review only those answers. Check each fix — re-run the scenario, never trust the claim — and each dismissal's reason. Report only a fix or a dismissal that is plainly wrong, or worse than the change it answered; a fix you would have written differently is not a finding.
|
||||
- Handed an earlier round's report, with each finding's fix or the reason it was dismissed: review only those answers. Check each fix — re-run the scenario, never trust the claim — and each dismissal's reason. Report only a fix or a dismissal that is plainly wrong, or worse than the change it answered; a fix you would have written differently is not a finding.
|
||||
- Handed a fix from another pass: review its diff as a first round would, limited to its lines.
|
||||
- Handed a report that asked to `Rerun one rung up`: confirm or dismiss each of its findings, with a reason, then review the rest of what your prompt scopes, as the round you rerun would. Do not ask again.
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 186`
|
||||
`Next ID = 188`
|
||||
|
||||
## Items
|
||||
|
||||
@@ -15,7 +15,6 @@
|
||||
| 180 | 0.2.0 | | question | **Settle how a trigger that fires after a step-up at an interrupt or a quoted answer steps the worker, which today it never does.** | 2 | 2 | 2 | 4 | 3 | 1.1 by 3 | 4.1 |
|
||||
| 185 | 0.2.0 | | decision | **Let todo.md say whether a release whose last items moved to other releases was renamed or deferred, so the tools stop guessing and a deferred release stops the loop for its cut and calls its scoring run.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 4, 1.2 by 4 | 9.1 |
|
||||
| 177 | 0.2.0 | | | **Move the next chunk's pick, from `LastCut` and `standing` to `chunk_item` and `default_plan`, into one module named for it, so `git_repo` only runs git and `todo_table` only parses the plan.** | 3 | 1 | 3 | 5 | 4 | 1.1 by 4 | 3.6 |
|
||||
| 73 | 0.2.0 | | | **Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.** | 3 | 2 | 3 | 6 | 5 | 1.4 by 4 | 1.1 |
|
||||
| 159 | 0.2.0 | | decision | **Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.** | 4 | 2 | 3 | 7 | 6 | 1.4 by 7 | 1.1 |
|
||||
| 80 | 0.3.0 | | defect | **State in README → Install that every machine taking this checkout must be a sandbox, as `settings.json`'s auto mode assumes.** | 1 | 2 | 1 | 2 | 1 | 2 by 4 | 17.5 |
|
||||
| 21 | 0.3.0 | | defect | **Give `settings.json`'s model one writer, and sort its keys.** | 3 | 3 | 3 | 7 | 3 | 2 by 5 | 13.7 |
|
||||
@@ -36,6 +35,7 @@
|
||||
| 106 | 0.3.0 | | | **Make larv-review's "Do not over-read a clean result" cite a rule that says Renovate or Dependabot owns dependency CVEs, or drop the citation.** | 1 | 2 | 1 | 2 | 1 | 1.1 by 5 | 10.9 |
|
||||
| 167 | 0.3.0 | | | **Refuse a `/security-review` or `/claude-security` Skill call below the project's rung, as rule-hook refuses such an Agent call.** | 3 | 2 | 3 | 6 | 2 | 1.1 by 4 | 7.3 |
|
||||
| 176 | 0.3.0 | | | **Give `loop-claude --help`'s routing paragraph sub-headings, such as Routing, Judge rung and Routing figures, so a reader finds a rule without reading 39 lines.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3, 1.4 by 2 | 7.3 |
|
||||
| 186 | 0.3.0 | | | **Point every file a session parks outside its checkout, such as the comprehension panel's detached worktree and a build probe's output, at a folder a headless session has, since loop-claude's sessions have no scratchpad.** | 1 | 2 | 2 | 3 | 2 | 1.1 by 3, 1.5 by 2 | 6.9 |
|
||||
| 172 | 0.3.0 | | | **Name the project's rung `project` across `bin/loop_claude`, in place of `anchor`, as `bin/model_routing/ladder.py` names it.** | 2 | 1 | 3 | 4 | 3 | 1.1 by 3, 1.4 by 3 | 6.2 |
|
||||
| 82 | 0.3.0 | | principle | **Sort `AGENTS.md`'s decision titles and `docs/decisions.md`'s entries by title.** | 2 | 2 | 1 | 3 | 4 | 1.1 by 4, 1.4 by 4 | 5.5 |
|
||||
| 173 | 0.3.0 | | | **Have rule-hook's `todo-rank-check` run a worktree's own `bin/rank_todo.py` when the worktree is of this repo, so a branch that changes `todo.md`'s shape commits without an override.** | 2 | 1 | 3 | 4 | 2 | 1.1 by 2, 1.4 by 2 | 5.5 |
|
||||
@@ -48,6 +48,7 @@
|
||||
| 169 | 0.3.0 | | | **Have rule-hook's `todo-rank-check` catch a `todo.md` that the same Bash call edits and then commits.** | 3 | 1 | 3 | 5 | 3 | 1.1 by 2 | 2.7 |
|
||||
| 183 | 0.3.0 | | | **Merge the two 2026-10-13 paragraphs of `docs/decisions.md` → `loop-claude: a judge decides each interrupt …` into one.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 1 | 2.2 |
|
||||
| 184 | 0.3.0 | | | **Rewrap the lines of `loop-claude --help` that run past the column the rest of it wraps at.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 1 | 2.2 |
|
||||
| 187 | 0.3.0 | | | **Count the tokens a worker spends reading reviewers' reports, so goal 1.4's signal shows whether report files lowered what a review costs the worker's context.** | 2 | 2 | 1 | 3 | 3 | 1.4 by 3 | 2.1 |
|
||||
| 174 | 0.3.0 | | | **In a chunk no item routed, ask the worker to name each todo.md item it takes, with its L and E + I, so loop-claude can route on an item it takes in.** | 1 | 2 | 3 | 4 | 2 | 1.1 by 1 | 1.8 |
|
||||
| 122 | 0.4.0 | | | **Assemble the judge's prompt in `bin/loop_claude/cycle/judge.py`, importing only wording the worker also sees, and call the worker "the worker" in it.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 5, 1.4 by 5 | 10.3 |
|
||||
| 158 | 0.4.0 | | decision | **Measure the fixed cost of a chunk from loop-claude's archive, in tokens and hours against its items' S, and set F in `Score = 100 * V / (2**S + F)` to it.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 5, 1.4 by 5 | 10.3 |
|
||||
@@ -114,19 +115,6 @@ Raised by PR #302's paired ruling, where two of four seats read the project as w
|
||||
- `bin/loop_claude/loop/main.py` → `run_loop`: routing reads the plan as chunk-preflight's fetch left it, an order only a comment states.
|
||||
- `bin/chunk_preflight/gathered.py`: the name covers the data, not the pure choices it also holds.
|
||||
|
||||
### 73. Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.
|
||||
|
||||
2026-10-08, lilleman: rescored from A10/G8 to A4/G4 on the measurement below, so other items that lower token use go first.
|
||||
|
||||
Measured over the 68 transcripts in loop-claude's archive on 2026-10-08, matching each Agent call to its tool result and to any `<task-notification>` naming its tool-use id. No report reached the worker twice: 813 arrived once as the hand-back, and 363 once as a notification after Claude Code ran the agent in the background. Reports took a median 19k tokens per chunk, 8% of the worker's peak context (24% at most). Lines carrying a severity mark are 9–24% of a reviewer's report, and the worker would still open the file for each 🔴 or 🟡 it fixes. A third of the report volume is the comprehension seats', whose votes and places the worker copies into the history. File reads fill more of the worker's context: `sed -n` output is 30% of its Bash results and `grep` output 11%, which `reading-cost` targets.
|
||||
|
||||
The reports would live in the session's scratchpad, such as `<scratchpad>/larv-review/prose-round-1.md`, so no project's checkout sees them. A later round gets the path of that pass's earlier report in place of the findings pasted in. How a report reaches the file:
|
||||
|
||||
- **Write access.** Grant each reviewer Write. No reviewer under `agents/` holds it today, an agent's tool list cannot confine Write to one folder, and `~/.claude/CLAUDE.md` → "Working with the codebase" denies edits to a reviewer that only reads. Check whether a PreToolUse hook can tell a subagent's call from the worker's and deny a Write outside the scratchpad.
|
||||
- **A wrapper.** A general-purpose subagent launches the reviewer, saves the report and returns only the summary lines. Claude Code's documentation says a subagent cannot start another; drop this route unless a test shows one can.
|
||||
|
||||
This risks goal 1.1: a worker fixing from a one-line summary can miss context it needs.
|
||||
|
||||
### 159. Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.
|
||||
|
||||
lilleman asked, on 2026-10-16, for the prose pass to run on the worker's rung or lower, and the same day, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. Today `bin/loop_claude/cycle/prompts.py`'s `review_launches` keeps every reviewer at or above the project's rung whatever the item's E + I, so a docs-only chunk such as item 140's, routed to Haiku 5.5, runs its reviewers on Opus 5.5.
|
||||
@@ -213,6 +201,10 @@ rule-hook checks Agent calls (item 163) but no Skill calls. A worker that invoke
|
||||
|
||||
Raised by the prose pass of PR #302: the help, the docstring of `bin/loop_claude/loop/main.py`, is 145 lines, and its routing paragraph one block of 39, from the ladder through the judge's rung to the routing figures. A reader looking for one rule, such as when the worker steps up, reads it all. The comprehension seats already found the help hard (`larv-review places`: "loop-claude --help": a 120-line contract with no pointers to the modules it maps onto).
|
||||
|
||||
### 186. Point every file a session parks outside its checkout, such as the comprehension panel's detached worktree and a build probe's output, at a folder a headless session has, since loop-claude's sessions have no scratchpad.
|
||||
|
||||
`skills/comprehension-panel/SKILL.md` puts the base version's detached worktree "in your scratchpad", and `CLAUDE.md` → "Working with the codebase" lets a build-probing subagent write "only to the session scratchpad". A probe on 2026-10-18 found no `scratchpad_dir` in a headless `claude -p` session's hook input, and neither the session nor its subagent knew a scratchpad path. larv-review now parks its reports and votes file in the chunk's review folder under `~/.local/state/larv-review/`, per `docs/decisions.md` → `review: each reviewer under agents/ writes its full report into the chunk's review folder`. Found by the architecture review of PR #310.
|
||||
|
||||
### 172. Name the project's rung `project` across `bin/loop_claude`, in place of `anchor`, as `bin/model_routing/ladder.py` names it.
|
||||
|
||||
The architecture pass of PR #296 found one rung under two names: `bin/model_routing/ladder.py` calls it `project`, while `bin/loop_claude/routing.py`, `cycle/prompts.py`, `cycle/run.py`'s `Cycle.anchor` and `loop/main.py` call it `anchor`, and gloss it as "anchor, the project's rung" in docstring after docstring. A reader works out that the two are one rung each time. Make `bin/loop_claude/routing.py`'s `first_route` easier to read.
|
||||
@@ -263,6 +255,10 @@ They say the same thing for `~/.claude/rules/` (item 95) and `~/.claude/skills/l
|
||||
|
||||
In `bin/loop_claude/loop/main.py`'s docstring, the line holding "of one PR, is finished" and the one ending "Each of these opens with its UTC time." run to 107 and 159 columns. Raised by the prose review of PR #308.
|
||||
|
||||
### 187. Count the tokens a worker spends reading reviewers' reports, so goal 1.4's signal shows whether report files lowered what a review costs the worker's context.
|
||||
|
||||
Before PR #310, reports reached the worker as the Agent call's result, a median 19k tokens per chunk (2026-10-08, 68 archived transcripts). Now a reviewer returns its verdict and one line per finding, and the worker reads a 🔴 or 🟡 finding's entry from the report file. `reading-cost` counts only tracked files (`bin/reading_cost/measure.py`), and `routing-figures` counts only the Agent call's result (`bin/routing_figures/measure.py`), so neither sees the report reads. Found by the architecture review of PR #310.
|
||||
|
||||
### 174. In a chunk no item routed, ask the worker to name each todo.md item it takes, with its L and E + I, so loop-claude can route on an item it takes in.
|
||||
|
||||
The principles pass of PR #301 found that only `routed_by` in `bin/loop_claude/cycle/prompts.py` asks the worker to name each item it takes with its L and E + I, and `fresh` uses it only where an item routed the chunk. In an unrouted chunk the judge has no levels to report, so neither the taken-in L nor the taken-in E + I lifts the worker, the reviewers or the judge, and the first prompt names no reviewers' rung for a higher E + I.
|
||||
|
||||
Reference in New Issue
Block a user