Answer architecture round 1: one module owns the judge's file lists, the reviewer agents read each principles file's Applies to line, logging applies to services; the audits' findings filed
This commit is contained in:
@@ -19,7 +19,7 @@ You consider internal logic **only** when it directly affects an external surfac
|
||||
|
||||
## Your Review Method
|
||||
|
||||
Before anything else, **read the shared principles** at `~/.claude/principles/`. Two of them are your territory: `rest-apis.md` and `frontend.md`. `technical-principles.md` applies where it surfaces externally (timestamps communicated as UTC, idempotent writes). Re-read each time — they change.
|
||||
Before anything else, **read the shared principles** at `~/.claude/principles/`. Your territory is each file whose `Applies to` line names an external surface; `technical-principles.md` applies where it surfaces externally (timestamps communicated as UTC, idempotent writes). Re-read each time — they change.
|
||||
|
||||
1. **Identify the change surface**: Read the recent changes and pinpoint what external behavior is affected. If the change is purely internal with no external impact, say so plainly and stop — don't manufacture concerns.
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ You are an elite stability-focused code reviewer with deep expertise in defensiv
|
||||
## First Steps
|
||||
|
||||
1. **Scope the change.** You review every PR; a fresh invocation has no prior context, so determine the diff yourself. Diff the PR branch against its base (`gh pr diff`, or `git merge-base <base> HEAD` then `git diff`), and review the changed lines **plus enough surrounding code and callers** to judge correctness — read the whole changed file when practical; a diff alone hides whether a caller breaks. Read the PR description and, when the project uses an issue tracker (project-specific — check its docs) and the change references a ticket, that ticket: the diff is judged against its stated intent, and a scope mismatch in either direction is a finding — claimed-but-unimplemented, and implemented-but-unasked (#2).
|
||||
2. **Read every file in the shared principles** at `~/.claude/principles/` — not just `technical-principles.md`, also `frontend.md`, `logging.md`, and `rest-apis.md`. These are mandatory org guidelines; all code MUST comply. Map the relevant file(s) to the change (frontend PR → `frontend.md`; API PR → `rest-apis.md`; etc.). Re-read each time — they change. Flag violations prominently.
|
||||
2. **Read every file in the shared principles** at `~/.claude/principles/`. These are mandatory org guidelines; all code MUST comply with each that applies. A file's `Applies to` line says what it covers; one without the line covers every change. Re-read each time — they change. Flag violations prominently.
|
||||
3. **Read README.md, AGENTS.md and `docs/decisions.md`** in the current project for project-specific conventions, build instructions, tooling, and the accepted trade-offs a reviewer should honor rather than re-flag. Note the documented test, lint, and static-analysis commands — you will run them.
|
||||
4. **Verify, don't just read.** Run the test suite and static analysis/linters through the project's container tooling, unless a decision puts them on the host; tear down test containers afterwards. A full-suite run catches broken callers that diff-reading misses. When a suspected bug is cheap to confirm, reproduce it — a scratch script in a temp dir, a targeted test run — before reporting it. Leave the working tree exactly as you found it. Anything you could not run goes in Coverage & Verification, never silently skipped.
|
||||
5. **Two passes minimum, then hunt absences.**
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
import dataclasses
|
||||
import typing
|
||||
|
||||
from loop_claude.project.records import ANSWER_DESTINATIONS, ANSWERS_RECORDED
|
||||
from failure import Failure
|
||||
|
||||
RECORD_PLACES = "README.md's Goals or Audience, AGENTS.md, docs/decisions.md, todo.md or a comment"
|
||||
ANSWERS_RECORDED = f"every answer the human typed in it that a later chunk could need is recorded in {RECORD_PLACES} at the one place it governs"
|
||||
JUDGE_RULES = (
|
||||
"Decide whether the human must now be interrupted. interrupt=true when any of these appear:\n"
|
||||
"- the agent asks the human anything, seeks approval, or says a decision is the human's, and neither "
|
||||
@@ -32,7 +31,7 @@ JUDGE_RULES = (
|
||||
"finished=true when the chunk is complete: merged, or nothing remains that the agent could do on "
|
||||
f"its own. Even then, it is finished only once {ANSWERS_RECORDED}. An answer counts as recorded "
|
||||
"where README.md or the records above hold it, ~/.claude/CLAUDE.md included. It also counts where "
|
||||
f"the transcript shows the agent writing it into {RECORD_PLACES}; writing it into "
|
||||
f"the transcript shows the agent writing it into {ANSWER_DESTINATIONS}; writing it into "
|
||||
"~/.claude/CLAUDE.md does not count. finished=false when work remains that a further run can "
|
||||
"continue: an open PR, a review not yet run, CI not yet reported, an answer not yet recorded."
|
||||
)
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
from loop_claude.cycle.judge_reply import ANSWERS_RECORDED, QuotedAnswer
|
||||
from loop_claude.project.records import ANSWERS_RECORDED, QUOTE_SOURCES
|
||||
from loop_claude.claude_code.entries import assistant_parts, ends_stream, hit_limit, limit_said, parse_entry, rejected_until
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.retry import Limited
|
||||
from loop_claude.claude_code.transcript import part_line
|
||||
from loop_claude.project.records import QUOTE_SOURCES
|
||||
from loop_claude.cycle.judge_reply import QuotedAnswer
|
||||
from loop_claude.claude_code.session import Session
|
||||
|
||||
PRINTED_TOOL_INPUT = 160
|
||||
|
||||
@@ -3,10 +3,14 @@ import pathlib
|
||||
from loop_claude.project.checkout import read_utf8
|
||||
from loop_claude.stop import Stop
|
||||
|
||||
# The records: the project's files the judge reads beside the README's premises and HUMAN_RULES.
|
||||
RECORD_FILES = ("AGENTS.md", "docs/decisions.md")
|
||||
HUMAN_RULES = "~/.claude/CLAUDE.md"
|
||||
# The files a quoted answer comes from, named as the human and the worker see them; of README.md the judge sees only the premises.
|
||||
# Where the judge may quote an answer from, named as the human and the worker see them; of README.md the judge sees only the premises.
|
||||
QUOTE_SOURCES = "README.md's Goals and Audience, " + ", ".join(RECORD_FILES) + " or " + HUMAN_RULES
|
||||
# Where the worker may write an answer the human typed, so it counts as recorded; HUMAN_RULES is the human's alone to write.
|
||||
ANSWER_DESTINATIONS = "README.md's Goals or Audience, " + ", ".join(RECORD_FILES) + ", todo.md or a comment"
|
||||
ANSWERS_RECORDED = f"every answer the human typed in it that a later chunk could need is recorded in {ANSWER_DESTINATIONS} at the one place it governs"
|
||||
|
||||
|
||||
def read_records(root: str, claude_md: pathlib.Path) -> str | Stop:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Logging
|
||||
|
||||
Applies to code that writes logs.
|
||||
Applies to a service that writes logs.
|
||||
|
||||
## Levels
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ A pass is one kind of review, a row of the table, named by its first column. Its
|
||||
|
||||
| Pass | Reviewer | Trigger |
|
||||
|------|----------|----------------------------------------|
|
||||
| Architecture | systems-architect, then principles-auditor and the scoring run | carries an architectural decision: a new, restructured or moved component, module, service or boundary; a new dependency or first use of an infrastructure pattern (queue, cron, cache, websocket, …); a new or changed external contract (routes, request/response shapes, events, webhooks, published file formats, CLI flags and exit codes); a schema change beyond an additive column; the chunk shipping its release's last item |
|
||||
| Architecture | systems-architect, then principles-auditor and the scoring run | carries an architectural decision: a new, restructured or moved component, module, service or boundary; a new dependency or first use of an infrastructure pattern (queue, cron, cache, websocket, …); a new or changed external contract (routes, request/response shapes, events, webhooks, published file formats, CLI flags and exit codes); a schema change beyond an additive column; or ships its release's last item |
|
||||
| Security | `/security-review` | changes code or config; a fix, only where it opens an untrusted-input boundary or touches authn/authz, secrets or crypto |
|
||||
| Full scan | the `claude-security` plugin at high effort: `/claude-security scan this branch's changes at high` | touches authn/authz, secrets or crypto, opens an untrusted-input boundary, or adds an external surface |
|
||||
| Stability | stability-reviewer | changes code, config or tests; a fix, only where it changes behaviour: what the code or its tests do |
|
||||
@@ -51,7 +51,7 @@ Give a reviewer without Bash the branch diff and `git ls-files`'s output in its
|
||||
**Round cap.**
|
||||
|
||||
- **Round.** A round is one launch of a pass's reviewer: the first, each re-entry, and the rerun after the architect returns Stage 0 questions ("Architecture" below).
|
||||
- **Architecture round.** The architecture pass's first round is the architect (unless skipped), the principles audit and the scoring run. A later round relaunches the scoring run and each reviewer whose 🔴 or 🟡 its fixes answer, or the whole round, the architect included, where the fixes fire the pass's trigger by more than shipping the release's last item.
|
||||
- **Architecture round.** The architecture pass's first round is the architect (unless skipped), the principles audit and the scoring run. A later round relaunches the scoring run and each reviewer whose 🔴 or 🟡 its fixes answer, or the whole round, the architect included, where the fixes carry an architectural decision.
|
||||
- **Auto-reviewer.** The external auto-reviewer (§4) is a pass of its own, each review it posts a round.
|
||||
- **Count.** Close each round with one commit that holds the round's fixes and is titled `Answer <pass> round <n>: …`, `<pass>` being the pass's name in lower case, such as `full scan`. A round that changed nothing gets an empty commit (`git commit --allow-empty`). Push the commits when §4 runs next, where there is a remote. A pass's count is the highest `<n>` among titles holding `Answer <pass> round <n>:` in `git log origin/<default branch>..HEAD` after a `git fetch`, or in `git log <default branch>..HEAD` with no remote. The auto-reviewer's count is the number of its reviews on the PR.
|
||||
- **Later rounds.** From round 2, prompt an agent under `~/.claude/agents/` with its own last round's findings verbatim and what changed since: `git diff <the commit that round reviewed> HEAD`, since the fixes may be unpushed.
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
`Bar = 9`
|
||||
|
||||
`Next ID = 60`
|
||||
`Next ID = 71`
|
||||
|
||||
| Goal | W |
|
||||
|---|---|
|
||||
@@ -24,21 +24,35 @@
|
||||
|
||||
| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| 64 | 0.1.0 | security | **Check the `settings.json` diff for secrets before the first push under `AGENTS.md`'s working rule.** | 2 | 2 | 6 | 7 | 3 | 20.4 |
|
||||
| 1 | 0.1.0 | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 3 | 5 | 8 | 9 | 2, 3 | 27.8 |
|
||||
| 63 | 0.1.0 | | **Give README → Install a command that checks the repo out into the `~/.claude` Claude Code already created.** | 2 | 2 | 7 | 8 | 1 | 27.5 |
|
||||
| 60 | 0.1.0 | | **Rerun the scoring run before merge whenever a fix landed after the last one, and count a score under the floor or the ratchet as the scoring run's 🔴.** | 3 | 3 | 6 | 7 | 4.1 | 16.1 |
|
||||
| 62 | 0.1.0 | defect | **Lift Self-sufficiency and the overall to the comprehension floor, at the places the seats name.** | 4 | 6 | 6 | 8 | 4.1 | 15.5 |
|
||||
| 70 | 0.1.0 | question | **Record whether `settings.json`'s `"defaultMode": "auto"` with the dangerous-mode prompt skipped is deliberate.** | 2 | 1 | 4 | 5 | 3 | 13.6 |
|
||||
| 65 | 0.1.0 | | **Archive each judge reply the loop rejects, not only the last.** | 3 | 4 | 4 | 5 | 2 | 12.9 |
|
||||
| 21 | 0.1.0 | | **Give `settings.json`'s model one writer, and sort its keys.** | 3 | 3 | 6 | 7 | 4.3 | 12.9 |
|
||||
| 49 | 0.1.0 | principle | **Check that `AGENTS.md` lists exactly the titles of `docs/decisions.md`.** | 1 | 2 | 5 | 6 | 4.3 | 12.5 |
|
||||
| 12 | 0.1.0 | | **Run the test suite in CI on every PR, or record in `docs/decisions.md` why the local run is enough.** | 2 | 4 | 4 | 6 | 4.1 | 11.7 |
|
||||
| 59 | 0.1.0 | | **Bound a run of unfinished verdicts on one chunk, so a worker and judge that disagree cannot cycle unattended without end.** | 3 | 3 | 6 | 6 | 4.4 | 10.9 |
|
||||
| 5 | 0.1.0 | decision | **Key a project on its checkout's full path, and refuse to start while another session uses that key.** | 5 | 5 | 5 | 6 | 4.1 | 10.5 |
|
||||
| 66 | 0.1.0 | principle | **Hold systems-architect's method thresholds only while the comprehension panel finds the flagged code hard.** | 3 | 3 | 4 | 5 | 4.1 | 9.9 |
|
||||
| 4 | 0.1.0 | | **Name each chunk's archive folder by the chunk's start time, so `<key>/` lists chunks in the order they ran.** | 3 | 2 | 4 | 3 | 2 | 9.8 |
|
||||
| 50 | 0.1.0 | principle | **Grant the bench's runs only the tools a review needs, in place of skipping every permission.** | 2 | 2 | 3 | 4 | 3 | 9.7 |
|
||||
| 69 | 0.1.0 | principle | **Measure test coverage, so a chunk that lowers it is seen.** | 3 | 4 | 4 | 5 | 4.1 | 9.6 |
|
||||
| 29 | 0.1.0 | | **Make the contributor a standing member of every project's audience, unless its `## Audience` says maintainability does not matter.** | 2 | 2 | 3 | 5 | 4.1, 4.4 | 9.1 |
|
||||
| 68 | 0.1.0 | principle | **Test `workflow-bench/compare.py`'s blinding and `bench.py`'s run plan.** | 2 | 4 | 3 | 5 | 4.1 | 8.6 |
|
||||
| 67 | 0.1.0 | principle | **Point stability-reviewer at the technical principles in place of its own restatement of them.** | 2 | 3 | 4 | 5 | 4.3 | 8.6 |
|
||||
| 52 | 0.1.0 | principle | **Record why this checkout lives in Claude Code's own directory, or move it out and link it in.** | 6 | 6 | 3 | 5 | 1 | 8.5 |
|
||||
| 6 | 0.1.0 | decision | **Move what `loop-claude` writes out of `~/.claude.json`, and ask the human about each write that must stay.** | 5 | 5 | 4 | 5 | 4.1 | 7.4 |
|
||||
| 51 | 0.1.0 | principle | **Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints.** | 3 | 5 | 3 | 5 | 4.1 | 7.4 |
|
||||
|
||||
## Details
|
||||
|
||||
### 64. Check the `settings.json` diff for secrets before the first push under `AGENTS.md`'s working rule.
|
||||
|
||||
Goal 3, raised by the principles audit of PR #215. Exposure: Claude Code writes `env` and `mcpServers` into `settings.json`, where a token lands. Path: `AGENTS.md` → "Working rules" commits uncommitted changes to that file on a branch, and larv-review §2 pushes it and opens the PR before `/security-review` runs, so a token reaches the remote unchecked.
|
||||
|
||||
### 1. Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.
|
||||
|
||||
It lives under `~/.claude`, which `.gitignore` keeps from the remote, so a lost disk loses every unattended run's record. Back it up only to a destination private to me and encrypted at rest: the worker runs with permissions bypassed, so the archived tool results can hold credentials it read.
|
||||
@@ -47,6 +61,26 @@ It lives under `~/.claude`, which `.gitignore` keeps from the remote, so a lost
|
||||
|
||||
The README line covers the bench's transcripts in `workflow-bench/logs/` too (goal 2).
|
||||
|
||||
### 63. Give README → Install a command that checks the repo out into the `~/.claude` Claude Code already created.
|
||||
|
||||
Goal 1, raised by the principles audit of PR #215: Claude Code creates `~/.claude` on its first run, `git clone` refuses a directory that is not empty, and Install names neither the remote nor another way in.
|
||||
|
||||
### 60. Rerun the scoring run before merge whenever a fix landed after the last one, and count a score under the floor or the ratchet as the scoring run's 🔴.
|
||||
|
||||
Raised by the systems-architect review of PR #215. A stability, product-owner, prose or auto-reviewer fix that neither carries an architectural decision nor answers an architecture 🔴 or 🟡 merges after the last scoring run, so the `comprehension floor:` entry records scores for an earlier commit, and the next chunk's ratchet inherits a fall it did not cause. "Round cap" → "Done" and "Re-entry" → "Rerun" count only 🔴 and 🟡, and nothing makes a score under the floor one. Exempt a commit that only writes records into `docs/decisions.md`, `todo.md` or `CHANGELOG.md`.
|
||||
|
||||
### 62. Lift Self-sufficiency and the overall to the comprehension floor, at the places the seats name.
|
||||
|
||||
Raised by the principles audit of PR #215 against KISS, in `principles/technical-principles.md`, and goal 4.1: the `comprehension floor:` entry in `docs/decisions.md` lists seven ships under the floor and 0.1.0 waits on it, yet no item names the places to fix. PR #215's seats named, under Self-sufficiency: `skills/larv-review/SKILL.md`, which needs `CLAUDE.md` → "Vocabulary" open beside it; the archive layout, defined only in `bin/loop_claude/loop/main.py`'s docstring and not at `cycle/archive.py`; `JUDGE_RULES`' precedence in `cycle/judge_reply.py`, restated by `verdict_from`'s branch order and the `Reply` field descriptions; the `sys.path` insert in `bin/loop-claude` that the `from failure import` lines rely on, unstated at the import sites.
|
||||
|
||||
### 70. Record whether `settings.json`'s `"defaultMode": "auto"` with the dangerous-mode prompt skipped is deliberate.
|
||||
|
||||
"Always grant explicit access to data", in `principles/technical-principles.md`: a classifier grants tool access across `~/code` with no explicit allow list, and no decision records why.
|
||||
|
||||
### 65. Archive each judge reply the loop rejects, not only the last.
|
||||
|
||||
Goal 2, raised by the principles audit of PR #215: the judge runs with `--no-session-persistence`, and the first two rejected replies reach only stderr, so the archive cannot answer the 3am question of why the loop rejected a verdict.
|
||||
|
||||
### 21. Give `settings.json`'s model one writer, and sort its keys.
|
||||
|
||||
Claude Code rewrites `"model"` in this tracked file whenever a model is picked, so one `/model` dirties the checkout and overrides the synced choice.
|
||||
@@ -69,6 +103,10 @@ Raised by the product-owner review of PR #211. The resumed worker gets the judge
|
||||
|
||||
The key names the project's session, archive folder and models entry. `project_key` drops the leading `/`, so `~/code/a` and `/a` share all three, and one loop can resume or delete the other's session.
|
||||
|
||||
### 66. Hold systems-architect's method thresholds only while the comprehension panel finds the flagged code hard.
|
||||
|
||||
KISS, in `principles/technical-principles.md`, which keeps a limit only while readers find the flagged code hard. `agents/systems-architect.md` makes cyclomatic 7 and 24 lines gate rules from the first commit, a 🔴 unless a reason is stated.
|
||||
|
||||
### 4. Name each chunk's archive folder by the chunk's start time, so `<key>/` lists chunks in the order they ran.
|
||||
|
||||
Today the folder is the SHA-256 of the transcript's first timestamped line, which tells the human nothing; that line carries the start time. Drop the fallback to the first line, which every chunk of a project shares, and give the tests' fake transcripts ISO timestamps.
|
||||
@@ -77,10 +115,22 @@ Today the folder is the SHA-256 of the transcript's first timestamped line, whic
|
||||
|
||||
"Always grant explicit access to data", in `principles/technical-principles.md`: `workflow-bench/bench.py` starts claude with `--dangerously-skip-permissions`, and no rule confines it to a sandbox, so a run on the host reaches the host's credentials and other checkouts.
|
||||
|
||||
### 69. Measure test coverage, so a chunk that lowers it is seen.
|
||||
|
||||
"Test coverage should not decline", in `principles/technical-principles.md`: nothing reports coverage, so a chunk that deletes a test or adds untested code passes unseen.
|
||||
|
||||
### 29. Make the contributor a standing member of every project's audience, unless its `## Audience` says maintainability does not matter.
|
||||
|
||||
State that in `CLAUDE.md`'s goals-and-audience rule, and have larv-review's comprehension floor name the contributor as the reader it serves. The floor then gates every chunk while the contributor stands, and a project that drops them skips it.
|
||||
|
||||
### 68. Test `workflow-bench/compare.py`'s blinding and `bench.py`'s run plan.
|
||||
|
||||
"Most, if not all, code should have automated tests", in `principles/technical-principles.md`: `compare.blind()` keeps the comparison blind, and a version name it misses invalidates the bench silently.
|
||||
|
||||
### 67. Point stability-reviewer at the technical principles in place of its own restatement of them.
|
||||
|
||||
DRY, in `principles/technical-principles.md`: `agents/stability-reviewer.md` restates principles in its own words, and the copy has drifted: it asks for "properly randomised UUIDs" where the principle requires UUID v7.
|
||||
|
||||
### 52. Record why this checkout lives in Claude Code's own directory, or move it out and link it in.
|
||||
|
||||
"One owner per value" and "Prefer the vendor-neutral option", in `principles/technical-principles.md`: git, Claude Code and loop-claude all write `~/.claude`, kept apart only by the `.gitignore` allowlist, and the repo's records live in a vendor directory.
|
||||
|
||||
Reference in New Issue
Block a user