Answer stability round 1: 0 🔴, 3 🟡 (0 misread); finish item 202, fix the breaches in four more files agents follow
CI / tests (pull_request) Successful in 2m3s
CI / tests (pull_request) Successful in 2m3s
This commit is contained in:
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- The prose reviewer now holds prose to the negation rule: write `doesn't cost anything`, never `costs nothing`, after any verb or its preposition.
|
||||
- `rule-hook` now checks files at `git commit`: each file that differs from HEAD or is untracked and not ignored, even one the commit leaves out. It holds each to `action-pin`, `image-pin`, `ignorefile-sorted` and `no-ai-attribution`, so a file a Bash command wrote is checked as an Edit or Write is. A `# rule-override` line in the file or the command overrides a breach.
|
||||
- `chunk-pr merge` now refuses while a scoring run the plan calls is unrecorded on the branch, and `larv-review needs-scoring-run` calls one after a release is cut with none.
|
||||
|
||||
|
||||
@@ -72,7 +72,7 @@
|
||||
- BANNED: a CLI that refuses to run until given `--data-path` and `--seed`, when nearly every run wants the shipped data and fresh randomness. GOOD: it uses the shipped data and fresh randomness unless a flag overrides them, and `--help` names those flags.
|
||||
* Deterministic order wherever order carries no meaning.
|
||||
- Sort such a list, on a key that does not move: the name a line introduces, never where it came from. Exception: `id` goes first. A sorted list is searchable, its diff shows only what changed, and two edits to it conflict only where they mean to.
|
||||
- An import sorts on its first binding, not its path, so moving or renaming a module reorders nothing. Where lines already group by kind — type imports, host packages — the key sorts within each group.
|
||||
- An import sorts on its first binding, not its path, so moving or renaming a module doesn't reorder anything. Where lines already group by kind — type imports, host packages — the key sorts within each group.
|
||||
* About seven things per level, so a reader new to the project can take in each level at once.
|
||||
- A level is what a reader looks at together: the parameters of a function and its branches (cyclomatic complexity), the methods of a class, the files in a folder and the folders beside them, the sections of a document, the components of a system, the hops of a call chain. Past about seven, group the things under names a reader can follow; each group then counts as one.
|
||||
- Parameters, branches and the values a function tracks act on each other, so a reader holds them in mind together, and a mind holds about four. Aim for four or five of them, and treat seven as the ceiling. Keep a function to one screen, about 24 lines of 80 characters.
|
||||
|
||||
@@ -15,7 +15,7 @@ independently, so every seat's report counts.
|
||||
|
||||
## What makes the result trustworthy
|
||||
|
||||
Hold all of these or the comparison proves nothing.
|
||||
Hold all of these or the comparison doesn't prove anything.
|
||||
|
||||
- **One variable.** Every panelist runs the same model with the same brief and the same report
|
||||
shape. Only the persona differs. A cheaper model for the juniors turns persona into a confound.
|
||||
@@ -162,7 +162,7 @@ with the same four dimensions and overall as the other seven.
|
||||
|
||||
Every panelist ends with a comprehension score, 1 to 10, for the project as a whole. The scale is
|
||||
absolute, so two panels run on two projects produce comparable numbers — a panelist scoring against
|
||||
their own expectations of *this* codebase has produced a number that means nothing next to
|
||||
their own expectations of *this* codebase has produced a number that doesn't mean anything next to
|
||||
another. Put these anchors in every brief, verbatim.
|
||||
|
||||
- **1 — Spaghetti.** No boundary holds. Finding where a behaviour lives is grep archaeology;
|
||||
|
||||
@@ -76,8 +76,8 @@ Give a reviewer without Bash `git ls-files`'s output in its prompt, since some h
|
||||
**Round cap.**
|
||||
|
||||
- **Round.** A round is one review by a pass's reviewer: the first and each re-entry. The rerun after the architect returns Stage 0 questions ("Architecture" below) and a relaunch one rung up ("Rung up" below) stay in their round.
|
||||
- **Auto-reviewer.** The external auto-reviewer (§4) is a pass of its own, each review it posts, up to the cap, a round. Its count is the number of its reviews on the PR, less its reviews of a `Record the comprehension check:` commit, and `round-count` doesn't list it. A commit that answers one of its rounds takes an ordinary title, not an `Answer <pass> round` one, and a round that changed nothing gets no commit.
|
||||
- **Count.** Close each round of a pass other than the auto-reviewer with one commit that holds the round's fixes. Title it `Answer <pass> round <n>: <k> 🔴, <m> 🟡 (<j> misread); …`: `<pass>` is the pass's name in lower case, such as `full scan`, `<k>` and `<m>` are the 🔴 and 🟡 findings all the round's reviewers reported, and `<j>` is how many of those they marked `[misread]` (review-findings.md → "Misread"); a reviewer that marks none, such as `/security-review`, adds 0. Example: `Answer stability round 1: 1 🔴, 2 🟡 (1 misread); flag a deleted test`. A round that changed nothing gets an empty commit (`git commit --allow-empty`). Push the commits when §4 runs next, where there is a remote. `~/.claude/bin/larv-review round-count` prints each pass's count and marks a pass at the cap.
|
||||
- **Auto-reviewer.** The external auto-reviewer (§4) is a pass of its own, each review it posts, up to the cap, a round. Its count is the number of its reviews on the PR, less its reviews of a `Record the comprehension check:` commit, and `round-count` doesn't list it. A commit that answers one of its rounds takes an ordinary title, not an `Answer <pass> round` one, and a round that didn't change anything gets no commit.
|
||||
- **Count.** Close each round of a pass other than the auto-reviewer with one commit that holds the round's fixes. Title it `Answer <pass> round <n>: <k> 🔴, <m> 🟡 (<j> misread); …`: `<pass>` is the pass's name in lower case, such as `full scan`, `<k>` and `<m>` are the 🔴 and 🟡 findings all the round's reviewers reported, and `<j>` is how many of those they marked `[misread]` (review-findings.md → "Misread"); a reviewer that marks none, such as `/security-review`, adds 0. Example: `Answer stability round 1: 1 🔴, 2 🟡 (1 misread); flag a deleted test`. A round that didn't change anything gets an empty commit (`git commit --allow-empty`). Push the commits when §4 runs next, where there is a remote. `~/.claude/bin/larv-review round-count` prints each pass's count and marks a pass at the cap.
|
||||
- **Done.** A pass is done when its latest round leaves no 🔴 or 🟡 standing: none was reported, or each was shown factually wrong, declined, filed, or named under the bar per "Findings", or settled in the PR summary per "Comprehension check" → "Places", or, for the prose pass and a pass at its cap, fixed. Resolve that round's 🟢s and ❓s per "Findings" too.
|
||||
- **Cap.** A pass takes at most two rounds per chunk, and the release review's architecture and principles passes five; don't ask the user for more. Answer its last round's findings per "Findings": those fixes merge once the gate passes, unreviewed by that pass. rule-hook's `round-cap` refuses a round commit that would take a pass past its cap, and no override waives it. A review the auto-reviewer posts after its second isn't a round: fix none of it and rerun no pass for it, but file a security finding it raises per `~/.claude/CLAUDE.md` → "Working loop", and any other 🔴 in `todo.md`.
|
||||
|
||||
@@ -92,7 +92,7 @@ Give a reviewer without Bash `git ls-files`'s output in its prompt, since some h
|
||||
| made rule-hook log an override | Overrides |
|
||||
| changes behaviour: what the code or its tests do | Stability |
|
||||
|
||||
A fix meets every row it fits. Prose has no row, so a comment or doc edit inside a fix reruns nothing. Example: a fix for a stability 🔴 that changes code and a comment reruns Principles and Stability.
|
||||
A fix meets every row it fits. Prose has no row, so a comment or doc edit inside a fix doesn't rerun any pass. Example: a fix for a stability 🔴 that changes code and a comment reruns Principles and Stability.
|
||||
|
||||
**What a round reviews.** An agent under `~/.claude/agents/` reviews the branch in round 1. From round 2 it reviews only the answers to its last round's findings, and each fix from another pass that re-entered its pass. Prompt it with the path of its last round's report and each finding's answer: the fix's diff, or the reason it was dismissed. Give it each other pass's fix as `git diff <the commit before that fix> <the fix's commit>`, since the fixes may be unpushed, with the finding it answered. It reports only an answer that is plainly wrong or worse than the change it answered, so its pass converges in its second round. Other reviewers review the branch as their tool does.
|
||||
|
||||
@@ -159,7 +159,7 @@ The comprehension check records how hard the project is for its contributor to r
|
||||
|
||||
### Security
|
||||
|
||||
`/security-review` over the branch diff is the pass of record, except where the full scan runs on the same change and replaces it. Where the full scan fires but does not run — skipped on a precedent, declined at its cost prompt, or unavailable — `/security-review` runs. Launch a general-purpose subagent that invokes it through the Skill tool, edits nothing, and replies with the skill's report, or that the skill is unavailable. Invoked in your own context, the skill ends your turn on that report. Open this subagent's prompt, and that of any subagent that runs the full scan, with `You are the security pass for this chunk.`: rule-hook reads that sentence as a security launch.
|
||||
`/security-review` over the branch diff is the pass of record, except where the full scan runs on the same change and replaces it. Where the full scan fires but does not run — skipped on a precedent, declined at its cost prompt, or unavailable — `/security-review` runs. Launch a general-purpose subagent that invokes it through the Skill tool, doesn't edit anything, and replies with the skill's report, or that the skill is unavailable. Invoked in your own context, the skill ends your turn on that report. Open this subagent's prompt, and that of any subagent that runs the full scan, with `You are the security pass for this chunk.`: rule-hook reads that sentence as a security launch.
|
||||
|
||||
A verified HIGH inside this chunk's blast radius is fixed here, outside it on its own branch and PR with this chunk waiting behind it. Say which.
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Words: `~/.claude/CLAUDE.md` → "Vocabulary".
|
||||
|
||||
A reviewer reads and reports; it edits nothing. It writes no file but its report, except the scratch files that a reviewer holding Bash makes with Bash. Read `docs/decisions.md` and `todo.md` first: a finding declined or disproved in `docs/decisions.md` or in a comment at the code it concerns, or filed in `todo.md`, is not raised again, unless the decline works against a goal or a technical principle and names no `todo.md` item ending it.
|
||||
A reviewer reads and reports; it doesn't edit anything. It writes no file but its report, except the scratch files that a reviewer holding Bash makes with Bash. Read `docs/decisions.md` and `todo.md` first: a finding declined or disproved in `docs/decisions.md` or in a comment at the code it concerns, or filed in `todo.md`, is not raised again, unless the decline works against a goal or a technical principle and names no `todo.md` item ending it.
|
||||
|
||||
## Severity
|
||||
|
||||
|
||||
@@ -68,7 +68,6 @@
|
||||
| 181 | 0.9.0 | | principle | **Have the test of a quoted answer's step-up assert the stepped rung's facts as literals, not through `prompts.rungs_held`.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 1.0 |
|
||||
| 193 | 0.9.0 | | | **Narrow `readme_goals`'s mark to `Mark` by a typed lookup in place of `typing.cast`, so a mark added to the regex but not the lookup fails at run time, and one added to the lookup but not `Mark` fails mypy.** | 1 | 1 | 3 | 3 | 1 | 1.1 by 2 | 1.0 |
|
||||
| 194 | 0.9.0 | | | **Make `test_waits_out_a_usage_limit_which_ends_a_run_of_failures` expect a usage-limit wait independent of the wall-clock time that passes during the run.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 1.0 |
|
||||
| 202 | 0.9.0 | | | **Add `CLAUDE.md` → "Prose"'s rule "Write negations as people say them", with its examples, to `skills/larv-review/standards/prose.md`, so the prose reviewer holds prose to it.** | 1 | 2 | 2 | 3 | 1 | 1.1 by 2 | 1.0 |
|
||||
| 265 | 0.9.0 | | | **Archive each wait that costs a chunk wall time, for a usage limit, low memory, a failed worker start or a heavy slot, with its start and end stamps, so goal 1.5's hours split waiting from work.** | 3 | 2 | 2 | 5 | 4 | 1.5 by 3 | 1.0 |
|
||||
| 169 | 0.9.0 | | | **Have rule-hook's `todo-rank-check` catch a `todo.md` that the same Bash call edits and then commits.** | 3 | 1 | 3 | 5 | 3 | 1.1 by 2 | 0.8 |
|
||||
| 174 | 0.9.0 | | | **In a chunk no item routed, ask the worker to name each todo.md item it takes, with its L and E + I, so loop-claude can route on an item it takes in.** | 1 | 2 | 3 | 4 | 1 | 1.1 by 1 | 0.5 |
|
||||
@@ -430,10 +429,6 @@ Found by review of PR #317. `bin/release_plan/ranking.py`'s `readme_goals` write
|
||||
|
||||
Found by review of PR #317. The black-box test, `bin/loop_claude/test_black_box.py:609`, not the unit test of the same name in `test_loop_claude.py`, which runs on a fake clock, sets `resets_at` from the test's `time.time()` and expects the wait, rounded to tens, to be 3720. When more than about 5 s pass between that read and the loop's, the wait rounds to 3710 and the test fails with no bug. Item 102 covers only replacing the `time.sleep` swap; this item changes the expected value, for example by asserting the wait against `resets_at + 120` minus the time the loop read, or a bound on it.
|
||||
|
||||
### 202. Add `CLAUDE.md` → "Prose"'s rule "Write negations as people say them", with its examples, to `skills/larv-review/standards/prose.md`, so the prose reviewer holds prose to it.
|
||||
|
||||
Found by the prose review of PR #329. `prose.md` opens by calling itself the full rules that `CLAUDE.md` → "Prose" keeps a core of, yet it lacks that rule, so a prose reviewer whose bar is the standards files never sees it.
|
||||
|
||||
### 265. Archive each wait that costs a chunk wall time, for a usage limit, low memory, a failed worker start or a heavy slot, with its start and end stamps, so goal 1.5's hours split waiting from work.
|
||||
|
||||
No record holds these waits today. An outcomes file holds verdicts, stops, step-ups and the judge's cost (`bin/transcript_archive/outcomes.py`), and `~/.local/state/run-heavy/` holds only the slots' lock files. loop-claude waits out a usage limit and low memory in `bin/loop_claude/cycle/retry.py`; run-heavy waits for a slot in `bin/run_heavy/`. A heavy slot's wait happens inside a worker's Bash call, so loop-claude doesn't see it; run-heavy would write it where the archive can find it.
|
||||
|
||||
Reference in New Issue
Block a user