Merge remote-tracking branch 'origin/main' into item-293
# Conflicts: # AGENTS.md # agents/stability-reviewer.md # docs/reading-cost.md # todo.md
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
# This repo keeps no `AGENTS.md`
|
||||
|
||||
`~/.claude` is a checkout of this repo, so an `AGENTS.md` here would load into every other project's sessions. Where a rule names a project's `AGENTS.md`, read it here as:
|
||||
|
||||
- the project principles, the title of each decision no project principle covers, and the working rules: the head of `docs/decisions.md`, before `# Decisions`;
|
||||
- the scoring-run values: `skills/comprehension-panel/seat-prompts.md` → "claude-config's values".
|
||||
@@ -1,3 +1,4 @@
|
||||
{
|
||||
"model": "claude-opus-5-5"
|
||||
"model": "claude-opus-5-5",
|
||||
"claudeMdExcludes": ["**/.claude/CLAUDE.md"]
|
||||
}
|
||||
|
||||
+7
-3
@@ -6,7 +6,6 @@
|
||||
|
||||
!.gitattributes
|
||||
!.gitignore
|
||||
!AGENTS.md
|
||||
!CHANGELOG.md
|
||||
!CLAUDE.md
|
||||
!README.md
|
||||
@@ -15,6 +14,8 @@
|
||||
!todo.md
|
||||
|
||||
!.claude/
|
||||
!.claude/rules/
|
||||
!.claude/rules/*.md
|
||||
!.claude/settings.json
|
||||
!.gitea/
|
||||
!.gitea/workflows/
|
||||
@@ -25,9 +26,9 @@
|
||||
!bin/branch_chunks/
|
||||
!bin/branch_chunks/*.py
|
||||
!bin/chunk-pr
|
||||
!bin/chunk-preflight
|
||||
!bin/chunk_pr/
|
||||
!bin/chunk_pr/*.py
|
||||
!bin/chunk-preflight
|
||||
!bin/chunk_preflight/
|
||||
!bin/chunk_preflight/*.py
|
||||
!bin/claude_code/
|
||||
@@ -50,12 +51,12 @@
|
||||
!bin/larv_review/
|
||||
!bin/larv_review/*.py
|
||||
!bin/loop-claude
|
||||
!bin/loop-inbox
|
||||
!bin/loop_claude/
|
||||
!bin/loop_claude/**/
|
||||
!bin/loop_claude/**/*.py
|
||||
!bin/loop_effects/
|
||||
!bin/loop_effects/*.py
|
||||
!bin/loop-inbox
|
||||
!bin/loop_inbox/
|
||||
!bin/loop_inbox/*.py
|
||||
!bin/loop_state/
|
||||
@@ -115,6 +116,9 @@
|
||||
!bin/transcript_archive/*.py
|
||||
!docs/
|
||||
!docs/*.md
|
||||
!judge-field-trials/
|
||||
!judge-field-trials/*.py
|
||||
!judge-field-trials/README.md
|
||||
!presentation/
|
||||
!presentation/README.md
|
||||
!presentation/build-pdf.py
|
||||
|
||||
@@ -1,81 +0,0 @@
|
||||
# Working here
|
||||
|
||||
Decisions live in `docs/decisions.md`:
|
||||
|
||||
- bin/: an outcome is an entry of a `.outcomes.md` file in loop-claude's archive, a record is a document tools or people keep entries in, and a place is what a comprehension seat names
|
||||
- bin/: the tools run on the host's Python, with no container, and so do their tests outside CI
|
||||
- bin/: the tools support only the latest stable Python, and the tests, written in plain unittest, run on it in four pytest-xdist workers
|
||||
- chunks: a chunk takes every unblocked item of its release that fires the same review passes, whatever its purpose, and an item above R 5 ships alone
|
||||
- comprehension floor: every dimension and the overall at 7.0 or above is the target, and blocks no merge or release
|
||||
- deploy: `deploy-release --cut` cuts a release through a pull request merged fast-forward only, then tags the merged commit and moves `~/.claude` to it; the human runs it
|
||||
- deploy: `~/.claude` holds only a release, which `deploy-release` moves it to; this repo is developed in `~/code/claude-config`, where loop-claude runs its chunks like any project's
|
||||
- larv-review: within a chunk lifting a project towards the comprehension floor, work on the project's lowest-scoring dimension comes first, and the floor doesn't block anything
|
||||
- loop-claude: a `claude` that cannot start is a failed worker run, retried
|
||||
- loop-claude: a judge decides each interrupt and each quoted answer from the transcript, the README's premises, `AGENTS.md`, `docs/decisions.md`, `~/.claude/CLAUDE.md`, `~/.claude/rules/` and larv-review's standards, and holds a chunk open while an answer the human typed in it is recorded nowhere; it interrupts for a reserved choice those don't settle, or an ask that carries no recommendation
|
||||
- loop-claude: every claude it or loop-inbox opens on a terminal has `CLAUDE_CODE_SANDBOXED` set, so Claude Code skips its trust dialog, and neither tool writes `~/.claude.json`
|
||||
- loop-claude: each project's model lives in `~/.claude/loop-claude-models.json`, which only loop-claude writes
|
||||
- loop-claude: every claude it starts bypasses permissions inside its sandbox
|
||||
- loop-claude: one session per project, its id derived from the checkout's path relative to `~/code`, or to `/` outside it
|
||||
- loop-claude: the archive keeps the raw transcript, and a failed archive stops the loop
|
||||
- loop-claude: the worker's claude compacts its context before it fills a 200k window, at about 170k tokens, and `CLAUDE.md` → "Compact instructions" says what the summary keeps
|
||||
- loop-inbox: a loop-claude lists itself while it runs, and its waiting question, as files per session under `~/.local/state/loop-claude/`, and takes the first answer, from its terminal or from loop-inbox, that names that question
|
||||
- machine load: a heavy command waits for one of the machine's heavy slots, one for every 4 CPUs, which every session shares, and loop-claude starts a worker only while at least a quarter of the machine's memory is available
|
||||
- machine load: every container a session starts has a memory limit and a swap limit as large as its work needs; rule-hook's `docker-run-memory` blocks a `docker run` or `docker create` without them outside run-heavy, under which a container that lacks them gets the slot's share of three quarters of the memory; and no daemon-wide cap limits all containers together
|
||||
- principles: about seven things per level, four or five where they act on each other, more where a reader finds them by name; crossing a limit asks for a grouping or a recorded reason
|
||||
- reading cost and `score-outcomes`: each waits for 10 finished chunks before it shows a median
|
||||
- reading cost: a rise is the last 5 chunks' median above the 5 before them, and files or refreshes one plan item
|
||||
- reading cost: larv-review measures each finished chunk from loop-claude's archive and appends the figure to `docs/reading-cost.md` in the next chunk's PR
|
||||
- review: `/security-review` is the security pass of record, except where the full scan runs on the same change and replaces it
|
||||
- review: a comment is prose in any file; a branch changing only Markdown no agent follows runs only the prose pass and, for README or docs an outside reader lands on, the product-owner pass; plan rows a chunk files from a finding, re-scores or moves get no review
|
||||
- review: a fix for a 🔴 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in `larv-review` §3's re-entry table the fix meets; from round 2 a pass reviews only the answers to its last round's findings and any other pass's fix that re-entered it; a fix for a 🟡, a 🟢, a ❓ or a prose finding reruns none
|
||||
- review: a pass gets at most two rounds per chunk, and the release review's architecture and principles passes five, with no more and no ask for more; the fixes answering a pass's last round merge once the gate passes
|
||||
- review: a pass skips only on a precedent: a decision, or a plan item crediting the pass, that the repo held before the chunk began
|
||||
- review: a project decision may opt every chunk out of the architect, the product-owner pass or the comprehension check, and a project opting out of the comprehension check voids code health
|
||||
- review: a project decision may suspend the comprehension check for the plan items it names
|
||||
- review: a release review fixes its findings in the release it reviews, and files under that release only a finding that needs a human answer, adds a feature or outlasts its five rounds
|
||||
- review: a reviewer or comprehension seat that rules the work below standard may ask to rerun once, one rung up, in the same round; the ruling that stands is an escalation trigger for the worker unless the fix changes only prose, and never for the judge
|
||||
- review: a reviewer under `agents/` marks a 🔴 or 🟡 `[misread]` where the implementer misread code that had already shipped, each round title counts the marked findings, and `score-outcomes` lists them per chunk
|
||||
- review: an ask no user answers leaves the PR unmerged
|
||||
- review: each pass keeps a reviewer of its own
|
||||
- review: each reviewer under `agents/` writes its full report into the chunk's review folder under `~/.local/state/larv-review/` and returns the worker its verdict and one line per finding; rule-hook refuses its Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`
|
||||
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
|
||||
- review: the architecture and principles passes run only in the release review, a release's last chunk, which must land before the release is ready to cut; between releases the product-owner pass reviews changes to what consumers rely on
|
||||
- review: the comprehension check records how hard a project is to read and blocks no merge: a paired ruling against the last measured commit in the release review, and the four-seat scoring run, once when an item names the floor and twice on one head when a release's items run out
|
||||
- review: the premises chunk's whole review is the human's approval
|
||||
- review: the principles audit runs only in the release review, and reads the project whole with one auditor checking the goals and every principles file that applies
|
||||
- review: mutate-changes mutation-tests the lines a chunk changes before review, in place of measuring test coverage
|
||||
- routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass or the branch changes only Markdown no agent follows, which reviews on the project's rung; the seats and the security pass always on the project's
|
||||
- routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and only where the E + I steps no review up and the worker runs no higher than the project's rung, then even below the worker's; no pass has such a rung
|
||||
- routing: a worker delegates a task only where it needs just the task's conclusion, and each subagent runs on the rung its task's L picks from the project's rung, never below the project's rung when it files or scores todo.md items
|
||||
- routing: a worker hands a task one rung up on each escalation trigger, up to two rungs above the rung its item picks unless the human agrees, and opens the subagent's prompt with the trigger so the routing figures count it
|
||||
- routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, steps one rung up as the judge interrupts or quotes an answer, or next cycle, up to two rungs above the rung its item picks
|
||||
- routing: loop-claude measures review rounds and unfinished verdicts per rung every 20 cycles, and calls for a routing change when, of two rungs with 5 or more chunks each, the lower took more of either than the one above, or when 5 or more subagents of one type had reports longer than the tool results they read
|
||||
- routing: on a model judge-replay covered, the judge runs on the rung a review would get from the highest E + I among its chunk's items, never below the project's rung
|
||||
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
|
||||
- routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, or when the judge reports a taken-in item that picks a higher rung or an escalation trigger the worker cites
|
||||
- rule-hook: a PreToolUse hook enforces each rule a program can check, a `# rule-override <rule>: <reason>` marker overrides any rule but those `rule-hook --help` marks as taking none, and larv-review's overrides pass rules on each override before merge
|
||||
- rules/: `CLAUDE.md` holds the worker's rules that no file path can scope, and each topic's rules for the worker live in a `rules/` file that Claude Code loads by its `paths:` frontmatter
|
||||
- settings.json: Claude Code runs in auto mode with the dangerous-mode prompt skipped
|
||||
- settings.json: each machine owns its model, and git's clean filter `bin/settings_filter.py` keeps it, and Claude Code's key order, out of every commit
|
||||
- tests: CI runs the suite through `ci.sh` on every pull request, in a pinned Python container, and `main`'s branch protection holds a merge until it passes
|
||||
- tests: each test earns its keep as the technical principles' test bullet says, and the worker takes a test's expected values from its item, never from the code
|
||||
- the repo may name bender, bender-infra and ai-sandbox
|
||||
- todo.md: one table of items, scored by the formula it states, held to its bar and ranked by `bin/rank_todo.py`, which ranks each blocker above the items it blocks; each achieve goal weighs by its position, code health first and four times as heavy while the comprehension panel's overall is under 7.0 or unmeasured, then README.md's goals, and a maintain goal weighs 0.5 per point of the confidence an item's test, gate leg or probe adds to it; the table's order is the plan's only order
|
||||
- todo.md: only a session or subagent on the project's rung or above scores or reshapes the plan; one below it drafts new items and changes to filed ones for a subagent at or above that rung to apply and score, and removes finished ones
|
||||
- worker: a chunk's worker writes the simplest change that solves its item, test first wherever a test can check the change, and reads a technical principle only where the item, a finding or its own record names it; larv-review's reviewers hold the change to the principles and to the standards in `skills/larv-review/standards/`
|
||||
- workflow-bench/: the remote may show the bench cases' names and commit SHAs
|
||||
|
||||
## Working rules
|
||||
|
||||
- `routing-calls-raised.md` sits at the root of loop-claude's archive (`$LOOP_CLAUDE_TRANSCRIPTS`, default `~/.claude/loop-claude-transcripts`). Each of its lines is a call or a `Reset`, with a stamp. A call is new when it is stamped after the latest `Reset` and after the latest stamp that `todo.md` or `docs/decisions.md` → `routing:` cites. When there are new calls, file one `todo.md` item to settle them, or refresh the item already filed. Its detail quotes those calls with their stamps, and the routed-chunks table of `routing-figures.md` beside it, never the project names its other tables hold. A change to the ladder, its steps or the delegation rule is the human's. The chunk that settles the calls cites the latest stamp it settled in `docs/decisions.md` → `routing:`, whatever the human decided, so they are not filed again. Once a chunk that changes the ladder, its steps or the delegation rule has merged, append `- <UTC time> Reset` to `routing-calls-raised.md`: the figures then count only chunks finished after it, and its calls may be raised again.
|
||||
- When `~/.claude/settings.json` has uncommitted changes, carry them into `~/code/claude-config` on a branch of their own and merge them through a PR: Claude Code writes that file from any session, and `deploy-release` refuses to move `~/.claude` until a release holds them. Read their diff first: `git -C ~/.claude diff HEAD -- settings.json`. An empty diff means only the model or the key order changed, which `deploy-release` takes as no change. When the diff holds a credential, such as a token in `env` or `mcpServers`, ask the human to move it out of the file, and commit only once it is gone. A token in `env` moves to the shell profile; a server in `mcpServers` moves to `~/.claude.json` with `claude mcp add --scope user`.
|
||||
|
||||
## Scoring run
|
||||
|
||||
The project values for `skills/comprehension-panel/seat-prompts.md`:
|
||||
|
||||
- `{language}`: Python
|
||||
- `{kind}`: a Claude Code configuration with an agent loop and review skills
|
||||
- `{domain}`: driving headless Claude Code sessions, agent review workflows
|
||||
- `{domain docs}`: the Claude Code documentation
|
||||
- `{3am question}`: a loop-claude run exits 1 overnight after the judge returned a verdict the loop rejected, or a larv-review run looped a reviewer a third time
|
||||
@@ -1,5 +1,14 @@
|
||||
# Changelog
|
||||
|
||||
## Unreleased
|
||||
|
||||
- loop-claude now asks each worker to name the items its chunk takes on one line, as in `Chunk items: 12 (L 3, E + I 5), 14 (L 2, E + I 4)`, and scores them from `todo.md` on the default branch. That line, not the judge, now sets the items loop-inbox lists, and how far an item beyond the one the chunk was routed on lifts the worker, its reviewers and the judge. The judge still reports escalations and whether a release is ready to cut.
|
||||
- `~/.claude` holds no `AGENTS.md` any more, so each session in your projects starts about 3,600 tokens smaller, and a session in `~/code/claude-config` loads its `CLAUDE.md` once instead of twice.
|
||||
- A project's `AGENTS.md` may group decision titles into project principles, and each new decision is checked against them: a principle covers it, it breaks one, or it suggests a new one for you to approve.
|
||||
- The prose reviewer now holds prose to the negation rule: write `doesn't cost anything`, never `costs nothing`, after any verb or its preposition.
|
||||
- `rule-hook` now checks files at `git commit`: each file that differs from HEAD or is untracked and not ignored, even one the commit leaves out. It holds each to `action-pin`, `image-pin`, `ignorefile-sorted` and `no-ai-attribution`, so a file a Bash command wrote is checked as an Edit or Write is. A `# rule-override` line in the file or the command overrides a breach.
|
||||
- `chunk-pr merge` now refuses while a scoring run the plan calls is unrecorded on the branch, and `larv-review needs-scoring-run` calls one after a release is cut with none.
|
||||
|
||||
## 0.7.0
|
||||
|
||||
- README's test, CI and mutation-test commands now have their own section, README → Test.
|
||||
|
||||
@@ -8,16 +8,16 @@ How work proceeds in any repo, unless the repo's `AGENTS.md` says otherwise. The
|
||||
- **Plan `todo.md` as one scored table, in the shape whose example `~/.claude/bin/rank_todo.py --help` shows**: every item names the release that ships it. Until `todo.md` has that shape, reshaping it is the only work, and the human approves it.
|
||||
- **When the next release has no items and no work has landed towards it, plan it as this chunk.** Move the highest-scoring items of later releases into it, and get the human's approval of that set before writing any code.
|
||||
- **When the next release has no items left and work has landed towards it, its release review is the chunk** (`larv-review` §3 → "Release review"). **Once that has merged and the release holds no item, the release is finished: stop.** Report what it holds and that it is ready to cut. Under `/loop`, this ends the loop. Until the human cuts it, only a security fix, or an item its release review filed, lands, into that release; never start a later release's items or plan a new one.
|
||||
- **Open every chunk with one line:** the last version cut, how many changes are unreleased (the entries under `CHANGELOG.md` → `## Unreleased`, or else the commits since the last tag), and the release this chunk ships into. A repo that publishes nothing says so once.
|
||||
- **Open every chunk with one line:** the last version cut, how many changes are unreleased (the entries under `CHANGELOG.md` → `## Unreleased`, or else the commits since the last tag), and the release this chunk ships into. A repo that doesn't publish anything says so once.
|
||||
- **Read the README's `## Goals` and `## Audience` before starting a chunk, unless your prompt quotes them.** The chunk's items set its scope. The goals choose between ways to deliver those items, and never add to them.
|
||||
- **Read the plan and the decisions only as far as the chunk needs.** Where your prompt quotes `chunk-preflight`'s facts, they hold the next item's L, E + I, tagline and detail, and each item that may join it with its L, E + I, S, tagline and detail; open `todo.md` only for what they leave out. Otherwise run `grep -n '^## Details' todo.md`, then read `todo.md` with the Read tool up to the line it prints, or whole when it prints none. Then read the details of only the items the chunk takes. `AGENTS.md` lists every decision's title. Read the entry under a title in `docs/decisions.md` only where the title leaves open what the chunk or the review needs.
|
||||
- **Read the plan and the decisions only as far as the chunk needs.** Where your prompt quotes `chunk-preflight`'s facts, they hold the next item's L, E + I, tagline and detail, and each item that may join it with its L, E + I, S, tagline and detail; open `todo.md` only for what they leave out. Otherwise run `grep -n '^## Details' todo.md`, then read `todo.md` with the Read tool up to the line it prints, or whole when it prints none. Then read the details of only the items the chunk takes. `AGENTS.md` lists the project principles and the title of each decision no project principle covers. Read an entry in `docs/decisions.md` only where the project principle or the title leaves open what the chunk or the review needs.
|
||||
- **Work one chunk at a time, and put every item that can share its review into it.** Every review has a fixed cost, so a chunk takes the next plan item plus every other item for which three things hold: they name one release; they fire the same review passes (`larv-review` §3); and no item outside the chunk blocks one of them. The items need not share a purpose. An item whose risk R, in `todo.md`, is above 5 ships alone: it joins no chunk, and none joins it. The next plan item is the first that does not wait: an item waits while it is marked `question`, until the human answers, or while an item that blocks it is still in the plan. Tiny items, a few lines of change each, need not fire the same passes: bundle up to ten that name one release, and that no item outside the chunk blocks, into one chunk. Split a milestone into items in `todo.md` before starting it. Fix a finding in the chunk's own change within the chunk. File a finding about anything else in `todo.md`.
|
||||
- **Ask, don't guess.** Three choices are always the human's: a goal; a release, meaning the items planned into it and when it is cut; and a change to loop-claude's routing ladder, its rungs included. Any other rule that asks the human's approval reserves that choice too. Ask about a reserved choice. For any other choice, first check whether the goals already answer it. If they nearly do, propose a change to the goals and show how it answers this case: an answer settles one question, a goal settles every question like it. Otherwise take the option you would recommend, and record it with its reason: in the item's detail, or, where it governs later work, as the next sentence says. A recommendation you can give a reason for is not a guess. Ask only where you have no recommendation. Write every answer once, where it belongs, never only in chat: a goal in `README.md`, a working rule in `AGENTS.md`, a standing choice in a comment at the one place it governs, or in `docs/decisions.md` once made in more than one place. When an answer holds a goal and a choice, write the goal in the README, and record as a choice only what the goal leaves open.
|
||||
- **Bench a syntax with the people who write it.** Before a new or changed spelling of something users read or write ships — markdown, HTML, a config key, a CLI flag — a panel of fresh-context readers, seated as those users from the README's `## Audience`, picks it.
|
||||
- **Implement the simplest change that solves the item, and ship it with `/larv-review`.** Whenever a test can check the change, write that test first and see it fail. Take its expected values from the item, never from the code. larv-review's reviewers hold the change to the technical principles, and a finding against one names it. Read a file under `~/.claude/principles/` only when the item changes it, a finding or a record you write names it, or you decline a finding.
|
||||
- **Remove a finished item's row and detail from `todo.md`, and revise any decision that cites it.** If the repo publishes something others consume — a package, an image, an API — reword the item for them and move it to `CHANGELOG.md` under `## Unreleased`. A changelog is for the public, never for the next agent. Otherwise delete it; git history is the record. `~/.claude/bin/todo-item finish <ID>` removes the row and detail, and adds the bullet you pass with `--changelog <bullet>`; it refuses while a decision cites the item.
|
||||
- **Releases are the human's.** The version bump and the publishing credentials are theirs, whatever another rule says. Prepare everything else, so cutting a release is one decision.
|
||||
- **Open each `todo.md` item or ticket with a bold one-sentence tagline**, so the taglines alone say what a release holds. The tagline states the target state, checkable when done: an imperative verb plus what holds afterwards. A present-tense fact reads as the current state and says nothing about what to do. Add "today X" only where the direction is otherwise ambiguous. GOOD: `**Accept undefined in MetadataValue and drop such keys on output.**` BANNED: `MetadataValue accepts undefined; such keys are dropped on output`.
|
||||
- **Open each `todo.md` item or ticket with a bold one-sentence tagline**, so the taglines alone say what a release holds. The tagline states the target state, checkable when done: an imperative verb plus what holds afterwards. A present-tense fact reads as the current state and doesn't say anything about what to do. Add "today X" only where the direction is otherwise ambiguous. GOOD: `**Accept undefined in MetadataValue and drop such keys on output.**` BANNED: `MetadataValue accepts undefined; such keys are dropped on output`.
|
||||
- **File a security finding the moment it is found.** A security finding — a leaked credential, secret or PII, an auth bypass, an injection — goes under the next release, marked `security`, naming the exposure and the path it reaches. `~/.claude/bin/rank_todo.py` ranks it first in its release, after only the open questions, which wait for me. Never among additive or nice-to-have items, never only in a chunk's summary, and never as a question about where it belongs: file it, say you did, and I will unmark it if I disagree. Fixing it inside an unrelated change is a separate call — say whether it is in the blast radius.
|
||||
|
||||
## Shared goal
|
||||
@@ -42,7 +42,7 @@ Everything written — chat, commit messages, comments, docs, code — is in **E
|
||||
|
||||
Write the thing, not an introduction to it. State what holds and stop; name a rejected alternative only where that stops a reader choosing it.
|
||||
|
||||
Write negations as people say them. BANNED: `stops nothing`, `changes nothing`, `reveals nothing`, `depends on no one`. GOOD: `doesn't stop anything`, `doesn't change anything`, `doesn't reveal anything`, `doesn't depend on anyone`.
|
||||
Write negations as people say them: after a verb, or after its preposition, write `doesn't <verb> anything` or `anyone`, never `<verb> nothing` or `no one`. BANNED: `stops nothing`, `depends on no one`, `so answering in time costs nothing`. GOOD: `doesn't stop anything`, `doesn't depend on anyone`, `so answering in time doesn't cost anything`.
|
||||
|
||||
Reviewers hold prose to the full rules, with examples: `~/.claude/skills/larv-review/standards/prose.md`.
|
||||
|
||||
@@ -81,7 +81,7 @@ The rules only reviewers apply, for quality, comments, READMEs and prose, live i
|
||||
When one fires, hand the hard part, edits included, to a subagent on the rung loop-claude's prompt names, one above yours. Open the subagent's prompt with the marker loop-claude's prompt gives, then the trigger's evidence. Each later subagent on that task runs on that rung or higher, and each further escalation trigger moves the task one rung higher, up to the highest rung loop-claude's prompt names. Ask the human before going higher. Where loop-claude's prompt names no escalation, as on the ladder's top rung or off it, keep the task in your own thread.
|
||||
- Give a subagent only the access its job needs, and state the limits in its prompt. One that only reads, such as a prose review, gets no shell and no edits. A reviewer under `~/.claude/agents/` writes only its report, under `~/.local/state/larv-review/`, and rule-hook refuses it every other Write, every Edit and every MultiEdit. One that probes a build tags the image as throwaway and removes it afterwards, writes only to a folder it makes under the chunk's review folder (`~/.claude/bin/larv-review review-folder` prints it), and removes no image or file it did not create.
|
||||
- Check `README.md` first for build, test, install, and codegen commands, and fix one that misbehaves in the same change. Refresh `README.md` whenever you change what it describes, keeping it true, not longer.
|
||||
- Where `~/.claude/bin/rule-hook` does not check it, such as an image named in test code, a Kubernetes manifest, a `docker run` line or a file a Bash command writes, pin a container image to its full patch version and a GitHub Action to a semver tag.
|
||||
- Where `~/.claude/bin/rule-hook` does not check it, such as an image named in test code, a Kubernetes manifest or a `docker run` line, pin a container image to its full patch version and a GitHub Action to a semver tag.
|
||||
|
||||
## Naming files & folders
|
||||
- Name by what's inside. BANNED: `scripts`, `utils`, `helpers`, `misc`, `common`, `lib`, `stuff`. GOOD: `e2e-tests/`, `db-migrations/`, `release-tooling/`.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Claude Code configuration
|
||||
|
||||
My personal `~/.claude`, version controlled: `CLAUDE.md`, `AGENTS.md` and `rules/` for instructions, the words under `CLAUDE.md` → "Vocabulary"; `agents/` and `skills/`; `principles/` and `skills/larv-review/standards/` for what the reviewer agents hold projects to; `docs/` for decisions and the comprehension history; `settings.json` for the harness and `.claude/settings.json` for the harness inside this repo; `CHANGELOG.md` and `todo.md`; the tools in `bin/`; `workflow-bench/`, the scripts and prompts of the 2026-09-30 bench of larv-review; `reviewer-merge-trials/`, the 2026-11-09 trials of merging reviewers, with their method, results and scripts; and `presentation/`, a page that explains this setup to newcomers, with the script that prints it to PDF.
|
||||
My personal `~/.claude`, version controlled: `CLAUDE.md` and `rules/` for instructions, the words under `CLAUDE.md` → "Vocabulary"; `.claude/rules/` for the rules that load in this repo alone; `agents/` and `skills/`; `principles/` and `skills/larv-review/standards/` for what the reviewer agents hold projects to; `docs/` for decisions and the comprehension history; `settings.json` for the harness and `.claude/settings.json` for the harness inside this repo; `CHANGELOG.md` and `todo.md`; the tools in `bin/`; `workflow-bench/`, the scripts and prompts of the 2026-09-30 bench of larv-review; `reviewer-merge-trials/`, the 2026-11-09 trials of merging reviewers, with their method, results and scripts; `judge-field-trials/`, the 2026-11-11 trials of which facts code could read in place of loop-claude's judge, with their results and scripts; and `presentation/`, a page that explains this setup to newcomers, with the script that prints it to PDF.
|
||||
|
||||
## Goals
|
||||
|
||||
@@ -77,7 +77,9 @@ Move `~/.claude` to the newest release:
|
||||
~/code/claude-config/bin/deploy-release
|
||||
```
|
||||
|
||||
It refuses while a file `~/.claude` tracks has an uncommitted change, such as a hook Claude Code wrote into `settings.json`: merge that per `AGENTS.md` → "Working rules", then cut a release. A model picked with `/model` is no such change. It prints `CHANGELOG.md`'s section for each version it moved past: do what each asks, such as a new step under Install. `deploy-release --help` says what else it checks.
|
||||
It prints `CHANGELOG.md`'s section for each version it moved past: do what each asks, such as a new step under Install. `deploy-release --help` says what else it checks.
|
||||
|
||||
It refuses while a file `~/.claude` tracks has an uncommitted change, such as a hook Claude Code wrote into `settings.json`; a model picked with `/model` is no such change. Read the change first, as in `git -C ~/.claude diff HEAD -- settings.json`. When it holds a credential, such as a token in `env` or `mcpServers`, move that out of the file: a token in `env` to the shell profile, a server in `mcpServers` to `~/.claude.json` with `claude mcp add --scope user`. Then carry the change into `~/code/claude-config` on a branch of its own, merge it through a PR, and cut a release.
|
||||
|
||||
## Release
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ Run them in this order. A fix for a later check never breaks an earlier one.
|
||||
|
||||
1. **Easy to understand.** Read each sentence once, cold. A sentence you had to reread, a term used before it is explained, or an "it" or "them" that could point at two things is a finding. Fix: **Rewrite**, giving the plain sentence.
|
||||
2. **Most important first.** Name the document's reader and what they want to know first. Check that the sections run in that order, then the paragraphs within each section, then the items within each list. If not, fix: **Reorder**, naming the new order. A section so long that a reader has to search it gets sub-headings or an index. Fix: **Split**.
|
||||
3. **Default to delete.** A line the reader does not need fails. Fix: **Delete** what the code, a type, a test name, the README or another rule already says. Fix: **Move** a choice made in one place to a comment there, one made in more places to `docs/decisions.md` with its title in `AGENTS.md`, and a goal to the README. A pointer standing where a definition belongs (`~/.claude/skills/larv-review/standards/comments.md` → "Links and pointers", "A pointer is not a definition") gets the fix **Move back**: inline the definition, and leave the reasoning where it is.
|
||||
3. **Default to delete.** A line the reader does not need fails. Fix: **Delete** what the code, a type, a test name, the README or another rule already says. Fix: **Move** a choice made in one place to a comment there, one made in more places to `docs/decisions.md` with its title in `AGENTS.md` unless a project principle there covers it, and a goal to the README. A pointer standing where a definition belongs (`~/.claude/skills/larv-review/standards/comments.md` → "Links and pointers", "A pointer is not a definition") gets the fix **Move back**: inline the definition, and leave the reasoning where it is.
|
||||
4. **Short.** Words that do not clearly help 1–3 fail. Fix: **Cut**.
|
||||
|
||||
Also report a list whose order carries no meaning and is not sorted (`~/.claude/principles/technical-principles.md` → "Deterministic order"), a sentence contradicting another, a claim the code shows false, and a word used against `CLAUDE.md` → "Vocabulary".
|
||||
|
||||
@@ -12,13 +12,13 @@ You are an elite stability-focused code reviewer with deep expertise in defensiv
|
||||
1. **Scope the change.** You review every PR; a fresh invocation has no prior context, so determine the diff yourself. Diff the PR branch against its base (`gh pr diff`, or `git merge-base <base> HEAD` then `git diff`), and review the changed lines **plus enough surrounding code and callers** to judge correctness — read the whole changed file when practical; a diff alone hides whether a caller breaks. Read the PR description and, when the project uses an issue tracker (project-specific — check its docs) and the change references a ticket, that ticket: the diff is judged against its stated intent, and a scope mismatch in either direction is a finding — claimed-but-unimplemented, and implemented-but-unasked (#2).
|
||||
2. **Read every file under `~/.claude/principles/`**, the shared principles, and `~/.claude/skills/larv-review/standards/quality.md`. These are mandatory org guidelines; all code MUST comply with each that applies. A file's `Applies to` line says what it covers; one without the line covers every change. Re-read each time — they change. Flag violations prominently.
|
||||
3. **Read README.md, AGENTS.md and `docs/decisions.md`** in the current project for project-specific conventions, build instructions, tooling, and the accepted trade-offs a reviewer should honor rather than re-flag. Note the documented test, lint, and static-analysis commands — you will run them.
|
||||
4. **Verify, don't just read.** Run the test suite and static analysis/linters through the project's container tooling, unless a decision puts them on the host; tear down test containers afterwards. Run the full suite under `~/.claude/bin/run-heavy`. Run it in the background or with the Bash tool's longest timeout: that timeout includes the wait for a slot. A full-suite run catches broken callers that diff-reading misses. When a suspected bug is cheap to confirm, reproduce it — a scratch script, a targeted test run — before reporting it. Make scratch work with Bash, since Write is for your report alone. Put it in a directory you create for this run, and delete it when done; delete nothing else, since a directory the prompt names may hold other agents' files. Leave the working tree exactly as you found it. Anything you could not run goes in Coverage & Verification, never silently skipped.
|
||||
4. **Verify, don't just read.** Run the test suite and static analysis/linters through the project's container tooling, unless a decision puts them on the host; tear down test containers afterwards. Run the full suite under `~/.claude/bin/run-heavy`. Run it in the background or with the Bash tool's longest timeout: that timeout includes the wait for a slot. A full-suite run catches broken callers that diff-reading misses. When a suspected bug is cheap to confirm, reproduce it — a scratch script, a targeted test run — before reporting it. Make scratch work with Bash, since Write is for your report alone. Put it in a directory you create for this run, and delete it when done; don't delete anything else, since a directory the prompt names may hold other agents' files. Leave the working tree exactly as you found it. Anything you could not run goes in Coverage & Verification, never silently skipped.
|
||||
5. **Judge the surviving mutants.** Your prompt hands you what `~/.claude/bin/mutate-changes` printed on the change, and the commit it ran on: the mutants of changed lines that no test kills. Where it hands none, or names a commit other than the head, run it yourself in the change's checkout, through run-heavy as in step 4: `~/.claude/bin/run-heavy ~/.claude/bin/mutate-changes --prefix '<how the project runs its tools>'`. `~/.claude/bin/mutate-changes --help` says which tool it runs for each language and what it prints. Under run-heavy, a container that sets neither memory limit gets the slot's share, in MiB in `RUN_HEAVY_MEMORY`; set `--jobs` so that its parallel test runs together fit in that share. Report the wall time and mutant count in Coverage & Verification, or why nothing was mutated. Ignore a surviving mutant that leaves behaviour unchanged, or changes only message or log text no caller or user depends on. Report one 🟡 per test to add or strengthen, naming the surviving mutants it would catch.
|
||||
6. **Two passes minimum, then hunt absences.**
|
||||
- *Breadth:* every hunk in every changed file, no skimming. Track which files you have covered; batch large diffs, never truncate the review.
|
||||
- *Depth:* trace the riskiest data paths end-to-end (input → logic → persistence/response), the interactions between changed pieces, and between the change and config (timeout hierarchies, retries × idempotency, queue worker counts, env parity).
|
||||
- *Absences:* what should be in this diff but isn't — un-updated callers (grep every changed or renamed public symbol), missing migration for a model change, missing authz on a new path, missing tests, un-regenerated OpenAPI/docs, stale README/AGENTS.md, and deletions the change obsoletes but didn't make.
|
||||
- Repeat a targeted pass over the riskiest files; stop only when a pass yields nothing new.
|
||||
- Repeat a targeted pass over the riskiest files; stop only when a pass doesn't find anything new.
|
||||
|
||||
## Review Focus Areas
|
||||
|
||||
|
||||
@@ -127,7 +127,7 @@ You review the seams that make verification possible and the strategy that uses
|
||||
- **The pyramid follows the boundaries.** Coverage is end-to-end; the pyramid decides at which layer. Each suite earns its existence by naming the failure class only it can catch and its time and determinism budget — how fast it must run and how hermetic it must be. Core logic at the bottom, real-adapter integration at the seams (the actual DB, the actual queue) in the middle, a thin hermetic e2e layer (owned clock, seeded data, faked third parties) proving the wiring on a handful of critical journeys — an e2e failure is a deployment signal, not the primary bug detector. Anything testable at a cheaper layer moves down; an inverting pyramid signals boundaries the design failed to provide. A flaky test is a missing seam and a finding, never a retry policy. Prefer sociable tests with fakes over mock-heavy solitary ones — a suite that only passes by mocking internals is pinning structure, not behavior: the missing-seam smell again.
|
||||
- **Concurrency is testable or it is untested.** Name every place two flows touch shared state, the invariant that serializes each (transaction, queue, idempotency key, ordering guarantee), and how a test forces the interleaving. A race reachable only by stress-looping is a missing seam — flag the seam as the finding; probabilistic runs are scheduled, never gating.
|
||||
- **The external contract is executably pinned.** Contract tests fail when a published shape breaks — the lock on that one-way door. Price them like the versioning machinery (The External Contract): zero consumers outside your control, defer; many, mandatory.
|
||||
- **Load targets come from Stage 0.** Load and performance tests verify the recorded scale numbers; no recorded scale, nothing to verify — fix Stage 0 first (a rig for unrecorded load is "Hunt the imagined requirement" wearing a QA hat). Data and traffic are production-shaped — a load run against a 1000-row database measures nothing. Runs are scheduled outside the merge gate (see Delivery); determinism here means a repeatable method against fixed thresholds, not identical latencies.
|
||||
- **Load targets come from Stage 0.** Load and performance tests verify the recorded scale numbers; no recorded scale, nothing to verify — fix Stage 0 first (a rig for unrecorded load is "Hunt the imagined requirement" wearing a QA hat). Data and traffic are production-shaped — a load run against a 1000-row database doesn't measure anything. Runs are scheduled outside the merge gate (see Delivery); determinism here means a repeatable method against fixed thresholds, not identical latencies.
|
||||
- **Post-deploy production suites are a designed capability.** Tagged, idempotent synthetic traffic against external surfaces only (API, UI), built once and run twice: post-deploy smoke and scheduled monitoring. It asserts liveness of the deployed surface, not logic — logic determinism lives in the pre-merge suites. Self-cleaning means expendable-by-design: a synthetic tenant or TTL-tagged data reaped out-of-band, never teardown-in-the-test, and never a delete endpoint added to the contract only for tests. Side effects are sandboxed by design — a synthetic checkout must never charge a real card — and the synthetic marking propagates through every downstream sink (telemetry, billing, analytics) so real signals stay uncontaminated; a tag that dies at the API edge is a missing seam.
|
||||
|
||||
## Delivery (the path to production is architecture)
|
||||
|
||||
@@ -18,7 +18,9 @@ merge merges the pull request, and only while its head is still the commit ch
|
||||
push after the gate is never merged. It takes the first method the repository allows of:
|
||||
a merge commit, fast-forward only, a rebase followed by a merge commit, a rebase, a squash.
|
||||
GitHub offers only the merge commit, the rebase and the squash. --gate-passed says the
|
||||
merge gate passed on that commit; without it, chunk-pr refuses.
|
||||
merge gate passed on that commit; without it, chunk-pr refuses. It also refuses where
|
||||
`larv-review scoring-run-owed` exits 1, because the chunk owes a scoring run it has not
|
||||
recorded; no flag waives that.
|
||||
clean acts once the forge reports the branch's pull request merged at the commit checked out,
|
||||
by any merge method. It fetches origin, fast-forwards the main checkout to origin's
|
||||
default branch, removes the chunk's worktree, then deletes the local branch; origin's
|
||||
@@ -32,6 +34,7 @@ It prints what it did, exit 0, or why it failed, exit 1.
|
||||
import dataclasses
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import typing
|
||||
|
||||
@@ -40,6 +43,7 @@ from chunk_preflight.gathered import Forge
|
||||
from chunk_preflight.choices import forge_of
|
||||
from deploy_release.forge import gitea, merged_pull, open_pull, Run, runner
|
||||
|
||||
BIN = pathlib.Path(__file__).resolve().parents[1]
|
||||
USAGE = "usage: chunk-pr open <title> [<body>] | chunk-pr ready | chunk-pr merge --gate-passed | chunk-pr clean"
|
||||
|
||||
# Each forge's merge methods in the order merge prefers them, as the repository setting that allows the method and the method's name in the forge's merge call. A merge commit and a fast-forward keep the branch's commits as they are; a rebase or a squash rewrites them.
|
||||
@@ -157,6 +161,9 @@ def merged(run: Run) -> str | Failure:
|
||||
if isinstance(found, Failure):
|
||||
return found
|
||||
forge, number, head = found
|
||||
owed = run([str(BIN / "larv-review"), "scoring-run-owed"])
|
||||
if isinstance(owed, Failure):
|
||||
return owed
|
||||
method = merge_method(forge, run)
|
||||
if isinstance(method, Failure):
|
||||
return method
|
||||
|
||||
@@ -21,6 +21,7 @@ GH_LIST = ["gh", "pr", "list", "--head", "item-7", "--json", "number,headRefOid,
|
||||
TEA_LIST = ["tea", "api", "repos/{owner}/{repo}/pulls?state=open&limit=50"]
|
||||
GH_SETTINGS = ["gh", "repo", "view", "--json", "mergeCommitAllowed,rebaseMergeAllowed,squashMergeAllowed"]
|
||||
TEA_SETTINGS = ["tea", "api", "repos/{owner}/{repo}"]
|
||||
OWED = [str(pathlib.Path(__file__).resolve().parents[1] / "larv-review"), "scoring-run-owed"]
|
||||
|
||||
|
||||
class Checkout:
|
||||
@@ -58,7 +59,7 @@ class Checkout:
|
||||
|
||||
def acted(self) -> list[list[str]]:
|
||||
"""What it ran beyond reading the checkout, the pull requests and the repository's settings."""
|
||||
reads = [GH_LIST, TEA_LIST, GH_SETTINGS, TEA_SETTINGS]
|
||||
reads = [GH_LIST, TEA_LIST, GH_SETTINGS, TEA_SETTINGS, OWED]
|
||||
return [command for command in self.ran if tuple(command) not in self.answers and command not in reads]
|
||||
|
||||
|
||||
@@ -137,6 +138,14 @@ class Merge(unittest.TestCase):
|
||||
self.assertIsInstance(step(["merge"], checkout), Failure)
|
||||
self.assertEqual(checkout.ran, [])
|
||||
|
||||
def test_a_scoring_run_the_chunk_owes_and_has_not_recorded_refuses_the_merge(self):
|
||||
for origin in (GITHUB, GITEA):
|
||||
with self.subTest(origin=origin):
|
||||
checkout = Checkout(origin, open=True)
|
||||
checkout.answers[tuple(OWED)] = Failure("larv-review scoring-run-owed: the chunk owes 2 scoring runs")
|
||||
self.assertEqual(step(["merge", "--gate-passed"], checkout), Failure("larv-review scoring-run-owed: the chunk owes 2 scoring runs"))
|
||||
self.assertEqual(checkout.acted(), [])
|
||||
|
||||
def test_github_merges_with_a_merge_commit_pinned_to_the_head(self):
|
||||
checkout = Checkout(GITHUB, open=True, allowed=("squashMergeAllowed", "mergeCommitAllowed"))
|
||||
self.assertEqual(step(["merge", "--gate-passed"], checkout), "Merged pull request #12 (merge).")
|
||||
|
||||
@@ -123,7 +123,7 @@ class Main(unittest.TestCase):
|
||||
|
||||
@staticmethod
|
||||
def verdict(interrupt, evidence):
|
||||
output = {"items_taken_in": [], "answer": "", "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "evidence": evidence, "finished": not interrupt, "interrupt": interrupt, "question": "Which?" if interrupt else "", "answered_before": "", "left_open": "", "release_ready": False}
|
||||
output = {"answer": "", "escalation": "", "evidence": evidence, "finished": not interrupt, "interrupt": interrupt, "question": "Which?" if interrupt else "", "answered_before": "", "left_open": "", "release_ready": False}
|
||||
return {"type": "result", "structured_output": output}
|
||||
|
||||
def test_replays_the_models_projects_on_the_reference_then_from_the_projects_rung_down_and_prints_the_lowest_that_returns_all(self):
|
||||
|
||||
+27
-5
@@ -1,8 +1,8 @@
|
||||
"""Hold larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's rounds, the scoring run's trigger; the comprehension history's format, its latest places and its count of how often seats agree; and the header of each reviewer's prompt.
|
||||
|
||||
usage: larv-review review-folder | ruling-base | round-count | needs-scoring-run [<item ID>...] | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement | reviewer-header <pass> [--rerun]
|
||||
usage: larv-review review-folder | ruling-base | round-count | needs-scoring-run [<item ID>...] | scoring-run-owed | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement | reviewer-header <pass> [--rerun]
|
||||
|
||||
The comprehension check records how hard a project is to read, two ways, and blocks no merge. In a
|
||||
The comprehension check records how hard a project is to read, two ways, and no score blocks a merge. In a
|
||||
paired ruling, simulated readers, the seats, read the branch's head beside a base and vote better,
|
||||
same or worse, overall and on each dimension. In a scoring run, four seats score the project from 1
|
||||
to 10. Each seat names the places it found hard.
|
||||
@@ -11,7 +11,7 @@ Run it inside a project checkout or one of its worktrees. Each command but revie
|
||||
record-ruling, places and agreement first fetches origin, when the checkout has that remote, and
|
||||
takes origin's default branch, or else the local main or master, as the default branch. When the
|
||||
fetch fails, round-count and reviewer-header read origin as last fetched and say so on stderr; the
|
||||
other three exit 1.
|
||||
other four exit 1.
|
||||
The history is docs/comprehension-history.md in the checkout.
|
||||
|
||||
review-folder makes the chunk's review folder, where each reviewer writes its report, and prints its
|
||||
@@ -36,6 +36,18 @@ an item whose tagline names the comprehension floor. It is also due when the chu
|
||||
with no item in the checkout's todo.md, once the shipped items are set aside, and work has landed
|
||||
towards that release; bin/larv_review/plan.py's `work_landed` and `landed_towards` say when. A release the checkout's
|
||||
todo.md marks renamed in its ## Renamed section, as `- 0.2.1 renamed to 0.3.0`, is due none.
|
||||
A chunk is also due one while no scoring run in the default branch's history is newer than the
|
||||
commit of its second-newest v* tag, so a release cut unmeasured is caught up by the next chunk. A project whose
|
||||
docs/decisions.md opts out of the comprehension check is never due one.
|
||||
|
||||
scoring-run-owed prints `No scoring run owed.` or exits 1 naming the runs the chunk owes and has
|
||||
not recorded. chunk-pr merge runs it and refuses to merge on exit 1, so a due line read earlier in
|
||||
the chunk can't stand in for it. It applies needs-scoring-run's rule to the checkout's todo.md and the committed history, against the
|
||||
merge base with the default branch, so a default branch that has moved on adds nothing: the items
|
||||
the merge base's todo.md holds and the checkout's does not count as shipped. A chunk that empties a
|
||||
release owes two runs, any other due chunk one. The chunk has recorded the scoring runs that its
|
||||
committed docs/comprehension-history.md holds beyond the merge base's. A project that opts out or
|
||||
has no todo.md items table owes none.
|
||||
|
||||
A second scoring run of one head names itself `<PR or branch> rerun at <commit>`, so record-scoring
|
||||
does not refuse it as a repeat of the first.
|
||||
@@ -110,7 +122,7 @@ missing.
|
||||
Exit 1, saying why, on unknown arguments, an unknown item ID, a seats file it cannot read or that
|
||||
breaks the shape above, a todo.md that fails rank_todo.py after record-scoring records a run, or
|
||||
when git cannot start, finds no checkout or no default branch, cannot fetch origin for ruling-base,
|
||||
needs-scoring-run or record-scoring, or cannot read or write the history, todo.md or the review
|
||||
needs-scoring-run, scoring-run-owed or record-scoring, or cannot read or write the history, todo.md or the review
|
||||
folder.
|
||||
"""
|
||||
import pathlib
|
||||
@@ -126,12 +138,13 @@ from larv_review.history_changes import history_changes
|
||||
from review_rules.review_folder import made_review_folder
|
||||
from larv_review.ruling_base import measured_commit
|
||||
from larv_review.agreement import MIN_PAIRS
|
||||
from larv_review.owed import owed_scoring_run
|
||||
from review_rules.review_passes import PASSES, REVIEWERS
|
||||
from larv_review.plan import plan_reasons
|
||||
from larv_review.history_commands import record_ruling, record_scoring, show_agreement, show_places
|
||||
from larv_review.reviewer_header import reviewer_header
|
||||
|
||||
USAGE = "usage: larv-review review-folder | ruling-base | round-count | needs-scoring-run [<item ID>...] | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement | reviewer-header <pass> [--rerun]"
|
||||
USAGE = "usage: larv-review review-folder | ruling-base | round-count | needs-scoring-run [<item ID>...] | scoring-run-owed | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement | reviewer-header <pass> [--rerun]"
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
@@ -147,6 +160,8 @@ def main(argv: list[str]) -> int:
|
||||
done = with_default_branch(round_count, stale_ok=True)
|
||||
elif command == "needs-scoring-run" and all(re.fullmatch(r"[0-9]+", id_) for id_ in given):
|
||||
done = with_default_branch(lambda default: needs_scoring_run(default, [int(id_) for id_ in given]), stale_ok=False)
|
||||
elif command == "scoring-run-owed" and not given:
|
||||
done = with_default_branch(owed, stale_ok=False)
|
||||
elif command == "record-ruling" and len(given) == 1:
|
||||
done = record_ruling(pathlib.Path(given[0]))
|
||||
elif command == "record-scoring" and given and all(re.fullmatch(r"[0-9]+", id_) for id_ in given[1:]):
|
||||
@@ -223,6 +238,13 @@ def cap_note(count: int, most: int) -> str:
|
||||
return f", {'at' if count == most else 'past'} the cap: no more rounds"
|
||||
|
||||
|
||||
def owed(default: str) -> Failure | None:
|
||||
unrecorded = owed_scoring_run(default)
|
||||
if unrecorded is None:
|
||||
print("No scoring run owed.")
|
||||
return unrecorded
|
||||
|
||||
|
||||
def needs_scoring_run(default: str, ids: list[int]) -> Failure | None:
|
||||
reasons = plan_reasons(default, ids)
|
||||
if isinstance(reasons, Failure):
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
import pathlib
|
||||
|
||||
from larv_review.plan import EMPTIED_WORDS, plan_reasons
|
||||
from review_rules.comprehension_history import entries, HISTORY
|
||||
from failure import Failure
|
||||
from git_cli.git_repo import file_at, git, top
|
||||
from release_plan.todo_table import items, NoItemsTable
|
||||
|
||||
|
||||
def owed_scoring_run(default: str) -> Failure | None:
|
||||
"""A Failure saying which scoring runs the chunk owes and has not recorded; None where it owes none.
|
||||
|
||||
The chunk owes the runs `plan_reasons` calls, two where it empties a release, else one. It has recorded the scoring runs its committed history holds beyond those at the merge base, so a default branch that has moved on since the branch began counts for nothing. A project with no plan owes none. A missing run blocks the merge and a low score never does, which keeps the decision `review: the comprehension check records …`: it bars only scores from blocking.
|
||||
"""
|
||||
checkout = top()
|
||||
if isinstance(checkout, Failure):
|
||||
return checkout
|
||||
if not has_plan(checkout):
|
||||
return None
|
||||
base = git("merge-base", default, "HEAD", cwd=checkout)
|
||||
if isinstance(base, Failure):
|
||||
return base
|
||||
reasons = plan_reasons(base.strip(), None)
|
||||
if isinstance(reasons, Failure):
|
||||
return reasons
|
||||
if not reasons:
|
||||
return None
|
||||
before = file_at(base.strip(), HISTORY, checkout)
|
||||
if isinstance(before, Failure):
|
||||
return before
|
||||
after = file_at("HEAD", HISTORY, checkout)
|
||||
if isinstance(after, Failure):
|
||||
return after
|
||||
return shortfall(reasons, len(scoring_runs(after)) - len(scoring_runs(before)))
|
||||
|
||||
|
||||
def shortfall(reasons: list[str], recorded: int) -> Failure | None:
|
||||
"""Which scoring runs the plan's reasons call that recorded, the count the branch added to the history, leaves unrecorded; None where it holds them all."""
|
||||
needed = 2 if any(reason.endswith(EMPTIED_WORDS) for reason in reasons) else 1
|
||||
if recorded >= needed:
|
||||
return None
|
||||
return Failure(f"the chunk owes {needed} scoring run{'s' if needed > 1 else ''} and the branch's committed history records {max(recorded, 0)}: it {' and '.join(reasons)}. larv-review's SKILL.md → 'Scoring run' says how to run and record {'them' if needed > 1 else 'it'}{"; → 'Second run' says how the second is titled" if needed > 1 else ""}")
|
||||
|
||||
|
||||
def has_plan(checkout: str) -> bool:
|
||||
"""Whether the checkout has a todo.md holding an items table. A file that can't be read counts, so `plan_reasons` reports it."""
|
||||
try:
|
||||
return not isinstance(items(pathlib.Path(checkout, "todo.md").read_text(encoding="utf-8")), NoItemsTable)
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return True
|
||||
|
||||
|
||||
def scoring_runs(history: str) -> list[str]:
|
||||
return [entry.heading for entry in entries(history) if not entry.paired]
|
||||
+28
-10
@@ -2,28 +2,36 @@ import pathlib
|
||||
|
||||
from release_plan.release_standing import Cut, default_plan, LastCut, next_release, Unknown
|
||||
from failure import Failure
|
||||
from git_cli.git_repo import file_at, top
|
||||
from larv_review.scoring import FLOOR_ITEM
|
||||
from release_plan.todo_table import Item, items, NoItemsTable, renames
|
||||
from release_plan.ranking import OPTED_OUT
|
||||
from release_plan.release import Release, version
|
||||
from git_cli.git_repo import top
|
||||
from larv_review.unmeasured import unmeasured_cut
|
||||
|
||||
|
||||
def scoring_run_due(before: list[Item], after: list[Item], ships: list[int], landed_release: Release | None, renamed: set[Release]) -> list[str] | Failure:
|
||||
# The words that end the reason a chunk empties a release. Such a chunk is due two scoring runs; a chunk with any other reason, one.
|
||||
EMPTIED_WORDS = "with no plan item"
|
||||
|
||||
|
||||
def scoring_run_due(before: list[Item], after: list[Item], ships: list[int], landed_release: Release | None, renamed: set[Release], unmeasured: str | None = None) -> list[str] | Failure:
|
||||
"""Why the plan calls a scoring run, empty when it does not.
|
||||
|
||||
A chunk that ships an item naming the floor is due one, so it can prove the floor met. So is a chunk that empties a release after work has landed towards it, so no release ends unmeasured. A release the chunk's plan marks renamed is due none.
|
||||
A chunk that ships an item naming the floor is due one, so it can prove the floor met. So is a chunk that empties a release after work has landed towards it, so no release ends unmeasured. A release the chunk's plan marks renamed is due none. A chunk after a release `unmeasured_cut` names is due one too, so a release cut with no scoring run is caught up.
|
||||
|
||||
before is the default branch's plan and after the chunk's. ships holds the IDs of the items the chunk ships. landed_release is the next release where work has landed towards it, else None (`landed_towards`). renamed holds the releases the chunk's plan marks renamed.
|
||||
before is the default branch's plan and after the chunk's. ships holds the IDs of the items the chunk ships. landed_release is the next release where work has landed towards it, else None (`landed_towards`). renamed holds the releases the chunk's plan marks renamed. unmeasured is `unmeasured_cut`'s answer.
|
||||
"""
|
||||
plan = {item.id: item for item in before + after}
|
||||
if unknown := sorted(set(ships) - set(plan)):
|
||||
return Failure(f"neither plan holds item {', '.join(map(str, unknown))}")
|
||||
shipped = [plan[id_] for id_ in ships]
|
||||
reasons = [f"ships item {item.id}, which names the comprehension floor" for item in shipped if FLOOR_ITEM in item.tagline.lower()]
|
||||
if unmeasured:
|
||||
reasons.append(f"follows {unmeasured}")
|
||||
for release in emptied(before, after, shipped, renamed):
|
||||
if work_landed(release, shipped, landed_release):
|
||||
# SKILL.md → "Second run" keys on this wording.
|
||||
reasons.append(f"leaves release {version(release)} with no plan item")
|
||||
reasons.append(f"leaves release {version(release)} {EMPTIED_WORDS}")
|
||||
return reasons
|
||||
|
||||
|
||||
@@ -55,12 +63,17 @@ def emptied(before: list[Item], after: list[Item], shipped: list[Item], renamed:
|
||||
return sorted(named - {item.release for item in kept} - renamed)
|
||||
|
||||
|
||||
def plan_reasons(default: str, ids: list[int]) -> list[str] | Failure:
|
||||
"""Why the plan calls a scoring run, from the default branch's todo.md and the checkout's."""
|
||||
def plan_reasons(ref: str, ids: list[int] | None) -> list[str] | Failure:
|
||||
"""Why the plan calls a scoring run, from todo.md at ref and the checkout's; empty where the project opts out of the comprehension check. ref is the default branch, or the commit where the chunk began. Where ids is None, the chunk ships each item the plan at ref holds and the checkout's does not."""
|
||||
checkout = top()
|
||||
if isinstance(checkout, Failure):
|
||||
return checkout
|
||||
held = default_plan(checkout, default)
|
||||
decisions = file_at(ref, "docs/decisions.md", checkout)
|
||||
if isinstance(decisions, Failure):
|
||||
return decisions
|
||||
if OPTED_OUT.search(decisions):
|
||||
return []
|
||||
held = default_plan(checkout, ref)
|
||||
before = held.todo
|
||||
if isinstance(before, Failure):
|
||||
return before
|
||||
@@ -73,14 +86,19 @@ def plan_reasons(default: str, ids: list[int]) -> list[str] | Failure:
|
||||
if isinstance(planned, NoItemsTable):
|
||||
planned = []
|
||||
if isinstance(planned, Failure):
|
||||
return Failure(f"{default}'s todo.md: {planned.reason}")
|
||||
return Failure(f"{ref}'s todo.md: {planned.reason}")
|
||||
renamed = renames(after)
|
||||
if isinstance(renamed, Failure):
|
||||
return Failure(f"the checkout's todo.md: {renamed.reason}")
|
||||
landed_release = landed_towards(planned, held.cut)
|
||||
if isinstance(landed_release, Failure):
|
||||
return landed_release
|
||||
return scoring_run_due(planned, chunk, ids, landed_release, renamed)
|
||||
unmeasured = unmeasured_cut(ref, checkout)
|
||||
if isinstance(unmeasured, Failure):
|
||||
return unmeasured
|
||||
if ids is None:
|
||||
ids = sorted({item.id for item in planned} - {item.id for item in chunk})
|
||||
return scoring_run_due(planned, chunk, ids, landed_release, renamed, unmeasured)
|
||||
|
||||
|
||||
def checkout_plan(checkout: str) -> tuple[str, list[Item]] | Failure:
|
||||
|
||||
@@ -12,7 +12,7 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from larv_review.agreement import agreement_report, ruling_agreement, run_agreement
|
||||
from review_rules.comprehension_history import appended, entries, HistoryChange, latest_places, Place, recorded, ruling_entry, scoring_entry, ScoringRun, Seat
|
||||
from larv_review.plan import checkout_plan, landed_towards, plan_reasons, scoring_run_due
|
||||
from larv_review.plan import checkout_plan, EMPTIED_WORDS, landed_towards, plan_reasons, scoring_run_due
|
||||
from review_rules import comprehension_history
|
||||
from larv_review.ruling import counted, ruling, split, surplus, votes
|
||||
from larv_review.history_commands import decided
|
||||
@@ -23,6 +23,7 @@ from release_plan.todo_table import Item, items, NoItemsTable
|
||||
from release_plan.release_standing import LastCut
|
||||
from larv_review.ruling_base import measured_commit
|
||||
from larv_review.seats import score_line, scores, seats_file, SeatsFile
|
||||
from larv_review.owed import shortfall
|
||||
|
||||
TOOL = pathlib.Path(__file__).resolve().parents[1] / "larv-review"
|
||||
SCORING = "## 2026-10-08T12:00:00Z, PR #1 at aaaaaaa\n\n| Seat | Overall |\n|---|---|\n| Mean | 7.00 |\n"
|
||||
@@ -425,6 +426,20 @@ class LandedTowards(unittest.TestCase):
|
||||
self.assertEqual(landed_towards(plan, LastCut(Failure("fatal: bad object"), "", 3, [])), Failure("could not tell the next release: the default branch's tags cannot be read: fatal: bad object"))
|
||||
|
||||
|
||||
class Shortfall(unittest.TestCase):
|
||||
def test_skill_md_names_the_words_that_end_the_reason_a_release_empties(self):
|
||||
self.assertIn(EMPTIED_WORDS, (pathlib.Path(__file__).resolve().parents[2] / "skills" / "larv-review" / "SKILL.md").read_text(encoding="utf-8"))
|
||||
|
||||
def test_a_chunk_that_empties_a_release_owes_two_runs_and_any_other_reason_one(self):
|
||||
emptied = ["leaves release 0.1.0 with no plan item"]
|
||||
floor = ["ships item 2, which names the comprehension floor"]
|
||||
self.assertIn("owes 2 scoring runs and the branch's committed history records 1", shortfall(emptied + floor, 1).reason)
|
||||
self.assertIn("owes 1 scoring run and", shortfall(floor, 0).reason)
|
||||
self.assertIn("records 0:", shortfall(floor, -1).reason)
|
||||
self.assertIsNone(shortfall(emptied, 2))
|
||||
self.assertIsNone(shortfall(floor, 1))
|
||||
|
||||
|
||||
class LarvReviewTool(unittest.TestCase):
|
||||
def setUp(self):
|
||||
temp = tempfile.TemporaryDirectory()
|
||||
@@ -502,6 +517,128 @@ class LarvReviewTool(unittest.TestCase):
|
||||
self.assertEqual(self.run_tool("needs-scoring-run").stdout, "No scoring run due.\n")
|
||||
self.assertIn("todo.md's ID column", self.run_tool("needs-scoring-run", "item-85").stderr)
|
||||
|
||||
def test_a_scoring_run_that_a_release_ending_chunk_owes_is_refused_until_the_branch_records_two(self):
|
||||
rest = ("| 2 | 0.2.0 | | **Then this.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |", "| 3 | 0.2.0 | | **And this.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |")
|
||||
self.commit("Ship item 1", {"todo.md": todo(*rest)})
|
||||
owed = self.run_tool("scoring-run-owed")
|
||||
self.assertEqual(owed.returncode, 1)
|
||||
self.assertIn("leaves release 0.1.0 with no plan item", owed.stderr)
|
||||
self.assertIn("records 0:", owed.stderr)
|
||||
history = "# Comprehension history\n\n" + SCORING
|
||||
self.commit("Record a run", {"docs/comprehension-history.md": history})
|
||||
self.assertIn("records 1:", self.run_tool("scoring-run-owed").stderr)
|
||||
self.commit("Record a rerun", {"docs/comprehension-history.md": history + "\n" + SCORING.replace("PR #1 at", "PR #1 rerun at")})
|
||||
done = self.run_tool("scoring-run-owed")
|
||||
self.assertEqual((done.returncode, done.stdout), (0, "No scoring run owed.\n"), done.stderr)
|
||||
|
||||
def test_a_scoring_run_the_floor_calls_is_owed_once(self):
|
||||
row = "| 1 | 0.1.0 | | **Do it.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |"
|
||||
other = "| 3 | 0.2.0 | | **And this.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |"
|
||||
self.commit("Plan the floor item", {"todo.md": todo(row, "| 2 | 0.2.0 | | **Reach the comprehension floor.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |", other)})
|
||||
self.git("push", "-q", "origin", "HEAD:main")
|
||||
self.commit("Ship it", {"todo.md": todo(row, other)})
|
||||
self.assertIn("names the comprehension floor", self.run_tool("scoring-run-owed").stderr)
|
||||
self.commit("Record a run", {"docs/comprehension-history.md": "# Comprehension history\n\n" + SCORING})
|
||||
self.assertEqual(self.run_tool("scoring-run-owed").returncode, 0)
|
||||
|
||||
ROW = "| 3 | 0.2.0 | | **And this.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |"
|
||||
|
||||
def ship_item_1(self):
|
||||
self.commit("Ship item 1", {"todo.md": todo(self.ROW)})
|
||||
|
||||
def test_no_scoring_run_is_owed_where_none_is_due(self):
|
||||
self.commit("Ship item 2", {"todo.md": todo("| 1 | 0.1.0 | | **Do it.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |", self.ROW)})
|
||||
self.assertEqual(self.run_tool("scoring-run-owed").stdout, "No scoring run owed.\n")
|
||||
self.ship_item_1()
|
||||
self.assertEqual(self.run_tool("scoring-run-owed").returncode, 1)
|
||||
|
||||
def test_a_run_counts_only_as_committed_and_a_ruling_is_no_run(self):
|
||||
self.ship_item_1()
|
||||
(self.checkout / "docs").mkdir()
|
||||
(self.checkout / "docs" / "comprehension-history.md").write_text("# Comprehension history\n\n" + SCORING)
|
||||
self.assertIn("records 0", self.run_tool("scoring-run-owed").stderr)
|
||||
self.commit("Record a ruling", {"docs/comprehension-history.md": "# Comprehension history\n\n" + paired("2026-10-08T12:00:00Z", "better")})
|
||||
self.assertIn("records 0", self.run_tool("scoring-run-owed").stderr)
|
||||
|
||||
def test_a_project_that_opts_out_owes_none_and_is_never_due_one_but_a_branch_cannot_opt_itself_out(self):
|
||||
self.ship_item_1()
|
||||
self.commit("Opt out on the branch", {"docs/decisions.md": "## review opt-out: the comprehension check\n\n2026-11-10, lilleman.\n"})
|
||||
self.assertEqual(self.run_tool("scoring-run-owed").returncode, 1)
|
||||
self.git("switch", "-q", "main")
|
||||
self.commit("Opt out", {"docs/decisions.md": "## review opt-out: the comprehension check\n\n2026-11-10, lilleman.\n"})
|
||||
self.git("push", "-q", "origin", "HEAD:main")
|
||||
self.git("switch", "-q", "chunk")
|
||||
self.git("merge", "-q", "main")
|
||||
self.assertEqual(self.run_tool("scoring-run-owed").stdout, "No scoring run owed.\n")
|
||||
self.assertEqual(self.run_tool("needs-scoring-run", "1").stdout, "No scoring run due.\n")
|
||||
|
||||
def test_a_project_with_no_plan_or_no_items_table_owes_none(self):
|
||||
self.ship_item_1()
|
||||
self.assertEqual(self.run_tool("scoring-run-owed").returncode, 1)
|
||||
self.commit("Reshape the plan", {"todo.md": "# Plan\n"})
|
||||
self.assertEqual(self.run_tool("scoring-run-owed").stdout, "No scoring run owed.\n")
|
||||
self.git("rm", "-q", "todo.md")
|
||||
self.commit("Drop the plan")
|
||||
self.assertEqual(self.run_tool("scoring-run-owed").stdout, "No scoring run owed.\n")
|
||||
(self.checkout / "todo.md").write_bytes(b"\xff\xfe")
|
||||
self.git("add", "todo.md")
|
||||
self.commit("Break the plan")
|
||||
self.assertEqual(self.run_tool("scoring-run-owed").returncode, 1, "a plan that can't be read is no plan the gate may skip")
|
||||
|
||||
def test_a_default_branch_that_moved_on_since_the_branch_began_adds_no_owed_run(self):
|
||||
self.git("switch", "-q", "main")
|
||||
self.commit("Plan release 0.3.0", {"todo.md": todo("| 1 | 0.1.0 | | **Do it.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |", "| 2 | 0.2.0 | | **Then this.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |", self.ROW, "| 4 | 0.3.0 | | **Later.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |"), "docs/comprehension-history.md": "# Comprehension history\n\n" + SCORING})
|
||||
self.git("push", "-q", "origin", "HEAD:main")
|
||||
self.git("switch", "-q", "chunk")
|
||||
self.commit("Do some work")
|
||||
self.assertEqual(self.run_tool("needs-scoring-run").stdout, "No scoring run due.\n")
|
||||
self.assertEqual(self.run_tool("scoring-run-owed").stdout, "No scoring run owed.\n")
|
||||
self.ship_item_1()
|
||||
self.commit("Ship item 1 and record a run", {"docs/comprehension-history.md": "# Comprehension history\n\n" + SCORING.replace("PR #1", "PR #7")})
|
||||
self.assertIn("owes 2 scoring runs and the branch's committed history records 1:", self.run_tool("scoring-run-owed").stderr, "main's own run is no credit to the branch")
|
||||
|
||||
def release(self, tag, date, files=None):
|
||||
"""Cut tag on the default branch at a commit dated date."""
|
||||
self.env["GIT_COMMITTER_DATE"] = date
|
||||
self.addCleanup(self.env.pop, "GIT_COMMITTER_DATE", None)
|
||||
self.commit(f"Release {tag}", files)
|
||||
self.git("tag", tag)
|
||||
self.git("push", "-q", "origin", "HEAD:main")
|
||||
|
||||
def test_a_release_cut_with_no_scoring_run_since_the_one_before_it_calls_one_and_a_later_run_ends_that(self):
|
||||
self.release("v0.0.1", "2026-10-09T12:00:00Z")
|
||||
self.release("v0.0.2", "2026-10-20T12:00:00Z", {"docs/comprehension-history.md": "# Comprehension history\n\n" + SCORING})
|
||||
self.assertEqual(self.run_tool("needs-scoring-run", "2").stdout, "Scoring run due: the chunk follows v0.0.2, cut with no scoring run since v0.0.1.\n")
|
||||
self.assertEqual(self.run_tool("scoring-run-owed").returncode, 1)
|
||||
self.commit("Record a run", {"docs/comprehension-history.md": "# Comprehension history\n\n" + SCORING + "\n" + SCORING.replace("PR #1 at", "PR #2 at")})
|
||||
self.assertEqual(self.run_tool("scoring-run-owed").returncode, 0)
|
||||
self.release("v0.0.3", "2026-10-21T12:00:00Z", {"docs/comprehension-history.md": "# Comprehension history\n\n" + SCORING.replace("2026-10-08T12", "2026-10-20T18")})
|
||||
self.assertEqual(self.run_tool("needs-scoring-run", "2").stdout, "No scoring run due.\n")
|
||||
|
||||
def test_a_run_ends_an_unmeasured_release_only_when_it_is_a_scoring_run_newer_than_the_tag_with_a_time_that_reads(self):
|
||||
self.release("v0.0.1", "2026-10-09T12:00:00Z")
|
||||
history = "# Comprehension history\n\n"
|
||||
due = "Scoring run due: the chunk follows v0.0.2, cut with no scoring run since v0.0.1.\n"
|
||||
self.release("v0.0.2", "2026-10-20T12:00:00Z", {"docs/comprehension-history.md": history + paired("2026-10-21T12:00:00Z", "better") + "\n" + SCORING.replace("2026-10-08T12:00:00Z", "2026-10-09T12:00:00Z")})
|
||||
self.assertEqual(self.run_tool("needs-scoring-run", "2").stdout, due, "a ruling, and a run at the earlier tag's own second, end nothing")
|
||||
self.release("v0.0.3", "2026-10-21T12:00:00Z", {"docs/comprehension-history.md": history + SCORING.replace("2026-10-08T12:00:00Z", "2026-10-30")})
|
||||
self.assertEqual(self.run_tool("needs-scoring-run", "2").stdout, due.replace("v0.0.2", "v0.0.3").replace("v0.0.1", "v0.0.2"), "a run whose time doesn't parse ends nothing")
|
||||
|
||||
def test_an_unmeasured_release_is_judged_by_the_default_branch_alone_and_names_a_release_over_its_candidate(self):
|
||||
self.release("v0.0.1", "2026-10-09T12:00:00Z")
|
||||
self.release("v0.0.2-rc.1", "2026-10-10T12:00:00Z")
|
||||
self.release("v0.0.2", "2026-10-20T12:00:00Z")
|
||||
self.git("switch", "-q", "-c", "side")
|
||||
self.commit("Tag a side branch")
|
||||
self.git("tag", "v0.9.0")
|
||||
self.git("switch", "-q", "chunk")
|
||||
self.commit("Record a run on the branch", {"docs/comprehension-history.md": "# Comprehension history\n\n" + SCORING.replace("2026-10-08T12", "2026-10-25T12")})
|
||||
self.assertEqual(self.run_tool("needs-scoring-run", "2").stdout, "Scoring run due: the chunk follows v0.0.2, cut with no scoring run since v0.0.2-rc.1.\n")
|
||||
|
||||
def test_one_release_alone_calls_no_scoring_run_for_lacking_one(self):
|
||||
self.release("v0.0.1", "2026-10-09T12:00:00Z")
|
||||
self.assertEqual(self.run_tool("needs-scoring-run", "2").stdout, "No scoring run due.\n")
|
||||
|
||||
def test_calls_a_scoring_run_for_a_deferred_release_and_none_for_one_the_checkouts_plan_marks_renamed(self):
|
||||
rows = ("| 1 | 0.2.0 | | **Do it.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |", "| 2 | 0.2.0 | | **Then this.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |", "| 3 | 0.2.0 | | **And this.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |")
|
||||
self.commit("Move 0.1.0's item to 0.2.0", {"todo.md": todo(*rows)})
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
import datetime
|
||||
|
||||
from review_rules.comprehension_history import entries, HISTORY
|
||||
from failure import Failure
|
||||
from git_cli.git_repo import file_at, git
|
||||
|
||||
|
||||
def unmeasured_cut(default: str, checkout: str) -> str | None | Failure:
|
||||
"""Whether the default branch's newest release was cut unmeasured, as `v0.7.0, cut with no scoring run since v0.6.0`. It was where no scoring run in the default branch's history is newer than the second-newest v* tag. None where the branch holds fewer than two such tags, or a run since the earlier one.
|
||||
|
||||
The reason ends once a run follows the earlier tag, so it asks only after a release was cut with none; it lets the chunk that finds the gap record the run.
|
||||
|
||||
A scoring run counts by the time its history heading records, against the tag's commit time; a heading whose time doesn't parse counts for nothing, so the run is asked for again.
|
||||
"""
|
||||
# Sorts v1.0.0-rc.1 below v1.0.0, as release.newest_tag does.
|
||||
tags = git("-c", "versionsort.suffix=-", "tag", "--list", "v*", "--merged", default, "--sort=-v:refname", cwd=checkout)
|
||||
if isinstance(tags, Failure):
|
||||
return tags
|
||||
newest, earlier, *_ = [*tags.split(), None, None]
|
||||
if earlier is None:
|
||||
return None
|
||||
since = git("log", "-1", "--format=%ct", earlier, cwd=checkout)
|
||||
history = file_at(default, HISTORY, checkout)
|
||||
if isinstance(since, Failure) or isinstance(history, Failure):
|
||||
return since if isinstance(since, Failure) else history
|
||||
if any(at > int(since) for at in (seconds(entry.time) for entry in entries(history) if not entry.paired)):
|
||||
return None
|
||||
return f"{newest}, cut with no scoring run since {earlier}"
|
||||
|
||||
|
||||
def seconds(heading_time: str) -> float:
|
||||
"""A history heading's UTC time as seconds since the epoch; 0 where it doesn't parse."""
|
||||
try:
|
||||
return datetime.datetime.strptime(heading_time, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=datetime.timezone.utc).timestamp()
|
||||
except ValueError:
|
||||
return 0
|
||||
@@ -0,0 +1,35 @@
|
||||
"""The todo.md items a chunk takes: those the worker names on its latest `Chunk items:` line, scored as todo.md scores them (docs/decisions.md → `routing: loop-claude reads the items a chunk takes`)."""
|
||||
import re
|
||||
|
||||
from claude_code.entries import AssistantTurn, Entry, Text
|
||||
from failure import Failure
|
||||
from release_plan.todo_table import items
|
||||
from claude_code.transcript import plain
|
||||
from loop_claude.verdicts import TakenIn
|
||||
|
||||
# The label opening the line; prompts.NAME_THE_ITEMS asks the worker for it.
|
||||
CHUNK_ITEMS = "Chunk items:"
|
||||
# A line the worker opens with the label, after any `-` or `>` marker or a backtick; what follows names the items.
|
||||
LINE = re.compile(rf"[\s>`-]*{re.escape(CHUNK_ITEMS)}(.*)")
|
||||
# What the worker says of an item in parentheses, such as its L and E + I.
|
||||
SAID_OF_AN_ITEM = re.compile(r"\([^)]*\)")
|
||||
# An entry of the line, the text between commas or "and": an item's ID, after an optional "item" or "#".
|
||||
ITEM_ID = re.compile(r"\s*(?:item\s+)?#?(\d+)", re.IGNORECASE)
|
||||
|
||||
|
||||
def taken_in(entries: list[Entry], todo: str | Failure) -> TakenIn:
|
||||
"""The items named, with the highest L and the highest E + I that todo.md, as read when the cycle started, gives any of them. An item todo.md doesn't hold, or a todo.md that can't be read, adds no level, so the route holds."""
|
||||
ids = named(entries)
|
||||
found = items(todo) if isinstance(todo, str) else todo
|
||||
scored = [item for item in found if item.id in ids] if isinstance(found, list) else []
|
||||
return TakenIn(ids, max((item.likelihood for item in scored), default=0), max((item.escape_and_impact for item in scored), default=0))
|
||||
|
||||
|
||||
def named(entries: list[Entry]) -> tuple[int, ...]:
|
||||
"""The IDs on the worker's latest Chunk items: line, each once, in its order; empty where it wrote none. Only the worker's own text counts: a loop-claude prompt quotes the line's example, and a tool call may write the label into a file."""
|
||||
lines = [line for entry in entries if isinstance(entry, AssistantTurn) for part in entry.parts if isinstance(part, Text) for line in part.text.splitlines()]
|
||||
for line in reversed(lines):
|
||||
if found := LINE.match(plain(line)):
|
||||
each = re.split(r",|\band\b", SAID_OF_AN_ITEM.sub("", found[1]))
|
||||
return tuple(dict.fromkeys(int(item[1]) for named_item in each if (item := ITEM_ID.match(named_item))))
|
||||
return ()
|
||||
@@ -9,9 +9,7 @@ from loop_effects.effects import Effects
|
||||
from claude_code.entries import Entry, structured_output
|
||||
from loop_claude import escalation_mark
|
||||
from failure import Failure
|
||||
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, TakenIn, Unfinished, Verdict
|
||||
from release_plan.todo_table import LEVELS
|
||||
from model_routing.route import on_its_scale
|
||||
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict
|
||||
from claude_code.transcript import plain, verbatim
|
||||
from loop_claude.cycle.retry import retried
|
||||
from model_routing.ladder import Rung
|
||||
@@ -69,12 +67,6 @@ JUDGE_RULES = "\n\n".join((INTERRUPT_RULES, QUOTE_RULES, FINISHED_RULES))
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Reply:
|
||||
answer: str = dataclasses.field(metadata={"description": "With interrupt false, the passage of README.md or the records that answers the worker's question, quoted verbatim, without elisions. Empty otherwise."})
|
||||
# Routes the next cycle's worker. Counting the item the chunk was routed on is harmless: loop-claude never steps the worker down.
|
||||
highest_likelihood_taken_in: int = dataclasses.field(metadata={"description": "The highest L the worker states for a todo.md item it took into this chunk; 0 when it states none."})
|
||||
# Routes the next cycle's judge and reviewers, as highest_likelihood_taken_in routes the worker.
|
||||
highest_escape_and_impact_taken_in: int = dataclasses.field(metadata={"description": "The highest E + I the worker states for a todo.md item it took into this chunk; 0 when it states none."})
|
||||
# Lists the chunk's items in loop-inbox: running_loops.py.
|
||||
items_taken_in: list[int] = dataclasses.field(metadata={"description": "The IDs of the todo.md items the worker states this chunk takes, the item the chunk was routed on included, never an item it only filed in todo.md for later; empty when it states none."})
|
||||
# Routes the worker one rung up, as the judge interrupts or quotes an answer, or on the next cycle: loop_claude/escalation_mark.py.
|
||||
escalation: str = dataclasses.field(metadata={"description": "Empty unless the chunk is unfinished, interrupt is true or answer quotes a passage, and the worker's run ends asking loop-claude for a higher rung, citing an escalation trigger. Then: a verbatim quote, without elisions, of the transcript passage where the trigger fired, such as a review round's commit title or the worker's report of the finding or failing test. " + escalation_mark.QUOTE_AFTER_STEP + " Never a later mention of the trigger, and never the request itself."})
|
||||
evidence: str = dataclasses.field(metadata={"description": "A verbatim quote of the one transcript passage the verdict rests on, without elisions: what triggers the interrupt; for a quoted answer, the worker's question; else what shows the chunk finished or the work remaining."})
|
||||
@@ -88,8 +80,7 @@ class Reply:
|
||||
|
||||
|
||||
REPLY_TYPES = typing.get_type_hints(Reply)
|
||||
JSON_TYPES: dict[object, dict[str, typing.Any]] = {bool: {"type": "boolean"}, int: {"type": "integer"}, str: {"type": "string"}, list[int]: {"type": "array", "items": {"type": "integer"}}}
|
||||
HIGHEST_L = LEVELS["L"]
|
||||
JSON_TYPES: dict[object, dict[str, typing.Any]] = {bool: {"type": "boolean"}, int: {"type": "integer"}, str: {"type": "string"}}
|
||||
JUDGE_SCHEMA = {
|
||||
"properties": {field.name: {**field.metadata, **JSON_TYPES[REPLY_TYPES[field.name]]} for field in dataclasses.fields(Reply)},
|
||||
"required": list(REPLY_TYPES),
|
||||
@@ -128,43 +119,25 @@ def verdict_from(output: object) -> Verdict | Failure:
|
||||
return Failure("the judge returned no reply")
|
||||
for name, kind in REPLY_TYPES.items():
|
||||
if not fits(output.get(name), kind):
|
||||
return Failure(f"the judge's reply has no {name} of type {type_name(JSON_TYPES[kind])}")
|
||||
taken_in = taken_in_from(output)
|
||||
return Failure(f"the judge's reply has no {name} of type {JSON_TYPES[kind]['type']}")
|
||||
if output["interrupt"]:
|
||||
if not output["question"].strip():
|
||||
return Failure("the judge interrupted without a question")
|
||||
return Interrupt(output["evidence"], output["question"], escalation=output["escalation"], taken_in=taken_in, release_ready=output["release_ready"], answered_before=output["answered_before"], left_open=output["left_open"])
|
||||
return Interrupt(output["evidence"], output["question"], escalation=output["escalation"], release_ready=output["release_ready"], answered_before=output["answered_before"], left_open=output["left_open"])
|
||||
if output["answer"].strip():
|
||||
if not output["question"].strip():
|
||||
return Failure("the judge quoted an answer without a question")
|
||||
return QuotedAnswer(output["evidence"], output["question"], output["answer"], escalation=output["escalation"], taken_in=taken_in)
|
||||
return QuotedAnswer(output["evidence"], output["question"], output["answer"], escalation=output["escalation"])
|
||||
if output["finished"]:
|
||||
return Finished(output["evidence"])
|
||||
return Unfinished(output["evidence"], escalation=output["escalation"], taken_in=taken_in)
|
||||
|
||||
|
||||
def type_name(schema: dict[str, typing.Any]) -> str:
|
||||
return f"array of {schema['items']['type']}s" if schema["type"] == "array" else schema["type"]
|
||||
return Unfinished(output["evidence"], escalation=output["escalation"])
|
||||
|
||||
|
||||
def fits(value: object, kind: object) -> bool:
|
||||
"""Whether value, read from the judge's JSON, is of kind. type(), not isinstance(): JSON true is no integer."""
|
||||
if typing.get_origin(kind) is list:
|
||||
return type(value) is list and all(fits(each, typing.get_args(kind)[0]) for each in value)
|
||||
return type(value) is kind
|
||||
|
||||
|
||||
def taken_in_from(output: dict) -> TakenIn:
|
||||
"""The reply's readings of the items taken in. An ID below 1 is left out, and an L or E + I off its scale counts as 0. Neither fails the reply: three failures would stop an unattended loop, and holding the rung is safe."""
|
||||
likelihood = output["highest_likelihood_taken_in"]
|
||||
escape_and_impact = output["highest_escape_and_impact_taken_in"]
|
||||
return TakenIn(
|
||||
tuple(item for item in output["items_taken_in"] if item > 0),
|
||||
likelihood if 0 <= likelihood <= HIGHEST_L else 0,
|
||||
escape_and_impact if on_its_scale(escape_and_impact) else 0,
|
||||
)
|
||||
|
||||
|
||||
def judge(root: str, brief: Brief, rung: Rung, effects: Effects, keep: typing.Callable[[Rejected], Stop | None]) -> Verdict | Stop:
|
||||
return retried("judge", lambda: judge_once(root, brief, rung, effects), effects, lambda failure: keep(failure) if isinstance(failure, Rejected) else None)
|
||||
|
||||
@@ -211,13 +184,13 @@ def cost_of(reply: dict) -> float:
|
||||
def quotes_checked(verdict: Verdict, grounds: Grounds) -> Verdict | Failure:
|
||||
"""The verdict with each of its quotes checked against the text it must appear in, as written:
|
||||
|
||||
| Quote | Must be in | Where it isn't |
|
||||
|---|---|---|
|
||||
| the evidence | grounds.shown, the evidence's fragments in order | a Failure: run_judge rejects the reply, and judge retries it |
|
||||
| an interrupt's answered_before | grounds.shown | the interrupt stands, as if no earlier answer was found |
|
||||
| its left_open | answered_before and the question | the interrupt becomes an Unfinished that quotes answered_before back to the worker |
|
||||
| a quoted answer | grounds.quotable | the question goes to the human as an interrupt, which keeps the escalation the answer carried |
|
||||
| the escalation | grounds.shown and grounds.after_mark | the escalation is dropped, so the worker's rung holds, as escalation_mark.checked says |
|
||||
| Quote | Must be in | Where it isn't | Checked by |
|
||||
|---|---|---|---|
|
||||
| the evidence | grounds.shown, the evidence's fragments in order | a Failure: run_judge rejects the reply, and judge retries it | grounded |
|
||||
| an interrupt's answered_before | grounds.shown | the interrupt stands, as if no earlier answer was found | re_ask_checked |
|
||||
| its left_open | answered_before and the question | the interrupt becomes an Unfinished that quotes answered_before back to the worker | re_ask_checked |
|
||||
| a quoted answer | grounds.quotable | the question goes to the human as an interrupt, which keeps the escalation the answer carried | answer_checked |
|
||||
| the escalation | grounds.shown and grounds.after_mark | the escalation is dropped, so the worker's rung holds | escalation_mark.checked |
|
||||
|
||||
Only a misquoted evidence fails. A Failure for any other would exit an unattended loop, so each changes the verdict instead.
|
||||
"""
|
||||
@@ -228,13 +201,18 @@ def quotes_checked(verdict: Verdict, grounds: Grounds) -> Verdict | Failure:
|
||||
return escalation_mark.checked(verdict, after=grounds.after_mark, shown=grounds.shown)
|
||||
|
||||
|
||||
def re_ask_checked(verdict: Verdict, shown: str) -> Verdict:
|
||||
"""The verdict, or an Unfinished quoting the human's earlier answer back to the worker, where it is an Interrupt that re-asks what that answer settles: its left_open isn't in both the answer and the question."""
|
||||
if not isinstance(verdict, Interrupt) or not verbatim(verdict.answered_before, shown):
|
||||
return verdict
|
||||
if verbatim(verdict.left_open, verdict.answered_before) and verbatim(verdict.left_open, verdict.question):
|
||||
return verdict
|
||||
return Unfinished(verdict.answered_before, escalation=verdict.escalation, taken_in=verdict.taken_in)
|
||||
def grounded(evidence: str, rendered_transcript: str) -> bool:
|
||||
"""Whether the transcript holds the evidence word for word, once emphasis marks are stripped and whitespace is reflowed in both. The judge joins passages with an ellipsis though told to quote one, so each fragment between ellipses must come after the one before it."""
|
||||
transcript = plain(rendered_transcript)
|
||||
fragments = [fragment.strip() for fragment in re.split(r"\.\.\.|…", plain(evidence)) if fragment.strip()]
|
||||
after = 0
|
||||
for fragment in fragments:
|
||||
at = transcript.find(fragment, after)
|
||||
if at < 0:
|
||||
return False
|
||||
after = at + len(fragment)
|
||||
return bool(fragments)
|
||||
|
||||
|
||||
|
||||
def misquoted_evidence(verdict: Verdict) -> Failure:
|
||||
@@ -242,10 +220,19 @@ def misquoted_evidence(verdict: Verdict) -> Failure:
|
||||
return Failure(f"the judge quoted {verdict.evidence!r}, which is not in the transcript{asked}")
|
||||
|
||||
|
||||
def re_ask_checked(verdict: Verdict, shown: str) -> Verdict:
|
||||
"""The verdict, or an Unfinished quoting the human's earlier answer back to the worker, where it is an Interrupt that re-asks what that answer settles: its left_open isn't in both the answer and the question."""
|
||||
if not isinstance(verdict, Interrupt) or not verbatim(verdict.answered_before, shown):
|
||||
return verdict
|
||||
if verbatim(verdict.left_open, verdict.answered_before) and verbatim(verdict.left_open, verdict.question):
|
||||
return verdict
|
||||
return Unfinished(verdict.answered_before, escalation=verdict.escalation)
|
||||
|
||||
|
||||
def answer_checked(verdict: Verdict, quotable: str) -> Verdict:
|
||||
"""The verdict; where it is a quoted answer that README.md and the records don't hold, an Interrupt on the same question instead."""
|
||||
if isinstance(verdict, QuotedAnswer) and not verbatim(verdict.answer, quotable):
|
||||
return Interrupt(verdict.evidence, verdict.question, escalation=verdict.escalation, taken_in=verdict.taken_in)
|
||||
return Interrupt(verdict.evidence, verdict.question, escalation=verdict.escalation)
|
||||
return verdict
|
||||
|
||||
|
||||
@@ -266,17 +253,4 @@ def judge_prompt(brief: Brief) -> tuple[str, Grounds]:
|
||||
after_mark=escalation_mark.after_mark(brief.entries, brief.escalated_at),
|
||||
quotable=f"{brief.premises}\n\n{brief.records}",
|
||||
)
|
||||
return f"{head}{shown}\n\n{JUDGE_RULES}", grounds
|
||||
|
||||
|
||||
def grounded(evidence: str, rendered_transcript: str) -> bool:
|
||||
"""Whether the transcript holds the evidence word for word, once emphasis marks are stripped and whitespace is reflowed in both. The judge joins passages with an ellipsis though told to quote one, so each fragment between ellipses must come after the one before it."""
|
||||
transcript = plain(rendered_transcript)
|
||||
fragments = [fragment.strip() for fragment in re.split(r"\.\.\.|…", plain(evidence)) if fragment.strip()]
|
||||
after = 0
|
||||
for fragment in fragments:
|
||||
at = transcript.find(fragment, after)
|
||||
if at < 0:
|
||||
return False
|
||||
after = at + len(fragment)
|
||||
return bool(fragments)
|
||||
return f"{head}{shown}\n\n{JUDGE_RULES}", grounds
|
||||
@@ -1,5 +1,6 @@
|
||||
"""What loop-claude says to the worker each cycle. model_routing.review_choice chooses where the reviews run. review_launches and reruns put that choice into sentences, and escalation says where escalated subagents run; rung_sentences returns the sentences of all three. In a chunk's first prompt, routed_by places them after its routing sentences where an item routed the chunk, and fresh places them alone otherwise. rungs_held repeats them at the end of a prompt after a step-up or an unfinished verdict."""
|
||||
from model_routing.review_choice import chosen, ReviewChoice
|
||||
from loop_claude.cycle.chunk_items import CHUNK_ITEMS
|
||||
from model_routing.ladder import climbed, LADDER, position, Rung
|
||||
from model_routing.route import ESCALATION_MARKER, Route
|
||||
from loop_claude.routing import escalation_rungs
|
||||
@@ -21,6 +22,8 @@ THE_SECURITY_PASS = "the security pass's subagent"
|
||||
|
||||
# Workers only, not CLAUDE.md, which reviewers load too: a reviewer under a like rule missed findings.
|
||||
READING_RULE = "Read in small pieces: never read more than 80 lines of a file at once, nor take a tool output over 3,000 characters into your context."
|
||||
# chunk_items reads the latest such line; three fresh readers picked the label (docs/decisions.md → `routing: loop-claude reads the items a chunk takes`).
|
||||
NAME_THE_ITEMS = f"Name every todo.md item this chunk takes on one line, as in `{CHUNK_ITEMS} 297 (L 3, E + I 5), 298 (L 3, E + I 7)`, and write the whole line again whenever the set changes."
|
||||
|
||||
|
||||
AFTER_CODE_HEALTH = 'which rank after code health (~/.claude/CLAUDE.md → "Shared goal") unless README.md voids it'
|
||||
@@ -30,13 +33,12 @@ def fresh(task: str, premises: str, route: Route, project: Rung | None, facts: s
|
||||
"""A chunk's first prompt; facts is preflight_facts' or preflight_failed's text."""
|
||||
routed = f"{routed_by(route, project)} {recount(route)}" if route.item else " ".join(rung_sentences(route, project))
|
||||
opening = f"{routed}\n\n" if routed else ""
|
||||
return f"{task}\n\nContinue any open draft PR or worktree for the chunk before creating a new one. {READING_RULE}\n\n{opening}{facts}\n\nThe chunk's todo.md items, or else the task above, set its scope. The goals below, {AFTER_CODE_HEALTH}, choose between ways to deliver that scope, and never widen it.\n\nFrom README.md:\n{premises}"
|
||||
return f"{task}\n\nContinue any open draft PR or worktree for the chunk before creating a new one. {READING_RULE} {NAME_THE_ITEMS}\n\n{opening}{facts}\n\nThe chunk's todo.md items, or else the task above, set its scope. The goals below, {AFTER_CODE_HEALTH}, choose between ways to deliver that scope, and never widen it.\n\nFrom README.md:\n{premises}"
|
||||
|
||||
|
||||
def routed_by(route: Route, project: Rung | None) -> str:
|
||||
return " ".join((
|
||||
f"loop-claude runs you on {route.rung}, routed by item {route.item}'s L of {route.likelihood}.",
|
||||
f"Name each todo.md item this chunk takes, with its L and E + I, whether or not it is item {route.item}.",
|
||||
*rung_sentences(route, project),
|
||||
))
|
||||
|
||||
@@ -97,7 +99,7 @@ def shared_launches(choice: ReviewChoice, route: Route, project: Rung) -> list[s
|
||||
|
||||
|
||||
def taken_in_launches(choice: ReviewChoice) -> list[str]:
|
||||
"""The sentence naming the reviewers' rung each higher E + I picks, so the worker reviews an item it takes in on that rung in the same run, since the judge reports a taken-in item's E + I only as the cycle ends, and loop-claude routes on it from the next cycle. None where no higher E + I picks a higher rung.
|
||||
"""The sentence naming the reviewers' rung each higher E + I picks, so the worker reviews an item it takes in on that rung in the same run, since loop-claude reads a taken-in item's E + I only as the cycle ends, and routes on it from the next cycle. None where no higher E + I picks a higher rung.
|
||||
|
||||
For a worker on Opus 5.5 at medium, on an Opus 5.5 project, routed by an item of E + I 6, it reads: `Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8.`
|
||||
"""
|
||||
@@ -160,7 +162,7 @@ def resumed(unfinished_evidence: str, route: Route, project: Rung | None) -> str
|
||||
"""Names the worker's rung, so it learns whether its request for a higher rung went through, and the project's again, since a step-up between cycles may have taken the worker above it."""
|
||||
on = f" loop-claude runs you on {route.rung}." if project else ""
|
||||
ruled = f" A judge ruled the chunk unfinished, quoting this passage of your transcript as the work remaining:\n\n{quoted(unfinished_evidence)}\n\n" if unfinished_evidence else " "
|
||||
return f"Continue the task; the previous run ended before the chunk was finished.{ruled}The chunk is finished once it is merged, or nothing remains that you could do on your own. Even then, it is finished only once {ANSWERS_RECORDED}.{on}{rungs_held(route, project)}"
|
||||
return f"Continue the task; the previous run ended before the chunk was finished.{ruled}The chunk is finished once it is merged, or nothing remains that you could do on your own. Even then, it is finished only once {ANSWERS_RECORDED}. {NAME_THE_ITEMS}{on}{rungs_held(route, project)}"
|
||||
|
||||
|
||||
def rung_sentences(route: Route, project: Rung | None) -> list[str]:
|
||||
|
||||
@@ -6,6 +6,7 @@ from loop_claude.cycle.states import Active, Archived, Closing, Ended, Event, Hu
|
||||
from transcript_archive.write import archive
|
||||
from loop_claude.cycle.worker import AUTO_COMPACT_WINDOW, COMPACT_WINDOW, run_worker, worker_args
|
||||
from loop_claude.cycle.judge import Brief, judge, Rejected
|
||||
from loop_claude.cycle import chunk_items
|
||||
from claude_code.session import delete_session, Session, session_open, transcript_file
|
||||
from loop_effects.effects import Effects, say
|
||||
from failure import Failure
|
||||
@@ -42,9 +43,11 @@ class Cycle:
|
||||
root: str
|
||||
route: Route
|
||||
session: Session
|
||||
# Lists in loop-inbox the items the chunk takes, given the IDs an unfinished ruling found taken in.
|
||||
# Lists in loop-inbox the items the chunk takes, given the IDs on the worker's latest `Chunk items:` line.
|
||||
show_taken_in: typing.Callable[[tuple[int, ...]], None]
|
||||
task: str
|
||||
# todo.md on the default branch as the cycle starts, which scores the items the worker names.
|
||||
todo: str | Failure
|
||||
# Empty when the previous cycle's verdict was not Unfinished.
|
||||
unfinished_evidence: str
|
||||
|
||||
@@ -144,13 +147,15 @@ def worker_start(cycle: Cycle, answered: QuotedAnswer | TypedAnswer | None, step
|
||||
|
||||
|
||||
def rule(cycle: Cycle, transcript_path: str, quoted_answers: int) -> Ruled | Stop:
|
||||
"""The judge's ruling. An interrupt or a quoted answer is archived as it is ruled, on the route the worker ran on until then, and then any step-up it makes, so the figures charge each stretch of work to its own rung."""
|
||||
"""The judge's ruling. Unless it is Finished, code stamps it with the items the worker names on its latest `Chunk items:` line. An interrupt or a quoted answer is archived as it is ruled, on the route the worker ran on until then, and then any step-up it makes, so the figures charge each stretch of work to its own rung."""
|
||||
entries = read_transcript(transcript_path)
|
||||
if isinstance(entries, Failure):
|
||||
return Stop(1, entries.reason)
|
||||
verdict = judge(cycle.root, Brief(cycle.task, cycle.premises, cycle.records, entries, cycle.route.escalated_at), judge_of(cycle), cycle.effects, lambda rejected: keep_rejection(cycle, transcript_path, rejected))
|
||||
if isinstance(verdict, Stop):
|
||||
return verdict
|
||||
if not isinstance(verdict, Finished):
|
||||
verdict = dataclasses.replace(verdict, taken_in=chunk_items.taken_in(entries, cycle.todo))
|
||||
if not isinstance(verdict, Interrupt | QuotedAnswer):
|
||||
return Ruled(verdict)
|
||||
stepped = ruling_step_up(cycle, verdict)
|
||||
|
||||
@@ -7,7 +7,7 @@ from git_cli.git_repo import default_ref, file_at
|
||||
from loop_effects.effects import Effects, say
|
||||
from failure import Failure
|
||||
from loop_claude.routing import first_route, kept, routed_item
|
||||
from release_plan.todo_table import Item
|
||||
from release_plan.todo_table import Item, items
|
||||
from transcript_archive.read import latest_worker
|
||||
from loop_claude.loop.this_loop import Loop
|
||||
from loop_claude.cycle import prompts
|
||||
@@ -20,16 +20,18 @@ from loop_state.running_loops import taken_items
|
||||
ORIGIN_TIMEOUT = 60
|
||||
|
||||
|
||||
def shown_items(loop: Loop, routed: int) -> typing.Callable[[tuple[int, ...]], None]:
|
||||
"""A function that lists in loop-inbox the items a chunk takes, given the IDs the judge found taken in. routed is the item the chunk was routed on, 0 for none. shown_items reads todo.md once, as the cycle starts, and the function shows those taglines."""
|
||||
todo = fetched_todo(loop)
|
||||
def shown_items(loop: Loop, todo: str | Failure, routed: int) -> typing.Callable[[tuple[int, ...]], None]:
|
||||
"""A function that lists in loop-inbox the items a chunk takes, given the IDs the worker names, with their taglines in todo, as fetched_todo read it as the cycle started. routed is the item the chunk was routed on, 0 for none."""
|
||||
return lambda taken_in: loop.listed.show(taken_items(todo, routed, taken_in))
|
||||
|
||||
|
||||
def fetched_todo(loop: Loop) -> str | Failure:
|
||||
"""todo.md on the default branch, as last fetched. fetched_plan reads it too, but also the release tag, CHANGELOG.md and each todo.md since the tag, which the listing doesn't need."""
|
||||
"""todo.md on the default branch, as last fetched, which lists and scores the items a chunk takes. fetched_plan reads it too, but also the release tag, CHANGELOG.md and each todo.md since the tag, which listing and scoring the items don't need. Where it can't be read or scored, it says why."""
|
||||
ref = default_ref(loop.root, ORIGIN_TIMEOUT, run=loop.effects.git)
|
||||
return ref if isinstance(ref, Failure) else file_at(ref, "todo.md", loop.root, loop.effects.git)
|
||||
todo = ref if isinstance(ref, Failure) else file_at(ref, "todo.md", loop.root, loop.effects.git)
|
||||
if isinstance(scored := items(todo) if isinstance(todo, str) else todo, Failure):
|
||||
say(f"the items the worker names lift no rung: {scored.reason}", loop.effects)
|
||||
return todo
|
||||
|
||||
|
||||
def routed(route: Route | None, project: Rung | None, model: str, archived: typing.Callable[[], Route | None], item: typing.Callable[[], Item | None]) -> Route:
|
||||
|
||||
@@ -39,7 +39,7 @@ def reached_limit(loop_run: LoopRun) -> Stop | None:
|
||||
|
||||
|
||||
def after_verdict(loop_run: LoopRun, project: Rung | None, concluded: Concluded) -> LoopRun:
|
||||
"""The loop run the next cycle starts from. A finished chunk drops its route, its run of unfinished verdicts and the items it took in. An unfinished one takes the route routing.next_route makes of the route its cycle ended on, which an interrupt's or a quoted answer's escalation may have raised; the evidence the judge gave for the verdict and the items it found taken in; and counts the verdict."""
|
||||
"""The loop run the next cycle starts from. A finished chunk drops its route, its run of unfinished verdicts and the items it took in. An unfinished one takes the route routing.next_route makes of the route its cycle ended on, which an interrupt's or a quoted answer's escalation may have raised; the evidence the judge gave for the verdict, and the items the worker names on its latest `Chunk items:` line; and counts the verdict."""
|
||||
verdict = concluded.verdict
|
||||
if isinstance(verdict, Finished):
|
||||
return dataclasses.replace(loop_run, route=None, unfinished_evidence="", unfinished_run=0, items_taken_in=())
|
||||
|
||||
@@ -3,7 +3,7 @@ import pathlib
|
||||
import typing
|
||||
|
||||
from transcript_archive.location import archive_root, project_archive
|
||||
from loop_claude.loop.chunk_start import archived_route, chunk_start, routed, shown_items
|
||||
from loop_claude.loop.chunk_start import archived_route, chunk_start, fetched_todo, routed, shown_items
|
||||
from loop_claude.loop.routing_check import check_routing
|
||||
from loop_claude.project.checkout import checkout_root
|
||||
from loop_claude.loop.close import closed_cycle
|
||||
@@ -80,9 +80,10 @@ def next_cycle(loop_run: LoopRun, loop: Loop) -> Cycle | Stop:
|
||||
project = project_rung(models.worker)
|
||||
route = routed(loop_run.route, project, models.worker, lambda: archived_route(archive_folder, loop), item)
|
||||
say(f"\n=== cycle {loop_run.cycle_number}, worker on {route.rung} ===", loop.effects, stdout=True)
|
||||
show_taken_in = shown_items(loop, route.item)
|
||||
todo = fetched_todo(loop)
|
||||
show_taken_in = shown_items(loop, todo, route.item)
|
||||
show_taken_in(loop_run.items_taken_in)
|
||||
return Cycle(effects=loop.effects, facts=facts, judge_model=models.judge, premises=premises, project=project, project_archive=archive_folder, questions=loop.listed.folder, records=records, root=loop.root, route=route, session=loop.session, show_taken_in=show_taken_in, task=loop_run.task, unfinished_evidence=loop_run.unfinished_evidence)
|
||||
return Cycle(effects=loop.effects, facts=facts, judge_model=models.judge, premises=premises, project=project, project_archive=archive_folder, questions=loop.listed.folder, records=records, root=loop.root, route=route, session=loop.session, show_taken_in=show_taken_in, task=loop_run.task, todo=todo, unfinished_evidence=loop_run.unfinished_evidence)
|
||||
|
||||
|
||||
def loop_run_from(argv: list[str], cwd: pathlib.Path, effects: Effects) -> LoopRun | Stop:
|
||||
|
||||
@@ -63,7 +63,7 @@ def raise_calls(root: pathlib.Path, new: list[Call], rung: Rung, loop: Loop) ->
|
||||
unstarted = hand_over(ClaudeCommand(claude_args(rung, [prompt]), str(loop.paths.config_clone), {}), loop.effects)
|
||||
if unstarted:
|
||||
return Stop(1, f"could not start claude for the routing calls in {loop.paths.config_clone}, the setup's development clone: {unstarted.reason}")
|
||||
# Raised once the human has seen them, answered or not: an unanswered call reaches the plan through AGENTS.md, so it is not asked twice.
|
||||
# Raised once the human has seen them, answered or not: an unanswered call reaches the plan through docs/decisions.md → "Working rules", so it is not asked twice.
|
||||
if unmarked := state.mark_raised(root, new, utc(loop.effects.now())):
|
||||
say(f"routing figures: {unmarked.reason}", loop.effects)
|
||||
return None
|
||||
|
||||
@@ -67,9 +67,10 @@ the worker and the session /talk opens; with both set, nothing is asked.
|
||||
Where the project's model is on the routing ladder (bin/model_routing/ladder.py), the worker and the
|
||||
session /talk opens run on the rung that the chunk's todo.md item picks, and the judge of some
|
||||
models on another; a chunk no item routes runs on the project's model. The worker's prompts name the
|
||||
rungs it launches its reviews on. The worker's rung steps up as a chunk stays unfinished, or when
|
||||
the judge finds in the transcript an escalation trigger that fired and the worker asked for a higher
|
||||
rung. The rules are in bin/loop_claude/routing.py and bin/model_routing/ladder.py, the choice of
|
||||
rungs it launches its reviews on. The worker's rung steps up as a chunk stays unfinished, when the
|
||||
worker names on its `Chunk items:` line an item that picks a higher rung, or when the judge finds
|
||||
in the transcript an escalation trigger that fired and the worker asked for a higher rung. The
|
||||
rules are in bin/loop_claude/routing.py and bin/model_routing/ladder.py, the choice of
|
||||
each review's rung in bin/model_routing/review_choice.py, and the prompts' wording in
|
||||
bin/loop_claude/cycle/prompts.py. Each verdict, interrupt, quoted answer, stop and
|
||||
step-up in the archive ends with a Worker: line, the route as JSON (Route.worker_line in
|
||||
|
||||
@@ -137,6 +137,9 @@ PREMISES = "# Project\n\n## Goals\n\n1. Work.\n\n## Audience\n\nMe.\n"
|
||||
TODO = "## Items\n\n| ID | Release | Exempt | Item | L | E | I | R | S | Moves | Score |\n|---|---|---|---|---|---|---|---|---|---|---|\n| 4 | 0.1.0 | | **Do it.** | 4 | 2 | 3 | 7 | 2 | 1 by 9 | 25 |\n"
|
||||
# Item 4's L of 2 steps one rung down.
|
||||
CHEAP = TODO.replace("| 4 | 2 | 3 | 7 |", "| 2 | 1 | 2 | 3 |")
|
||||
# Item 9 follows CHEAP's item 4, and a worker routed by item 4 may take it in: at L 4 it picks a higher rung for the worker, and at E + I 6 for the judge and the reviewers. At E + I 3, or at L 2, it picks the rung item 4 picks.
|
||||
TAKEN_IN_AT_L_4 = CHEAP + "| 9 | 0.1.0 | | **Do more.** | 4 | 1 | 2 | 5 | 2 | 1 by 9 | 20 |\n"
|
||||
TAKEN_IN_AT_E_PLUS_I_6 = CHEAP + "| 9 | 0.1.0 | | **Do more.** | 2 | 3 | 3 | 6 | 2 | 1 by 9 | 20 |\n"
|
||||
|
||||
|
||||
# Waits for the interrupt's prompt before typing the answer, since loop-claude drops what was typed before it.
|
||||
@@ -147,8 +150,8 @@ def worker(**step):
|
||||
return {"role": "worker", **step}
|
||||
|
||||
|
||||
def judge(interrupt=False, finished=False, evidence="working", question="", items_taken_in=(), highest_likelihood_taken_in=0, highest_escape_and_impact_taken_in=0, escalation="", release_ready=False, **step):
|
||||
verdict = {"items_taken_in": list(items_taken_in), "answer": "", "highest_likelihood_taken_in": highest_likelihood_taken_in, "highest_escape_and_impact_taken_in": highest_escape_and_impact_taken_in, "escalation": escalation, "evidence": evidence, "finished": finished, "interrupt": interrupt, "question": question, "answered_before": "", "left_open": "", "release_ready": release_ready}
|
||||
def judge(interrupt=False, finished=False, evidence="working", question="", escalation="", release_ready=False, **step):
|
||||
verdict = {"answer": "", "escalation": escalation, "evidence": evidence, "finished": finished, "interrupt": interrupt, "question": question, "answered_before": "", "left_open": "", "release_ready": release_ready}
|
||||
return {"role": "judge", "verdict": verdict, **step}
|
||||
|
||||
|
||||
@@ -427,11 +430,11 @@ class Cycles(unittest.TestCase):
|
||||
def test_routes_the_worker_by_its_items_l_and_steps_it_up_within_the_chunk(self):
|
||||
checkout = Checkout(self)
|
||||
checkout.env["LOOP_CLAUDE_WORKER_MODEL"] = "claude-opus-5-5"
|
||||
checkout.commit_plan(CHEAP)
|
||||
checkout.commit_plan(TAKEN_IN_AT_L_4)
|
||||
run = checkout.run([
|
||||
worker(say="one"), judge(evidence="one"),
|
||||
worker(say="two"), judge(evidence="two"),
|
||||
worker(say="three"), judge(evidence="three", highest_likelihood_taken_in=4),
|
||||
worker(say="three\nChunk items: 4 (L 2, E + I 3), 9 (L 4, E + I 5)"), judge(evidence="three"),
|
||||
worker(say="four"), judge(finished=True, evidence="four"),
|
||||
worker(say="next chunk"),
|
||||
])
|
||||
@@ -445,7 +448,8 @@ class Cycles(unittest.TestCase):
|
||||
("claude-sonnet-5-5", "high"),
|
||||
])
|
||||
self.assertEqual({(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] == "judge"}, {("judge-model", None)})
|
||||
self.assertIn("loop-claude runs you on claude-sonnet-5-5 at high effort, routed by item 4's L of 2. Name each todo.md item this chunk takes, with its L and E + I, whether or not it is item 4.", argument(workers[0], "-p"))
|
||||
self.assertIn("loop-claude runs you on claude-sonnet-5-5 at high effort, routed by item 4's L of 2.", argument(workers[0], "-p"))
|
||||
self.assertIn("on one line, as in `Chunk items: ", argument(workers[0], "-p"))
|
||||
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
|
||||
self.assertIn('<UTC> Verdict: unfinished\n\nEvidence:\n> three\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "medium", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n\n<UTC> Stepped up\n\nWorker: {"model": "claude-opus-5-5", "effort": "high", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n', outcomes)
|
||||
self.assertIn('<UTC> Verdict: finished\n\nEvidence:\n> four\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "high", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n', outcomes)
|
||||
@@ -474,11 +478,11 @@ class Cycles(unittest.TestCase):
|
||||
self.assertIn("item 6, L 4, E + I 5", prompt)
|
||||
self.assertTrue((checkout.home / "code" / "proj-item-6").is_dir())
|
||||
|
||||
def test_lists_itself_in_loop_inbox_with_its_routed_item_then_those_the_judge_finds_taken_in_until_the_chunk_finishes(self):
|
||||
def test_lists_itself_in_loop_inbox_with_its_routed_item_then_those_the_worker_names_on_its_chunk_items_line_until_the_chunk_finishes(self):
|
||||
checkout = Checkout(self)
|
||||
checkout.env["LOOP_CLAUDE_WORKER_MODEL"] = "claude-opus-5-5"
|
||||
checkout.commit_plan(TODO)
|
||||
run = checkout.run([worker(), judge(items_taken_in=[7, 4]), worker(), judge(finished=True), worker()])
|
||||
run = checkout.run([worker(say="working\nChunk items: 7, 4"), judge(), worker(), judge(finished=True), worker()])
|
||||
self.assertEqual(run.code, 130)
|
||||
routed, taken_in = {"id": 4, "tagline": "Do it."}, {"id": 7, "tagline": ""}
|
||||
self.assertEqual([call["running"] for call in run.calls], [[[routed]], [[routed]], [[routed, taken_in]], [[routed, taken_in]], [[routed]], [[routed]]])
|
||||
@@ -540,9 +544,9 @@ class Cycles(unittest.TestCase):
|
||||
def test_routes_the_judge_and_the_reviewers_on_the_highest_e_plus_i_the_worker_takes_in(self):
|
||||
checkout = Checkout(self)
|
||||
checkout.env["LOOP_CLAUDE_WORKER_MODEL"] = checkout.env["LOOP_CLAUDE_JUDGE_MODEL"] = "claude-opus-5-5"
|
||||
checkout.commit_plan(CHEAP)
|
||||
checkout.commit_plan(TAKEN_IN_AT_E_PLUS_I_6)
|
||||
run = checkout.run([
|
||||
worker(say="one"), judge(evidence="one", highest_escape_and_impact_taken_in=6),
|
||||
worker(say="one\nChunk items: 4, 9"), judge(evidence="one"),
|
||||
worker(say="two"), judge(finished=True, evidence="two"),
|
||||
worker(say="next chunk"),
|
||||
])
|
||||
|
||||
@@ -5,6 +5,7 @@ import io
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
import stat
|
||||
import subprocess
|
||||
@@ -20,6 +21,7 @@ from routing_figures import archive, state
|
||||
from loop_state.waiting.ask import Ask, Asking, OpenInInbox, OpenOnTerminal
|
||||
from routing_figures.calls import Call
|
||||
from loop_claude.project import checkout, model_choice, records
|
||||
from loop_claude.cycle import chunk_items
|
||||
from loop_claude.loop import chunk_start, close
|
||||
from loop_state.waiting.answers import ClaudeEnded, ClaudeOpening, send, Typed
|
||||
from release_plan.release_standing import DefaultPlan, LastCut
|
||||
@@ -161,8 +163,8 @@ def ran(stdout="", code=0, stderr=""):
|
||||
return subprocess.CompletedProcess([], code, stdout, stderr)
|
||||
|
||||
|
||||
def judged(verdict, evidence="working", answer="", escalation="", release_ready=False, items_taken_in=(), cost=None):
|
||||
reply = {"items_taken_in": list(items_taken_in), "answer": answer, "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": escalation, "evidence": evidence, "finished": verdict == "finished", "interrupt": verdict == "interrupt", "question": "Which one?" if verdict in ("interrupt", "quoted") else "", "answered_before": "", "left_open": "", "release_ready": release_ready}
|
||||
def judged(verdict, evidence="working", answer="", escalation="", release_ready=False, cost=None):
|
||||
reply = {"answer": answer, "escalation": escalation, "evidence": evidence, "finished": verdict == "finished", "interrupt": verdict == "interrupt", "question": "Which one?" if verdict in ("interrupt", "quoted") else "", "answered_before": "", "left_open": "", "release_ready": release_ready}
|
||||
return ran(json.dumps({"structured_output": reply, **({} if cost is None else {"total_cost_usd": cost})}))
|
||||
|
||||
|
||||
@@ -268,7 +270,7 @@ class Cycles(unittest.TestCase):
|
||||
self.archive = project_archive(str(self.home / "code" / "proj"), self.home, {})
|
||||
|
||||
def cycle(self, fake, records="", unfinished_evidence="", facts="- A preflight fact."):
|
||||
return run.Cycle(effects=fake.effects(), facts=facts, judge_model="judge-model", premises="## Goals", project=None, project_archive=self.archive, questions=self.paths.waiting, records=records, root=str(self.home), route=Route(Rung("worker-model")), session=self.session, show_taken_in=lambda items: None, task="Do it.", unfinished_evidence=unfinished_evidence)
|
||||
return run.Cycle(effects=fake.effects(), facts=facts, judge_model="judge-model", premises="## Goals", project=None, project_archive=self.archive, questions=self.paths.waiting, records=records, root=str(self.home), route=Route(Rung("worker-model")), session=self.session, show_taken_in=lambda items: None, task="Do it.", todo="", unfinished_evidence=unfinished_evidence)
|
||||
|
||||
def archived_outcomes(self):
|
||||
return sorted(path.read_text() for path in self.archive.glob("*/*.outcomes.md"))
|
||||
@@ -284,16 +286,16 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: finished\n\nEvidence:\n> working\n\nJudge: judge-model\nJudge cost: 0.2500\nWorker: {OFF_LADDER}\n"])
|
||||
self.assertIsNone(session.transcript_file(self.session))
|
||||
|
||||
def test_shows_the_items_each_ruling_finds_the_chunk_takes_as_the_judge_gave_them(self):
|
||||
worker = Worker(self.paths.claude_code_transcripts.parent)
|
||||
fake = FakeEffects(streams=[worker], claude=[judged("unfinished", items_taken_in=(14, 3))])
|
||||
def test_shows_the_items_each_ruling_finds_the_worker_names_on_its_chunk_items_line(self):
|
||||
worker = Worker(self.paths.claude_code_transcripts.parent, text="working\nChunk items: 14 (L 2, E + I 3), 3 (L 1, E + I 2)")
|
||||
fake = FakeEffects(streams=[worker], claude=[judged("unfinished")])
|
||||
shown = []
|
||||
run.run_cycle(dataclasses.replace(self.cycle(fake), show_taken_in=shown.append))
|
||||
self.assertEqual(shown, [(14, 3)])
|
||||
|
||||
def test_shows_no_items_for_a_finished_chunk(self):
|
||||
worker = Worker(self.paths.claude_code_transcripts.parent)
|
||||
fake = FakeEffects(streams=[worker], claude=[judged("finished", items_taken_in=(3,))])
|
||||
worker = Worker(self.paths.claude_code_transcripts.parent, text="working\nChunk items: 3")
|
||||
fake = FakeEffects(streams=[worker], claude=[judged("finished")])
|
||||
shown = []
|
||||
run.run_cycle(dataclasses.replace(self.cycle(fake), show_taken_in=shown.append))
|
||||
self.assertEqual(shown, [])
|
||||
@@ -1181,6 +1183,14 @@ class RouteSources(unittest.TestCase):
|
||||
self.assertIsNone(chunk_start.plan_item(plan, FakeEffects().effects()))
|
||||
self.assertTrue(stderr.getvalue().startswith(f"{STAMP} {said}"), stderr.getvalue())
|
||||
|
||||
def test_says_why_the_items_the_worker_names_lift_no_rung(self):
|
||||
fetched = ["origin\n", "origin/main\n", "", "todo.md\n"]
|
||||
for git, said in ((fetched + [TODO.replace("| 4 | 2 | 3 |", "| x | 2 | 3 |")], "the items the worker names lift no rung: item 3's L 'x' is not a whole number from 1 to 4\n"), ([Failure("no git")], "the items the worker names lift no rung: no git\n"), (fetched + [TODO], "")):
|
||||
with self.subTest(said=said):
|
||||
with contextlib.redirect_stderr(io.StringIO()) as stderr:
|
||||
chunk_start.fetched_todo(a_loop(FakeEffects(git=git).effects()))
|
||||
self.assertEqual(stderr.getvalue(), f"{STAMP} {said}" if said else "")
|
||||
|
||||
def test_resumes_on_the_archived_route_and_says_why_it_routes_afresh(self):
|
||||
with tempfile.TemporaryDirectory() as home:
|
||||
opened = session.session_named("A proj loop-claude", pathlib.Path(home, "projects"))
|
||||
@@ -1340,9 +1350,9 @@ class WorkerArgs(unittest.TestCase):
|
||||
|
||||
def test_names_the_route_in_a_fresh_prompt_only_where_an_item_routed_it(self):
|
||||
routed = prompts.fresh("Do it.", "## Goals", Route(Rung("m", "high"), 3, 4, 5), None, self.FACTS)
|
||||
self.assertIn(f"Continue any open draft PR or worktree for the chunk before creating a new one. {prompts.READING_RULE}\n\nloop-claude runs you on m at high effort, routed by item 3's L of 4. Name each todo.md item this chunk takes, with its L and E + I, whether or not it is item 3.", routed)
|
||||
self.assertIn(f"Continue any open draft PR or worktree for the chunk before creating a new one. {prompts.READING_RULE} {prompts.NAME_THE_ITEMS}\n\nloop-claude runs you on m at high effort, routed by item 3's L of 4.", routed)
|
||||
unrouted = prompts.fresh("Do it.", "## Goals", Route(Rung("m")), None, self.FACTS)
|
||||
self.assertIn(f"before creating a new one. {prompts.READING_RULE}\n\n{self.FACTS}\n\nThe chunk's", unrouted)
|
||||
self.assertIn(f"before creating a new one. {prompts.READING_RULE} {prompts.NAME_THE_ITEMS}\n\n{self.FACTS}\n\nThe chunk's", unrouted)
|
||||
self.assertNotIn("loop-claude runs you on", unrouted)
|
||||
|
||||
def test_has_the_routed_worker_count_its_items_size_again_and_puts_the_facts_before_the_scope(self):
|
||||
@@ -1372,21 +1382,21 @@ class WorkerArgs(unittest.TestCase):
|
||||
def test_keeps_every_review_on_the_projects_rung_while_the_worker_runs_below_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
below = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 4), opus, self.FACTS)
|
||||
self.assertIn(f"whether or not it is item 3. Launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.{self.SCAN} Your rung is below the project's: launch each subagent that files or scores todo.md items on the rung `~/.claude/bin/subagent-rung --plan claude-opus-5-5 <L>` prints. Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5, `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8. When an escalation trigger", below)
|
||||
self.assertIn(f"item 3's L of 2. Launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.{self.SCAN} Your rung is below the project's: launch each subagent that files or scores todo.md items on the rung `~/.claude/bin/subagent-rung --plan claude-opus-5-5 <L>` prints. Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5, `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8. When an escalation trigger", below)
|
||||
self.assertNotIn("Launch each", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus, self.FACTS))
|
||||
|
||||
def test_launches_reviewers_on_the_rung_the_items_e_plus_i_picks_above_the_workers(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
level = prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 6), opus, self.FACTS)
|
||||
self.assertIn(f"whether or not it is item 3. Launch each reviewer with the Agent tool set to model opus, effort xhigh, the rung item 3's E + I of 6 picks. Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", level)
|
||||
self.assertIn(f"item 3's L of 3. Launch each reviewer with the Agent tool set to model opus, effort xhigh, the rung item 3's E + I of 6 picks. Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", level)
|
||||
below = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 5), opus, self.FACTS)
|
||||
self.assertIn(f"whether or not it is item 3. Launch each reviewer with the Agent tool set to model opus, effort high, the rung item 3's E + I of 5 picks.{self.SEATS}{self.SCAN} Your rung is below the project's:", below)
|
||||
self.assertIn(f"item 3's L of 2. Launch each reviewer with the Agent tool set to model opus, effort high, the rung item 3's E + I of 5 picks.{self.SEATS}{self.SCAN} Your rung is below the project's:", below)
|
||||
self.assertNotIn("Launch each reviewer", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 6), opus))
|
||||
|
||||
def test_names_the_reviewers_rung_each_higher_e_plus_i_an_item_taken_in_picks(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
taking_in = " Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5, `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8."
|
||||
self.assertIn(f"whether or not it is item 3.{taking_in} When an escalation trigger", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus, self.FACTS))
|
||||
self.assertIn(f"item 3's L of 3.{taking_in} When an escalation trigger", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus, self.FACTS))
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8.", prompts.resumed("", Route(opus, 3, 3, 6), opus))
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8.", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 4), opus))
|
||||
self.assertNotIn("Where an item you take into", prompts.resumed("", Route(opus, 3, 3, 8), opus))
|
||||
@@ -1401,15 +1411,15 @@ class WorkerArgs(unittest.TestCase):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
above = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-opus-5-5", "high"), 3, 4, 4), opus, self.FACTS)
|
||||
stepped_up = prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus)
|
||||
self.assertIn(f"{prompts.ANSWERS_RECORDED}. loop-claude runs you on claude-opus-5-5 at high effort.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", stepped_up)
|
||||
self.assertIn(f"{prompts.ANSWERS_RECORDED}. {prompts.NAME_THE_ITEMS} loop-claude runs you on claude-opus-5-5 at high effort.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", stepped_up)
|
||||
self.assertNotIn("the project's", prompts.resumed("", Route(opus, 3, 3, 4), opus))
|
||||
self.assertNotIn("runs you on", prompts.resumed("", Route(Rung("m")), None))
|
||||
self.assertIn(f"whether or not it is item 3.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", above)
|
||||
self.assertIn(f"item 3's L of 4.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", above)
|
||||
|
||||
def test_names_the_rungs_an_escalation_trigger_lets_the_worker_hand_a_task_to_at_most_two_above_its_items(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
fresh = prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus, self.FACTS)
|
||||
self.assertIn("whether or not it is item 3. Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5, `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", fresh)
|
||||
self.assertIn("item 3's L of 3. Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5, `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", fresh)
|
||||
self.assertIn(" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort xhigh`, one rung above yours; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus))
|
||||
self.assertIn(" An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `model opus, effort max`, one rung above yours, only once the human agrees; loop-claude will not resume you higher for a trigger either, so ask the human for that yes. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 3, 4), opus))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("claude-fable-5-1", "max"), 3, 4, 8), opus))
|
||||
@@ -1417,7 +1427,7 @@ class WorkerArgs(unittest.TestCase):
|
||||
self.assertIn("hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort max`, one rung above yours;", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 5), opus))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("m")), None))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("claude-opus-5-5", "low"), 3, 3, 4), opus))
|
||||
self.assertIn(f"{prompts.READING_RULE}\n\nWhere an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5, `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number. Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`. A reviewer", prompts.fresh("Do it.", "## Goals", Route(opus), opus, self.FACTS))
|
||||
self.assertIn(f"{prompts.READING_RULE} {prompts.NAME_THE_ITEMS}\n\nWhere an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5, `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number. Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`. A reviewer", prompts.fresh("Do it.", "## Goals", Route(opus), opus, self.FACTS))
|
||||
|
||||
def test_offers_the_worker_a_higher_rung_for_the_rest_of_the_chunk_up_to_the_escalation_cap(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
@@ -1566,65 +1576,75 @@ class Grounded(unittest.TestCase):
|
||||
self.assertTrue(judge.grounded("a ... b ... c", "x a b a c"))
|
||||
|
||||
|
||||
# Item 297 scores the higher L, item 298 the higher E + I.
|
||||
SCORED = "## Items\n\n| ID | Release | Exempt | Item | L | E | I | R | S | Moves | Score |\n|---|---|---|---|---|---|---|---|---|---|---|\n| 297 | 0.8.0 | | **Do it.** | 4 | 2 | 3 | 7 | 2 | 1 by 9 | 25 |\n| 298 | 0.8.0 | | **Do more.** | 3 | 3 | 4 | 8 | 2 | 1 by 9 | 20 |\n"
|
||||
|
||||
|
||||
def said(*texts):
|
||||
return entries.AssistantTurn(parts=tuple(entries.Text(text) for text in texts))
|
||||
|
||||
|
||||
class ChunkItems(unittest.TestCase):
|
||||
def test_names_the_items_on_the_workers_latest_chunk_items_line(self):
|
||||
earlier = said("Chunk items: 297 (L 3, E + I 5)")
|
||||
latest = said("Starting.\n**Chunk items:** 297 (L 4, E + I 5), item 298 (L 3, E + I 7), #12, 298\nOn to the tests.")
|
||||
self.assertEqual(chunk_items.named([earlier, latest]), (297, 298, 12))
|
||||
self.assertEqual(chunk_items.named([latest, earlier]), (297,))
|
||||
self.assertEqual(chunk_items.named([earlier, said("Chunk items: none")]), ())
|
||||
self.assertEqual(chunk_items.named([said("Chunk items: 297 and 298 (L 3, E + I 7)")]), (297, 298))
|
||||
|
||||
def test_reads_the_example_line_the_prompt_gives_the_worker(self):
|
||||
example = re.search(r"`(Chunk items:[^`]*)`", prompts.NAME_THE_ITEMS)[1]
|
||||
self.assertEqual(chunk_items.named([said(example)]), (297, 298))
|
||||
|
||||
def test_every_prompt_that_starts_or_resumes_a_chunk_asks_for_the_line(self):
|
||||
self.assertIn("on one line, as in `Chunk items: ", prompts.resumed("", Route(Rung("m")), None))
|
||||
|
||||
def test_reads_only_a_line_the_worker_opens_with_the_label_in_its_own_text(self):
|
||||
prompt = entries.LoopPrompt(text="Chunk items: 297 (L 3, E + I 5)")
|
||||
written = entries.AssistantTurn(parts=(entries.ToolCall("t", "Write", {"content": "Chunk items: 9"}),))
|
||||
mentioned = said("Once I know them, I write `Chunk items: 5`.")
|
||||
self.assertEqual(chunk_items.named([prompt, written, mentioned]), ())
|
||||
|
||||
def test_takes_the_highest_l_and_e_plus_i_todo_md_scores_among_the_named_items(self):
|
||||
self.assertEqual(chunk_items.taken_in([said("Chunk items: 297 (L 1, E + I 2), 298, 400")], SCORED), TakenIn((297, 298, 400), 4, 7))
|
||||
self.assertEqual(chunk_items.taken_in([said("Chunk items: 400")], SCORED), TakenIn((400,)))
|
||||
self.assertEqual(chunk_items.taken_in([], SCORED), TakenIn())
|
||||
|
||||
def test_names_the_items_at_no_level_where_todo_md_cannot_be_read(self):
|
||||
self.assertEqual(chunk_items.taken_in([said("Chunk items: 297")], Failure("no todo.md")), TakenIn((297,)))
|
||||
self.assertEqual(chunk_items.taken_in([said("Chunk items: 297")], "no items table"), TakenIn((297,)))
|
||||
|
||||
|
||||
class Verdict(unittest.TestCase):
|
||||
def test_needs_every_field_the_schema_requires(self):
|
||||
self.assertEqual(verdict_from({"items_taken_in": [], "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": False, "evidence": "e", "finished": True, "question": "", "answered_before": "", "left_open": "", "release_ready": False}), Finished("e"))
|
||||
self.assertEqual(verdict_from({"items_taken_in": [], "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": False, "evidence": "e", "finished": False, "question": "ignored", "answered_before": "", "left_open": "", "release_ready": False}), Unfinished("e"))
|
||||
self.assertEqual(verdict_from({"items_taken_in": [], "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "e", "finished": True, "question": "Which one?", "answered_before": "", "left_open": "", "release_ready": False}), Interrupt("e", "Which one?"))
|
||||
self.assertEqual(verdict_from({"items_taken_in": [], "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "e", "finished": False, "question": "Cut 0.2.0?", "answered_before": "", "left_open": "", "release_ready": True}), Interrupt("e", "Cut 0.2.0?", release_ready=True))
|
||||
self.assertEqual(verdict_from({"items_taken_in": [], "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answered_before": "", "left_open": "", "answer": "", "interrupt": True, "evidence": "e", "finished": False, "question": "Cut 0.2.0?"}), Failure("the judge's reply has no release_ready of type boolean"))
|
||||
self.assertEqual(verdict_from({"items_taken_in": [], "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "A.", "interrupt": False, "evidence": "e", "finished": False, "question": "Which one?", "answered_before": "", "left_open": "", "release_ready": False}), QuotedAnswer("e", "Which one?", "A."))
|
||||
self.assertEqual(verdict_from({"items_taken_in": [], "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "A.", "interrupt": True, "evidence": "e", "finished": False, "question": "Which one?", "answered_before": "", "left_open": "", "release_ready": False}), Interrupt("e", "Which one?"))
|
||||
self.assertEqual(verdict_from({"items_taken_in": [], "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "A.", "interrupt": False, "evidence": "e", "finished": True, "question": "", "answered_before": "", "left_open": "", "release_ready": False}), Failure("the judge quoted an answer without a question"))
|
||||
self.assertEqual(verdict_from({"items_taken_in": [], "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": "no", "evidence": "", "finished": True, "question": "", "answered_before": "", "left_open": "", "release_ready": False}), Failure("the judge's reply has no interrupt of type boolean"))
|
||||
self.assertEqual(verdict_from({"items_taken_in": [], "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": False, "evidence": None, "finished": True, "question": "", "answered_before": "", "left_open": "", "release_ready": False}), Failure("the judge's reply has no evidence of type string"))
|
||||
self.assertEqual(verdict_from({"items_taken_in": [], "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "x", "finished": False, "question": " ", "answered_before": "", "left_open": "", "release_ready": False}), Failure("the judge interrupted without a question"))
|
||||
self.assertEqual(verdict_from({"escalation": "", "answer": "", "interrupt": False, "evidence": "e", "finished": True, "question": "", "answered_before": "", "left_open": "", "release_ready": False}), Finished("e"))
|
||||
self.assertEqual(verdict_from({"escalation": "", "answer": "", "interrupt": False, "evidence": "e", "finished": False, "question": "ignored", "answered_before": "", "left_open": "", "release_ready": False}), Unfinished("e"))
|
||||
self.assertEqual(verdict_from({"escalation": "", "answer": "", "interrupt": True, "evidence": "e", "finished": True, "question": "Which one?", "answered_before": "", "left_open": "", "release_ready": False}), Interrupt("e", "Which one?"))
|
||||
self.assertEqual(verdict_from({"escalation": "", "answer": "", "interrupt": True, "evidence": "e", "finished": False, "question": "Cut 0.2.0?", "answered_before": "", "left_open": "", "release_ready": True}), Interrupt("e", "Cut 0.2.0?", release_ready=True))
|
||||
self.assertEqual(verdict_from({"escalation": "", "answered_before": "", "left_open": "", "answer": "", "interrupt": True, "evidence": "e", "finished": False, "question": "Cut 0.2.0?"}), Failure("the judge's reply has no release_ready of type boolean"))
|
||||
self.assertEqual(verdict_from({"escalation": "", "answer": "A.", "interrupt": False, "evidence": "e", "finished": False, "question": "Which one?", "answered_before": "", "left_open": "", "release_ready": False}), QuotedAnswer("e", "Which one?", "A."))
|
||||
self.assertEqual(verdict_from({"escalation": "", "answer": "A.", "interrupt": True, "evidence": "e", "finished": False, "question": "Which one?", "answered_before": "", "left_open": "", "release_ready": False}), Interrupt("e", "Which one?"))
|
||||
self.assertEqual(verdict_from({"escalation": "", "answer": "A.", "interrupt": False, "evidence": "e", "finished": True, "question": "", "answered_before": "", "left_open": "", "release_ready": False}), Failure("the judge quoted an answer without a question"))
|
||||
self.assertEqual(verdict_from({"escalation": "", "answer": "", "interrupt": "no", "evidence": "", "finished": True, "question": "", "answered_before": "", "left_open": "", "release_ready": False}), Failure("the judge's reply has no interrupt of type boolean"))
|
||||
self.assertEqual(verdict_from({"escalation": "", "answer": "", "interrupt": False, "evidence": None, "finished": True, "question": "", "answered_before": "", "left_open": "", "release_ready": False}), Failure("the judge's reply has no evidence of type string"))
|
||||
self.assertEqual(verdict_from({"escalation": "", "answer": "", "interrupt": True, "evidence": "x", "finished": False, "question": " ", "answered_before": "", "left_open": "", "release_ready": False}), Failure("the judge interrupted without a question"))
|
||||
self.assertEqual(verdict_from(None), Failure("the judge returned no reply"))
|
||||
|
||||
def test_carries_the_earlier_answer_an_interrupt_names(self):
|
||||
reply = {"items_taken_in": [], "answer": "", "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "interrupt": True, "evidence": "e", "finished": False, "release_ready": False, "answered_before": "Grant one more round.", "left_open": "one more", "question": "Which one?"}
|
||||
reply = {"answer": "", "escalation": "", "interrupt": True, "evidence": "e", "finished": False, "release_ready": False, "answered_before": "Grant one more round.", "left_open": "one more", "question": "Which one?"}
|
||||
self.assertEqual(verdict_from(reply), Interrupt("e", "Which one?", answered_before="Grant one more round.", left_open="one more"))
|
||||
self.assertEqual(verdict_from({**reply, "interrupt": False, "finished": True}), Finished("e"))
|
||||
|
||||
def test_takes_the_highest_l_taken_in_on_its_scale_and_counts_any_other_as_none(self):
|
||||
reply = {"items_taken_in": [], "answer": "", "highest_escape_and_impact_taken_in": 0, "escalation": "", "interrupt": False, "evidence": "e", "finished": False, "question": "", "answered_before": "", "left_open": "", "release_ready": False}
|
||||
self.assertEqual(verdict_from({**reply, "highest_likelihood_taken_in": 4}), Unfinished("e", taken_in=TakenIn(highest_likelihood=4)))
|
||||
self.assertEqual(verdict_from({**reply, "highest_likelihood_taken_in": True}), Failure("the judge's reply has no highest_likelihood_taken_in of type integer"))
|
||||
for highest_likelihood_taken_in in (5, 13, -1):
|
||||
with self.subTest(highest_likelihood_taken_in=highest_likelihood_taken_in):
|
||||
self.assertEqual(verdict_from({**reply, "highest_likelihood_taken_in": highest_likelihood_taken_in}), Unfinished("e"))
|
||||
|
||||
|
||||
def test_takes_the_highest_e_plus_i_taken_in_on_its_scale_and_counts_any_other_as_none(self):
|
||||
reply = {"items_taken_in": [], "answer": "", "highest_likelihood_taken_in": 0, "escalation": "", "interrupt": False, "evidence": "e", "finished": False, "question": "", "answered_before": "", "left_open": "", "release_ready": False}
|
||||
self.assertEqual(verdict_from({**reply, "highest_escape_and_impact_taken_in": 6}), Unfinished("e", taken_in=TakenIn(highest_escape_and_impact=6)))
|
||||
self.assertEqual(verdict_from({**reply, "highest_escape_and_impact_taken_in": 2}), Unfinished("e", taken_in=TakenIn(highest_escape_and_impact=2)))
|
||||
self.assertEqual(verdict_from({**reply, "highest_escape_and_impact_taken_in": False}), Failure("the judge's reply has no highest_escape_and_impact_taken_in of type integer"))
|
||||
for escape_and_impact in (1, 9, -1):
|
||||
with self.subTest(escape_and_impact=escape_and_impact):
|
||||
self.assertEqual(verdict_from({**reply, "highest_escape_and_impact_taken_in": escape_and_impact}), Unfinished("e"))
|
||||
|
||||
def test_takes_the_items_taken_in_as_the_judge_gives_them_leaving_out_ids_below_1(self):
|
||||
reply = {"items_taken_in": [], "answer": "", "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "interrupt": False, "evidence": "e", "finished": False, "question": "", "answered_before": "", "left_open": "", "release_ready": False}
|
||||
self.assertEqual(verdict_from({**reply, "items_taken_in": [12, 14, 12]}), Unfinished("e", taken_in=TakenIn((12, 14, 12))))
|
||||
self.assertEqual(verdict_from({**reply, "items_taken_in": [7, 0]}), Unfinished("e", taken_in=TakenIn((7,))))
|
||||
self.assertEqual(verdict_from({**reply, "items_taken_in": [3], "interrupt": True, "question": "Which one?"}), Interrupt("e", "Which one?", taken_in=TakenIn((3,))))
|
||||
self.assertEqual(verdict_from({**reply, "items_taken_in": [3], "answer": "A.", "question": "Which one?"}), QuotedAnswer("e", "Which one?", "A.", taken_in=TakenIn((3,))))
|
||||
self.assertEqual(verdict_from({**reply, "items_taken_in": [3], "finished": True}), Finished("e"))
|
||||
self.assertEqual(verdict_from({**reply, "items_taken_in": "3"}), Failure("the judge's reply has no items_taken_in of type array of integers"))
|
||||
self.assertEqual(verdict_from({**reply, "items_taken_in": ["3"]}), Failure("the judge's reply has no items_taken_in of type array of integers"))
|
||||
|
||||
def test_takes_an_escalation_quote_on_an_unfinished_verdict(self):
|
||||
reply = {"items_taken_in": [], "answer": "", "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "round 2: 🔴", "interrupt": False, "evidence": "e", "finished": False, "question": "", "answered_before": "", "left_open": "", "release_ready": False}
|
||||
reply = {"answer": "", "escalation": "round 2: 🔴", "interrupt": False, "evidence": "e", "finished": False, "question": "", "answered_before": "", "left_open": "", "release_ready": False}
|
||||
self.assertEqual(verdict_from(reply), Unfinished("e", "round 2: 🔴"))
|
||||
self.assertEqual(verdict_from({**reply, "finished": True}), Finished("e"))
|
||||
self.assertEqual(verdict_from({**reply, "escalation": None}), Failure("the judge's reply has no escalation of type string"))
|
||||
|
||||
def test_takes_an_escalation_quote_on_an_interrupt(self):
|
||||
reply = {"items_taken_in": [], "answer": "", "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "round 2: 🔴", "interrupt": True, "evidence": "e", "finished": False, "question": "Which one?", "answered_before": "", "left_open": "", "release_ready": False}
|
||||
reply = {"answer": "", "escalation": "round 2: 🔴", "interrupt": True, "evidence": "e", "finished": False, "question": "Which one?", "answered_before": "", "left_open": "", "release_ready": False}
|
||||
self.assertEqual(verdict_from(reply), Interrupt("e", "Which one?", "round 2: 🔴"))
|
||||
self.assertEqual(verdict_from({**reply, "highest_likelihood_taken_in": 4}), Interrupt("e", "Which one?", "round 2: 🔴", TakenIn(highest_likelihood=4)))
|
||||
self.assertEqual(verdict_from({**reply, "highest_likelihood_taken_in": 5}), Interrupt("e", "Which one?", "round 2: 🔴"))
|
||||
|
||||
def test_drops_an_escalation_quote_the_transcript_does_not_hold_word_for_word_from_an_interrupt_or_an_unfinished_verdict(self):
|
||||
shown = "The agent: round 2: 🔴. Which one?"
|
||||
@@ -1639,8 +1659,8 @@ class Verdict(unittest.TestCase):
|
||||
asking = "Grant a third round? Your answer left open: a 🔴 survives it"
|
||||
kept = Interrupt("[step] Round 2", asking, answered_before=answer, left_open="a 🔴 survives it")
|
||||
self.assertEqual(judge.quotes_checked(kept, grounds), kept)
|
||||
re_asked = Interrupt("[step] Round 2", "Grant a third round?", "Round 2: a 🔴", TakenIn((3,)), answered_before=answer, left_open="a 🔴 survives it")
|
||||
self.assertEqual(judge.quotes_checked(re_asked, grounds), Unfinished(answer, "Round 2: a 🔴", TakenIn((3,))))
|
||||
re_asked = Interrupt("[step] Round 2", "Grant a third round?", "Round 2: a 🔴", answered_before=answer, left_open="a 🔴 survives it")
|
||||
self.assertEqual(judge.quotes_checked(re_asked, grounds), Unfinished(answer, "Round 2: a 🔴"))
|
||||
self.assertEqual(judge.quotes_checked(dataclasses.replace(kept, left_open=""), grounds), Unfinished(answer))
|
||||
self.assertEqual(judge.quotes_checked(dataclasses.replace(kept, left_open="a 🔴 survived"), grounds), Unfinished(answer))
|
||||
invented = dataclasses.replace(re_asked, answered_before="Grant two more rounds.")
|
||||
@@ -1648,10 +1668,9 @@ class Verdict(unittest.TestCase):
|
||||
unnamed = Interrupt("[step] Round 2", "Grant a third round?")
|
||||
self.assertEqual(judge.quotes_checked(unnamed, grounds), unnamed)
|
||||
|
||||
def test_takes_an_escalation_quote_and_the_highest_l_taken_in_on_a_quoted_answer(self):
|
||||
reply = {"items_taken_in": [], "answer": "A.", "highest_likelihood_taken_in": 4, "highest_escape_and_impact_taken_in": 0, "escalation": "round 2: 🔴", "interrupt": False, "evidence": "e", "finished": False, "question": "Which one?", "answered_before": "", "left_open": "", "release_ready": False}
|
||||
self.assertEqual(verdict_from(reply), QuotedAnswer("e", "Which one?", "A.", "round 2: 🔴", TakenIn(highest_likelihood=4)))
|
||||
self.assertEqual(verdict_from({**reply, "highest_likelihood_taken_in": 5}), QuotedAnswer("e", "Which one?", "A.", "round 2: 🔴"))
|
||||
def test_takes_an_escalation_quote_on_a_quoted_answer(self):
|
||||
reply = {"answer": "A.", "escalation": "round 2: 🔴", "interrupt": False, "evidence": "e", "finished": False, "question": "Which one?", "answered_before": "", "left_open": "", "release_ready": False}
|
||||
self.assertEqual(verdict_from(reply), QuotedAnswer("e", "Which one?", "A.", "round 2: 🔴"))
|
||||
|
||||
def test_drops_an_escalation_quote_the_transcript_does_not_hold_word_for_word_from_a_quoted_answer(self):
|
||||
shown = "The agent: round 2: 🔴. Which one?"
|
||||
@@ -1659,8 +1678,7 @@ class Verdict(unittest.TestCase):
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "A.", "round 2: … 🔴"), judge.Grounds(shown, shown, "A.")), QuotedAnswer("e", "q", "A."))
|
||||
|
||||
def test_puts_a_quoted_answer_the_records_do_not_hold_to_the_human_with_the_escalation_it_carried(self):
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("s", "q", "not there", taken_in=TakenIn((3,))), judge.Grounds("shown", "shown", "A.")), Interrupt("s", "q", taken_in=TakenIn((3,))))
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "not there", "round 2: 🔴", TakenIn(highest_likelihood=4)), judge.Grounds("The agent: round 2: 🔴.", "The agent: round 2: 🔴.", "A.")), Interrupt("e", "q", "round 2: 🔴", TakenIn(highest_likelihood=4)))
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "not there", "round 2: 🔴"), judge.Grounds("The agent: round 2: 🔴.", "The agent: round 2: 🔴.", "A.")), Interrupt("e", "q", "round 2: 🔴"))
|
||||
|
||||
def test_puts_a_quoted_answer_the_records_do_not_hold_word_for_word_to_the_human(self):
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("s", "q", "not there"), judge.Grounds("shown", "shown", "the records")), Interrupt("s", "q"))
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
"""The verdicts cycle.judge.verdict_from builds from the judge's reply. They sit outside cycle/, since routing reads them and cycle imports routing."""
|
||||
"""The verdicts cycle.judge.verdict_from builds from the judge's reply; cycle.run.rule then stamps every verdict but Finished with the items the chunk takes. They sit outside cycle/, since routing reads them and cycle imports routing."""
|
||||
import dataclasses
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class TakenIn:
|
||||
"""What the judge read in the transcript about the todo.md items the worker took into the chunk, the item the chunk was routed on included."""
|
||||
"""The todo.md items the worker names on its latest `Chunk items:` line, as cycle.chunk_items reads them. The item the chunk was routed on counts only where the worker names it."""
|
||||
# Their IDs; loop-inbox lists them.
|
||||
items: tuple[int, ...] = ()
|
||||
# The highest L among them, 0 for none: an escalation steps up from the rung that L picks, and an unfinished verdict also raises the worker to it.
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
A running loop holds its session's lock (session_lock.py) in Paths.waiting for its whole run. Beside the lock it keeps
|
||||
<session>.running, which holds the loop's checkout and each item its chunk takes, by ID and by its tagline on the default
|
||||
branch as last fetched. The items are the one the chunk was routed on, then those the judge's latest ruling found the
|
||||
worker took in, each listed once. loop-inbox lists a loop, and its question (waiting/), only while the loop holds its
|
||||
branch as last fetched. The items are the one the chunk was routed on, then those the worker names on its latest
|
||||
`Chunk items:` line, each listed once. loop-inbox lists a loop, and its question (waiting/), only while the loop holds its
|
||||
lock. So a killed loop drops off the list, though its files stay until loop-inbox removes them: <session>.running
|
||||
through drop_ended_loops here, and the question through waiting.listing.drop_ended.
|
||||
"""
|
||||
|
||||
@@ -43,7 +43,7 @@ class Route:
|
||||
likelihood: int = 0
|
||||
escape_and_impact: int = 0
|
||||
unfinished: int = 0
|
||||
# The highest E + I the judge saw the worker take in, kept only where it picks a higher review step than the item's; 0 otherwise.
|
||||
# The highest E + I among the items the worker names on its `Chunk items:` line, kept only where it picks a higher review step than the item's; 0 otherwise.
|
||||
escape_and_impact_taken_in: int = 0
|
||||
# The escalation mark, as loop_claude/escalation_mark.py says: how many transcript entries the session held when an escalation last stepped the worker up, 0 before any.
|
||||
escalated_at: int = 0
|
||||
@@ -59,7 +59,7 @@ class Route:
|
||||
return levels_fit(fields.get("item", 0), fields.get("likelihood", 0), fields.get("escape_and_impact", 0), fields.get("escape_and_impact_taken_in", 0))
|
||||
|
||||
def taking_in(self, escape_and_impact: int) -> "Route":
|
||||
"""The route after the judge saw the worker take in an item of this E + I, 0 for none: it keeps the E + I only where REVIEW_STEPS steps it further than reviewed_on, so the archive records a step-up only where REVIEW_STEPS steps the review floor further."""
|
||||
"""The route after the worker takes in an item of this E + I, 0 for none: it keeps the E + I only where REVIEW_STEPS steps it further than reviewed_on, so the archive records a step-up only where REVIEW_STEPS steps the review floor further."""
|
||||
return dataclasses.replace(self, escape_and_impact_taken_in=escape_and_impact) if REVIEW_STEPS.get(escape_and_impact, 0) > REVIEW_STEPS.get(self.reviewed_on(), 0) else self
|
||||
|
||||
def reviewed_on(self) -> int:
|
||||
|
||||
@@ -50,10 +50,14 @@ class Citations(unittest.TestCase):
|
||||
def test_every_folder_the_judge_reads_rules_from_exists(self):
|
||||
self.assertEqual([folder for folder in HUMAN_RULE_DIRS if not (ROOT / folder).is_dir()], [])
|
||||
|
||||
def test_agents_md_lists_exactly_the_decision_titles(self):
|
||||
listed = (ROOT / "AGENTS.md").read_text().split("Decisions live in `docs/decisions.md`:\n\n")[1].split("\n\n")[0]
|
||||
# The two files order the titles differently, so they are compared sorted.
|
||||
self.assertEqual(sorted(LISTED_TITLE.findall(listed)), sorted(DECISION_TITLE.findall((ROOT / "docs/decisions.md").read_text())))
|
||||
def test_every_decision_title_the_log_lists_at_its_head_is_a_title_in_it(self):
|
||||
log = (ROOT / "docs/decisions.md").read_text()
|
||||
listed = LISTED_TITLE.findall(log.split("# Decisions no project principle covers\n\n")[1].split("\n\n")[0])
|
||||
self.assertTrue(listed)
|
||||
self.assertEqual([title for title in listed if title not in set(DECISION_TITLE.findall(log))], [])
|
||||
|
||||
def test_the_repo_keeps_no_agents_md_since_claude_code_would_load_the_releases_copy_into_every_project(self):
|
||||
self.assertFalse((ROOT / "AGENTS.md").exists())
|
||||
|
||||
def test_every_todo_citation_of_a_decision_names_a_title(self):
|
||||
titles = DECISION_TITLE.findall((ROOT / "docs/decisions.md").read_text())
|
||||
|
||||
@@ -9,7 +9,8 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from repo_checks import ROOT
|
||||
|
||||
TRIALS = ROOT / "reviewer-merge-trials"
|
||||
# The trials folders whose scripts import from bin/.
|
||||
TRIALS = [ROOT / "judge-field-trials", ROOT / "reviewer-merge-trials"]
|
||||
|
||||
|
||||
def bin_imports(source: str) -> list[tuple[str, str | None]]:
|
||||
@@ -39,11 +40,11 @@ def in_bin(module: str) -> bool:
|
||||
|
||||
|
||||
class TrialsImports(unittest.TestCase):
|
||||
"""docs/decisions.md says to rerun the trials' scripts when a new model joins the ladder or a reviewer's definition changes, so a refactor of bin/ must not break what they import."""
|
||||
"""docs/decisions.md says when to rerun each trials folder's scripts, so a refactor of bin/ must not break what they import."""
|
||||
|
||||
def test_each_name_a_trials_script_imports_from_bin_exists(self):
|
||||
scripts = sorted(TRIALS.glob("*.py"))
|
||||
self.assertTrue(scripts)
|
||||
scripts = sorted(script for folder in TRIALS for script in folder.glob("*.py"))
|
||||
self.assertTrue(all(any(folder.glob("*.py")) for folder in TRIALS))
|
||||
for script in scripts:
|
||||
for module, name in bin_imports(script.read_text(encoding="utf-8")):
|
||||
with self.subTest(script=script.name, module=module, name=name):
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""What loop-claude keeps at the archive's root between measurements: docs/decisions.md → `routing: loop-claude measures review rounds…`.
|
||||
|
||||
STATE holds, as fields loop-claude reads back, the cycle it last measured at and each call it raised. RAISED lists the raised calls for the human to read. A chunk appends a reset to RAISED by hand, as AGENTS.md says, and loop-claude reads the resets from there."""
|
||||
STATE holds, as fields loop-claude reads back, the cycle it last measured at and each call it raised. RAISED lists the raised calls for the human to read. A chunk appends a reset to RAISED by hand, as docs/decisions.md → "Working rules" says, and loop-claude reads the resets from there."""
|
||||
import dataclasses
|
||||
import json
|
||||
import pathlib
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
import dataclasses
|
||||
import posixpath
|
||||
|
||||
from rule_hook.rules import Breach, file_breaches
|
||||
from failure import Failure
|
||||
from git_cli import git_repo
|
||||
from rule_hook.git import guarded_git
|
||||
from rule_hook.file_text import read
|
||||
|
||||
# settings.json gives rule-hook 30 seconds; a file past this size is not source code, and reading it would use them up.
|
||||
MAX_BYTES = 1_000_000
|
||||
|
||||
|
||||
def committed_file_breaches(directory: str) -> list[Breach]:
|
||||
"""The breaches of file_breaches in the files a commit in directory's checkout would commit.
|
||||
|
||||
A file a Bash command wrote reaches no Edit or Write check, so the commit checks the file instead. The files are those that differ from HEAD in the index or the work tree, or that are untracked and not ignored: the hook runs before a `git add` earlier in the same command, so the index alone would miss those files. A tracked file is checked as the index holds it and as the work tree does, since the commit takes the index. A deleted file, or one that is not UTF-8 text, has no added text. A breach carries the lines its file adds, so a marker among them overrides it, as in a Write; a marker the file already held doesn't."""
|
||||
top = guarded_git(directory, "rev-parse", "--show-toplevel")
|
||||
if isinstance(top, Failure):
|
||||
return []
|
||||
tracked = tracked_names(top)
|
||||
untracked = guarded_git(top, "ls-files", "-z", "--others", "--exclude-standard")
|
||||
breaches: list[Breach] = []
|
||||
for name in sorted(tracked | (set() if isinstance(untracked, Failure) else names_in(untracked))):
|
||||
path = posixpath.join(top, name)
|
||||
# An untracked file has neither a version in HEAD nor one in the index. A tracked file costs two git calls, about 4 ms each: a commit of 3000 changed files uses 13 of the hook's 30 seconds.
|
||||
before = blob(top, "HEAD", name) if name in tracked else ""
|
||||
for after in dict.fromkeys([read(path) if size(path) <= MAX_BYTES else "", blob(top, "", name) if name in tracked else ""]):
|
||||
known = set(before.splitlines())
|
||||
added = "\n".join(line for line in after.splitlines() if line not in known)
|
||||
breaches += [dataclasses.replace(breach, marked=added) for breach in file_breaches(path, before, added, after)]
|
||||
# The index copy and the work-tree copy of a file can breach the same way; the first stands.
|
||||
return list({(breach.rule, breach.detail): breach for breach in reversed(breaches)}.values())[::-1]
|
||||
|
||||
|
||||
def tracked_names(top: str) -> set[str]:
|
||||
"""The tracked files that differ from HEAD in the index or the work tree, deleted ones left out. It lists with diff-index, which neither hashes the work tree nor enters a submodule: either would run a clean filter that the checkout's config names, before the call is allowed."""
|
||||
listings = [guarded_git(top, "diff-index", "-z", "--name-only", "--no-renames", "--diff-filter=d", "--ignore-submodules", *cached, "HEAD") for cached in ([], ["--cached"])]
|
||||
if any(isinstance(listing, Failure) for listing in listings):
|
||||
# A checkout with no commit has no HEAD: every file in its index is new.
|
||||
listing = guarded_git(top, "ls-files", "-z")
|
||||
return set() if isinstance(listing, Failure) else names_in(listing)
|
||||
return {name for listing in listings if not isinstance(listing, Failure) for name in names_in(listing)}
|
||||
|
||||
|
||||
def names_in(listing: str) -> set[str]:
|
||||
return {name for name in listing.split("\0") if name}
|
||||
|
||||
|
||||
def blob(top: str, commit: str, name: str) -> str:
|
||||
"""name's text in the commit, or in the index where commit is empty; "" where there is none."""
|
||||
shown = git_repo.git("show", f"{commit}:{name}", cwd=top, guarded=True)
|
||||
return "" if isinstance(shown, Failure) or len(shown) > MAX_BYTES else shown
|
||||
|
||||
|
||||
def size(path: str) -> int:
|
||||
try:
|
||||
return posixpath.getsize(path)
|
||||
except OSError:
|
||||
return 0
|
||||
@@ -0,0 +1,12 @@
|
||||
import os
|
||||
import stat
|
||||
|
||||
|
||||
def read(path: str) -> str:
|
||||
"""path's text, or "" for anything but a regular file: a FIFO would hang the hook and a device such as /dev/zero would flood it."""
|
||||
try:
|
||||
# O_NONBLOCK opens a FIFO without waiting for a writer; fstat then judges the file opened, not one swapped in after a stat.
|
||||
with open(os.open(path, os.O_RDONLY | os.O_NONBLOCK), encoding="utf-8") as opened:
|
||||
return opened.read() if stat.S_ISREG(os.fstat(opened.fileno()).st_mode) else ""
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
+11
-14
@@ -8,14 +8,18 @@ subagent. The rules:
|
||||
|
||||
{rules}
|
||||
|
||||
A file a Bash command writes, such as through `cat > Dockerfile`, is not checked.
|
||||
|
||||
A call overrides a rule with `# rule-override <rule>: <reason>`:
|
||||
|
||||
- anywhere in a Bash command, such as a trailing comment
|
||||
- in a file, as a comment line in the text the edit writes
|
||||
- in Markdown, as `<!-- rule-override <rule>: <reason> -->`
|
||||
|
||||
A file a Bash command writes, such as through `cat > Dockerfile`, is checked when a `git commit` runs, not when written.
|
||||
The commit checks every file that differs from HEAD, in the index or the work tree, and every untracked file that is
|
||||
not ignored, even one the commit leaves out. It holds each to the rules that name a file (action-pin, image-pin,
|
||||
ignorefile-sorted, no-ai-attribution), judged on the lines the file adds. A marker among those lines, or in the commit
|
||||
command, overrides the file's breach.
|
||||
|
||||
Each override that let a call through is appended to rule-overrides.tsv in the git directory of
|
||||
the checkout it touched: UTC time, branch, the commit checked out, rule, reason, and the file
|
||||
written or the command's first line. An override outside any git checkout is not logged.
|
||||
@@ -31,7 +35,6 @@ cannot read.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
import sys
|
||||
import time
|
||||
import typing
|
||||
@@ -39,8 +42,10 @@ import typing
|
||||
from rule_hook.call import applied, Call, FILE_TOOLS, parsed_call, text
|
||||
from rule_hook.rules import Breach, check, Checked, command_breaches, file_breaches, rules_help
|
||||
from git_cli.git_command import commits
|
||||
from rule_hook.committed_files import committed_file_breaches
|
||||
from failure import Failure
|
||||
from rule_hook.override_log import Log, log_path, logged, print_overrides, write_override
|
||||
from rule_hook.file_text import read
|
||||
from rule_hook.repeat_read import read_breaches
|
||||
from rule_hook.reviewer_report import report_breaches
|
||||
from rule_hook.reviewer_rung import reviewer_breaches, skill_breaches
|
||||
@@ -110,14 +115,6 @@ def answer(checked: Checked, what: str, now: float) -> int:
|
||||
|
||||
def bash_breaches(command: str, cwd: str, home: str) -> list[Breach]:
|
||||
simples = simple_commands(command, cwd, home)
|
||||
return command_breaches(command, simples, home) + [breach for commit in commits(simples, home) for breach in round_cap_breaches(commit) + todo_breaches(commit.directory)]
|
||||
|
||||
|
||||
def read(path: str) -> str:
|
||||
"""path's text, or "" for anything but a regular file: a FIFO would hang the hook and a device such as /dev/zero would flood it."""
|
||||
try:
|
||||
# O_NONBLOCK opens a FIFO without waiting for a writer; fstat then judges the file opened, not one swapped in after a stat.
|
||||
with open(os.open(path, os.O_RDONLY | os.O_NONBLOCK), encoding="utf-8") as opened:
|
||||
return opened.read() if stat.S_ISREG(os.fstat(opened.fileno()).st_mode) else ""
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
found = commits(simples, home)
|
||||
in_files = [breach for directory in dict.fromkeys(commit.directory for commit in found) for breach in committed_file_breaches(directory)]
|
||||
return command_breaches(command, simples, home) + [breach for commit in found for breach in round_cap_breaches(commit) + todo_breaches(commit.directory)] + in_files
|
||||
|
||||
@@ -104,6 +104,8 @@ class Breach:
|
||||
detail: str
|
||||
# The checkout whose log records an override of this breach.
|
||||
directory: str
|
||||
# The lines a commit adds to the breach's file: its markers override this breach, as a Write's text overrides its own file's.
|
||||
marked: str = ""
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -124,7 +126,9 @@ def overrides(written: str) -> tuple[dict[str, str], list[str]]:
|
||||
|
||||
def check(breaches: list[Breach], written: str, markdown: bool, bash: bool) -> Checked:
|
||||
granted, malformed = overrides(written)
|
||||
blocking = [breach for breach in breaches if breach.rule not in granted]
|
||||
# A marker overrides a breach when it is in the call's own text or in the breach's file.
|
||||
grants = [granted | overrides(breach.marked)[0] for breach in breaches]
|
||||
blocking = [breach for breach, grant in zip(breaches, grants) if breach.rule not in grant]
|
||||
messages = [f"`{marker}` names no rule below, or gives no reason." for marker in malformed] if blocking else []
|
||||
where = "the command" if bash else "the text you write, as a comment line"
|
||||
for rule in sorted({breach.rule for breach in blocking}):
|
||||
@@ -134,7 +138,7 @@ def check(breaches: list[Breach], written: str, markdown: bool, bash: bool) -> C
|
||||
continue
|
||||
marker = f"<!-- rule-override {rule}: <reason> -->" if markdown else f"# rule-override {rule}: <reason>"
|
||||
messages.append(f"Blocked by rule {rule}: {details}. The rule: {RULES[rule].holds}. Where it does not fit this case, add `{marker}` to {where}; larv-review rules on the reason.")
|
||||
return Checked(messages, bool(blocking), [(breach, granted[breach.rule]) for breach in breaches if breach.rule in granted])
|
||||
return Checked(messages, bool(blocking), [(breach, grant[breach.rule]) for breach, grant in zip(breaches, grants) if breach.rule in grant])
|
||||
|
||||
|
||||
def command_breaches(command: str, simples: list[Simple], home: str) -> list[Breach]:
|
||||
|
||||
@@ -913,6 +913,127 @@ class TodoRankCheck(unittest.TestCase):
|
||||
self.assertNotIn("own script", " ".join(reasons))
|
||||
|
||||
|
||||
class CommittedFiles(unittest.TestCase):
|
||||
"""A git commit holds the files it would commit to the file rules, as an Edit or Write of each would be."""
|
||||
UNPINNED = "jobs:\n a:\n steps:\n - uses: actions/checkout@main\n"
|
||||
|
||||
def setUp(self):
|
||||
self.repo = tempfile.mkdtemp()
|
||||
self.addCleanup(subprocess.run, ["rm", "-rf", self.repo])
|
||||
git(self.repo, "init", "-q", "-b", "main")
|
||||
git(self.repo, "commit", "-q", "--allow-empty", "-m", "base")
|
||||
|
||||
def write(self, name: str, content: str) -> None:
|
||||
path = pathlib.Path(self.repo, name)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(content)
|
||||
|
||||
def commit(self, command: str = "git add -A && git commit -m x") -> subprocess.CompletedProcess:
|
||||
return hook({"cwd": self.repo, "tool_input": {"command": command}, "tool_name": "Bash"}, self.repo)
|
||||
|
||||
def test_a_file_a_command_wrote_is_checked_at_the_commit(self):
|
||||
self.write(".github/workflows/ci.yml", self.UNPINNED)
|
||||
self.write("Dockerfile", "FROM node:24\n")
|
||||
self.write(".gitignore", "b\na\n")
|
||||
self.write("NOTES.md", "Co-Authored-" + "By: Claude <noreply@anthropic.com>\n")
|
||||
ran = self.commit()
|
||||
self.assertEqual(ran.returncode, 2)
|
||||
for rule in ["action-pin", "image-pin", "ignorefile-sorted", "no-ai-attribution"]:
|
||||
with self.subTest(rule):
|
||||
self.assertIn(f"Blocked by rule {rule}: ", ran.stderr)
|
||||
|
||||
def test_a_staged_file_is_checked_and_a_clean_tree_passes(self):
|
||||
self.assertEqual(self.commit().returncode, 0)
|
||||
self.write(".github/workflows/ci.yml", self.UNPINNED)
|
||||
git(self.repo, "add", "-A")
|
||||
self.assertEqual(self.commit("git commit -m x").returncode, 2)
|
||||
|
||||
def test_a_breach_the_head_already_holds_is_left_alone(self):
|
||||
self.write("Dockerfile", "FROM node:24\n")
|
||||
self.write(".gitignore", "b\na\n")
|
||||
git(self.repo, "add", "-A")
|
||||
git(self.repo, "commit", "-qm", "x", "--no-verify")
|
||||
self.write("Dockerfile", "FROM node:24\nRUN true\n")
|
||||
self.write(".gitignore", "b\na\n\n# more\nc\n")
|
||||
self.assertEqual(self.commit().returncode, 0)
|
||||
self.write("Dockerfile", "FROM node:24\nRUN true\nFROM node:25\n")
|
||||
self.assertIn("image-pin: ", self.commit().stderr)
|
||||
|
||||
def test_a_marker_in_the_command_or_the_file_lets_the_commit_through(self):
|
||||
self.write("Dockerfile", "FROM vendor/tool:2\n")
|
||||
self.assertEqual(self.commit("git add -A && git commit -m x # rule-override image-pin: the vendor tags no patch versions").returncode, 0)
|
||||
self.write("Dockerfile", "# rule-override image-pin: the vendor tags no patch versions\nFROM vendor/tool:2\n")
|
||||
self.assertEqual(self.commit().returncode, 0)
|
||||
|
||||
def test_a_staged_breach_blocks_the_commit_after_the_work_tree_is_fixed(self):
|
||||
self.write("Dockerfile", "FROM node:24\n")
|
||||
git(self.repo, "add", "Dockerfile")
|
||||
self.write("Dockerfile", "FROM node:24.18.1\n")
|
||||
self.assertIn("image-pin: ", self.commit("git commit -m x").stderr)
|
||||
|
||||
def test_a_stat_dirty_file_runs_no_clean_filter(self):
|
||||
self.write("a.txt", "a\n")
|
||||
self.write(".gitattributes", "a.txt filter=planted\n")
|
||||
git(self.repo, "add", "-A")
|
||||
git(self.repo, "commit", "-qm", "x", "--no-verify")
|
||||
git(self.repo, "config", "filter.planted.clean", f"touch {self.repo}/ran; cat")
|
||||
pathlib.Path(self.repo, "ran").unlink(missing_ok=True)
|
||||
os.utime(pathlib.Path(self.repo, "a.txt"), (1, 1))
|
||||
self.assertEqual(self.commit("git commit -m x").returncode, 0)
|
||||
self.assertFalse(pathlib.Path(self.repo, "ran").exists())
|
||||
|
||||
def test_a_submodule_runs_no_clean_filter(self):
|
||||
inner = tempfile.mkdtemp()
|
||||
self.addCleanup(subprocess.run, ["rm", "-rf", inner])
|
||||
git(inner, "init", "-q", "-b", "main")
|
||||
pathlib.Path(inner, "b.txt").write_text("b\n")
|
||||
pathlib.Path(inner, ".gitattributes").write_text("b.txt filter=planted\n")
|
||||
git(inner, "add", "-A")
|
||||
git(inner, "commit", "-qm", "x", "--no-verify")
|
||||
git(self.repo, "-c", "protocol.file.allow=always", "submodule", "add", "-q", inner, "sub")
|
||||
git(self.repo, "commit", "-qm", "x", "--no-verify")
|
||||
git(f"{self.repo}/sub", "config", "filter.planted.clean", f"touch {self.repo}/ran; cat")
|
||||
os.utime(pathlib.Path(self.repo, "sub", "b.txt"), (1, 1))
|
||||
self.assertEqual(self.commit("git commit -m x").returncode, 0)
|
||||
self.assertFalse(pathlib.Path(self.repo, "ran").exists())
|
||||
|
||||
def test_a_checkout_with_no_commit_checks_its_index_and_untracked_files(self):
|
||||
fresh = tempfile.mkdtemp()
|
||||
self.addCleanup(subprocess.run, ["rm", "-rf", fresh])
|
||||
git(fresh, "init", "-q", "-b", "main")
|
||||
pathlib.Path(fresh, "Dockerfile").write_text("FROM node:24\n")
|
||||
git(fresh, "add", "Dockerfile")
|
||||
pathlib.Path(fresh, "NOTES.md").write_text(TRAILER + "\n")
|
||||
ran = hook({"cwd": fresh, "tool_input": {"command": "git commit -m x"}, "tool_name": "Bash"}, fresh)
|
||||
for rule in ["image-pin", "no-ai-attribution"]:
|
||||
self.assertIn(f"Blocked by rule {rule}: ", ran.stderr)
|
||||
|
||||
def test_a_change_that_only_reorders_an_ignore_file_is_checked(self):
|
||||
self.write(".gitignore", "a\nb\nc\n")
|
||||
git(self.repo, "add", "-A")
|
||||
git(self.repo, "commit", "-qm", "x", "--no-verify")
|
||||
self.write(".gitignore", "c\nb\na\n")
|
||||
self.assertIn("ignorefile-sorted: ", self.commit().stderr)
|
||||
|
||||
def test_a_marker_the_head_already_holds_does_not_override_a_new_breach(self):
|
||||
self.write("Dockerfile", "# rule-override image-pin: the vendor tags no patch versions\nFROM vendor/tool:2\n")
|
||||
git(self.repo, "add", "-A")
|
||||
git(self.repo, "commit", "-qm", "x", "--no-verify")
|
||||
self.write("Dockerfile", "# rule-override image-pin: the vendor tags no patch versions\nFROM vendor/tool:2\nFROM node:24\n")
|
||||
self.assertIn("image-pin: ", self.commit().stderr)
|
||||
|
||||
def test_a_marker_in_another_file_does_not_override_a_breach(self):
|
||||
self.write("Dockerfile", "FROM vendor/tool:2\n")
|
||||
self.write("NOTES.md", "# rule-override image-pin: quoted in a note\n")
|
||||
self.assertIn("Blocked by rule image-pin: ", self.commit().stderr)
|
||||
|
||||
def test_an_ignored_or_binary_file_is_left_alone(self):
|
||||
self.write(".gitignore", "build/\n")
|
||||
self.write("build/Dockerfile", "FROM node:24\n")
|
||||
pathlib.Path(self.repo, "image.png").write_bytes(b"\xff\xfe\x00FROM node:24\n")
|
||||
self.assertEqual(self.commit().returncode, 0)
|
||||
|
||||
|
||||
class RuleNames(unittest.TestCase):
|
||||
def test_every_rule_name_has_a_rule_and_every_rule_a_name(self):
|
||||
self.assertEqual(set(RULES), set(typing.get_args(RuleName)))
|
||||
|
||||
@@ -3435,3 +3435,96 @@ Inherited architect, decided by:
|
||||
- Locality, Shape: `bin/loop_claude/routing.py:84` `next_route`: each verdict kind has its own function, and `steps` and `below_cap` name the two conditions
|
||||
- Locality: `bin/model_routing/route.py:88` `route_of`: the archive line and the reload hand-over decode a Route through one decoder
|
||||
- Shape: `bin/loop_claude/loop/main.py:66` `next_cycle`: signatures of seven or eight positional parameters shrink to `(loop_run, loop)`
|
||||
|
||||
## 2026-11-10T23:11:34Z, PR #473 at 13d1550
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Junior A | 7 | 5.5 | 6 | 6 | 6 |
|
||||
| Mid A | 7 | 5.5 | 6 | 6.5 | 6 |
|
||||
| Maintainability senior | 7 | 5.5 | 6 | 6 | 6 |
|
||||
| Inherited architect | 7 | 5.5 | 6 | 7 | 6 |
|
||||
| Mean | 7.00 | 5.50 | 6.00 | 6.38 | 6.00 |
|
||||
|
||||
Junior A, hardest first:
|
||||
|
||||
- Locality, Shape: `bin/loop_claude/routing.py:93` `unfinished_route`: the floor rung, the escalation mark, the unfinished count and the cap are held at once, with escalation_mark.py open beside it
|
||||
- Locality, Shape: `bin/larv_review/plan.py:66` `plan_reasons`: eight failure branches mix git reads, file reads and logic, and a parameter named `default` receives a merge-base commit
|
||||
- Locality: `bin/larv_review/owed.py:37` `shortfall`: whether one or two runs are owed hangs on the wording of an English sentence that SKILL.md also keys on
|
||||
- Shape: `bin/loop_claude/cycle/run.py:59` `run_cycle`: four near-identical archive helpers and a route replaced partway through the loop
|
||||
- Shape: `bin/release_plan/ranking.py:56` `rank`: nearly every function forks on the table's two shapes
|
||||
- Self-sufficiency: `skills/larv-review/SKILL.md:147` "Scoring run": dense rule prose with many cross-references, three sections to rebuild one rule
|
||||
|
||||
Mid A, hardest first:
|
||||
|
||||
- Locality, Shape: `bin/larv_review/plan.py:66` `plan_reasons`: `default` holds a merge-base commit on one call path and `after` means the checkout's text, then its items
|
||||
- Locality: `bin/loop_claude/routing.py:93` `unfinished_route`: six concepts meet in about twelve lines and need four modules open beside it
|
||||
- Locality: `bin/larv_review/owed.py:37` `shortfall`: a rule keyed on the text of a sentence that plan.py builds, and `scoring_runs` defines a scoring run differently from `Entry.scored`
|
||||
- Locality: `bin/loop_claude/cycle/judge.py:211` `quotes_checked`: a verdict is rewritten four times, half the behaviour in prompt prose and half in code
|
||||
- Shape: `bin/release_plan/ranking.py:304` `score`: every function branches on the old and the current table shape
|
||||
|
||||
Maintainability senior, hardest first:
|
||||
|
||||
- Locality, Shape: `bin/larv_review/plan.py:66` `plan_reasons`: a merge-base commit enters the parameter named for the default branch, and its error text would print the commit as a branch
|
||||
- Locality: `bin/larv_review/owed.py:37` `shortfall`: the one-run-or-two rule is spread over three files and a skill document
|
||||
- Locality, Shape: `bin/loop_claude/routing.py:93` `unfinished_route`: the six inputs interlock and a wrong answer runs the worker on the wrong rung without failing
|
||||
- Locality: `bin/model_routing/route.py:88` `route_of`: a route has two serializations, a flat archive line and a nested handover
|
||||
- Self-sufficiency: `bin/larv_review/plan.py:66` `plan_reasons`: its docstring doesn't say the first argument may be a commit
|
||||
|
||||
Inherited architect, would restructure first:
|
||||
|
||||
- Navigation, Shape: `bin/larv_review/owed.py:10` `owed_scoring_run`: the name gives the merge gate's question, not the module's subject, and the scoring-run logic sits in three files whose imports read as circular
|
||||
- Locality: `bin/larv_review/plan.py:14` `EMPTIED_WORDS`: prose and code share an exact string that SKILL.md keys on
|
||||
- Locality: `bin/loop_claude/routing.py:93` `unfinished_route`: routing logic spreads over about eight places, so a new trigger touches five files in three packages
|
||||
- Shape: `bin/rule_hook/rules.py:25` `RULES`: rules sit inline in one module while five others have a module each
|
||||
- Navigation: `bin/` README.md:116 "bin/": 56 entries in one flat list, the worst fan-out level
|
||||
|
||||
## 2026-11-11T12:55:06Z, PR #475 at 52da15e
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Junior A | 7 | 5.5 | 6 | 6 | 6 |
|
||||
| Mid A | 7.5 | 6 | 6.5 | 6.5 | 6.5 |
|
||||
| Maintainability senior | 7 | 6 | 6.5 | 6 | 6.5 |
|
||||
| Inherited architect | 7 | 6.5 | 6 | 7 | 6.5 |
|
||||
| Mean | 7.13 | 6.00 | 6.25 | 6.38 | 6.38 |
|
||||
|
||||
Junior A, hardest first:
|
||||
|
||||
- Locality, Shape: `bin/loop_claude/routing.py:93` `unfinished_route`: the floor, the escalation cap, the escalation mark, the unfinished count and the taken-in E + I are held at once
|
||||
- Locality: `bin/loop_claude/cycle/judge.py:211` `quotes_checked`: the judge's contract is split over prose constants, schema descriptions and post-hoc checks that change a verdict's type
|
||||
- Locality: `bin/loop_claude/cycle/run.py:146` `rule`: archive order carries meaning across `run_cycle` and `loop/close.py:18` `closed_cycle`
|
||||
- Shape: `bin/release_plan/release_standing.py:145` `next_release`: tag, changelog, before-first-release and renamed branches meet in the `is_earliest` expression
|
||||
- Shape: `bin/release_plan/ranking.py:56` `rank`: `OLD_SHAPE` and `SHAPE` forks run through nearly every function
|
||||
- Self-sufficiency: `bin/loop_claude/cycle/prompts.py:59` `review_launches`: the worker's prompt is built from about ten conditional sentence builders
|
||||
- Locality: `skills/larv-review/SKILL.md:76` "Round cap": the cap is enforced in four units in two places
|
||||
|
||||
Mid A, hardest first:
|
||||
|
||||
- Locality, Shape: `bin/loop_claude/routing.py:93` `unfinished_route`: five things applied in a fixed order across about six helpers; the reason for the floor and cap sits in the decision log
|
||||
- Locality: `bin/loop_claude/cycle/run.py:146` `rule`: the order of ruling, rejection and step-up archive writes is stated piecemeal
|
||||
- Shape: `bin/release_plan/ranking.py:315` `weight_of`: old-shape branches double the module, and the constants are magic until rules/todo-items.md is open
|
||||
- Shape: `bin/release_plan/release_standing.py:145` `next_release`: a three-way boolean on line 167 with no comment for the `None` and `nothing_cut` case
|
||||
- Locality: `bin/rule_hook/repeat_read.py:119` `shown_lines`: rebuilds which lines are in the agent's context from transcript entries
|
||||
- Locality: `bin/loop_claude/cycle/judge.py:211` `quotes_checked`: half of the judge's rules are English constants edited in step with Python
|
||||
- Locality: `bin/loop_claude/cycle/prompts.py:119` `escalation`: about 20 sentence builders restate the routing logic
|
||||
|
||||
Maintainability senior, hardest first:
|
||||
|
||||
- Shape, Locality: `bin/release_plan/release_standing.py:145` `next_release`: four fallbacks held at once, and `is_earliest` took three reads
|
||||
- Locality: `bin/loop_claude/routing.py:93` `unfinished_route`: the escalation mark's meaning lives in `escalation_mark.py`, which has to stay open beside it
|
||||
- Locality: `bin/loop_claude/cycle/judge.py:211` `quotes_checked`: a bad quote silently changes the verdict's type
|
||||
- Shape: `bin/release_plan/ranking.py:304` `score`: every path forks on `SHAPE` versus `OLD_SHAPE`
|
||||
- Locality: `bin/rule_hook/reviewer_rung.py:40` `skill_breaches`: three cases rest on assumptions made in another function
|
||||
- Locality: `bin/loop_claude/cycle/run.py:146` `rule`: order-dependent archive writes that nothing enforces
|
||||
- Shape: `bin/review_rules/comprehension_history.py:127` `entries`: a line-by-line parser that tolerates two legacy formats
|
||||
- Self-sufficiency: `skills/larv-review/SKILL.md:76` "Round cap": dense cross-referencing rules, with the enforcing code in other modules
|
||||
|
||||
Inherited architect, would restructure first:
|
||||
|
||||
- Navigation, Shape: `README.md:116` `bin/`: 55 entries at one level, about 31 packages, the worst fan-out level
|
||||
- Locality: `bin/larv_review/reviewer_header.py:49` `unanswered_round`: the round cap counts answer commits, so a worker that relaunches without committing gets the same round number, and the rule is split over SKILL.md and three modules
|
||||
- Locality, Shape: `bin/loop_claude/routing.py:93` `unfinished_route`: seven counters and limits across five files, with field names frozen by the reload handover
|
||||
- Locality: `bin/chunk_pr/main.py:42` `forge`: the forge helpers live in `chunk_preflight` and `deploy_release`, so a second forge needs edits in four packages
|
||||
- Shape: `bin/records/__init__.py:1` `records`: one name for three concepts, as `cut` and `Plan` also name several
|
||||
- Locality: `bin/loop_claude/cycle/run.py:86` `archived`: the archive helpers that `loop/close.py` imports sit in the cycle runner
|
||||
|
||||
+74
-10
@@ -1,3 +1,49 @@
|
||||
# Project principles
|
||||
|
||||
Each project principle states one rule that the decisions below apply; their entries hold the specifics and the evidence. The numbers name the principles, and don't rank them.
|
||||
|
||||
1. **Cut cost only where a measurement of archived chunks shows quality holds.** Chunk bundling, the round cap, which fixes rerun a pass, a reviewer of its own for each pass, the review a branch changing only Markdown no agent follows gets, the passes kept for the release review, review opt-outs, the security pass, compaction and the routing ladder all apply it.
|
||||
2. **A measurement files or ranks work, and never blocks a merge.** The comprehension floor and check, and a reading-cost rise, apply it. A merge waits on a scoring run the plan calls for only until the run is recorded, whatever it scores.
|
||||
3. **Each file has one writer.** `~/.claude/loop-claude-models.json` is loop-claude's, `~/.claude.json` is Claude Code's, and `~/.claude` moves only through `deploy-release`. The exception is `~/.claude/settings.json`, which Claude Code writes from any session. Git's clean filter `bin/settings_filter.py` keeps each machine's model and Claude Code's key order out of every commit, and `README.md` → "Update" carries its other changes into this repo.
|
||||
4. **A rule a program can check is enforced by a program, and reviewers rule on the rest.** rule-hook and its overrides, `docker-run-memory`, the refused writes of reviewers under `agents/`, and CI with `main`'s branch protection apply it.
|
||||
5. **A chunk can't vouch for itself.** What a chunk writes is approved by a review pass, by the human, or by a precedent the repo held before the chunk began. So an unanswered ask leaves the PR unmerged, the human approves the premises chunk and runs `deploy-release --cut`, and the judge holds a chunk open while an answer the human typed isn't recorded anywhere.
|
||||
6. **An unattended run never waits on a prompt, and a lasting failure stops it.** Every claude loop-claude starts bypasses permissions inside its sandbox, each claude it or loop-inbox opens on a terminal has `CLAUDE_CODE_SANDBOXED` set, and interactive sessions run in auto mode with the dangerous-mode prompt skipped. A failed archive stops the loop, while a `claude` that cannot start, as during an update, is retried.
|
||||
7. **One word names one thing, and one rule has one home.** In `bin/`, an outcome is an entry of a `.outcomes.md` file in loop-claude's archive, a record is a document tools or people keep entries in, and a place is what a comprehension seat names. `CLAUDE.md` holds only the rules no path can scope, and each `rules/` file loads by its `paths:`.
|
||||
|
||||
# Decisions no project principle covers
|
||||
|
||||
- bin/: the tools run on the host's Python, with no container, and so do their tests outside CI
|
||||
- bin/: the tools support only the latest stable Python, and the tests, written in plain unittest, run on it in four pytest-xdist workers
|
||||
- deploy: this repo keeps no `AGENTS.md`, since `~/.claude` is its checkout; its principles and decision titles head `docs/decisions.md`, its working rules follow them there, and its `.claude/settings.json` excludes every `.claude/CLAUDE.md`, so a session in the clone loads only the clone's `CLAUDE.md`
|
||||
- loop-claude: one session per project, its id derived from the checkout's path relative to `~/code`, or to `/` outside it
|
||||
- loop-inbox: a loop-claude lists itself while it runs, and its waiting question, as files per session under `~/.local/state/loop-claude/`, and takes the first answer, from its terminal or from loop-inbox, that names that question
|
||||
- machine load: a heavy command waits for one of the machine's heavy slots, one for every 4 CPUs, which every session shares, and loop-claude starts a worker only while at least a quarter of the machine's memory is available
|
||||
- principles: about seven things per level, four or five where they act on each other, more where a reader finds them by name; crossing a limit asks for a grouping or a recorded reason
|
||||
- review: a release review fixes its findings in the release it reviews, and files under that release only a finding that needs a human answer, adds a feature or outlasts its five rounds
|
||||
- review: a reviewer or comprehension seat that rules the work below standard may ask to rerun once, one rung up, in the same round; the ruling that stands is an escalation trigger for the worker unless the fix changes only prose, and never for the judge
|
||||
- review: a reviewer under `agents/` marks a 🔴 or 🟡 `[misread]` where the implementer misread code that had already shipped, each round title counts the marked findings, and `score-outcomes` lists them per chunk
|
||||
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
|
||||
- review: mutate-changes mutation-tests the lines a chunk changes before review, in place of measuring test coverage
|
||||
- routing: a worker delegates a task only where it needs just the task's conclusion, and each subagent runs on the rung its task's L picks from the project's rung, never below the project's rung when it files or scores todo.md items
|
||||
- routing: a worker hands a task one rung up on each escalation trigger, up to two rungs above the rung its item picks unless the human agrees, and opens the subagent's prompt with the trigger so the routing figures count it
|
||||
- routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, steps one rung up as the judge interrupts or quotes an answer, or next cycle, up to two rungs above the rung its item picks
|
||||
- routing: loop-claude reads the items a chunk takes from the worker's latest `Chunk items:` line and scores them by `todo.md` as the default branch held it when the cycle started; the judge reports the escalation and whether a release is ready to cut
|
||||
- routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds
|
||||
- tests: each test earns its keep as the technical principles' test bullet says, and the worker takes a test's expected values from its item, never from the code
|
||||
- the repo may name bender, bender-infra and ai-sandbox
|
||||
- todo.md: one table of items, scored by the formula it states, held to its bar and ranked by `bin/rank_todo.py`, which ranks each blocker above the items it blocks; each achieve goal weighs by its position, code health first and four times as heavy while the comprehension panel's overall is under 7.0 or unmeasured, then README.md's goals, and a maintain goal weighs 0.5 per point of the confidence an item's test, gate leg or probe adds to it; the table's order is the plan's only order
|
||||
- todo.md: only a session or subagent on the project's rung or above scores or reshapes the plan; one below it drafts new items and changes to filed ones for a subagent at or above that rung to apply and score, and removes finished ones
|
||||
- worker: a chunk's worker writes the simplest change that solves its item, test first wherever a test can check the change, and reads a technical principle only where the item, a finding or its own record names it; larv-review's reviewers hold the change to the principles and to the standards in `skills/larv-review/standards/`
|
||||
- workflow-bench/: the remote may show the bench cases' names and commit SHAs
|
||||
|
||||
# Working rules
|
||||
|
||||
**Routing calls: file them in the plan.** A routing call is a change to the routing ladder, its steps or the delegation rule that loop-claude's figures call for. loop-claude puts each call to the human once, and appends it to `routing-calls-raised.md`, at the root of its archive (`$LOOP_CLAUDE_TRANSCRIPTS`, default `~/.claude/loop-claude-transcripts`). Each line of that file is a call or a `Reset`, with a stamp. A call is new when it is stamped after the latest `Reset` and after the latest stamp that `todo.md` or `docs/decisions.md` → `routing:` cites.
|
||||
|
||||
When there are new calls, file one `todo.md` item to settle them, or refresh the item already filed. Its detail quotes the new calls with their stamps. It also quotes the routed-chunks table of `routing-figures.md`, which sits in the same folder, and none of that file's other tables, since they name other projects.
|
||||
|
||||
Settling a call is the human's. The chunk that settles the calls cites, in `docs/decisions.md` → `routing:`, the latest stamp it settled, whatever the human decided, so the calls are not filed again. Once a chunk that changes the ladder, its steps or the delegation rule has merged, append `- <UTC time> Reset` to `routing-calls-raised.md`. The figures then count only chunks finished after the reset, and a call settled before it may be raised again.
|
||||
|
||||
# Decisions
|
||||
|
||||
## bin/: an outcome is an entry of a `.outcomes.md` file in loop-claude's archive, a record is a document tools or people keep entries in, and a place is what a comprehension seat names
|
||||
@@ -24,7 +70,7 @@ Valid while the archive's file names, the Vocabulary and the panel keep these wo
|
||||
|
||||
The cap is 5, `bin/release_plan/todo_table.py`'s `BUNDLE_MAX_R`, from the same chunks: by the highest R among a chunk's items, the median was 0.9M to 1.5M tokens at R 1 to 5, and 2.7M at R 6, where the 90th percentile was 8.8M. A risky item that runs long re-runs review over every item bundled with it. R 5 is also the least R an I 4 item gets, so an item that could take everything down shares a chunk only where its L + E is 3 or less, and an irreversible I 5 item, at R 7 or more, never does. Valid while a review's cost is mostly fixed per chunk; revisit once 30 chunks have shipped bundles under it.
|
||||
|
||||
## comprehension floor: every dimension and the overall at 7.0 or above is the target, and blocks no merge or release
|
||||
## comprehension floor: every dimension and the overall at 7.0 or above is the target, and no score blocks a merge or release
|
||||
|
||||
2026-09-27, lilleman, restating the 2026-09-24 target and extending it to the overall. Serves goal 1. 2026-10-13, lilleman: the floor blocks nothing, and `review: the comprehension check records …` says why. Valid while the scoring run keeps its four seats.
|
||||
|
||||
@@ -32,6 +78,12 @@ The cap is 5, `bin/release_plan/todo_table.py`'s `BUNDLE_MAX_R`, from the same c
|
||||
|
||||
2026-10-29, lilleman. Serves goal 1.3 and goal 2. `~/.claude` was this repo's development checkout, so every merge reached every session on the machine at once, its unfinished work included, while other machines took only tags. Now each machine takes only releases: what a release holds is what every session runs, and cutting one is the human's step that ships it. `deploy-release` automates README → Update. It refuses an uncommitted change to a tracked file that the new tag doesn't hold, such as a hook Claude Code wrote into `settings.json`, and a move that would drop commits no release holds; a model pick, which the clean filter drops, is no change. Valid while Claude Code reads its configuration from one directory per machine.
|
||||
|
||||
## deploy: this repo keeps no `AGENTS.md`, since `~/.claude` is its checkout; its principles and decision titles head `docs/decisions.md`, its working rules follow them there, and its `.claude/settings.json` excludes every `.claude/CLAUDE.md`, so a session in the clone loads only the clone's `CLAUDE.md`
|
||||
|
||||
2026-11-11, lilleman through item 314. Serves goal 1.4. Claude Code loaded `~/.claude/AGENTS.md` into every other project's sessions, and each 1,000 tokens every thread starts with cost $68.60 of the $9,154 spent from 2026-09-30 to 2026-11-11: the 14.4 KB file cost about 2.1% of spend, and put this repo's rules in front of other projects. Other projects keep the name `AGENTS.md`, so the setup stays portable to other harnesses. The working rules sit there too, rather than in `.claude/rules/`, since loop-claude's judge reads the decision log and no project's `.claude/rules/`. In the clone, the release's `~/.claude/CLAUDE.md` and the clone's both loaded, about 0.9% of spend; the exclude drops the release's, so a chunk runs under the rules it is changing. A session working in `~/.claude` itself loads no `CLAUDE.md`, since there both are the same file; none works there, as the entry above says.
|
||||
|
||||
Valid while `~/.claude` is a checkout of this repo.
|
||||
|
||||
## deploy: `deploy-release --cut` cuts a release through a pull request merged fast-forward only, then tags the merged commit and moves `~/.claude` to it; the human runs it
|
||||
|
||||
2026-11-08, lilleman, todo item 271. Serves goal 1.5 and goal 2. Nobody can push to `main` on the remote, and it merges only by fast-forward, so a cut that commits on `main` and pushes no longer works. The tool cuts only the release `chunk-preflight` would call ready to cut, through `bin/release_plan/release_standing.py`, so the two never disagree on which release is next or whether it is ready; the version is that release's, never the tool's choice. It builds the cut commit in a worktree of its own, so the clone's checkout stays as it is. It finds the merged commit on `main` by its title, `Cut release <version>`, since GitHub's nearest merge, a rebase merge, may give the commit a new hash. It takes only a pull request from origin's own branch whose head is the commit that branch points at, and has the forge merge it only while its head is still that commit, so a fork's pull request from a branch of the same name, or a push after the checks passed, is never merged and deployed. Each step first checks whether an earlier run did it, so a rerun continues. Releases stay the human's, so the human runs it, and the forge credentials stay theirs. Valid while `main` takes changes only through a pull request.
|
||||
@@ -103,7 +155,7 @@ The trials: four stability rounds and five principles rounds archived on 2026-10
|
||||
- A cap on tool results. Claude Code caps a Bash result at about 30k characters already. A PostToolUse hook capping it at 12k, which `updatedToolOutput` allows, left the stability reviewers 3 of 6 findings, two lost in one round. Read results, 59.9M characters in subagents' threads with 58% of their characters in results over 12k, are out of such a hook's reach. Not worth an item.
|
||||
- A scout: Haiku 5.5 reading raw content, whole files or long output, and returning facts as path:line and the ranges worth reading, so the agent's own context holds the summary and the chosen lines. In session, as an agent under `agents/`, its first request writes 37k tokens, $0.005, and a read of a 38KB file costs $0.008 in 8 seconds. As a slim headless `claude -p` with its own system prompt, no setting sources and no dynamic sections, which keeps the subscription login, it starts at 6k tokens and $0.0003 in 3 to 5 seconds, its prefix cached across calls, and the same read costs $0.004 in 7 seconds; a direct API call and `claude --bare` need an API key, so they are out. Stability reviewers with the agent scout found 5 of 6 findings and read 29% fewer characters at 6% less cost and 14% more time; with the CLI scout 4 of 6, 38% fewer, 17% less and 39% more; the auditors with the agent scout 3 of 6, the 🔴 among the lost, at the same cost and twice the time. Workers offered the scout never called it.
|
||||
- What a reviewer reads: the auditors read the project whole by design. Of their 🔴 and 🟡 findings since the reset, 47% named a file the chunk edited and 51% another file, so reading only the change would drop half of them (`review: the principles audit runs only in the release review, and reads the project whole with one auditor checking the goals and every principles file that applies`).
|
||||
- The base context: every thread starts with `~/.claude/CLAUDE.md` (19k characters), `~/.claude/AGENTS.md` (11k, this repo's decisions, which Claude Code loads as user instructions into every project's sessions), the project's `AGENTS.md`, and, for the worker and general-purpose subagents, the skill listing (18k) and the agent listing (12k). Subagents' first messages wrote $97 of it. Trimming the listings would save about $20, 2%, and is not worth an item: `disableBundledSkills` would also drop `/security-review`, the security pass of record.
|
||||
- The base context: every thread starts with `~/.claude/CLAUDE.md` (19k characters), `~/.claude/AGENTS.md` (11k, this repo's decisions, which Claude Code loaded as user instructions into every project's sessions until 2026-11-11, when they moved out of `~/.claude`), the project's `AGENTS.md`, and, for the worker and general-purpose subagents, the skill listing (18k) and the agent listing (12k). Subagents' first messages wrote $97 of it. Trimming the listings would save about $20, 2%, and is not worth an item: `disableBundledSkills` would also drop `/security-review`, the security pass of record.
|
||||
|
||||
Quality. Under the 200k window, every figure's mean per chunk fell. Measured on 2026-11-04, the first 20 chunks whose worker started under it, from 2026-10-28, took a mean 4.0 review rounds past each pass's first against 5.8 for the 20 before them, from 2026-10-23 to 2026-10-26; 0.20 unfinished verdicts against 0.30; 1.15 🔴 against 1.90; 11.9 🟡 against 19.4; 0.11 misread findings against 0.25; $22 against $34; and 9.6 hours against 42. Only the median dollars went up, $15.17 against $11.72: the 20 before held four chunks of $94 to $167, and the 20 after none above $76. This decision weighs all spend, so the mean is the figure that counts. The two groups also differ in projects and routing, so how much of the fall the window caused is unknown. Before the window, 19 chunks compacted between 2026-10-10 and 2026-10-22, at the compaction points Claude Code set then, 214k to 972k tokens. Per 100 worker messages they took a median 3.6 review rounds past each pass's first against 4.0 for the 68 uncompacted chunks of 100 messages or more, and 13.9 🔴 and 🟡 findings against 16.6. 4 of 19 had an unfinished verdict against 3 of 68, each 8 to 10 hours after the compaction and followed by a finished verdict within minutes; in those four verdicts the judge quoted the worker's own passages and the human's typed answers, never a summary. One cause was possible and is closed here: Claude Code writes the summary as a user entry, which `bin/claude_code/entries.py` read as the human's words, so the judge would see a `[human]` line of thousands of words and `/talk` would count it as typed; it now reads as Claude Code's. Two summaries read whole kept the chunk's intent, files, errors and next step in 12k to 20k characters, and the worker resumed without re-reading.
|
||||
|
||||
@@ -120,7 +172,7 @@ Valid while cache reads cost a twentieth of input and Claude Code's compaction k
|
||||
- loop-inbox runs the claude command the loop wrote, `bypassPermissions` included, so that claude runs in the same sandbox as the loop. Since loop-inbox runs a command read from a file, the folder is private to the user.
|
||||
- Files, read by polling, need no daemon and only the standard library, and a loop with no loop-inbox running is answered on its terminal as before.
|
||||
- loop-inbox lists a question only while its loop holds the session's lock file with flock, and removes a question whose lock is free. The loop holds the lock for its whole run. The kernel frees the lock however the loop ends, killed included. A process id could not tell a running loop from a killed one: on 2026-10-29 this machine's kernel handed out its 32,768 ids again every 30 to 75 minutes, so a killed loop's question came back whenever another process got its id. The lock file stays once the loop ends, one per project: removing it while another process has it open could let loop-inbox lock a file no loop holds. PR #396.
|
||||
- A loop also lists itself as running, with the items its chunk takes, in `<session>.running`, under the same lock: one lock tells loop-inbox both that the loop runs and that its question stands. The judge names the items the worker took in, as an array of IDs in its reply's `items_taken_in`. PR #397.
|
||||
- A loop also lists itself as running, with the items its chunk takes, in `<session>.running`, under the same lock: one lock tells loop-inbox both that the loop runs and that its question stands. The items are the one the chunk was routed on, then those the worker names on its latest `Chunk items:` line (`routing: loop-claude reads the items a chunk takes`).
|
||||
|
||||
A bench of three readers seated as the README's audience picked the name `loop-inbox`, 3–0.
|
||||
|
||||
@@ -254,7 +306,7 @@ Valid while Claude Code passes `agent_type` to hooks inside a subagent. Without
|
||||
|
||||
2026-10-28, lilleman: prose findings stay, since Claude's prose needs them, though prose's 105 first rounds since 2026-10-21 raised 498 🟡 and only 3 🔴.
|
||||
|
||||
## review: the comprehension check records how hard a project is to read and blocks no merge: a paired ruling against the last measured commit in the release review, and the four-seat scoring run, once when an item names the floor and twice on one head when a release's items run out
|
||||
## review: the comprehension check records how hard a project is to read, and no score blocks a merge: a paired ruling against the last measured commit in the release review, and the four-seat scoring run, once when an item names the floor and twice on one head when a release's items run out
|
||||
|
||||
2026-10-08, lilleman. Serves goals 1.1 and 1.4. Two independent four-seat runs moved up to 0.4 on near-identical code, so a fall within that noise forced rounds that changed nothing (PR #224). A seat reading both versions judges the difference on the same material, so the noise the versions share cancels. No conventional significance level lets a sign test settle with four votes, so the human chose a lead of two votes over every other ruling, same included. Two seats that agree settle it, so a PR that leaves the reading as it was costs two seats. A third seat alone can never settle it, so the third and fourth launch together. The scoring run also runs on a chunk shipping an item that names the floor, so the chunk that lifts a project to the floor can prove it. A chunk that removes a release's last item without shipping it runs the scoring run too, so no release ends unmeasured, unless `todo.md` marks that release renamed in a `## Renamed` section (`bin/release_plan/todo_table.py` → `renames`). Valid while seats reading both versions agree more often than two absolute runs do.
|
||||
|
||||
@@ -264,6 +316,8 @@ The base is the last measured commit: the default branch's newest merged change
|
||||
|
||||
2026-10-17, lilleman. Serves goal 1.1. Nothing checked the 2026-10-08 validity condition. The history's successive scoring runs of one PR read changed code, not near-identical code, so they cannot stand for two absolute runs. So each release's scoring run is done twice on the same head, the second recorded under `<PR> rerun at <commit>`, and `larv-review agreement` compares two rates: how often Mid A and the inherited architect vote alike overall in paired rulings, and how often their overall scores move alike between two scoring runs on one commit. It gives no verdict until the history holds 10 such pairs of runs, as `score-outcomes` waits for 10 chunks before it shows a median. The human chose the second run's cost, four more seats per release (goal 1.4), over a weaker check on the old pairs.
|
||||
|
||||
2026-11-11, lilleman. Serves goal 1.1. Item 296 made `chunk-pr merge` refuse while a scoring run the plan calls is unrecorded on the branch, so a missing run can block a merge. No score or ruling does, and the titles say so: "no score blocks a merge".
|
||||
|
||||
## review: the premises chunk's whole review is the human's approval
|
||||
|
||||
2026-09-20, lilleman. Serves goal 1.1. A reviewer's finding on that chunk could only be declined by citing the goals the chunk exists to write. Valid while every decline must name a goal or a technical principle.
|
||||
@@ -304,7 +358,7 @@ Only chunks run below the project's rung can prove that a cheaper worker holds q
|
||||
|
||||
While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers never run below the worker's rung while it runs above the project's: a worker climbs because its chunk proved harder than its score said, which is when a cheaper reviewer is the bigger risk. A reviewer or seat that rules the work below standard may rerun one rung up: `review: a reviewer or comprehension seat that rules the work below standard`.
|
||||
|
||||
On 2026-10-16 lilleman routed the reviewers on their item's E + I as well, on either side of the project's rung, which may lift them above both the worker's rung and the project's: how far a defect would escape, and what it would break, set how hard review must look (goal 1.1). loop-claude names that rung in the worker's prompt. It hands the E + I to the worker's claude beside the project's rung and the worker's, so rule-hook works out each review's rung as the prompt does and holds each launch to it. A low E + I never steps a reviewer below the project's rung: only a replay lowers a pass, as `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds…` records. An item the worker takes in beyond the routed one counts too, since its E + I scores its defects as the routed item's scores theirs. The worker's prompt names the reviewers' rung each higher E + I picks, so the worker reviews a taken-in item on it in the same run. The judge reports the highest E + I the worker states for a taken-in item. From the next cycle loop-claude routes on that E + I, and archives it with the route so a rerun keeps it.
|
||||
On 2026-10-16 lilleman routed the reviewers on their item's E + I as well, on either side of the project's rung, which may lift them above both the worker's rung and the project's: how far a defect would escape, and what it would break, set how hard review must look (goal 1.1). loop-claude names that rung in the worker's prompt. It hands the E + I to the worker's claude beside the project's rung and the worker's, so rule-hook works out each review's rung as the prompt does and holds each launch to it. A low E + I never steps a reviewer below the project's rung: only a replay lowers a pass, as `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds…` records. An item the worker takes in beyond the routed one counts too, since its E + I scores its defects as the routed item's scores theirs. The worker's prompt names the reviewers' rung each higher E + I picks, so the worker reviews a taken-in item on it in the same run. loop-claude scores each item the worker names on its `Chunk items:` line by `todo.md` (`routing: loop-claude reads the items a chunk takes`), routes on their highest E + I from the next cycle, and archives it with the route so a rerun keeps it.
|
||||
|
||||
Valid until the trial on real chunks shows whether a worker below the project's rung holds quality.
|
||||
|
||||
@@ -356,7 +410,7 @@ A run that asks for a higher rung may also ask a question. The judge then interr
|
||||
|
||||
The judge may quote a different passage for the trigger that already stepped the rung, and loop-claude cannot tell from the quote alone whether the trigger is new. So it goes by where the quoted passage sits in the transcript. When an escalation steps the worker, the route marks how many transcript entries the session held: the mark. loop-claude then accepts an escalation quote only from the worker's own lines after the mark, never from its own prompts or the human's lines, which repeat earlier passages. The judge reads a `[step]` line at the mark saying the triggers before it are spent, so it quotes a passage the check accepts. The session's transcript only grows, so the mark holds across cycles. The trigger that bought a step, and any rewording of it before the mark, cannot buy another. A trigger that fires after the step steps the worker again, in the same cycle or a later one. One risk remains: a worker on the stepped rung may restate an old trigger after the mark, and the judge may quote that. The cap two rungs above the rung the item picks limits what this costs. Before the mark, once an interrupt or a quoted answer had taken an escalation, no later ruling of that cycle carried one, so a trigger that fired after the answer was lost. lilleman chose the mark on 2026-10-18 over two alternatives: lifting that rule when a worker run ended would have let one trigger buy two steps, and telling the worker its request was dropped would still have lost the step.
|
||||
|
||||
The step is one rung up from the rung the worker would otherwise hold: its own, or the rung an item the judge saw it take in picks, whichever is higher. It resets the count of unfinished verdicts, so the step every second one takes counts afresh; when both steps fall on one verdict, the worker steps once. It stops at the limit escalated subagents stop at, two rungs above the rung the chunk started on. Past that limit the rung holds: loop-claude takes no yes from the human between cycles, so a worker there asks the human for an escalated subagent instead. The judge keeps its rung, as `routing: on a model judge-replay covered` says.
|
||||
The step is one rung up from the rung the worker would otherwise hold: its own, or the rung an item it names on its `Chunk items:` line picks, whichever is higher. It resets the count of unfinished verdicts, so the step every second one takes counts afresh; when both steps fall on one verdict, the worker steps once. It stops at the limit escalated subagents stop at, two rungs above the rung the chunk started on. Past that limit the rung holds: loop-claude takes no yes from the human between cycles, so a worker there asks the human for an escalated subagent instead. The judge keeps its rung, as `routing: on a model judge-replay covered` says.
|
||||
|
||||
The archive quotes the judge's passage under `Escalation:` in the unfinished verdict, the interrupt or the quoted answer. The step an unfinished verdict causes is archived after it. The step an interrupt or a quoted answer causes is archived with it, at once: the interrupt or answer names the rung the worker ran on until then, and a step-up archived beside it names the rung it runs on from then. So the routing figures charge each stretch to its own rung, and a loop that dies while the human answers resumes on the stepped rung. Each archived worker line names the mark once there is one, so a resumed loop keeps it. `bin/routing_figures/` counts none of these steps as an escalation by trigger, which it counts only from subagent prompts.
|
||||
|
||||
@@ -370,6 +424,16 @@ The cost of a cheaper worker shows as extra review rounds and unfinished verdict
|
||||
|
||||
Since 2026-10-16 the reviewers may run above the worker's rung, on the one the highest E + I among their chunk's items picks, so chunks whose worker ran on a lower rung may take more rounds from stronger reviewers, and a call could blame the worker's rung for them; `todo.md` item 171 groups the rounds by the reviewers' rung too. Valid while loop-claude's archive holds every finished chunk's review rounds and unfinished verdicts.
|
||||
|
||||
## routing: loop-claude reads the items a chunk takes from the worker's latest `Chunk items:` line and scores them by `todo.md` as the default branch held it when the cycle started; the judge reports the escalation and whether a release is ready to cut
|
||||
|
||||
2026-11-11, claude. Serves goal 1.4 and code health. The judge's reply held three facts beside its verdict: the items the worker took into the chunk, with their highest L and E + I; whether a release was ready to cut; and any escalation the worker asked for. Routing lifts the worker's, the reviewers' and the judge's rungs by the items' levels. Code now reads the items: each chunk's first prompt, and each prompt resuming it, asks the worker for a line opening `Chunk items:`, `bin/loop_claude/cycle/chunk_items.py` reads the latest such line in the worker's own text, and `release_plan.todo_table` scores the items. So the judge's reply is smaller, and every L and E + I that routing reads comes from `todo.md`, through one module.
|
||||
|
||||
The judge field trials (`judge-field-trials/`) read each fact by code over the 709 archived cases. Code was wrong in each of the 39 cases where it and the judge disagreed on an escalation, and read a release ready to cut in 3 of the 31 interrupts that asked about a cut, so the judge keeps both. Workers stated an item with its L and E + I in 413 cases, but also stated items they filed, finished or only mentioned, so the worker writes the items on a line that holds only them, which code reads. Three fresh readers picked its label `Chunk items:`, 3 to 0, because it says the items are the chunk's.
|
||||
|
||||
An item the chunk re-scores on its branch routes on its planned levels until the branch merges; where the new L picks a higher rung, the worker cites escalation trigger 1, which the judge finds. Where the worker writes no line, loop-claude counts no item as taken in, so the route holds. Rerun `judge-field-trials/release_ready.py` when the way `chunk-preflight` detects a release review changes.
|
||||
|
||||
Valid while workers write the line when asked, and while `todo.md` on the default branch scores each item a chunk takes.
|
||||
|
||||
## routing: on a model judge-replay covered, the judge runs on the rung a review would get from the highest E + I among its chunk's items, never below the project's rung
|
||||
|
||||
2026-10-15, lilleman's plan, settled by the replay below. Serves goals 1.2 and 1.4, under the ladder and steps of `routing: one ladder`.
|
||||
@@ -402,17 +466,17 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
|
||||
|
||||
Valid while the index ranks these models, and while no newer model joins the catalog.
|
||||
|
||||
## routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, or when the judge reports a taken-in item that picks a higher rung or an escalation trigger the worker cites
|
||||
## routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds
|
||||
|
||||
2026-10-14, lilleman. Serves goal 1.4.
|
||||
|
||||
Because a model switch inside a session discards its prompt cache, the worker changes rung only as the chunk starts, after every second unfinished verdict, when the judge reports a taken-in item whose L picks a higher rung, or when the judge finds the escalation trigger a worker cites in asking for a higher rung (`routing: a worker whose run ends asking for a higher rung`). loop-claude routes on the first row of the next release in `todo.md`'s table that does not wait: an item waits while it is marked `question` or a blocker of it is still in the table. Since `todo.md` holds no order but its table's (`todo.md: one table`), that row is the item the worker takes. An item the worker takes in beyond that row lifts the worker's rung once the judge reports it, after the first cycle.
|
||||
Because a model switch inside a session discards its prompt cache, the worker changes rung only as the chunk starts, after every second unfinished verdict, when the worker names an item on its `Chunk items:` line whose L picks a higher rung (`routing: loop-claude reads the items a chunk takes`), or when the judge finds the escalation trigger a worker cites in asking for a higher rung (`routing: a worker whose run ends asking for a higher rung`). loop-claude routes on the first row of the next release in `todo.md`'s table that does not wait: an item waits while it is marked `question` or a blocker of it is still in the table. Since `todo.md` holds no order but its table's (`todo.md: one table`), that row is the item the worker takes. An item the worker takes in beyond that row lifts the worker's rung once the cycle in which the worker names it ends.
|
||||
|
||||
Valid while a model switch inside a session discards its prompt cache.
|
||||
|
||||
## rule-hook: a PreToolUse hook enforces each rule a program can check, a `# rule-override <rule>: <reason>` marker overrides any rule but those `rule-hook --help` marks as taking none, and larv-review's overrides pass rules on each override before merge
|
||||
|
||||
2026-10-12, lilleman. Serves goals 1.2 and 1.3. Anthropic's best-practices page calls `CLAUDE.md` advisory and hooks deterministic, so a rule that must hold every time moves into `bin/rule-hook`, and `CLAUDE.md` keeps it only where the hook cannot check it. A Claude Code hook ties the check to one vendor ("Prefer the vendor-neutral option", `principles/technical-principles.md`), which is the target itself here: this repo configures Claude Code. What the tie buys: the hook blocks before a write lands, Edit and Write calls included, in every project with no per-repo install; a git `commit-msg` hook sees only commits, and CI sees them only after the push. The hook checks each write, so the agent fixes the line it just wrote; checking the staged diff at commit time would also cover files a Bash command writes, at the cost of later feedback. The marker's spelling was picked by a panel of three fresh readers, two to one. Each override is logged in the checkout's git directory, which is never pushed (goal 4). The plan had loop-claude's judge to rule on overrides inside a loop; lilleman approved larv-review's overrides pass in every run, because it rules before merge and an outrageous override still reaches the human through its ask. 2026-10-16, lilleman, answering plan item 166: no marker overrides `reviewer-rung`. 2026-11-04, lilleman: the title points at `rule-hook --help`, which lists every rule no marker overrides from the code, so the title can't drift from it; the comment above each in `bin/rule_hook/rules.py` says why. Valid while a PreToolUse hook that exits 2 blocks the call and hands its stderr to the model, and while an agent commits through Claude Code's Bash tool.
|
||||
2026-10-12, lilleman. Serves goals 1.2 and 1.3. Anthropic's best-practices page calls `CLAUDE.md` advisory and hooks deterministic, so a rule that must hold every time moves into `bin/rule-hook`, and `CLAUDE.md` keeps it only where the hook cannot check it. A Claude Code hook ties the check to one vendor ("Prefer the vendor-neutral option", `principles/technical-principles.md`), which is the target itself here: this repo configures Claude Code. What the tie buys: the hook blocks before a write lands, Edit and Write calls included, in every project with no per-repo install; a git `commit-msg` hook sees only commits, and CI sees them only after the push. The hook checks each write, so the agent fixes the line it just wrote; checking the staged diff at commit time would also cover files a Bash command writes, at the cost of later feedback. The marker's spelling was picked by a panel of three fresh readers, two to one. Each override is logged in the checkout's git directory, which is never pushed (goal 4). The plan had loop-claude's judge to rule on overrides inside a loop; lilleman approved larv-review's overrides pass in every run, because it rules before merge and an outrageous override still reaches the human through its ask. 2026-10-16, lilleman, answering plan item 166: no marker overrides `reviewer-rung`. 2026-11-04, lilleman: the title points at `rule-hook --help`, which lists every rule no marker overrides from the code, so the title can't drift from it; the comment above each in `bin/rule_hook/rules.py` says why. Valid while a PreToolUse hook that exits 2 blocks the call and hands its stderr to the model, and while an agent commits through Claude Code's Bash tool. 2026-11-11, lilleman: the hook also checks, at a `git commit`, each file the commit would commit against the rules that name a file, since a file a Bash command writes reaches no write check; the feedback comes at the commit, not at the write.
|
||||
|
||||
## rules/: `CLAUDE.md` holds the worker's rules that no file path can scope, and each topic's rules for the worker live in a `rules/` file that Claude Code loads by its `paths:` frontmatter
|
||||
|
||||
@@ -502,7 +566,7 @@ Three places carry the rule. `rules/todo-items.md` states it and loads with `tod
|
||||
|
||||
2026-10-13, lilleman through todo item 104, implemented by claude. Serves goals 1.4 and 1.5. Like red, green, refactor: the worker makes the item work, and the review passes raise it to the bar. A loop-claude worker loaded `CLAUDE.md` and every file under `rules/`, 4,247 words, and `CLAUDE.md` told it to read `technical-principles.md` too, 2,426 words more. The reviewers read the principles anyway, and a finding against one names it.
|
||||
|
||||
The worker keeps what it needs to write its own work. Each project's decision titles stay in its `AGENTS.md`, since a worker that misses a decision builds the wrong behaviour. A core of "Prose" stays in `CLAUDE.md`, since no reviewer reads the worker's chat. lilleman chose both on 2026-10-13. The rules for writing Goals and Audience, `todo.md` items and decisions stay in `rules/`, since writing those is the worker's job. `bin/rule-hook` stays: it blocks what a program can check, at no context cost. "Quality", the rules for comments and READMEs, and the examples behind "Prose" moved to `skills/larv-review/standards/`, which no path loads. loop-claude's judge and the prose reviewer read them, and the stability reviewer reads "Quality". They sit beside larv-review, whose passes apply them; `principles/` stays where it is, since the principles auditor reads it per file.
|
||||
The worker keeps what it needs to write its own work. Each project's decision titles stay in its `AGENTS.md`, since a worker that misses a decision builds the wrong behaviour. 2026-11-11, lilleman: a project principle in `AGENTS.md` now stands in for the titles of the decisions it covers. A core of "Prose" stays in `CLAUDE.md`, since no reviewer reads the worker's chat. lilleman chose both on 2026-10-13. The rules for writing Goals and Audience, `todo.md` items and decisions stay in `rules/`, since writing those is the worker's job. `bin/rule-hook` stays: it blocks what a program can check, at no context cost. "Quality", the rules for comments and READMEs, and the examples behind "Prose" moved to `skills/larv-review/standards/`, which no path loads. loop-claude's judge and the prose reviewer read them, and the stability reviewer reads "Quality". They sit beside larv-review, whose passes apply them; `principles/` stays where it is, since the principles auditor reads it per file.
|
||||
|
||||
A finding against a principle names it, so the worker reads only that one when it fixes or declines the finding. The worker still writes a failing test first wherever a test can check the change, since no reviewer can make a test come first. On 2026-10-13 lilleman widened it from code changes to anything there is to test. A chunk that changes no code, config or test, such as one that only plans, meets no principles review; the chunk that implements its items does. The prose reviewer checks that a doc sorts a list whose order carries no meaning.
|
||||
|
||||
|
||||
@@ -1597,9 +1597,9 @@ Read most (opens and searches):
|
||||
- `bin/loop_claude/loop/usage.py` 3
|
||||
- `skills/comprehension-panel/seat-prompts.md` 3
|
||||
- `.gitignore` 2
|
||||
- `AGENTS.md` 2
|
||||
- `CLAUDE.md` 2
|
||||
- `bin/loop_claude/cycle/judge.py` 2
|
||||
- `bin/loop_claude/project/records.py` 2
|
||||
|
||||
## Reading cost, chunk 47c4f6512691, finished 2026-11-11T17:21:45Z
|
||||
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
# Judge field trials
|
||||
|
||||
These trials ask one question: which of the facts loop-claude's judge reports could code read instead?
|
||||
The judge is the headless claude that rules on each worker run; its reply named the items the chunk took, whether a release was ready to cut, and any escalation the worker asked for.
|
||||
They ran on 2026-11-11, over the 709 cases archived by then. `docs/decisions.md` → `routing: loop-claude reads the items a chunk takes` records what was decided from them.
|
||||
|
||||
## Terms
|
||||
|
||||
- **Case:** an outcomes file in `~/.claude/loop-claude-transcripts/`, its first verdict, and the transcript the judge read for it.
|
||||
- **Final message:** the last text the worker wrote after the last prompt loop-claude or the human gave it.
|
||||
|
||||
## Results
|
||||
|
||||
| Field | What code tried | Result |
|
||||
|---|---|---|
|
||||
| Escalation | "higher rung", "rung up" or "one rung above" in the final message | Code is wrong in each of the 39 disagreements: 37 where only code finds a request, 2 where only the judge does. Code matched a worker asking for the human's yes at the top rung and a review report quoting "rerun one rung up", and missed requests made before the final message. The judge keeps the field. |
|
||||
| Release ready to cut | The plan as the default branch and tags stood at the verdict, read by `release_plan.release_standing` | Of 240 interrupts, 31 ask about a cut, and code reads ReadyToCut for 3. Two misses are code's: in two projects a release review had merged without the `Release review of <release>` commit that code looks for. The rest asked under older rules or after the cut. The judge keeps the field, and a `todo.md` item files the fix. |
|
||||
| Items the chunk takes | An item stated with its L and E + I anywhere in the worker's text | 413 of 709 cases state one, 77 of them more than one, and workers also state items they file, finish or only mention. Code reads the items from a line that holds only them, which workers write only since loop-claude asks for it, so the archive can't measure it. |
|
||||
|
||||
## Running
|
||||
|
||||
Each script reads the archive on this machine through `cases.py` and prints its counts last; run it from the checkout's root, as in `python3 judge-field-trials/escalations.py`.
|
||||
The lines above the counts quote transcripts and name projects, so they stay on this machine.
|
||||
|
||||
- `escalations.py` prints each escalation disagreement with the words around it.
|
||||
- `release_ready.py` prints each interrupt's question beside whether code reads the release ready to cut.
|
||||
- `items.py` prints a few cases stating more than one item.
|
||||
@@ -0,0 +1,41 @@
|
||||
"""Each archived case: an outcomes file's first verdict, with the transcript copy the judge read for it."""
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1] / "bin"))
|
||||
|
||||
from transcript_archive.outcomes import ANY_VERDICT_AT, OUTCOME_AT, task_of
|
||||
from claude_code.entries import AssistantTurn, HumanTurn, LoopPrompt, Text
|
||||
from claude_code.transcript import read_transcript
|
||||
from transcript_archive.location import transcript_of
|
||||
|
||||
ARCHIVE = pathlib.Path.home() / ".claude/loop-claude-transcripts"
|
||||
|
||||
|
||||
def first_block(outcomes: str) -> tuple[str, str, str] | None:
|
||||
"""The first verdict's stamp, its heading, such as `Verdict: interrupt`, and its whole block, up to the next outcome."""
|
||||
found = ANY_VERDICT_AT.search(outcomes)
|
||||
if not found:
|
||||
return None
|
||||
ends = [start.start() for start in OUTCOME_AT.finditer(outcomes) if start.start() > found.start()]
|
||||
return found[1], found[2], outcomes[found.start():ends[0] if ends else len(outcomes)]
|
||||
|
||||
|
||||
def cases():
|
||||
"""Each outcomes file in the archive whose task and first verdict can be read, with that verdict and its transcript's entries."""
|
||||
for outcomes in sorted(ARCHIVE.glob("*/*/*.outcomes.md")):
|
||||
text = outcomes.read_text(encoding="utf-8")
|
||||
block = first_block(text)
|
||||
entries = read_transcript(str(transcript_of(outcomes)))
|
||||
if block is not None and task_of(text) is not None and isinstance(entries, list):
|
||||
yield outcomes, block, entries
|
||||
|
||||
|
||||
def worker_texts(entries) -> list[str]:
|
||||
return [part.text for entry in entries if isinstance(entry, AssistantTurn) for part in entry.parts if isinstance(part, Text)]
|
||||
|
||||
|
||||
def last_run_text(entries) -> list[str]:
|
||||
"""The worker's text after the last prompt loop-claude or the human gave it: what the judge rules on."""
|
||||
at = max((index for index, entry in enumerate(entries) if isinstance(entry, LoopPrompt | HumanTurn)), default=-1)
|
||||
return worker_texts(entries[at + 1:])
|
||||
@@ -0,0 +1,20 @@
|
||||
"""Could code find an escalation request where the judge did? Counts, per verdict kind, whether the worker's final message asks for a higher rung and whether the judge's verdict carried an escalation, and prints each disagreement with the words around it."""
|
||||
import collections
|
||||
import re
|
||||
|
||||
from cases import cases, last_run_text
|
||||
|
||||
ASKS = re.compile(r"higher rung|rung up|one rung above", re.IGNORECASE)
|
||||
|
||||
table = collections.Counter()
|
||||
for outcomes, (stamp, kind, block), entries in cases():
|
||||
texts = last_run_text(entries)
|
||||
final = texts[-1] if texts else ""
|
||||
found = ASKS.search(final)
|
||||
escalated = "\nEscalation:\n" in block
|
||||
table[(kind, bool(found), escalated)] += 1
|
||||
if bool(found) != escalated:
|
||||
around = final[max(0, found.start() - 200):found.end() + 150] if found else final[-300:]
|
||||
print(f"## {outcomes.parent.parent.name} {stamp} {kind} code={bool(found)} judge={escalated}\n {around!r}\n")
|
||||
for (kind, code, judge), count in sorted(table.items()):
|
||||
print(f"{kind}: code {code}, judge {judge}: {count}")
|
||||
@@ -0,0 +1,19 @@
|
||||
"""How do workers name the items a chunk takes, unasked? Counts, per verdict kind, the cases whose worker states an item with its L and E + I, and those stating more than one, and prints a few of the latter."""
|
||||
import collections
|
||||
import re
|
||||
|
||||
from cases import cases, worker_texts
|
||||
|
||||
STATED = re.compile(r"item (\d+)\b[^.\n]{0,60}?\bL (\d)\b[^.\n]{0,40}?E \+ I (\d+)", re.IGNORECASE)
|
||||
|
||||
table = collections.Counter()
|
||||
shown = 0
|
||||
for outcomes, (stamp, kind, block), entries in cases():
|
||||
found = [match for text in worker_texts(entries) for match in STATED.finditer(text)]
|
||||
ids = sorted({int(match[1]) for match in found})
|
||||
table[(kind, bool(found), len(ids) > 1)] += 1
|
||||
if len(ids) > 1 and shown < 12:
|
||||
shown += 1
|
||||
print(outcomes.parent.parent.name, stamp, ids, [match[0][:120] for match in found[:4]])
|
||||
for (kind, stated, several), count in sorted(table.items()):
|
||||
print(f"{kind}: states an item {stated}, more than one {several}: {count}")
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Could code find a release ready to cut where the judge did? For each archived interrupt, prints whether the plan, as the project's default branch and tags stood at the verdict's stamp, read ReadyToCut, and whether the judge's question is about a cut."""
|
||||
import collections
|
||||
import datetime
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
from cases import cases
|
||||
from transcript_archive.project_key import checkout_of
|
||||
from release_plan.release_standing import chunk_item, default_plan, ReadyToCut
|
||||
from git_cli.git_repo import default_ref, git
|
||||
from release_plan.todo_table import items
|
||||
|
||||
ABOUT_CUT = re.compile(r"\bcut\b|ready to (?:cut|release|ship)|tag (?:it|v\d)|version bump", re.IGNORECASE)
|
||||
|
||||
|
||||
def as_of(stamp: str):
|
||||
"""A git runner that lists only the tags created by stamp, so a later cut doesn't count."""
|
||||
def run(*args, cwd=None, **kwargs):
|
||||
out = git(*args, cwd=cwd, **kwargs)
|
||||
if "tag" in args and "--list" in args and isinstance(out, str):
|
||||
dates = git("for-each-ref", "--format=%(refname:short) %(creatordate:iso-strict)", "refs/tags", cwd=cwd)
|
||||
created = dict(line.split(" ", 1) for line in dates.splitlines()) if isinstance(dates, str) else {}
|
||||
out = "\n".join(tag for tag in out.split() if utc(created.get(tag, "")) <= stamp)
|
||||
return out
|
||||
return run
|
||||
|
||||
|
||||
def utc(iso: str) -> str:
|
||||
"""iso in the verdict stamp's form, or "" for no date, which sorts before every stamp."""
|
||||
return datetime.datetime.fromisoformat(iso).astimezone(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") if iso else ""
|
||||
|
||||
|
||||
table = collections.Counter()
|
||||
for outcomes, (stamp, kind, block), entries in cases():
|
||||
if kind != "Verdict: interrupt":
|
||||
continue
|
||||
project = outcomes.parent.parent.name
|
||||
root = str(checkout_of(project, pathlib.Path.home()))
|
||||
question = block.split("\n\n", 2)[1] if block.count("\n\n") >= 2 else ""
|
||||
about = bool(ABOUT_CUT.search(question))
|
||||
ref = default_ref(root, 10) if pathlib.Path(root).is_dir() else None
|
||||
commit = git("rev-list", "-1", f"--before={stamp}", ref, cwd=root) if isinstance(ref, str) else None
|
||||
if not isinstance(commit, str) or not commit.strip():
|
||||
table[("no checkout or commit", about)] += 1
|
||||
continue
|
||||
plan = default_plan(root, commit.strip(), as_of(stamp))
|
||||
found = items(plan.todo) if isinstance(plan.todo, str) else plan.todo
|
||||
pick = chunk_item(found, plan.cut) if isinstance(found, list) else found
|
||||
ready = isinstance(pick, ReadyToCut)
|
||||
table[(ready, about)] += 1
|
||||
print(f"{'ready' if ready else 'not '} {'cut?' if about else ' '} {project} {stamp} {type(pick).__name__}: {question[:220]!r}")
|
||||
for (ready, about), count in sorted(table.items(), key=str):
|
||||
print(f"ReadyToCut {ready}, asks about a cut {about}: {count}")
|
||||
@@ -72,7 +72,7 @@
|
||||
- BANNED: a CLI that refuses to run until given `--data-path` and `--seed`, when nearly every run wants the shipped data and fresh randomness. GOOD: it uses the shipped data and fresh randomness unless a flag overrides them, and `--help` names those flags.
|
||||
* Deterministic order wherever order carries no meaning.
|
||||
- Sort such a list, on a key that does not move: the name a line introduces, never where it came from. Exception: `id` goes first. A sorted list is searchable, its diff shows only what changed, and two edits to it conflict only where they mean to.
|
||||
- An import sorts on its first binding, not its path, so moving or renaming a module reorders nothing. Where lines already group by kind — type imports, host packages — the key sorts within each group.
|
||||
- An import sorts on its first binding, not its path, so moving or renaming a module doesn't reorder anything. Where lines already group by kind — type imports, host packages — the key sorts within each group.
|
||||
* About seven things per level, so a reader new to the project can take in each level at once.
|
||||
- A level is what a reader looks at together: the parameters of a function and its branches (cyclomatic complexity), the methods of a class, the files in a folder and the folders beside them, the sections of a document, the components of a system, the hops of a call chain. Past about seven, group the things under names a reader can follow; each group then counts as one.
|
||||
- Parameters, branches and the values a function tracks act on each other, so a reader holds them in mind together, and a mind holds about four. Aim for four or five of them, and treat seven as the ceiling. Keep a function to one screen, about 24 lines of 80 characters.
|
||||
|
||||
+3
-1
@@ -6,6 +6,8 @@ paths:
|
||||
|
||||
# Decisions: one home, each linked to a goal or a technical principle
|
||||
|
||||
**The decision log has one home: `docs/decisions.md`**. A title states the whole decision in one line: what holds at the commit it ships in, in words a reader finds in the code or the plan. `AGENTS.md` lists every title, so a session works by the decisions without loading their background. The entry holds the background: its date, who made it, the reasoning, and the premise it rests on ("valid while X") so it can expire honestly when that premise dies.
|
||||
**The decision log has one home: `docs/decisions.md`**. A title states the whole decision in one line: what holds at the commit it ships in, in words a reader finds in the code or the plan. `AGENTS.md` lists the project principles and the title of each decision no project principle covers, so a session works by the decisions without loading their background. The entry holds the background: its date, who made it, the reasoning, and the premise it rests on ("valid while X") so it can expire honestly when that premise dies.
|
||||
|
||||
**Every decision links to a goal or a technical principle.** An entry names the goal it serves or the technical principle under `~/.claude/principles/` it applies; one that can do neither is a missing goal, and an entry already in the repo with neither behind it is a finding. Propose the goal, never a reason of your own — `~/.claude/CLAUDE.md` → "Working loop", "Ask, don't guess" says how. An entry working against a goal or a technical principle stands only while it names the `todo.md` item that ends it.
|
||||
|
||||
**Check each new decision against the project's principles.** A project principle, listed in the project's `AGENTS.md` under `## Project principles`, states one rule that several decisions apply. Each time a decision is recorded, ask three things. Does a project principle already cover it? Then it adds no title to `AGENTS.md`, and needs no ask. Does it break one? Then either the principle or the decision is wrong. Does it apply the same rule as decisions already listed? Then that rule is a new project principle, which covers those decisions too. A break, a new project principle or a change to one is the human's: ask, with your recommendation, and leave the project principles as they are until the human answers. Once the human approves a new one, the titles it covers leave `AGENTS.md`.
|
||||
|
||||
@@ -15,7 +15,7 @@ independently, so every seat's report counts.
|
||||
|
||||
## What makes the result trustworthy
|
||||
|
||||
Hold all of these or the comparison proves nothing.
|
||||
Hold all of these or the comparison doesn't prove anything.
|
||||
|
||||
- **One variable.** Every panelist runs the same model with the same brief and the same report
|
||||
shape. Only the persona differs. A cheaper model for the juniors turns persona into a confound.
|
||||
@@ -162,7 +162,7 @@ with the same four dimensions and overall as the other seven.
|
||||
|
||||
Every panelist ends with a comprehension score, 1 to 10, for the project as a whole. The scale is
|
||||
absolute, so two panels run on two projects produce comparable numbers — a panelist scoring against
|
||||
their own expectations of *this* codebase has produced a number that means nothing next to
|
||||
their own expectations of *this* codebase has produced a number that doesn't mean anything next to
|
||||
another. Put these anchors in every brief, verbatim.
|
||||
|
||||
- **1 — Spaghetti.** No boundary holds. Finding where a behaviour lives is grep archaeology;
|
||||
|
||||
@@ -93,3 +93,11 @@ Report, in this shape:
|
||||
2. The places that decided them, most decisive first. Each opens with the names from your rulings line that it moves, joined by ", " and followed by ": ", such as `Locality, Shape: `. Then `path:line` and the unit's name, each in backticks, in the version the place is in, and one or two sentences on what reads easier or harder there than in the other version. A dimension's ruling that no place opens with counts as same, and the overall ruling does unless you name a place.
|
||||
3. Where a place you name backs a worse ruling and you want a more capable reader to check that ruling, the line `Rerun one rung up`.
|
||||
```
|
||||
|
||||
## claude-config's values
|
||||
|
||||
- `{language}`: Python
|
||||
- `{kind}`: a Claude Code configuration with an agent loop and review skills
|
||||
- `{domain}`: driving headless Claude Code sessions, agent review workflows
|
||||
- `{domain docs}`: the Claude Code documentation
|
||||
- `{3am question}`: a loop-claude run exits 1 overnight after the judge returned a verdict the loop rejected, or a larv-review run looped a reviewer a third time
|
||||
|
||||
@@ -78,8 +78,8 @@ Give a reviewer without Bash `git ls-files`'s output in its prompt, since some h
|
||||
**Round cap.**
|
||||
|
||||
- **Round.** A round is one review by a pass's reviewer: the first and each re-entry. The rerun after the architect returns Stage 0 questions ("Architecture" below) and a relaunch one rung up ("Rung up" below) stay in their round.
|
||||
- **Auto-reviewer.** The external auto-reviewer (§4) is a pass of its own, each review it posts, up to the cap, a round. Its count is the number of its reviews on the PR, less its reviews of a `Record the comprehension check:` commit, and `round-count` doesn't list it. A commit that answers one of its rounds takes an ordinary title, not an `Answer <pass> round` one, and a round that changed nothing gets no commit.
|
||||
- **Count.** Close each round of a pass other than the auto-reviewer with one commit that holds the round's fixes. Title it `Answer <pass> round <n>: <k> 🔴, <m> 🟡 (<j> misread); …`: `<pass>` is the pass's name in lower case, such as `full scan`, `<k>` and `<m>` are the 🔴 and 🟡 findings all the round's reviewers reported, and `<j>` is how many of those they marked `[misread]` (review-findings.md → "Misread"); a reviewer that marks none, such as `/security-review`, adds 0. Example: `Answer stability round 1: 1 🔴, 2 🟡 (1 misread); flag a deleted test`. A round that changed nothing gets an empty commit (`git commit --allow-empty`). Push the commits when §4 runs next, where there is a remote. `~/.claude/bin/larv-review round-count` prints each pass's count and marks a pass at the cap.
|
||||
- **Auto-reviewer.** The external auto-reviewer (§4) is a pass of its own, each review it posts, up to the cap, a round. Its count is the number of its reviews on the PR, less its reviews of a `Record the comprehension check:` commit, and `round-count` doesn't list it. A commit that answers one of its rounds takes an ordinary title, not an `Answer <pass> round` one, and a round that didn't change anything gets no commit.
|
||||
- **Count.** Close each round of a pass other than the auto-reviewer with one commit that holds the round's fixes. Title it `Answer <pass> round <n>: <k> 🔴, <m> 🟡 (<j> misread); …`: `<pass>` is the pass's name in lower case, such as `full scan`, `<k>` and `<m>` are the 🔴 and 🟡 findings all the round's reviewers reported, and `<j>` is how many of those they marked `[misread]` (review-findings.md → "Misread"); a reviewer that marks none, such as `/security-review`, adds 0. Example: `Answer stability round 1: 1 🔴, 2 🟡 (1 misread); flag a deleted test`. A round that didn't change anything gets an empty commit (`git commit --allow-empty`). Push the commits when §4 runs next, where there is a remote. `~/.claude/bin/larv-review round-count` prints each pass's count and marks a pass at the cap.
|
||||
- **Done.** A pass is done when its latest round leaves no 🔴 or 🟡 standing: none was reported, or each was shown factually wrong, declined, filed, or named under the bar per "Findings", or settled in the PR summary per "Comprehension check" → "Places", or, for the prose pass and a pass at its cap, fixed. Resolve that round's 🟢s and ❓s per "Findings" too.
|
||||
- **Cap.** A pass takes at most two rounds per chunk, and the release review's architecture and principles passes five; don't ask the user for more. Answer its last round's findings per "Findings": those fixes merge once the gate passes, unreviewed by that pass. rule-hook's `round-cap` refuses a round commit that would take a pass past its cap, and no override waives it. A review the auto-reviewer posts after its second isn't a round: fix none of it and rerun no pass for it, but file a security finding it raises per `~/.claude/CLAUDE.md` → "Working loop", and any other 🔴 in `todo.md`.
|
||||
|
||||
@@ -94,7 +94,7 @@ Give a reviewer without Bash `git ls-files`'s output in its prompt, since some h
|
||||
| made rule-hook log an override | Overrides |
|
||||
| changes behaviour: what the code or its tests do | Stability |
|
||||
|
||||
A fix meets every row it fits. Prose has no row, so a comment or doc edit inside a fix reruns nothing. Example: a fix for a stability 🔴 that changes code and a comment reruns Principles and Stability.
|
||||
A fix meets every row it fits. Prose has no row, so a comment or doc edit inside a fix doesn't rerun any pass. Example: a fix for a stability 🔴 that changes code and a comment reruns Principles and Stability.
|
||||
|
||||
**What a round reviews.** An agent under `~/.claude/agents/` reviews the branch in round 1. From round 2 it reviews only the answers to its last round's findings, and each fix from another pass that re-entered its pass. Prompt it with the path of its last round's report and each finding's answer: the fix's diff, or the reason it was dismissed. Give it each other pass's fix as `git diff <the commit before that fix> <the fix's commit>`, since the fixes may be unpushed, with the finding it answered. It reports only an answer that is plainly wrong or worse than the change it answered, so its pass converges in its second round. Other reviewers review the branch as their tool does.
|
||||
|
||||
@@ -108,6 +108,7 @@ A fix meets every row it fits. Prose has no row, so a comment or doc edit inside
|
||||
|
||||
- **Outside the chunk.** A finding outside the chunk's change, whatever its pass or severity, is scored and filed in `todo.md` per `~/.claude/rules/todo-items.md`. A 🔴 is marked `defect`. A 🟡 against a file under `~/.claude/principles/` is marked `principle`, its detail naming the principle, where no other reason fits. One under the bar and not exempt is named in the summary with its score. A verified security HIGH is handled per "Security". In the release review the whole project is the chunk's change, so nothing lies outside it; "Release review" → "Findings" says which of its findings are filed.
|
||||
- **Goal or decision.** A fix that would change a goal, or a `docs/decisions.md` entry the repo held before the chunk began, is asked; no user reachable → filed, marked `question`, the goal or decision left as is.
|
||||
- **New decision.** Each decision a fix or an answer records in `docs/decisions.md` is checked against the project principles per `~/.claude/rules/decisions.md` → "Decisions: one home, each linked to a goal or a technical principle", "Check each new decision against the project's principles", before the commit that records it.
|
||||
- **🔴.** A 🔴 is fixed, or shown factually wrong, and what disproved it recorded per `~/.claude/skills/larv-review/standards/comments.md` → "Where a choice goes": in a comment at the code the finding concerns, or in `docs/decisions.md` when it concerns more than one place. Never declined.
|
||||
- **🟡.** A 🟡 is fixed, shown factually wrong like a 🔴, or declined. Decline one only where the reason is sound, consistent with the goals and the technical principles, and confidence is very high; the decline and its reason are recorded the same way. A reason neither speaks to is a gap in the goals: fix the finding, or ask per `~/.claude/CLAUDE.md` → "Working loop", "Ask, don't guess"; no user reachable → fix. Never drop one silently, and never decline one for convenience or the size of the fix.
|
||||
- **🟢 and ❓.** A 🟢 or ❓ is fixed, answered, declined in the summary with its reason, or filed. Answer one where the code, a test, a goal or a decision the repo held before the chunk began settles it, and quote what in the summary. A ❓ only the user can answer is asked; no user reachable → filed, marked `question`.
|
||||
@@ -138,7 +139,7 @@ Launch one principles-auditor on the trigger in §3, so the project is read once
|
||||
|
||||
### Comprehension check
|
||||
|
||||
The comprehension check records how hard the project is for its contributor to read (`~/.claude/rules/goals-and-audience.md`), and blocks no merge. Simulated readers, the seats, launched per `~/.claude/skills/comprehension-panel/SKILL.md`, each read the whole project, not the diff. It reports no findings and is no pass: it runs once, in §5, after every pass is done, the auto-reviewer included, so it reads the branch as review left it. It runs a paired ruling in the release review, even where an opt-out skips its architect, and a scoring run where one is due. It skips only as "Opt-out" or "Suspended" below says.
|
||||
The comprehension check records how hard the project is for its contributor to read (`~/.claude/rules/goals-and-audience.md`), and no score blocks a merge; only a run the plan calls and the branch lacks does, per "Scoring run". Simulated readers, the seats, launched per `~/.claude/skills/comprehension-panel/SKILL.md`, each read the whole project, not the diff. It reports no findings and is no pass: it runs once, in §5, after every pass is done, the auto-reviewer included, so it reads the branch as review left it. It runs a paired ruling in the release review, even where an opt-out skips its architect, and a scoring run where one is due. It skips only as "Opt-out" or "Suspended" below says.
|
||||
|
||||
**Paired ruling.** Each seat votes whether the project reads better, the same or worse at this branch's head than at the base.
|
||||
|
||||
@@ -146,7 +147,7 @@ The comprehension check records how hard the project is for its contributor to r
|
||||
- **Head.** The head holds `origin/<default branch>`, merged in at §4 step 1, so it differs from the default branch only by this branch's change.
|
||||
- **Seats.** Mid A and the inherited architect. Write what they read and their replies into a votes file in the chunk's review folder, in the shape `~/.claude/bin/larv-review --help` shows, and run `~/.claude/bin/larv-review record-ruling <votes file>`. On `Split:`, launch the maintainability senior and Junior A together, add their lines, and run it again.
|
||||
|
||||
**Scoring run**, when `~/.claude/bin/larv-review needs-scoring-run <the IDs of the items the chunk ships, if any>` prints `Scoring run due:`. Run `needs-scoring-run` before removing the chunk's items from `todo.md`.
|
||||
**Scoring run**, when `~/.claude/bin/larv-review needs-scoring-run <the IDs of the items the chunk ships, if any>` prints `Scoring run due:`. Run `needs-scoring-run` before removing the chunk's items from `todo.md`. `chunk-pr merge` refuses until the branch's committed history records every run the plan calls, which `larv-review scoring-run-owed` counts at merge time from the merge base; an earlier due line is only a snapshot.
|
||||
|
||||
- **Seats.** Junior A, Mid A, the maintainability senior and the inherited architect each score the project at the head from 1 to 10 overall and on four dimensions: Navigation, Locality, Shape and Self-sufficiency. Write their lines into a scores file and run `~/.claude/bin/larv-review record-scoring <scores file> <the same item IDs>`, which appends the run to the history and prints the four seats' means.
|
||||
- **Floor item.** Where `record-scoring` prints `Floor:`, a `Next:` line follows: file one floor item, or refresh the open one. A floor item is a plan item meant to bring every mean to 7.0. File it as one ambitious refactor, not touch-ups. It restructures the area behind the places `larv-review places` prints for this run, chosen where they bear on the lowest dimension, sized to bring every mean to 7.0, and quotes those places. Its tagline holds the words `comprehension floor`, so its own chunk runs a scoring run; while a mean stays under 7.0, that run's `Next:` line asks for the next floor item. Mark it `owner`, since the human asked that such a project always hold one, so no score deletes it. Put it in the earliest release the plan names after the one the chunk ships into, or else in the next version after that one that `~/.claude/rules/todo-items.md` allows, so the floor doesn't hold up any release. While one is open, refresh its places and aim from the newest run instead of filing another; the item the chunk ships doesn't count as open. Commit it with the run, as "Record" says.
|
||||
@@ -161,7 +162,7 @@ The comprehension check records how hard the project is for its contributor to r
|
||||
|
||||
### Security
|
||||
|
||||
`/security-review` over the branch diff is the pass of record, except where the full scan runs on the same change and replaces it. Where the full scan fires but does not run — skipped on a precedent, declined at its cost prompt, or unavailable — `/security-review` runs. Launch a general-purpose subagent that invokes it through the Skill tool, edits nothing, and replies with the skill's report, or that the skill is unavailable. Invoked in your own context, the skill ends your turn on that report. Open this subagent's prompt, and that of any subagent that runs the full scan, with `You are the security pass for this chunk.`: rule-hook reads that sentence as a security launch.
|
||||
`/security-review` over the branch diff is the pass of record, except where the full scan runs on the same change and replaces it. Where the full scan fires but does not run — skipped on a precedent, declined at its cost prompt, or unavailable — `/security-review` runs. Launch a general-purpose subagent that invokes it through the Skill tool, doesn't edit anything, and replies with the skill's report, or that the skill is unavailable. Invoked in your own context, the skill ends your turn on that report. Open this subagent's prompt, and that of any subagent that runs the full scan, with `You are the security pass for this chunk.`: rule-hook reads that sentence as a security launch.
|
||||
|
||||
A verified HIGH inside this chunk's blast radius is fixed here, outside it on its own branch and PR with this chunk waiting behind it. Say which.
|
||||
|
||||
@@ -196,7 +197,7 @@ Launch prose-reviewer with what the diff touched and every section this review c
|
||||
1. Read the finished PR against the README goals and code health, unless the README voids it: does anything in it work against one? It breaks a maintain goal → do not merge; redesign the change or drop its item. It works against an achieve goal → do not merge; ask the human, stating the conflict.
|
||||
2. Run the comprehension check (§3) where it runs; it runs once. When `origin/<default branch>` moves after that, return to §4, and keep the ruling and scoring run already recorded.
|
||||
3. Remove the chunk's items from `todo.md` per `~/.claude/CLAUDE.md` → "Working loop" in the branch, commit and push that, and wait for the forge's checks (§4 step 4); the commit re-enters no pass.
|
||||
4. Aligned, every pass done and everything green → **merge** with `~/.claude/bin/chunk-pr merge --gate-passed`, which takes the first merge method the repository allows on its forge. A repo whose `AGENTS.md` names another method merges that way with the forge's CLI instead. A repo withholds this in its `AGENTS.md` → leave the PR ready and unmerged, and say which line withheld it.
|
||||
4. Aligned, every pass done and everything green → **merge** with `~/.claude/bin/chunk-pr merge --gate-passed`, which takes the first merge method the repository allows on its forge. A repo whose `AGENTS.md` names another method merges that way with the forge's CLI instead, after `~/.claude/bin/larv-review scoring-run-owed` exits 0. A repo withholds this in its `AGENTS.md` → leave the PR ready and unmerged, and say which line withheld it.
|
||||
5. Merged → run `~/.claude/bin/chunk-pr clean` in the chunk's worktree, and put a failure it prints in the summary.
|
||||
6. Stop. The next chunk gets a fresh session.
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Words: `~/.claude/CLAUDE.md` → "Vocabulary".
|
||||
|
||||
A reviewer reads and reports; it edits nothing. It writes no file but its report, except the scratch files that a reviewer holding Bash makes with Bash. Read `docs/decisions.md` and `todo.md` first: a finding declined or disproved in `docs/decisions.md` or in a comment at the code it concerns, or filed in `todo.md`, is not raised again, unless the decline works against a goal or a technical principle and names no `todo.md` item ending it.
|
||||
A reviewer reads and reports; it doesn't edit anything. It writes no file but its report, except the scratch files that a reviewer holding Bash makes with Bash. Read `docs/decisions.md` and `todo.md` first: a finding declined or disproved in `docs/decisions.md` or in a comment at the code it concerns, or filed in `todo.md`, is not raised again, unless the decline works against a goal or a technical principle and names no `todo.md` item ending it.
|
||||
|
||||
## Severity
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ The reader is a competent reader of THIS codebase. A comment or a paragraph earn
|
||||
|
||||
## Where a choice goes
|
||||
|
||||
**A choice that shapes what the code does, made in one place, is a comment there, with its reason. Made in more than one place, it is a decision, and the decision log owns it.** A decision's rejected alternative, or what it deliberately does not cover, stays out of every call site. A choice found anywhere else — an `AGENTS.md` body, a comment away from the code it governs, or a README section that is neither a goal nor how one is met — moves: to a comment at its one place, or, once made in more than one place, to `docs/decisions.md`, leaving its title in `AGENTS.md`'s list.
|
||||
**A choice that shapes what the code does, made in one place, is a comment there, with its reason. Made in more than one place, it is a decision, and the decision log owns it.** A decision's rejected alternative, or what it deliberately does not cover, stays out of every call site. A choice found anywhere else — an `AGENTS.md` body, a comment away from the code it governs, or a README section that is neither a goal nor how one is met — moves: to a comment at its one place, or, once made in more than one place, to `docs/decisions.md`, leaving its title among `AGENTS.md`'s decision titles unless a project principle there covers it.
|
||||
|
||||
## Comments
|
||||
|
||||
@@ -22,7 +22,7 @@ Default to NO comment. Delete one that is any of:
|
||||
|
||||
**Density is a review signal.** Prose comment lines much past ~10% of a file's non-blank lines mean something is wrong — narrating comments, or code that fails to carry its own meaning. A defect to investigate.
|
||||
|
||||
## Negation, history and limitations: name the action, or say nothing
|
||||
## Negation, history and limitations: name the action, or leave it out
|
||||
|
||||
Prose describing the code as it stands — a comment, a README, an `AGENTS.md`, an API spec — may narrate what is *not* done (a removed step, a call we don't make, a superseded version `we used to X, now Y`, a limitation) in one shape only: where a reader would otherwise **do** something harmful, and the sentence says what to do instead. GOOD: `wrap a direct ./tests/run.sh to put it under the same lock`. BANNED: `run.sh takes no host lock`. A limitation with nothing to do instead belongs in the PR or ticket, because **documenting a limitation is not fixing it**.
|
||||
|
||||
|
||||
@@ -9,5 +9,6 @@ Everything written — chat, commit messages, comments, docs, code — is in **E
|
||||
3. **Default to delete.** Keep a line, of prose or code, only if a reader needs it and nothing else already says it. Adding words is a deliberate act: name what a reader could not understand without them.
|
||||
4. **Short.** Cut whatever does not clearly help 1–3.
|
||||
|
||||
- **Write the thing, not an introduction to it.** A sentence announcing what follows, or flagging that it deserves attention, carries nothing — if it matters, it is in what follows. A bare label is fine. BANNED: `Here's the new audience text, which is the part that needs your eye:`, `One thing I changed that deserves your attention.`, `One judgment call I'd flag rather than bury.`, `Three things came along that would otherwise have been left false.` GOOD: `New audience text:`, or the list alone.
|
||||
- **Write the thing, not an introduction to it.** A sentence announcing what follows, or flagging that it deserves attention, doesn't carry anything — if it matters, it is in what follows. A bare label is fine. BANNED: `Here's the new audience text, which is the part that needs your eye:`, `One thing I changed that deserves your attention.`, `One judgment call I'd flag rather than bury.`, `Three things came along that would otherwise have been left false.` GOOD: `New audience text:`, or the list alone.
|
||||
- **Write negations as people say them.** After a verb, or after its preposition, write `doesn't <verb> anything` or `anyone`, never `<verb> nothing` or `no one`; each breach is a 🟡. BANNED: `stops nothing`, `depends on no one`, `so answering in time costs nothing`. GOOD: `doesn't stop anything`, `doesn't depend on anyone`, `so answering in time doesn't cost anything`.
|
||||
- **No contrastive tail.** `X rather than Y`, `not Y but X`, `instead of Y`: state X and stop. The rejected alternative is noise unless naming it stops a reader choosing Y. BANNED: `verified in CI rather than asserted`. GOOD: `verified in CI`.
|
||||
|
||||
@@ -6,29 +6,46 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 300`
|
||||
`Next ID = 325`
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 296 | 0.8.0 | | | **Run the scoring run that releases 0.5.0 to 0.7.0 skipped, and make a due scoring run fail a check the worker cannot dismiss.** | 2 | 2 | 3 | 5 | 3 | code health by 4 | 7.2 |
|
||||
| 294 | 0.8.0 | | | **Check the files a commit stages against rule-hook's file rules, so a file a Bash command wrote is held to them as an Edit or Write is.** | 2 | 2 | 3 | 5 | 4 | 1.3 by 5 | 5.2 |
|
||||
| 297 | 0.8.0 | | | **Have loop-claude compute the judge's taken-in items, release-ready and escalation fields from `chunk-preflight`, git and the worker's structured lines, and leave the judge only the verdict kind.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 2, code health by 2 | 3.7 |
|
||||
| 293 | 0.8.0 | | | **Run mutation tests on a chunk's changed lines in the worker's pre-review gate, through one command that picks the tool by language, and hand stability-reviewer the surviving mutants.** | 4 | 2 | 3 | 7 | 5 | 1.1 by 5, 1.4 by 4, 1.5 by 2 | 2.8 |
|
||||
| 284 | 0.8.0 | | owner | **Script each step that 0.7.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.6 |
|
||||
| 270 | 0.8.0 | | | **Move each check a review pass repeats across chunks, other than the mutation tests `bin/mutate-changes` runs, into the worker's pre-review gate, starting with the passes most often taken to a second round.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 3, 1.4 by 3 | 2.5 |
|
||||
| 292 | 0.8.0 | | owner | **Rebuild the routing ladder for Fable 5.2, placing each of its efforts by the Artificial Analysis Intelligence Index beside Opus 5.5's and Sonnet 5.5's, and replace Fable 5.1 wherever it is named.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 4, 1.4 by 3 | 2.4 |
|
||||
| 320 | 0.8.0 | | | **Bench a worker below the project's rung with the advisor tool against the same worker alone and the project's rung, and file the switch if it wins.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3, 1.1 by 2 | 2.0 |
|
||||
| 315 | 0.8.0 | | | **Start each reviewer under `agents/` without `CLAUDE.md`, and hand it the vocabulary, the prose rules and the project's decision titles.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 6 | 2.0 |
|
||||
| 312 | 0.8.0 | | | **Research the tooling agents read, search and run commands with, and file an item for each tool that cuts tokens a replay shows keep quality.** | 1 | 2 | 1 | 2 | 3 | 1.4 by 3, 1.5 by 2 | 2.0 |
|
||||
| 305 | 0.8.0 | | | **Record with each archived verdict the claude-config commit that ran it, the raw tokens each model spent by kind, and each choice of the worker's that the human reversed, so a change to the setup can be judged before and after.** | 3 | 2 | 3 | 6 | 5 | 1.4 by 4, 1.2 by 3 | 1.8 |
|
||||
| 318 | 0.8.0 | | | **Let `review-replay` qualify a rung whose recall is no lower than the project's rung's own replay, and replay Opus 5.5 at low effort on the prose pass.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 5 | 1.7 |
|
||||
| 308 | 0.8.0 | 293 | | **Skip a stability round 2 whose every must-fix finding named a surviving mutant or a failing test that the pre-review gate now kills.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3, 1.5 by 2 | 1.6 |
|
||||
| 295 | 0.8.0 | | | **Have `chunk-pr merge` wait for the forge's required checks itself, and give workers one command that waits for a background log or report file.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 2, 1.5 by 2 | 1.6 |
|
||||
| 319 | 0.8.0 | | | **Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 2, 1.4 by 2 | 1.4 |
|
||||
| 309 | 0.8.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.3 |
|
||||
| 299 | 0.8.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.2 |
|
||||
| 298 | 0.8.0 | 297 | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 1.0 |
|
||||
| 311 | 0.8.0 | | | **Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.2 |
|
||||
| 316 | 0.8.0 | | | **Launch comprehension seats as an agent type of their own under `agents/`, without `CLAUDE.md` and the skill and agent listings a general-purpose subagent carries.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.2 |
|
||||
| 310 | 0.8.0 | | | **Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 1.0 |
|
||||
| 298 | 0.8.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 1.0 |
|
||||
| 313 | 0.8.0 | | | **Launch the first agent of each type in a parallel wave one message before the others of that type, so they read its prompt cache.** | 1 | 2 | 2 | 3 | 2 | 1.4 by 2 | 0.9 |
|
||||
| 317 | 0.8.0 | | | **Cut the descriptions of stability-reviewer, product-owner-reviewer and systems-architect to one line each.** | 1 | 1 | 1 | 1 | 1 | 1.4 by 1 | 0.5 |
|
||||
| 303 | 0.9.0 | | defect | **Archive each judge run's prompt and json reply beside the verdict it gave, so judge cost, replays and backups cover the judge as they cover the worker.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3, 3 by 2 | 4.8 |
|
||||
| 304 | 0.9.0 | 303 | defect | **Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 2, 3 by 2 | 4.4 |
|
||||
| 306 | 0.9.0 | | | **Set a strategy that keeps loop-claude's archive and larv-review's review folders backed up and small over the years, compressing older entries without losing any.** | 3 | 2 | 4 | 7 | 5 | 3 by 5 | 3.9 |
|
||||
| 302 | 0.9.0 | | | **Call a scoring run for an item whose tagline makes the comprehension floor its target, and not for one that only names the floor, such as in a quoted decision title.** | 2 | 2 | 3 | 5 | 3 | code health by 2 | 3.6 |
|
||||
| 287 | 0.9.0 | | | **Move the forge helpers `Forge`, `forge_of`, `runner`, `open_pull`, `merged_pull`, `pull_at` and `gitea` into one module that `chunk_preflight`, `chunk_pr`, `deploy_release` and `larv_review` all import from, so no tool imports another tool's internals for them.** | 3 | 1 | 3 | 5 | 5 | code health by 3 | 3.4 |
|
||||
| 284 | 0.9.0 | | owner | **Script each step that 0.7.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.6 |
|
||||
| 288 | 0.9.0 | | | **Point the two reading-cost decisions at code health and goal 1.4's counter-signal, in place of the goal 1.1 they cite.** | 1 | 2 | 1 | 2 | 1 | code health by 1, 1.4 by 1 | 2.6 |
|
||||
| 270 | 0.9.0 | | | **Move each check a review pass repeats across chunks, other than the mutation tests `bin/mutate-changes` runs, into the worker's pre-review gate, starting with the passes most often taken to a second round.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 3, 1.4 by 3 | 2.5 |
|
||||
| 77 | 0.9.0 | | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 4 | 2 | 1 | 5 | 6 | 1.1 by 7, 1.4 by 7 | 2.4 |
|
||||
| 321 | 0.9.0 | 318 | | **Evaluate the review passes that run on a rung item 318 qualified, against the chunks before them, and revert any pass whose findings or later fixes worsened.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3, 1.4 by 2 | 2.3 |
|
||||
| 273 | 0.9.0 | | decision | **Re-tune the E + I steps of `REVIEW_STEPS` for 6 to 8 once 10 finished chunks have scored an E + I of 6 or more, from how well E + I orders their review rounds and 🔴 findings.** | 1 | 2 | 2 | 3 | 2 | 1.1 by 2, 1.4 by 3 | 2.3 |
|
||||
| 268 | 0.9.0 | | | **End a run whose release is ready to cut with a notice on loop-claude's terminal, in place of an interrupt that waits for an answer.** | 2 | 2 | 3 | 5 | 3 | 1 by 5 | 2.2 |
|
||||
| 171 | 0.9.0 | | decision | **Group the routing figures' review rounds by the reviewers' rung as well as the worker's, so a call blames a rung only for rounds its own reviews raised.** | 3 | 2 | 2 | 5 | 4 | 1.2 by 3, 1.4 by 3 | 2.0 |
|
||||
| 300 | 0.9.0 | | owner | **Restructure the places the comprehension seats found hardest to follow, so the project reads at 7.0 or above on every dimension, Locality first: today the comprehension floor run at 52da15e scored Locality 6.00, Shape 6.25, Self-sufficiency 6.38 and Overall 6.38.** | 4 | 2 | 3 | 7 | 8 | code health by 8 | 2.0 |
|
||||
| 264 | 0.9.0 | | | **Report the comprehension seats and the escalated subagents as subagent types of their own in `routing-figures.md`, with their dollars.** | 2 | 2 | 1 | 3 | 3 | 1.4 by 5 | 2.0 |
|
||||
| 323 | 0.9.0 | | | **Count a release review as landed once its pull request has merged, whatever its commits are titled.** | 3 | 2 | 3 | 6 | 3 | 1.4 by 3, 1.5 by 2 | 2.0 |
|
||||
| 149 | 0.9.0 | | decision | **Fill each release by priority up to a capacity it states, in place of the bar, and move what does not fit to a later release.** | 4 | 2 | 3 | 7 | 5 | 1.1 by 7 | 1.8 |
|
||||
| 165 | 0.9.0 | | | **Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 4, 1.4 by 3 | 1.8 |
|
||||
| 151 | 0.9.0 | | decision | **Keep 10 to 20 settled decisions per project as its goals' test set, and rerun them after a goal changes.** | 3 | 2 | 2 | 5 | 5 | 1.2 by 7 | 1.8 |
|
||||
@@ -44,58 +61,164 @@
|
||||
| 201 | 0.9.0 | | owner | **Save the human's tmux setups and restore them after a reboot, so work is up and running again quickly.** | 3 | 2 | 1 | 4 | 4 | 1.5 by 4 | 1.3 |
|
||||
| 267 | 0.9.0 | | | **Keep waiting when an interrupt gets an empty answer, at the terminal or in the claude opened for it, in place of stopping the loop.** | 2 | 1 | 3 | 4 | 2 | 1.5 by 3 | 1.3 |
|
||||
| 4 | 0.9.0 | | | **Name each chunk's archive folder by the chunk's start time, so `<key>/` lists chunks in the order they ran.** | 3 | 2 | 3 | 6 | 3 | 1.1 by 3 | 1.3 |
|
||||
| 324 | 0.9.0 | | | **Count in the routing figures the finished chunks whose worker wrote no `Chunk items:` line.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 2, 1.4 by 1 | 1.2 |
|
||||
| 187 | 0.9.0 | | | **Count the tokens a worker spends reading reviewers' reports, so goal 1.4's signal shows whether report files lowered what a review costs the worker's context.** | 2 | 2 | 1 | 3 | 3 | 1.4 by 3 | 1.2 |
|
||||
| 266 | 0.9.0 | | | **Move the chunks archived under the key `home/lilleman/.claude` to the key `claude-config`, so the routing figures and `score-outcomes` count this repo's chunks as one project.** | 3 | 2 | 2 | 5 | 3 | 1.4 by 3 | 1.2 |
|
||||
| 181 | 0.9.0 | | principle | **Have the test of a quoted answer's step-up assert the stepped rung's facts as literals, not through `prompts.rungs_held`.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 1.0 |
|
||||
| 192 | 0.9.0 | | principle | **Sort `.gitignore`'s allowlist block, from `!.claude/` to `!workflow-bench/test_review_git.py`, by code point, so rule-hook's `ignorefile-sorted` checks later edits to it again.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 1.0 |
|
||||
| 193 | 0.9.0 | | | **Narrow `readme_goals`'s mark to `Mark` by a typed lookup in place of `typing.cast`, so a mark added to the regex but not the lookup fails at run time, and one added to the lookup but not `Mark` fails mypy.** | 1 | 1 | 3 | 3 | 1 | 1.1 by 2 | 1.0 |
|
||||
| 194 | 0.9.0 | | | **Make `test_waits_out_a_usage_limit_which_ends_a_run_of_failures` expect a usage-limit wait independent of the wall-clock time that passes during the run.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 1.0 |
|
||||
| 202 | 0.9.0 | | | **Add `CLAUDE.md` → "Prose"'s rule "Write negations as people say them", with its examples, to `skills/larv-review/standards/prose.md`, so the prose reviewer holds prose to it.** | 1 | 2 | 2 | 3 | 1 | 1.1 by 2 | 1.0 |
|
||||
| 265 | 0.9.0 | | | **Archive each wait that costs a chunk wall time, for a usage limit, low memory, a failed worker start or a heavy slot, with its start and end stamps, so goal 1.5's hours split waiting from work.** | 3 | 2 | 2 | 5 | 4 | 1.5 by 3 | 1.0 |
|
||||
| 169 | 0.9.0 | | | **Have rule-hook's `todo-rank-check` catch a `todo.md` that the same Bash call edits and then commits.** | 3 | 1 | 3 | 5 | 3 | 1.1 by 2 | 0.8 |
|
||||
| 174 | 0.9.0 | | | **In a chunk no item routed, ask the worker to name each todo.md item it takes, with its L and E + I, so loop-claude can route on an item it takes in.** | 1 | 2 | 3 | 4 | 1 | 1.1 by 1 | 0.5 |
|
||||
| 197 | 0.9.0 | | | **Let the human edit what they type at loop-claude's terminal questions with the arrow keys, so Left moves the cursor in place of printing `^[[D`.** | 2 | 2 | 3 | 5 | 2 | 1.5 by 1 | 0.4 |
|
||||
| 1 | 0.10.0 | | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 1 | 2 | 1 | 2 | 1 | | 0.0 |
|
||||
|
||||
## Details
|
||||
|
||||
### 296. Run the scoring run that releases 0.5.0 to 0.7.0 skipped, and make a due scoring run fail a check the worker cannot dismiss.
|
||||
|
||||
The last scoring run in `docs/comprehension-history.md` is from 2026-10-29: overall 6.00, Locality 5.63, under the 7.0 floor. v0.5.0, v0.6.0 and v0.7.0 were cut without one, and `larv-review places` still prints that run's places, 254 commits stale. Workers twice dismissed `needs-scoring-run`'s due line as preflight reading a stale branch's plan; find why before choosing the check. Without a run, the floor item the scoring run files under 7.0 never fires.
|
||||
|
||||
### 294. Check the files a commit stages against rule-hook's file rules, so a file a Bash command wrote is held to them as an Edit or Write is.
|
||||
|
||||
From 2026-11-08 to 2026-11-10, workers in other projects wrote 1,266 files through Bash (a python heredoc, `sed -i`, `cat >`) against 260 through Edit and Write, and `rule-hook --help` says it doesn't check a file a Bash command writes. So `action-pin`, `image-pin`, `ignorefile-sorted` and the attribution check miss most edits. rule-hook already checks at `git commit` for `todo-rank-check`, so checking staged files there follows a pattern the repo has. Ship item 192 in the same chunk: its unsorted `.gitignore` block is one symptom of this gap.
|
||||
|
||||
### 297. Have loop-claude compute the judge's taken-in items, release-ready and escalation fields from `chunk-preflight`, git and the worker's structured lines, and leave the judge only the verdict kind.
|
||||
|
||||
`bin/loop_claude/cycle/judge.py` already checks quotes in code (`quotes_checked`, `grounded`, `re_ask_checked`). The `Reply` fields `items_taken_in`, `release_ready` and `escalation` state facts the plan and git already hold, so code can derive them with certainty, and the judge's prompt shrinks to what needs judgement. Before switching, recompute each field for the 668 archived verdicts offline and count where code and judge disagree; read each disagreement and keep the judge for a field where code is wrong. Seats found `quotes_checked` hard to read (PR #397's scoring run: the misquote rule is split across `judge.py` and `escalation_mark.py`); make it easier to read where the change touches it.
|
||||
|
||||
### 293. Run mutation tests on a chunk's changed lines in the worker's pre-review gate, through one command that picks the tool by language, and hand stability-reviewer the surviving mutants.
|
||||
|
||||
Of stability-reviewer's 428 round-1 findings from 2026-11-02 to 2026-11-10, 223 (52%) were test gaps, such as a surviving mutant or an unpinned boundary, and 31 of its 50 round 2s held one. 133 of its 144 reports ran mutation tests by hand: today the reviewer picks and runs mutmut, Stryker, cargo-mutants or Infection itself, per `agents/stability-reviewer.md` step 5. A command run once before review, on the lines the branch changed, finds the same survivors deterministically, so the worker kills them before review and the reviewer judges only what remains. This takes over the stability half of item 270. Ship it alone: the nearest finished items, 227, 210, 180 and 73, changed 261 to 402 lines and took 3 to 8 rounds past the first.
|
||||
|
||||
L is 4, re-scored on 2026-11-11 after the first read: the change is S 6 or more, spans several components (a new `bin/` tool, `skills/larv-review/SKILL.md`, `agents/stability-reviewer.md`, `bin/rule_hook/heavy_command.py`) and adds a new tool. mutmut 3 doesn't print the source line a mutant changes. In its `mutants/` copy, each mutant `x_<function>__mutmut_<n>` keeps the original's lines, tabs included, so the first line other than the `def` where it differs from `x_<function>__mutmut_orig` sits at the same offset from the source function's `def`; on 2026-11-11 that mapped `records/markdown_text.py`'s four survivors to their lines.
|
||||
|
||||
### 284. Script each step that 0.7.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.
|
||||
|
||||
lilleman asked, on 2026-10-16, for a standing audit each release, so the loop keeps moving work a program does as well from the model to scripts (goals 1.4 and 1.5). This audit reads the archived transcripts of the chunks that shipped release 0.7.0 for every step that chunks repeated by hand, before, during or after review. It keeps the steps whose answer a program computes the same way each time, such as a lookup, a check, a prompt assembled for a reviewer, a cleanup. File one item per such step, with the time and tokens it cost, into the next release with items left. Leave a step that needs judgement with the model. Run item 283's tool first, once it has shipped.
|
||||
|
||||
The chunk that ships this item files a copy of this audit in the release after this item's own, to audit this item's release, rewording "0.7.0" to the release that copy audits, and marks the copy `owner`, so the audit recurs each release until lilleman drops it.
|
||||
|
||||
### 270. Move each check a review pass repeats across chunks, other than the mutation tests `bin/mutate-changes` runs, into the worker's pre-review gate, starting with the passes most often taken to a second round.
|
||||
|
||||
The review folders under `~/.local/state/larv-review/`, review replays left out, hold a stability round 2 beside 137 of 211 round-1 reports (65%), product-owner 73 of 173 (42%) and prose 8 of 277 (3%), on 2026-11-08. The architecture and principles passes run up to five rounds in a release review, so they don't compare. From 2026-11-02 to 2026-11-10, 223 of stability's 428 round-1 findings (52%) were test gaps, and 31 of its 50 round 2s held one; `bin/mutate-changes` now runs the mutation tests that find most of them before review. Check which kinds of finding the other round 2s repeat, such as an unhandled error or a product-owner wording fix. Move only a check a program or the worker can run before review; leave judgement with the reviewer.
|
||||
|
||||
### 292. Rebuild the routing ladder for Fable 5.2, placing each of its efforts by the Artificial Analysis Intelligence Index beside Opus 5.5's and Sonnet 5.5's, and replace Fable 5.1 wherever it is named.
|
||||
|
||||
Fable 5.2 (`claude-fable-5-2`) has been released. The ladder in `bin/model_routing/ladder.py` tops out at `Rung("claude-fable-5-1", "max")`, and the decision `routing: one ladder of (model, effort) rungs` holds "while no newer model joins the catalog", so it is due for a rebuild.
|
||||
|
||||
Read the index again (https://artificialanalysis.ai/leaderboards/models), at each effort it scores, with its cost per task. Rebuild the ladder by the decision's cut: a rung stays only where no cheaper rung scores as high. Fable 5.2 may earn more than one rung, and a cheaper effort of it may push an Opus 5.5 rung off. Where the order changes which rung a project's model starts on, or how far the L and E + I steps reach, say so in the decision. The rungs are the human's choice (`CLAUDE.md` → "Working loop", "Ask, don't guess"), so propose the new ladder with the index's figures and get the human's approval before changing code.
|
||||
|
||||
Then replace `claude-fable-5-1` where it is named: `ladder.py`, `bin/routing_figures/prices.py` (Fable 5.2's prices), `settings.json` → `modelSettings`, the tests in `bin/model_routing/`, `bin/loop_claude/`, `bin/judge_replay/` and `bin/rule_hook/`, `AGENTS.md`, `docs/decisions.md` and `presentation/index.html`. Keep Fable 5.1 in `prices.py` while archived chunks that ran on it are still priced from there.
|
||||
Then replace `claude-fable-5-1` where it is named: `ladder.py`, `bin/routing_figures/prices.py` (Fable 5.2's prices), `settings.json` → `modelSettings`, the tests in `bin/model_routing/`, `bin/loop_claude/`, `bin/judge_replay/` and `bin/rule_hook/`, `docs/decisions.md` and `presentation/index.html`. Keep Fable 5.1 in `prices.py` while archived chunks that ran on it are still priced from there.
|
||||
|
||||
### 320. Bench a worker below the project's rung with the advisor tool against the same worker alone and the project's rung, and file the switch if it wins.
|
||||
|
||||
Anthropic reports Sonnet 4.6 with an Opus 4.6 advisor scored 2.7 points more on SWE-bench Multilingual at 11.9% less cost than Sonnet alone (the vendor's own run; https://claude.com/blog/the-advisor-strategy). The docs say `/advisor` works with `-p`, but not outright that the advisor is called during a `-p` run, so probe that first. Each advisor call hands the advisor the whole conversation, so count its cost. Bench on `workflow-bench` with archived L 1 and L 2 items: Sonnet with the advisor, Sonnet alone, Opus at medium. lilleman approved on 2026-11-11.
|
||||
|
||||
### 315. Start each reviewer under `agents/` without `CLAUDE.md`, and hand it the vocabulary, the prose rules and the project's decision titles.
|
||||
|
||||
`omitClaudeMd: true` in an agent's frontmatter (Claude Code 2.1.271 or later) starts it without the user, project and local instruction files. About 77% of `CLAUDE.md` is the worker's: "Working loop", "Working with the codebase", "Git", "Handoffs", "Compact instructions". Over the archive, each 1,000 prefix tokens cost $31 across the six reviewers, so this saves about 1.9% of spend. A reviewer still needs "Vocabulary", "Prose" and the project's decision titles: put the first two in the agent bodies or a `SubagentStart` hook's `additionalContext`, and have `larv-review reviewer-header` print the titles.
|
||||
|
||||
Gate it per pass: keep it only where `review-replay` with the changed definition (through `--agents`) on 10 archived rounds finds what two replays of the unchanged reviewer find; the unchanged reviewer re-finds only 69 to 73% of its own findings, so one baseline doesn't show the noise. Revise the decision `rules/: CLAUDE.md holds the worker's rules that no file path can scope`. lilleman approved on 2026-11-11.
|
||||
|
||||
### 312. Research the tooling agents read, search and run commands with, and file an item for each tool that cuts tokens a replay shows keep quality.
|
||||
|
||||
lilleman, 2026-11-11: besides an output ceiling, keep evaluating better tooling. The token research of that day found tool results carried in context cost about 21% of spend, reads by subagents the most, and that the tools themselves move fast: deferred tool loading, output saved to files, code-execution over tools, LSP and repo-map navigation, structured test output. Item 231 tried a scout, an outline script and LSP in October and none kept every finding.
|
||||
|
||||
Run a research round like that of 2026-11-11: a lead on the project's rung plans the searches, cheaper subagents search and distil sourced findings, and the lead checks every claim it relies on against this archive's measured spend. File an item for each tool that a replay could test. lilleman, 2026-11-11: repeat the round once per claude-config release, not per project. When this item's chunk finishes, file the next round into the next claude-config release with items left, so the check comes back periodically.
|
||||
|
||||
### 305. Record with each archived verdict the claude-config commit that ran it, the raw tokens each model spent by kind, and each choice of the worker's that the human reversed, so a change to the setup can be judged before and after.
|
||||
|
||||
Three records the archive lacks, asked for by lilleman on 2026-11-11 after the token research of that day.
|
||||
|
||||
1. **The setup that ran.** A verdict names the worker's model, effort and item, but not the `~/.claude` tag and commit the chunk ran under. Judging any change to the setup means comparing chunks before and after a deploy, and deploys lag behind cuts, so timestamps can't tell. Add a line such as `Config: v0.7.0 (<sha>)` to each verdict, interrupt and answer.
|
||||
2. **Spend in raw tokens.** Record what each cycle spent as raw token counts per model and per kind: input, cache write 5-minute, cache write 1-hour, cache read, output, thinking where known. Do it for the worker's main thread, each subagent (by agent type and description) and the judge. Dollars are derived later from model and tokens; as a secondary bonus, record the price table in force when the spend was made (from `bin/routing_figures/prices.py`), so a later price change doesn't rewrite history. The token research measured this by hand on 2026-11-11 and had to deduplicate cumulative worker snapshots by `message.id`: count each message once.
|
||||
3. **Reversals.** The decision of 2026-11-08 (item 262) lets a worker take its own recommendation on an unreserved choice "while the human reverses few of the choices workers take", and no tool counts those reversals. Have the judge flag an answer the human typed that overrides a choice the worker took and recorded, and count them per chunk in `score-outcomes` beside later fixes and misread findings. This is the quality signal a cheaper setup could quietly worsen.
|
||||
|
||||
Build on item 303's archive layout for the judge's part.
|
||||
|
||||
### 318. Let `review-replay` qualify a rung whose recall is no lower than the project's rung's own replay, and replay Opus 5.5 at low effort on the prose pass.
|
||||
|
||||
Today a rung qualifies only if it re-finds every finding the project's rung found. The project's rung itself re-finds only 69 to 73% of its own findings on a replay (29 of 40, 27 of 39, 20 of 29), so even a reviewer exactly as good would fail, and "no pass has such a rung" says less than it seems. Qualify a rung when its share of the archived findings is no lower than the project's rung's own replay share, within a margin, over enough rounds, and it misses no 🔴 that both project-rung runs found. Then replay Opus 5.5 at low effort on the prose pass first, the pass whose misses cost least; the `effort` frontmatter key sets it per reviewer.
|
||||
|
||||
lilleman approved on 2026-11-11 as somewhat risky, with a close evaluation in the next release: item 321. Revise the decision `routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds…`, and its title at the head of `docs/decisions.md`.
|
||||
|
||||
### 308. Skip a stability round 2 whose every must-fix finding named a surviving mutant or a failing test that the pre-review gate now kills.
|
||||
|
||||
Rounds past the first cost $846 of the $9,154 the archive measured from 2026-09-30 to 2026-11-11; stability's were $281 over 440 runs. Once item 293 runs the mutation tests in the worker's gate, a 🔴 that named a surviving mutant or a failing test has a mechanical answer: the gate's run on the fix kills the mutant, or the test passes. A round 2 whose every 🔴 and 🟡 is of that kind needs no reviewer.
|
||||
|
||||
Round 2 still matters elsewhere: 53 of 241 archived stability reports of round 2 or later open with BLOCK. So first count, from the archived round 2s, the BLOCKs the gate's run on the fix would not have raised, and ship the skip only if none falls in a round it would skip. Evidence: one review round scored F1 0.376 and a second 0.26 to 0.30 with more false positives (arXiv 2603.16244, `claude -p` with Opus 4.6, 30 artifacts). Revise larv-review §3 → "Re-entry" and the decision `review: a fix for a 🔴 reruns the pass that found it…`.
|
||||
|
||||
### 295. Have `chunk-pr merge` wait for the forge's required checks itself, and give workers one command that waits for a background log or report file.
|
||||
|
||||
In 0.7.0's 35 chunks, workers made 65 `tea api` calls in 26 chunks, mostly polling commit status and trying merge methods in turn, and ran 25 `until … sleep` loops that took 1,744 seconds. `chunk-pr merge` takes `--gate-passed` on the worker's word; waiting for the required checks itself makes that flag a fact, not a claim. This is the kind of step item 284 files, and can count as its first.
|
||||
|
||||
### 319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
|
||||
|
||||
On SWE-bench Verified with Haiku 4.5 and Opus 4.7, handing the strong model the weak model's transcript solved 69.2% at $1.61 a task; starting the strong model fresh solved 79.2% at $0.90 (arXiv 2608.24358, The Handoff Tax). A model switch also rereads the whole context uncached (Claude Code prompt caching docs). A worker that steps from Sonnet 5.5 to Opus 5.5 by `--resume` is the raw handoff; an effort-only step within one model keeps the cache and isn't. The escalated subagent, which starts fresh with the trigger's evidence, already follows the pattern that won.
|
||||
|
||||
Bench it first on `workflow-bench`: resume against a fresh start with a brief. The brief is the chunk's item, its branch and diff, open findings and the trigger. lilleman approved on 2026-11-11. Revise the decision `routing: a worker whose run ends asking for a higher rung…`.
|
||||
|
||||
### 309. Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.
|
||||
|
||||
Seats cost $1,143 of the $9,154 measured from 2026-09-30 to 2026-11-11 (12.5%), scoring runs $470 of that. A scoring run today is four seats, run twice on one head: eight seat runs. Two seats on the second pass make it six, and two on both passes four.
|
||||
|
||||
Why it should hold quality: the seats are one model with different personas, so their errors are correlated, and a correlated extra seat adds little. `larv-review agreement` showed on 2026-11-11 that Mid A and the inherited architect voted alike in 58 of 75 paired rulings, while their scores moved alike between two runs on one commit in only 1 of 4 pairs: seats agree with each other more than a run agrees with its rerun, so the rerun carries more new information than the third and fourth seat. Nine judges from seven model families carried about two independent votes (arXiv 2605.29800), and ten with error correlation 0.21 carried 3.5 (arXiv 2609.22512).
|
||||
|
||||
Decide once `agreement` has its 10 commits scored twice. Keep four seats on any pass that files a floor item, since its places come from seat reports. Revise the decision `review: the comprehension check records how hard a project is to read…`, which names the four-seat run.
|
||||
|
||||
### 299. Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.
|
||||
|
||||
The judge's verdict kind is the one judge decision that item 297 leaves an LLM. No single rule found so far separates interrupts: a "?" in the worker's final message appears in 48 interrupts and misses 66. A smaller LLM already failed the replay (`docs/decisions.md` → `routing: on a model judge-replay covered, …`: Sonnet returned another kind on 3 interrupts, Haiku missed 7). Untested are combined code rules, a decision table, and a small classifier trained and run on this machine on the archived verdicts (372 finished, 234 interrupt, 42 unfinished, 20 quoted). Run each offline, nothing sent to an outside service (goal 4), and count disagreements by kind; the bar is zero missed interrupts. A hosted decision model is out of scope, since it would send transcripts to an outside service.
|
||||
The judge's LLM still decides three things: the verdict kind, the escalation it quotes, and whether a release is ready to cut (`docs/decisions.md` → `routing: loop-claude reads the items a chunk takes …`). This item replays the verdict kind. No single rule found so far separates interrupts: a "?" in the worker's final message appears in 48 interrupts and misses 66. A smaller LLM already failed the replay (`docs/decisions.md` → `routing: on a model judge-replay covered, …`: Sonnet returned another kind on 3 interrupts, Haiku missed 7). Untested are combined code rules, a decision table, and a small classifier trained and run on this machine on the archived verdicts (372 finished, 234 interrupt, 42 unfinished, 20 quoted). Run each offline, nothing sent to an outside service (goal 4), and count disagreements by kind; the bar is zero missed interrupts. A hosted decision model is out of scope, since it would send transcripts to an outside service.
|
||||
|
||||
### 311. Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.
|
||||
|
||||
Workers took 7,870 Bash results over 3,000 characters, $305 of carried context, against the reading rule in their prompt; stability reviewers another $145. `BASH_MAX_OUTPUT_LENGTH` (or `bashOutputMaxChars`) makes Claude Code save a longer result to a file and hand the model its path and the first 2,000 characters. loop-claude can hand it as it hands `CLAUDE_CODE_AUTO_COMPACT_WINDOW`. A ceiling of 6,000 characters would have removed $228 of carried output, 10,000 characters $122. Reads of old session transcripts cost $46; a rule-hook rule can refuse them.
|
||||
|
||||
Subagents inherit the variable, and the decision of 2026-10-27 found a 12k cap cost stability reviewers a finding (3 of 6 against 4 of 6). That cap left no file to read, and the difference is inside the noise that decision reports. So keep the ceiling only where a replay of the archived stability rounds finds what two baseline replays find, and join item 260's comparison.
|
||||
|
||||
### 316. Launch comprehension seats as an agent type of their own under `agents/`, without `CLAUDE.md` and the skill and agent listings a general-purpose subagent carries.
|
||||
|
||||
Seats run as general-purpose subagents, so each starts with the skill and agent listings, about 7,500 tokens, at $9.00 per 1,000 prefix tokens over the archive measured from 2026-09-30 to 2026-11-11: about 1.3% of spend. An agent under `agents/` with Read, Grep, Glob and Bash, `omitClaudeMd: true`, and the seat's persona in its prompt drops both; a seat reads the project as a newcomer does, which is what it simulates. Keep each seat's prompt from `skills/comprehension-panel/seat-prompts.md`. Check that a scoring run's scores on one head stay within the spread two runs of today's seats show. lilleman approved on 2026-11-11.
|
||||
|
||||
### 310. Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.
|
||||
|
||||
281 worker resumes after the 1-hour cache expired rewrote a whole context, $379 of the $9,154 measured from 2026-09-30 to 2026-11-11 (4.1%). A compaction while the cache is warm reads the context at a twentieth of the input price and leaves about 20k tokens to rewrite later.
|
||||
|
||||
lilleman, 2026-11-11: do it on a timer, so an answer that comes in time doesn't cost anything. When loop-claude interrupts, start a timer a margin short of the cache lifetime (about 50 minutes of a 1-hour cache; read the lifetime from the session's last `cache_creation` usage). If no answer has come when it fires, compact the session, then keep waiting. Skip it below about 60k tokens of context, where the rewrite costs less than the compaction. A wait for a usage limit can't be helped this way.
|
||||
|
||||
Probe first that `claude -p "/compact" --resume <id>` compacts a session: the Agent SDK documents `/compact` as a prompt, the CLI pages don't show it. Claude Code compacts idle interactive sessions itself ("Compacted while idle"), but not `-p` runs. Check: the cost entries of 20 chunks with an interrupt, before and after.
|
||||
|
||||
### 298. Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.
|
||||
|
||||
153 `Judge cost:` entries total $109.76, a median of about $0.76 a verdict at every rung, and the cost follows the size of the records, not the transcript: about 76k characters in log and 188k in claude-config. In 224 of 251 recent verdicts (89%) the evidence sits in the worker's final message. A missed interrupt is the costly error, so the bar is zero changed interrupts and zero new rejections across the replay. It waits for item 297, which moves the judge's factual fields into code, since that shrinks the prompt first and changes what this cut measures. The judge is about 4.5% of dollars, so the saving is small; each cycle also gets faster.
|
||||
153 `Judge cost:` entries total $109.76, a median of about $0.76 a verdict at every rung, and the cost follows the size of the records, not the transcript: about 76k characters in log and 188k in claude-config. In 224 of 251 recent verdicts (89%) the evidence sits in the worker's final message. A missed interrupt is the costly error, so the bar is zero changed interrupts and zero new rejections across the replay. The judge is about 4.5% of dollars, so the saving is small; each cycle also gets faster.
|
||||
|
||||
### 313. Launch the first agent of each type in a parallel wave one message before the others of that type, so they read its prompt cache.
|
||||
|
||||
A cache entry is readable only once the first response that wrote it has started ("If you need cache hits for parallel requests, wait for the first response before sending subsequent requests", Anthropic's prompt caching docs). Between 2026-09-30 and 2026-11-11, 748 comprehension seats started within 90 seconds of another seat in the same chunk and each still wrote its own prefix, $73; so did 153 security passes ($17) and 130 general-purpose subagents ($9). Different agent types share no prefix, so only agents of one type gain.
|
||||
|
||||
Change `skills/comprehension-panel/SKILL.md` → "Parallel" and `skills/larv-review/SKILL.md` → "Waves": launch one agent of a type, and the others of that type in the next message. Quality doesn't change, since every agent gets the same prompt and model; the cost is one extra worker turn, about $0.02, and a few seconds. Check: the first-turn `cache_read_input_tokens` of seats two to four in the next scoring run.
|
||||
|
||||
### 317. Cut the descriptions of stability-reviewer, product-owner-reviewer and systems-architect to one line each.
|
||||
|
||||
Their descriptions, 7.5 KB together, sit in the agent listing of every worker and general-purpose subagent in every project, about 0.7% of spend. larv-review launches them by name, so its routing doesn't depend on them. The one downside: a session outside larv-review relies on a description to decide when to launch an agent unasked, so a one-line description makes that less likely. Keep the line specific enough that a reader knows when to pick each. lilleman approved on 2026-11-11.
|
||||
|
||||
### 303. Archive each judge run's prompt and json reply beside the verdict it gave, so judge cost, replays and backups cover the judge as they cover the worker.
|
||||
|
||||
Today no judge transcript reaches loop-claude's archive. `bin/claude_code/reply.py` runs every headless reply, the judge's included, with `--no-session-persistence`, so its runs stay out of the session picker; Claude Code then writes no session file for the archive to copy. No decision chose to leave the judge out: `loop-claude: the archive keeps the raw transcript` was written for the worker's `.jsonl`. Goal 3 asks for the transcripts of unattended runs to be backed up, and the judge is one. The gap also leaves the judge's spend measurable only from the `Judge cost:` lines, which older verdicts lack ($136 recorded over 2026-09-30 to 2026-11-11), and leaves `judge-replay` rebuilding each prompt from today's records: of the 39 verdicts it replayed on 2026-10-15, 8 may rest on records that had changed since.
|
||||
|
||||
The judge runs with no tools and one turn (`judge_args` in `bin/loop_claude/cycle/judge.py`), so its whole transcript is the prompt it read and the json reply. Recommended: keep `--no-session-persistence`, and have the archive write the prompt and the reply beside the outcomes entry of the verdict, interrupt or rejected reply they produced, deduplicated by content hash as the worker's copies are. A failed write stops the loop, as that decision says. Persisting the session and copying its file would also work, but fills the picker and needs a cleanup. The prompt repeats the worker's transcript, which the archive already holds: measure the size per cycle first, and if it outgrows "small enough to copy whole", keep the prompt without the transcript part and name the worker copy and entry range it held. Then let `judge-replay` read the archived prompt where one exists, and price the judge from the archived reply's usage. Update that decision entry to say the judge's runs are archived too.
|
||||
|
||||
### 304. Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.
|
||||
|
||||
Today `review-replay` and `judge-replay` leave no transcript: both run claude through `claude_reply` in `bin/claude_code/reply.py`, whose `--no-session-persistence` makes Claude Code write no session file, as item 303 describes for the judge. A replay decides which rung a review pass or the judge may run on (`routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows…`, and the judge's rung on `JUDGE_REPLAYED`), so the runs behind a routing decision can't be read again, re-priced or re-scored once the decision is questioned. Goal 3 asks for the transcripts of unattended runs, the bench's included, to be backed up; `workflow-bench/bench.py` already keeps its claude stream in `logs/`.
|
||||
|
||||
Unlike the judge, a replayed reviewer uses tools over several turns, so its transcript is more than its prompt and reply. Recommended: have `claude_reply` take `--output-format stream-json --verbose` in place of `json`, keep `--no-session-persistence`, write the stream where the replay keeps its result, and read the reply from the stream's last `result` event. That also gives item 303 the judge's stream for free, so build on 303's archive layout and do both callers through the one runner.
|
||||
|
||||
### 306. Set a strategy that keeps loop-claude's archive and larv-review's review folders backed up and small over the years, compressing older entries without losing any.
|
||||
|
||||
Measured on 2026-11-11: loop-claude's archive is 6.3 GB (12,721 files from October alone, 0.84 GB older than 30 days), and `~/.local/state/larv-review/` is 997 MB, 984 MB of it under `code/`, with no pruning: `skills/larv-review/SKILL.md` → "Reports" keeps every folder after the merge, because `review-replay` replays rounds from them. The largest archived worker transcript, 23.4 MB, gzips to 9.1 MB at level 6. The disk holds 65 GB free today.
|
||||
|
||||
Write the strategy as a decision, then build it:
|
||||
- **Backup.** Item 1 waits on ai-sandbox's data volume backup; check that both `~/.claude/loop-claude-transcripts` and `~/.local/state/larv-review` sit on that volume, since the review folders live outside `~/.claude`.
|
||||
- **Retention.** Keep everything: the archive is the evidence every routing and replay decision rests on. Compress entries older than a threshold (such as 30 days, or once a chunk is finished and measured) with a format the tools read transparently, such as zstd. Compress, verify the round trip, then remove the original, never the reverse; a failure keeps the original and stops, as a failed archive does.
|
||||
- **Readers.** `bin/transcript_archive`, `routing_figures`, `score-outcomes`, `review-replay`, `judge-replay` and `reading-cost` must read compressed entries; test each on one.
|
||||
- Revise `loop-claude: the archive keeps the raw transcript`, whose validity is "while transcripts stay small enough to copy whole".
|
||||
|
||||
### 302. Call a scoring run for an item whose tagline makes the comprehension floor its target, and not for one that only names the floor, such as in a quoted decision title.
|
||||
|
||||
`bin/larv_review/scoring.py` sets `FLOOR_ITEM = "comprehension floor"` (line 7) and tests it as a substring of the tagline at line 35 and in `bin/larv_review/plan.py:28`. So any item whose tagline holds those words counts as a floor item. Item 301's tagline quoted the title of the decision "comprehension floor: …", so its chunk was due a scoring run it did not need. The deployed `chunk-pr` (release 0.7.0) did not refuse the merge, so it merged without one, and the run was recorded late, in PR #476.
|
||||
|
||||
Recommendation: a floor item's tagline opens with one fixed phrase, `Reach the comprehension floor:`, and both checks test that the tagline, with its leading `**` stripped, starts with it, in place of the substring test. Reason: a quoted title or a passing mention never sits at a tagline's start, so the check needs no parsing and keeps one constant. An Exempt reason such as `floor` was the alternative; it would change `rank_todo.py`'s schema and every project's table for the same effect.
|
||||
|
||||
The same chunk names the phrase where a floor item is filed: `skills/larv-review/SKILL.md` → "Floor item", the `Next:` lines `floor_step` prints, and `larv-review --help` (`bin/larv_review/main.py`, lines 35 and 89). It rewords open floor item 300's tagline to open with the phrase, and updates the tests in `bin/larv_review/test_larv_review.py` that use floor taglines (lines 120, 361, 435, 537), adding one where a tagline quotes the floor mid-sentence and is due no run.
|
||||
|
||||
### 287. Move the forge helpers `Forge`, `forge_of`, `runner`, `open_pull`, `merged_pull`, `pull_at` and `gitea` into one module that `chunk_preflight`, `chunk_pr`, `deploy_release` and `larv_review` all import from, so no tool imports another tool's internals for them.
|
||||
|
||||
@@ -103,22 +226,12 @@ Today the helpers live in two tools. `Forge` is defined in `bin/chunk_preflight/
|
||||
|
||||
Both comprehension seats of PR #453's paired ruling (2026-11-09) named this web as a place that cost them time, and neither saw a reason for it: a second forge would need edits in `chunk_preflight`, `deploy_release`, `chunk_pr` and `larv_review`.
|
||||
|
||||
Put the seven helpers in a folder named by what it holds, such as `bin/forge/`, and import them from there. Keep each helper's behaviour and its tests. `chunk_preflight/choices.py` keeps `pulls_command`, which is preflight's own. Check `AGENTS.md` and `docs/decisions.md` for a mention of where a helper lives, and update it.
|
||||
|
||||
### 284. Script each step that 0.7.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.
|
||||
|
||||
lilleman asked, on 2026-10-16, for a standing audit each release, so the loop keeps moving work a program does as well from the model to scripts (goals 1.4 and 1.5). This audit reads the archived transcripts of the chunks that shipped release 0.7.0 for every step that chunks repeated by hand, before, during or after review. It keeps the steps whose answer a program computes the same way each time, such as a lookup, a check, a prompt assembled for a reviewer, a cleanup. File one item per such step, with the time and tokens it cost, into the next release with items left. Leave a step that needs judgement with the model. Run item 283's tool first, once it has shipped.
|
||||
|
||||
The chunk that ships this item files a copy of this audit in the release after this item's own, to audit this item's release, rewording "0.7.0" to the release that copy audits, and marks the copy `owner`, so the audit recurs each release until lilleman drops it.
|
||||
Put the seven helpers in a folder named by what it holds, such as `bin/forge/`, and import them from there. Keep each helper's behaviour and its tests. `chunk_preflight/choices.py` keeps `pulls_command`, which is preflight's own. Check `docs/decisions.md` for a mention of where a helper lives, and update it.
|
||||
|
||||
### 288. Point the two reading-cost decisions at code health and goal 1.4's counter-signal, in place of the goal 1.1 they cite.
|
||||
|
||||
Both `docs/decisions.md` entries under `reading cost: a rise is …` and `reading cost: larv-review measures each finished chunk …` open "Serves goal 1.1: comprehension …". No goal says that any more: code health (`CLAUDE.md` → "Shared goal") holds comprehension, and since PR #455 a rising reading cost is goal 1.4's first counter-signal. Raised by the stability review of PR #455 as a 🟢 outside that change.
|
||||
|
||||
### 270. Move each check a review pass repeats across chunks, other than the mutation tests `bin/mutate-changes` runs, into the worker's pre-review gate, starting with the passes most often taken to a second round.
|
||||
|
||||
The review folders under `~/.local/state/larv-review/`, review replays left out, hold a stability round 2 beside 137 of 211 round-1 reports (65%), product-owner 73 of 173 (42%) and prose 8 of 277 (3%), on 2026-11-08. The architecture and principles passes run up to five rounds in a release review, so they don't compare. From 2026-11-02 to 2026-11-10, 223 of stability's 428 round-1 findings (52%) were test gaps, and 31 of its 50 round 2s held one; `bin/mutate-changes` now runs the mutation tests that find most of them before review. Check which kinds of finding the other round 2s repeat, such as an unhandled error or a product-owner wording fix. Move only a check a program or the worker can run before review; leave judgement with the reviewer.
|
||||
|
||||
### 77. Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.
|
||||
|
||||
2026-10-08, lilleman: no metric enters review until a rerun on more data shows it tracking the panel.
|
||||
@@ -149,6 +262,10 @@ Claude's misses ranked the named units at a coin toss. Top k scores 0.50 by chan
|
||||
|
||||
Claude's answers overlapped the truth by a median Jaccard index of 0.76. A run took in 62,643 new input tokens and cost $0.70; it did not record which model answered. Grep answers these questions whatever the code's shape, so no question enters review; a new question bench would need questions that take understanding the code, not finding it.
|
||||
|
||||
### 321. Evaluate the review passes that run on a rung item 318 qualified, against the chunks before them, and revert any pass whose findings or later fixes worsened.
|
||||
|
||||
lilleman, 2026-11-11: item 318's new bar is a risk, so check it closely a release later. Compare the first 20 chunks reviewed on each newly qualified rung with the 20 before: review rounds past the first, 🔴 and 🟡 per chunk, misread findings, and later fixes that `score-outcomes` counts (defects that escaped review). Revert a pass to the project's rung where any of those worsened beyond the spread of the 20 before.
|
||||
|
||||
### 273. Re-tune the E + I steps of `REVIEW_STEPS` for 6 to 8 once 10 finished chunks have scored an E + I of 6 or more, from how well E + I orders their review rounds and 🔴 findings.
|
||||
|
||||
`docs/decisions.md` → `routing: one ladder…` records why lilleman kept the steps on 2026-11-08. Of the 36 chunks it measured, 1 scored an E + I of 6 or more and none 7 or more, so the steps for 6 to 8 are untested. Start once `score-outcomes` shows 10 finished chunks with an E + I of 6 or more. Then read its rank correlation for E + I, and the routed-chunks table of `routing-figures.md` in loop-claude's archive (`$LOOP_CLAUDE_TRANSCRIPTS`, default `~/.claude/loop-claude-transcripts`). Propose new steps to lilleman, whose call they are.
|
||||
@@ -163,10 +280,43 @@ Releases are the human's, and `CLAUDE.md` → "Working loop" already ends the lo
|
||||
|
||||
Raised by the goals audit of PR #296. `bin/routing_figures/calls.py` groups a chunk's review rounds by the rung its worker started on, and calls for a routing change when a lower rung took more of them. Since PR #296 the reviewers may run above the worker's rung, on the one their item's E + I picks, and no figure groups rounds by it, though the archived worker line names the E + I it follows from. A cheap worker on an item of high E + I gets stronger reviewers, who may raise more rounds, and a call could then blame the worker's rung for them: an interrupt of lilleman that settles nothing (goal 1.2).
|
||||
|
||||
### 300. Restructure the places the comprehension seats found hardest to follow, so the project reads at 7.0 or above on every dimension, Locality first: today the comprehension floor run at 52da15e scored Locality 6.00, Shape 6.25, Self-sufficiency 6.38 and Overall 6.38.
|
||||
|
||||
Filed by the comprehension check of PR #473 at 13d1550 (`larv-review record-scoring`), which printed `Floor:` and `Next:`, and refreshed from the run of PR #475 at 52da15e. The lowest mean is Locality, 6.00. Refresh this item's places and aim from the newest run before the chunk starts; do not file a second floor item.
|
||||
|
||||
Aim: one ambitious refactor of the areas below, sized so every mean reaches 7.0 on the chunk's own scoring run. Where a place is already easier to read when the chunk starts, drop it.
|
||||
|
||||
Places the seats named, hardest first (`larv-review places`):
|
||||
|
||||
- Locality, Shape: `bin/loop_claude/routing.py:93` `unfinished_route`: the floor rung, the escalation cap, the escalation mark, the unfinished count and the taken-in E + I are held at once, across about six helpers in five files. The mark's meaning lives in `escalation_mark.py`, and the reason for the floor and the cap sits in the decision log.
|
||||
- Locality, Shape: `bin/release_plan/release_standing.py:145` `next_release`: the tag, changelog, before-first-release and renamed branches meet in the `is_earliest` expression, which took a senior three reads. Nothing comments the `None` and `nothing_cut` case.
|
||||
- Locality: `bin/loop_claude/cycle/judge.py:211` `quotes_checked`: the judge's contract is split over prose constants, schema descriptions and checks after the fact. A bad quote silently changes the verdict's type.
|
||||
- Locality, Shape: `bin/loop_claude/cycle/run.py:146` `rule`, with `loop/close.py:18` `closed_cycle`: the order of the ruling, rejection and step-up archive writes carries meaning, and nothing enforces it. Four near-identical archive helpers, `run.py:86` `archived` among them, sit in the cycle runner though `loop/close.py` imports them, and `run_cycle` replaces the route partway through its loop.
|
||||
- Locality, Self-sufficiency: `bin/larv_review/reviewer_header.py:49` `unanswered_round` and `skills/larv-review/SKILL.md:76` "Round cap": the cap counts answer commits, so a worker that relaunches without committing gets the same round number. The cap is enforced in four units, split over SKILL.md and three modules.
|
||||
- Locality, Shape: `bin/larv_review/plan.py:66` `plan_reasons`: a merge-base commit enters the parameter that item 296's chunk renamed `ref`, eight failure branches mix git reads, file reads and logic, and `after` means the checkout's text and then its items.
|
||||
- Locality: `bin/larv_review/owed.py:37` `shortfall`: the one-run-or-two rule is keyed on the wording `EMPTIED_WORDS` that plan.py builds and SKILL.md quotes, and `scoring_runs` defines a scoring run differently from `Entry.scored`. Give the scoring-run logic one module, and have the number of runs owed come from a value, not from a sentence.
|
||||
- Locality: `bin/chunk_pr/main.py:42` `forge`: the forge helpers live in `chunk_preflight` and `deploy_release`, so a second forge needs edits in four packages.
|
||||
- Locality: `bin/rule_hook/reviewer_rung.py:40` `skill_breaches`: three cases rest on assumptions made in another function.
|
||||
- Locality: `bin/rule_hook/repeat_read.py:119` `shown_lines`: it rebuilds from transcript entries which lines are in the agent's context.
|
||||
- Locality: `bin/model_routing/route.py:88` `route_of`: a route has two serializations, a flat archive line and a nested handover.
|
||||
- Self-sufficiency, Locality: `bin/loop_claude/cycle/prompts.py:59` `review_launches` and `:119` `escalation`: the worker's prompt is built from about 20 conditional sentence builders, which restate the routing logic.
|
||||
- Shape: `bin/release_plan/ranking.py:56` `rank`, `:304` `score` and `:315` `weight_of`: nearly every function forks on `SHAPE` versus `OLD_SHAPE`, which doubles the module; item 156 removes the old shape. The constants read as magic until `rules/todo-items.md` is open.
|
||||
- Shape: `bin/review_rules/comprehension_history.py:127` `entries`: a line-by-line parser that tolerates two legacy formats.
|
||||
- Shape: `bin/records/__init__.py:1` `records`: one name for three concepts, as `cut` and `Plan` also name several.
|
||||
- Shape: `bin/rule_hook/rules.py:25` `RULES`: rules sit inline in one module while five others have a module each.
|
||||
- Navigation, Shape: `README.md:116` `bin/`: about 55 entries in one flat list, some 31 of them packages, the worst fan-out level.
|
||||
- Self-sufficiency: `skills/larv-review/SKILL.md:147` "Scoring run": dense rule prose with many cross-references; three sections rebuild one rule.
|
||||
|
||||
Scored L 4 because the refactor touches several components that hold known defects, E 2 because review and a manual check see a regression, I 3 because every session runs these tools. The chunk runs a scoring run, since its tagline names the comprehension floor.
|
||||
|
||||
### 264. Report the comprehension seats and the escalated subagents as subagent types of their own in `routing-figures.md`, with their dollars.
|
||||
|
||||
Seats launch as `general-purpose`, so they hide inside that type: $301.24 of the $962.17 the figures of 2026-11-04 count, from 221 subagents. By 2026-11-10 general-purpose was the second-largest spend, $585 of about $2,385, and by transcript size about two thirds of it were seats (291 launches), then 177 security passes. Escalations are counted by trigger only, never in dollars. `bin/routing_figures/chunk.py` groups subagents by their `subagent_type`; a seat is told apart by its prompt, and an escalated subagent by the escalation marker its prompt opens with.
|
||||
|
||||
### 323. Count a release review as landed once its pull request has merged, whatever its commits are titled.
|
||||
|
||||
`release_reviewed` in `bin/release_plan/release_standing.py` counts a release review only by a commit titled `Release review of <release>` on the default branch. In a replay of archived interrupts (`judge-field-trials/`), in two projects whose release review had merged, the default branch held no such commit, so `chunk-preflight` and routing read ReviewDue, and the next loop would run the release review again. `skills/larv-review/SKILL.md` opens the review's branch with an empty commit of that title, but a worker may skip it, and a rebase or squash merge, which `chunk-pr` uses where the forge allows no merge commit, may drop it. Detect the review by something every merge method keeps, and check the fix against both projects' histories. Until then the judge keeps `release_ready` (docs/decisions.md → `routing: loop-claude reads the items a chunk takes …`).
|
||||
|
||||
### 149. Fill each release by priority up to a capacity it states, in place of the bar, and move what does not fit to a later release.
|
||||
|
||||
A release's capacity is a sum of its items' cost, as the formula counts it. Each project states its own; the human sets the first value per project. Nothing is deleted by score.
|
||||
@@ -241,6 +391,10 @@ Weigh the widely used tmux-resurrect and tmux-continuum plugins against a script
|
||||
|
||||
Today the folder is the SHA-256 of the transcript's first timestamped line, which tells the human nothing; that line carries the start time. Drop the fallback to the first line, which every chunk of a project shares, and give the tests' fake transcripts ISO timestamps. `bin/reading-cost` takes a chunk's id from the folder's first 12 characters and skips a chunk whose id `docs/reading-cost.md` holds, so give it a way to match a figure written under the old name.
|
||||
|
||||
### 324. Count in the routing figures the finished chunks whose worker wrote no `Chunk items:` line.
|
||||
|
||||
`docs/decisions.md` → `routing: loop-claude reads the items a chunk takes …` holds only while workers write the line when asked, and nothing counts a chunk whose worker skipped it. There loop-claude scores no item beyond the one the chunk was routed on, so the route holds, and loop-inbox lists only that item. Count in `bin/routing_figures/`, beside the chunks counted, those whose worker's transcript holds no such line, so the human sees when the decision's premise stops holding.
|
||||
|
||||
### 187. Count the tokens a worker spends reading reviewers' reports, so goal 1.4's signal shows whether report files lowered what a review costs the worker's context.
|
||||
|
||||
Before PR #310, reports reached the worker as the Agent call's result, a median 19k tokens per chunk (2026-10-08, 68 archived transcripts). Now a reviewer returns its verdict and one line per finding, and the worker reads a 🔴 or 🟡 finding's entry from the report file. `reading-cost` counts only tracked files (`bin/reading_cost/measure.py`), and `routing-figures` counts only the Agent call's result (`bin/routing_figures/chunk.py`), so neither sees the report reads. Found by the architecture review of PR #310.
|
||||
@@ -253,10 +407,6 @@ The archive holds 277 chunk folders under `home/lilleman/.claude`, from 2026-09-
|
||||
|
||||
`bin/loop_claude/test_loop_claude.py`'s `test_a_quoted_answer_that_cites_an_escalation_trigger_steps_the_worker_up_before_it_resumes_and_archives_the_step` builds its expected prompt from `prompts.rungs_held`, the code under test, so a bug there passes it. The technical principles' test bullet bans that shape. Assert the new rung and one review launch as literals, as the typed answer's step-up test does. Raised by the principles audit of PR #308.
|
||||
|
||||
### 192. Sort `.gitignore`'s allowlist block, from `!.claude/` to `!workflow-bench/test_review_git.py`, by code point, so rule-hook's `ignorefile-sorted` checks later edits to it again.
|
||||
|
||||
Found by review of PR #317. The block, `.gitignore:17-142` on 2026-11-10, is out of order in several places: `!bin/chunk-preflight` sits below `!bin/chunk_pr/`, and `!bin/loop-inbox` below `!bin/loop_claude/`, though `-` sorts before `_`. `LC_ALL=C sort -c` on the block names the first one. `ignore_breaches` in `bin/rule_hook/rules.py` skips a block that was already unsorted before an edit, so while this one stays unsorted no later edit to it is checked. The unsorted block breaches the technical principle "Deterministic order wherever order carries no meaning".
|
||||
|
||||
### 193. Narrow `readme_goals`'s mark to `Mark` by a typed lookup in place of `typing.cast`, so a mark added to the regex but not the lookup fails at run time, and one added to the lookup but not `Mark` fails mypy.
|
||||
|
||||
Found by review of PR #317. `bin/release_plan/ranking.py`'s `readme_goals` writes `typing.cast(Mark, mark or None)`, which asserts the type `goal_weights` branches on without checking it: a mark added to the regex reaches `goal_weights` as a value `Mark` lacks. mypy does not read the regex, so no lookup lets it catch that. Replace the cast with a lookup typed `dict[str, Mark]`, from each mark the regex captures, and `""` for none, to its `Mark` value. A mark added to the regex but not the lookup then raises `KeyError` at run time, and a value added to the lookup but not `Mark` fails mypy.
|
||||
@@ -265,10 +415,6 @@ Found by review of PR #317. `bin/release_plan/ranking.py`'s `readme_goals` write
|
||||
|
||||
Found by review of PR #317. The black-box test, `bin/loop_claude/test_black_box.py:609`, not the unit test of the same name in `test_loop_claude.py`, which runs on a fake clock, sets `resets_at` from the test's `time.time()` and expects the wait, rounded to tens, to be 3720. When more than about 5 s pass between that read and the loop's, the wait rounds to 3710 and the test fails with no bug. Item 102 covers only replacing the `time.sleep` swap; this item changes the expected value, for example by asserting the wait against `resets_at + 120` minus the time the loop read, or a bound on it.
|
||||
|
||||
### 202. Add `CLAUDE.md` → "Prose"'s rule "Write negations as people say them", with its examples, to `skills/larv-review/standards/prose.md`, so the prose reviewer holds prose to it.
|
||||
|
||||
Found by the prose review of PR #329. `prose.md` opens by calling itself the full rules that `CLAUDE.md` → "Prose" keeps a core of, yet it lacks that rule, so a prose reviewer whose bar is the standards files never sees it.
|
||||
|
||||
### 265. Archive each wait that costs a chunk wall time, for a usage limit, low memory, a failed worker start or a heavy slot, with its start and end stamps, so goal 1.5's hours split waiting from work.
|
||||
|
||||
No record holds these waits today. An outcomes file holds verdicts, stops, step-ups and the judge's cost (`bin/transcript_archive/outcomes.py`), and `~/.local/state/run-heavy/` holds only the slots' lock files. loop-claude waits out a usage limit and low memory in `bin/loop_claude/cycle/retry.py`; run-heavy waits for a slot in `bin/run_heavy/`. A heavy slot's wait happens inside a worker's Bash call, so loop-claude doesn't see it; run-heavy would write it where the archive can find it.
|
||||
@@ -277,10 +423,6 @@ No record holds these waits today. An outcomes file holds verdicts, stops, step-
|
||||
|
||||
The hook judges a call before it runs, and `changed` in `bin/rule_hook/todo_check.py` compares `todo.md` with the index and HEAD as they stand then. A call that edits `todo.md` and commits it, such as a heredoc script followed by `git commit`, passes unchecked: PR #296's commit 149108e did, and the override marker it carried was never logged. larv-review's gate still runs `rank_todo.py --check`, so an unranked plan stops at merge rather than at commit. A git pre-commit hook would see the staged file.
|
||||
|
||||
### 174. In a chunk no item routed, ask the worker to name each todo.md item it takes, with its L and E + I, so loop-claude can route on an item it takes in.
|
||||
|
||||
The principles pass of PR #301 found that only `routed_by` in `bin/loop_claude/cycle/prompts.py` asks the worker to name each item it takes with its L and E + I, and `fresh` uses it only where an item routed the chunk. In an unrouted chunk the judge has no levels to report, so neither the taken-in L nor the taken-in E + I lifts the worker or the judge.
|
||||
|
||||
### 197. Let the human edit what they type at loop-claude's terminal questions with the arrow keys, so Left moves the cursor in place of printing `^[[D`.
|
||||
|
||||
`bin/loop_effects/effects.py` hands loop-claude Python's builtin `input`, and nothing under `bin/` imports `readline`, so the terminal gets no line editing: fixing a typo means erasing back to it. loop-claude asks two questions at its terminal. `effects.input` reads the pick of the project's model, in `project/model_choice.py`'s `ask_for_model`. `effects.input_or` reads the interrupt's answer, in `cycle/interrupt.py`'s `first_answer`, straight from stdin, since it also watches loop-inbox. Importing `readline` in `effects.py` gives `input` line editing, at two costs; `input_or` would also need to read through `input` while it watches. It moves the prompt from stderr to stdout, while the question above it stays on stderr. `test_black_box.py` waits for the prompt on stderr. `readline`'s bracketed paste takes a pasted block whole, so an answer can hold several lines. `prompts.typed_answer` ends the answer at its first blank line, so the judge would read the rest of such a paste as loop-claude's words. The product-owner review of PR #308 raised the interrupt's case. lilleman, 2026-10-18: "Not super-important, but a bit irritating."
|
||||
|
||||
Reference in New Issue
Block a user