47 KiB
todo
Scoring
Score = 100 * V / (2**S + 32)
Bar = 0.1
Next ID = 277
Items
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 0.7.0 | question | Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume. | 1 | 2 | 1 | 2 | 1 | 0.0 | ||
| 113 | 0.7.0 | Compute an item's S from the code it names, where it names any, and show it beside the finished items score-outcomes lists. |
3 | 2 | 2 | 5 | 4 | 1.1 by 5, 1.4 by 5 | 3.4 | ||
| 263 | 0.7.0 | Have the judge reject an interrupt that re-asks what the human answered earlier in the chunk, unless the ask quotes what that answer left open. | 2 | 2 | 3 | 5 | 3 | 1.2 by 6, 1.5 by 2 | 3.2 | ||
| 162 | 0.7.0 | owner | Script each step that 0.2.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release. | 2 | 2 | 1 | 3 | 3 | 1.4 by 4, 1.5 by 4 | 3.1 | |
| 221 | 0.7.0 | Parse each transcript entry once, where bin/claude_code/transcript.py reads it, into a typed value naming its kind, so no consumer works out an entry's kind from a raw dict. |
4 | 1 | 3 | 6 | 6 | code health by 4 | 3.0 | ||
| 105 | 0.7.0 | decision | Compare review rounds per pass and reading cost for the five chunks before and after the worker: decision moved the principles out of the worker's context, and file an item if either rose. |
1 | 2 | 1 | 2 | 2 | 1.4 by 6 | 2.7 | |
| 175 | 0.7.0 | Compare the first five chunks loop-claude starts with chunk-preflight against the five before them, and file an item where the preflight did not pay off. |
1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.6 | ||
| 112 | 0.7.0 | Count an item as shipped only where its chunk built it, so an item a chunk dropped from the plan gets no outcome. | 3 | 2 | 2 | 5 | 3 | 1.1 by 6 | 2.5 | ||
| 276 | 0.7.0 | Let a worker fix on its own rung a standards/quality.md finding whose fix changes only prose, so escalation trigger 4 calls for a higher rung only where code or config must change. |
2 | 2 | 3 | 5 | 3 | 1.2 by 4, 1.4 by 2 | 2.4 | ||
| 116 | 0.7.0 | Report L by the share of chunks with a 🔴, once 10 chunks that each shipped one item record finding counts and most record no 🔴. | 1 | 1 | 2 | 2 | 2 | 1.1 by 5 | 2.3 | ||
| 12 | 0.7.0 | Run the test suite in CI on every PR, or record in docs/decisions.md why the local run is enough. |
3 | 2 | 1 | 4 | 4 | 1.1 by 6 | 2.1 | ||
| 173 | 0.7.0 | Have rule-hook's todo-rank-check run a worktree's own bin/rank_todo.py when the worktree is of this repo, so a branch that changes todo.md's shape commits without an override. |
2 | 1 | 3 | 4 | 2 | 1.1 by 2, 1.4 by 2 | 1.8 | ||
| 272 | 0.7.0 | owner | Make deploy-release --cut say when each step finishes, end on one line saying the release is cut and ~/.claude is on it, and delete the merged release branch. |
1 | 1 | 1 | 1 | 2 | 1.5 by 3 | 1.3 | |
| 274 | 0.7.0 | Let an item name a maintain goal it raises confidence in, such as a test leg that guards it, and weigh that move against the achieve goals in bin/release_plan/ranking.py. |
2 | 2 | 3 | 5 | 4 | 1 by 3 | 1.1 | ||
| 275 | 0.7.0 | Make the Main tests of judge-replay and review-replay fail when the replays ignore the project's model, its lowest worker rung or the round's checkout. |
1 | 2 | 1 | 2 | 2 | 1.1 by 2 | 0.9 | ||
| 77 | 0.8.0 | Teach bin/metric-bench Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel. |
4 | 2 | 1 | 5 | 6 | 1.1 by 7, 1.4 by 7 | 2.4 | ||
| 273 | 0.8.0 | decision | Re-tune the E + I steps of REVIEW_STEPS for 6 to 8 once 10 finished chunks have scored an E + I of 6 or more, from how well E + I orders their review rounds and 🔴 findings. |
1 | 2 | 2 | 3 | 2 | 1.1 by 2, 1.4 by 3 | 2.3 | |
| 171 | 0.8.0 | decision | Group the routing figures' review rounds by the reviewers' rung as well as the worker's, so a call blames a rung only for rounds its own reviews raised. | 3 | 2 | 2 | 5 | 4 | 1.2 by 3, 1.4 by 3 | 2.0 | |
| 149 | 0.8.0 | decision | Fill each release by priority up to a capacity it states, in place of the bar, and move what does not fit to a later release. | 4 | 2 | 3 | 7 | 5 | 1.1 by 7 | 1.8 | |
| 165 | 0.8.0 | Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant. | 3 | 2 | 1 | 4 | 5 | 1.1 by 4, 1.4 by 3 | 1.8 | ||
| 151 | 0.8.0 | decision | Keep 10 to 20 settled decisions per project as its goals' test set, and rerun them after a goal changes. | 3 | 2 | 2 | 5 | 5 | 1.2 by 7 | 1.8 | |
| 68 | 0.8.0 | principle | Test workflow-bench/compare.py's blinding and bench.py's run plan. |
2 | 1 | 1 | 2 | 4 | 1.1 by 5 | 1.8 | |
| 102 | 0.8.0 | Give loop-claude's retry delay, and the file it reads the machine's memory from, environment overrides, so the black-box test stops replacing time.sleep and machine_load.memory.MEMINFO through sitecustomize. |
1 | 1 | 2 | 2 | 3 | 1.1 by 4 | 1.7 | ||
| 135 | 0.8.0 | Fingerprint only the modules loop-claude imports, so editing another tool's code no longer reloads a running loop. | 2 | 2 | 3 | 5 | 3 | 1.1 by 4 | 1.7 | ||
| 270 | 0.8.0 | Compare each review pass's share of chunks that go to a second round, and move each check stability repeats across chunks into the worker's pre-review gate. | 2 | 2 | 3 | 5 | 3 | 1.1 by 2, 1.4 by 2 | 1.6 | ||
| 5 | 0.8.0 | decision | Key a project on its checkout's full path, and refuse to start while another session uses that key. | 4 | 2 | 3 | 7 | 5 | 1.1 by 6 | 1.6 | |
| 156 | 0.8.0 | decision | Drop rank_todo.py's ranking of an old-shape table once no project under ~/code holds one. |
2 | 1 | 3 | 4 | 4 | 1.1 by 4 | 1.4 | |
| 196 | 0.8.0 | decision | Replay the architecture, principles, overrides and product-owner passes with review-replay on Opus 5.5 projects, record each run in the review-replay routing decision, and, where a run qualifies a rung, add back the code that lowers a pass to it. |
2 | 2 | 3 | 5 | 2 | 1.4 by 3 | 1.3 | |
| 260 | 0.8.0 | decision | Compare the first 20 chunks whose worker got the reading rule with the 20 before them, on review rounds past each pass's first, unfinished verdicts, 🔴, 🟡 and misread findings, dollars and hours per chunk, and drop the rule where any rose. | 1 | 2 | 1 | 2 | 2 | 1.4 by 3 | 1.3 | |
| 269 | 0.8.0 | Measure the judge's cost per rung from the Judge cost: lines once 20 chunks hold them, and file an item where an xhigh verdict costs more than the interrupts it avoids. |
1 | 2 | 1 | 2 | 2 | 1.4 by 3 | 1.3 | ||
| 51 | 0.8.0 | principle | Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints. | 2 | 1 | 3 | 4 | 5 | 1.1 by 5 | 1.3 | |
| 268 | 0.8.0 | End a run whose release is ready to cut with a notice on loop-claude's terminal, in place of an interrupt that waits for an answer. | 2 | 2 | 3 | 5 | 3 | 1 by 3 | 1.3 | ||
| 201 | 0.8.0 | owner | Save the human's tmux setups and restore them after a reboot, so work is up and running again quickly. | 3 | 2 | 1 | 4 | 4 | 1.5 by 4 | 1.3 | |
| 267 | 0.8.0 | Keep waiting when an interrupt gets an empty answer, at the terminal or in the claude opened for it, in place of stopping the loop. | 2 | 1 | 3 | 4 | 2 | 1.5 by 3 | 1.3 | ||
| 4 | 0.8.0 | Name each chunk's archive folder by the chunk's start time, so <key>/ lists chunks in the order they ran. |
3 | 2 | 3 | 6 | 3 | 1.1 by 3 | 1.3 | ||
| 187 | 0.8.0 | Count the tokens a worker spends reading reviewers' reports, so goal 1.4's signal shows whether report files lowered what a review costs the worker's context. | 2 | 2 | 1 | 3 | 3 | 1.4 by 3 | 1.2 | ||
| 264 | 0.8.0 | Report the comprehension seats and the escalated subagents as subagent types of their own in routing-figures.md, with their dollars. |
2 | 2 | 1 | 3 | 3 | 1.4 by 3 | 1.2 | ||
| 266 | 0.8.0 | Move the chunks archived under the key home/lilleman/.claude to the key claude-config, so the routing figures and score-outcomes count this repo's chunks as one project. |
3 | 2 | 2 | 5 | 3 | 1.4 by 3 | 1.2 | ||
| 52 | 0.8.0 | principle | Move this checkout out of Claude Code's own directory, or record why it stays, since Claude Code loads the checkout's AGENTS.md as the user's instructions into every project's sessions. |
4 | 3 | 4 | 9 | 6 | 1.1 by 4, 1.4 by 2 | 1.0 | |
| 181 | 0.8.0 | principle | Have the test of a quoted answer's step-up assert the stepped rung's facts as literals, not through prompts.rungs_held. |
1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 1.0 | |
| 192 | 0.8.0 | principle | Sort .gitignore's allowlist block, from !.claude/ to !workflow-bench/test_review_git.py, by code point, so rule-hook's ignorefile-sorted checks later edits to it again. |
1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 1.0 | |
| 193 | 0.8.0 | Narrow readme_goals's mark to Mark by a typed lookup in place of typing.cast, so a mark added to the regex but not the lookup fails at run time, and one added to the lookup but not Mark fails mypy. |
1 | 1 | 3 | 3 | 1 | 1.1 by 2 | 1.0 | ||
| 194 | 0.8.0 | Make test_waits_out_a_usage_limit_which_ends_a_run_of_failures expect a usage-limit wait independent of the wall-clock time that passes during the run. |
1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 1.0 | ||
| 202 | 0.8.0 | Add CLAUDE.md → "Prose"'s rule "Write negations as people say them", with its examples, to skills/larv-review/standards/prose.md, so the prose reviewer holds prose to it. |
1 | 2 | 2 | 3 | 1 | 1.1 by 2 | 1.0 | ||
| 265 | 0.8.0 | Archive each wait that costs a chunk wall time, for a usage limit, low memory, a failed worker start or a heavy slot, with its start and end stamps, so goal 1.5's hours split waiting from work. | 3 | 2 | 2 | 5 | 4 | 1.5 by 3 | 1.0 | ||
| 169 | 0.8.0 | Have rule-hook's todo-rank-check catch a todo.md that the same Bash call edits and then commits. |
3 | 1 | 3 | 5 | 3 | 1.1 by 2 | 0.8 | ||
| 174 | 0.8.0 | In a chunk no item routed, ask the worker to name each todo.md item it takes, with its L and E + I, so loop-claude can route on an item it takes in. | 1 | 2 | 3 | 4 | 1 | 1.1 by 1 | 0.5 | ||
| 197 | 0.8.0 | Let the human edit what they type at loop-claude's terminal questions with the arrow keys, so Left moves the cursor in place of printing ^[[D. |
2 | 2 | 3 | 5 | 2 | 1.5 by 1 | 0.4 |
Details
1. Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.
It lives under ~/.claude, which .gitignore keeps from the remote, so a lost disk loses every unattended run's record. Back it up only to a destination private to me and encrypted at rest: the worker runs with permissions bypassed, so the archived tool results can hold credentials it read.
2026-10-02, the human: ai-sandbox is the VM loop-claude will run on, hosted on the server bender. bender-infra, the infrastructure-as-code project that manages bender, will back up ai-sandbox's data volume once ai-sandbox is rebuilt. The item waits for the human to say the rebuild has landed; do not remind them. Before writing the README line, check that ~/.claude lives on that volume and that the backup is private and encrypted at rest.
The README line covers the bench's transcripts in workflow-bench/logs/ too (goal 3).
113. Compute an item's S from the code it names, where it names any, and show it beside the finished items score-outcomes lists.
score-outcomes grounds L, E, I and S in finished chunks only. Where an item names files or units, their size bounds what the change can touch.
263. Have the judge reject an interrupt that re-asks what the human answered earlier in the chunk, unless the ask quotes what that answer left open.
43 of the 175 chunks with an interrupt asked two or more times, and hold 123 of the archive's 255 interrupts (2026-11-08). Examples: adf-codec asked 4 times between 19:04 and 20:54 on 2026-11-04, 3 of them on item 167; bender-infra asked 3 times on 2026-10-21 whether to grant principles and stability one more round; fejkdata asked twice in 4 minutes on 2026-10-12 to cut 0.1.0. The judge already reads the transcript (docs/decisions.md:45), so it sees the earlier answer. The change touches bin/loop_claude/cycle/judge.py quotes_checked and grounded; make them easier to read, as the seats named them.
162. Script each step that 0.2.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.
lilleman asked, on 2026-10-16, for a standing audit each release, so the loop keeps moving work a program does as well from the model to scripts (goals 1.4 and 1.5). Item 161 scripts the steps found before the first edit in item 140's chunk. This audit reads the archived transcripts of the chunks that shipped release 0.2.0 for every step repeated by hand across chunks whose answer a program computes the same way, before, during and after review: a lookup, a check, a prompt assembled for a reviewer, a cleanup. File one item per such step, with the time and tokens it cost, into the next release with items left. Leave a step that needs judgement with the model.
The chunk that ships this item files a copy of it in the release after the one it audits, rewording "0.2.0" to the release that copy audits, and marks the copy owner, so the audit recurs each release until lilleman drops it.
221. Parse each transcript entry once, where bin/claude_code/transcript.py reads it, into a typed value naming its kind, so no consumer works out an entry's kind from a raw dict.
"Validate data and build DTOs as early as possible", in principles/technical-principles.md, raised 🟢 by the principles audit of item 210's PR. read_transcript (bin/claude_code/transcript.py:16) returns raw dicts, so each consumer, such as render beside it and since_step in bin/loop_claude/escalation_mark.py, calls the helpers in bin/claude_code/entries.py (human_text, loop_text, assistant_parts, tool_call) to work out an entry's kind again. Parse each line once into types such as a human turn, a loop-claude prompt and an assistant turn, and have every consumer of claude_code.entries under bin/ take those types. The change touches the escalation mark, which the comprehension seats found hard at bin/loop_claude/cycle/judge.py:153 escalation_mark before item 210 moved it: make since_step in bin/loop_claude/escalation_mark.py, and Grounds.since_step and judge_prompt in bin/loop_claude/cycle/judge.py, easier to read.
105. Compare review rounds per pass and reading cost for the five chunks before and after the worker: decision moved the principles out of the worker's context, and file an item if either rose.
The worker: decision is valid while review rounds per pass do not rise. Count the rounds from each PR's Answer <pass> round <n>: commits, and take reading cost from docs/reading-cost.md. Run it once five chunks have merged after the one that recorded the decision.
175. Compare the first five chunks loop-claude starts with chunk-preflight against the five before them, and file an item where the preflight did not pay off.
Item 161 put chunk-preflight's facts into a fresh worker's prompt, so the worker no longer finds them itself. Before that, in item 140's chunk, the worker spent 5 min 46 s of 30 min, 16 of its 68 messages and 226k of its 491k fresh tokens before its first edit, through 12 Bash and 5 Read calls. Measure those figures from loop-claude's archive, as medians over the first five chunks that started with the preflight and the five before them: one chunk is too noisy a baseline, as docs/decisions.md → reading cost: a rise is … records. Measure the counter-signals beside them: goal 1.4's review rounds and unfinished verdicts, and goal 1.5's 🔴 findings, per chunk, since a worker that reads less may start sooner and need more rounds or let more defects through. Where time or tokens did not fall, or a counter-signal rose, file an item naming the facts the workers still found by hand, or what the preflight got wrong.
112. Count an item as shipped only where its chunk built it, so an item a chunk dropped from the plan gets no outcome.
score-outcomes counts every item a merge removed from todo.md as shipped. A merge that changed only the plan ships nothing, but one that built an item and also dropped another gives the dropped item the built one's outcome, as PR #253 did with item 69. Mark a dropped item in the change that drops it, such as by the commit or merge title, and leave it out.
276. Let a worker fix on its own rung a standards/quality.md finding whose fix changes only prose, so escalation trigger 4 calls for a higher rung only where code or config must change.
On 2026-11-08, PR #432's prose round 1 reported two findings against standards/quality.md → "A false claim is a bug": a docstring at bin/release_plan/todo_table.py:61 that named the wrong module, and a sentence in docs/decisions.md. That fired escalation trigger 4, and the worker asked to hand both fixes to an opus max subagent. The human answered: "Just prose changes, run the fix on the same rung." Write that into trigger 4 in CLAUDE.md → "Working with the codebase", into larv-review's SKILL.md §3 → "Rung up", and into the docs/decisions.md entry review: a reviewer or comprehension seat that rules the work below standard…, whose background says work below standard gets a more capable rung to fix it. A comment or docstring counts as prose, as larv-review §3 says.
116. Report L by the share of chunks with a 🔴, once 10 chunks that each shipped one item record finding counts and most record no 🔴.
score-outcomes reports the median 🔴 count at each guessed R, and how well L orders chunks by 🔴 findings; the part of R a 🔴 tests is L. Where most chunks have no 🔴, every median reads 0 and the rank correlation rests on the few that have one, which tells a scorer little. No round title held counts before PR #254, so how often a chunk has a 🔴 is not yet known.
12. Run the test suite in CI on every PR, or record in docs/decisions.md why the local run is enough.
The repo has no CI, so larv-review's gate is the local run, and larv-review's Gate skips even that when the last stability round left the suite green: an unattended chunk can merge on the worker's own word.
When CI runs a rule check, such as one of bin/rule-hook's, let <!-- rule-override <rule>: <reason> --> in the PR body override it.
173. Have rule-hook's todo-rank-check run a worktree's own bin/rank_todo.py when the worktree is of this repo, so a branch that changes todo.md's shape commits without an override.
Raised by the overrides pass of PR #296. The hook imports the main checkout's release_plan/ranking.py, so while a branch of this repo changes the table's shape, every commit touching todo.md needs a todo-rank-check override: 33 log lines across four branches, the score-outcomes, plan-136, plan-148 and item-146 ones. The hook then checks nothing, and the rule rests on the worker chaining the branch's own check. Run another repo's bin/rank_todo.py never: that would run its code before every commit.
272. Make deploy-release --cut say when each step finishes, end on one line saying the release is cut and ~/.claude is on it, and delete the merged release branch.
Cutting 0.6.0 printed waiting for the checks of pull request 418, then Cut v0.6.0. and Moved from v0.5.0 to v0.6.0., then the whole 0.6.0 changelog section. The two lines saying it worked scrolled off behind the changelog, and no line said the checks passed or the pull request merged. So the human couldn't tell whether the cut had finished or was still waiting. Print a line on stderr as each step finishes: branch pushed, pull request opened, checks passed, merged, tagged, tag pushed. End stdout with one line, after the changelog: Done: v<version> is cut and pushed, and <checkout> is on it. Once the tag is pushed, delete the remote branch release-<version>, which 0.6.0's cut left on origin. A failed delete only warns, since the release is cut by then. Update the tests in bin/deploy_release/test_deploy_release.py and deploy-release --help.
274. Let an item name a maintain goal it raises confidence in, such as a test leg that guards it, and weigh that move against the achieve goals in bin/release_plan/ranking.py.
Today only a defect item may name a maintain goal, and only to restore one the project breaks
(docs/decisions.md → the todo.md entry of 2026-10-16). A maintain goal otherwise adds no value to
any item, so work that makes a held goal safer to rely on scores V 0 and ranks last: adf-codec's
item 170, which bundles the package with four bundlers to back its goal 8, runs on any JavaScript
engine, scores 0.0 beside items that move an achieve goal by 2.
Settle with the maintainer, and record in docs/decisions.md:
- How an item says it: a Moves entry naming the maintain goal with how far the item raises confidence in it, 1 to 10, or a separate column.
- What it weighs. The maintain goal's weight now is the heaviest achieve goal's, which suits a broken goal and is too much for a confidence gain; it needs its own factor, which the item measures or asks for.
- Which items count: a test, a gate leg or a probe that would fail where the goal breaks. A change that only claims the goal does not.
Then change bin/release_plan/ranking.py and its tests, rank_todo.py --help, and rules/todo-items.md's sentences on
maintain goals and Moves. The decision of 2026-10-16 keeps its veto: an item that breaks a maintain
goal is still redesigned or dropped.
275. Make the Main tests of judge-replay and review-replay fail when the replays ignore the project's model, its lowest worker rung or the round's checkout.
Found by the stability review of item 219's chunk, findings 1 and 2; both gaps predate it. Mutants that survive today:
bin/judge_replay/main.pymain: passingNonein place oflowest_worker_rung(project)todown_from, since everyMaintest inbin/judge_replay/test_judge_replay.pyuses an Opus 5.5 project, whose lowest worker rung already is the ladder's bottom. No test checks that the report's first line names the model passed asreport(project.model, ...)either. Add aMaintest with a Fable 5.1 project asserting that the replays stop at Opus 5.5 at max effort and that the first line names the model.bin/review_replay/main.pyfindings_found: passingNonein place offound.roottomatch, since theMaintests' fake claude inbin/review_replay/test_review_replay.pyignores the folder a match call runs in. The modelmainpasses toreportis unchecked too. Record the folder each match call runs in and assert it is the round's checkout, and assert that the first output line names the model.
77. Teach bin/metric-bench Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.
2026-10-08, lilleman: no metric enters review until a rerun on more data shows it tracking the panel.
Before the rerun, set the AUC and top-k lift over chance that count as tracking, from a published threshold, so the rerun ends in a decision. The rerun needs places from a project in Go or TypeScript, the languages most other projects use, so the bench first measures those languages, run through each project's own tooling. Function and file length stay warnings to look into, never limits: a plain long function can be simple, and a hard one may need five-line pieces.
On 2026-10-08 only this repo's history named places: five scoring runs, all that day. The medians:
| Metric | AUC | Top k |
|---|---|---|
| Cognitive complexity | 0.66 | 0.26 |
| Length | 0.74 | 0.18 |
| Shared state | constant | constant |
| Change spread | 0.67 | 0.18 |
AUC is 0.5 for a coin toss, and top k scores 0.10 by chance. Shared state cannot be judged here: every unit in bin/ scores 0 on it. Every run's first place was a section of skills/larv-review/SKILL.md, which no code metric sees.
On 2026-10-09 a question bench asked a headless claude, allowed only Read, Grep and Glob, three questions a script answers from the syntax tree: which units call a unit, which units it calls, and which test files refer to it. It asked them about every unit the history names whose name is unique, plus as many units the history does not name. Five of the history's six scoring runs named such a unit. The tool is in commit 083cc78; it was deleted after the run.
Claude's misses ranked the named units at a coin toss. Top k scores 0.50 by chance on these units:
| Metric | AUC | Top k |
|---|---|---|
| Question misses | 0.49 | 0.50 |
| Cognitive complexity | 0.66 | 0.59 |
| Length | 0.75 | 0.69 |
| Change spread | 0.57 | 0.54 |
Claude's answers overlapped the truth by a median Jaccard index of 0.76. A run took in 62,643 new input tokens and cost $0.70; it did not record which model answered. Grep answers these questions whatever the code's shape, so no question enters review; a new question bench would need questions that take understanding the code, not finding it.
273. Re-tune the E + I steps of REVIEW_STEPS for 6 to 8 once 10 finished chunks have scored an E + I of 6 or more, from how well E + I orders their review rounds and 🔴 findings.
docs/decisions.md → routing: one ladder… records why lilleman kept the steps on 2026-11-08. Of the 36 chunks it measured, 1 scored an E + I of 6 or more and none 7 or more, so the steps for 6 to 8 are untested. Start once score-outcomes shows 10 finished chunks with an E + I of 6 or more. Then read its rank correlation for E + I, and the routed-chunks table of routing-figures.md in loop-claude's archive ($LOOP_CLAUDE_TRANSCRIPTS, default ~/.claude/loop-claude-transcripts). Propose new steps to lilleman, whose call they are.
171. Group the routing figures' review rounds by the reviewers' rung as well as the worker's, so a call blames a rung only for rounds its own reviews raised.
Raised by the goals audit of PR #296. bin/routing_figures/calls.py groups a chunk's review rounds by the rung its worker started on, and calls for a routing change when a lower rung took more of them. Since PR #296 the reviewers may run above the worker's rung, on the one their item's E + I picks, and no figure groups rounds by it, though the archived worker line names the E + I it follows from. A cheap worker on an item of high E + I gets stronger reviewers, who may raise more rounds, and a call could then blame the worker's rung for them: an interrupt of lilleman that settles nothing (goal 1.2).
149. Fill each release by priority up to a capacity it states, in place of the bar, and move what does not fit to a later release.
A release's capacity is a sum of its items' cost, as the formula counts it. Each project states its own; the human sets the first value per project. Nothing is deleted by score.
165. Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant.
A test audit across lilleman's projects on 2026-10-16 found 10 of 33 sampled tests in this repo pinning wording that is not the contract. Examples of these and other weak tests: bin/loop_claude/test_loop_claude.py:165, :271 and most asserts in :664 to :687 pin whole sentences of the question, archive or prompt text, up to about 750 characters, and :864 asserts help-text wording. :906 and :936 parse the source with ast and list each Stop(...) and Failure(...) call by function name. bin/model_routing/test_ladder.py:16 asserts len(LADDER) == 11. Rewrite each to the technical principles' test bullet: assert the facts the text must carry, such as an item ID, a rung or a model argument, and delete a test no real bug could fail. Before and after rewriting a test, run the mutation tool on the code it covers: every mutant the old test killed, the new one kills.
151. Keep 10 to 20 settled decisions per project as its goals' test set, and rerun them after a goal changes.
Each case is a question the project settled and the answer it chose. A fresh-context agent given only the README's premises answers each case; a case it answers differently says the goals no longer decide it. Written guidance can fail to change behaviour, which is why the goals get a test.
68. Test workflow-bench/compare.py's blinding and bench.py's run plan.
"Most, if not all, code should have automated tests", in principles/technical-principles.md: compare.blind() keeps the comparison blind, and a version name it misses invalidates the bench silently.
The tests also cover workflow-bench/bench.py's permission_flags, with a probe of a headless claude given its flags. It is denied a read outside the granted trees, a bare git status, and wrapper calls with --output, --no-index, a glued -aF/abs, --fil=/abs or a path outside the clone. A cd <elsewhere> && python3 <wrapper> … acts on the clone only. It is allowed the wrapper's log, diff, show, status, ls-files, add and commit -m. workflow-bench/test_review_git.py tests the wrapper's refusals directly.
102. Give loop-claude's retry delay, and the file it reads the machine's memory from, environment overrides, so the black-box test stops replacing time.sleep and machine_load.memory.MEMINFO through sitecustomize.
Found by the principles audit of PR #242. bin/loop_claude/test_black_box.py:117-118 swaps the global time.sleep, the shape "Functional core, imperative shell" bans, and the audit of PR #376 found the same block swapping machine_load.memory.MEMINFO at lines 120-123. An override of each lets the test run the code as production runs it.
135. Fingerprint only the modules loop-claude imports, so editing another tool's code no longer reloads a running loop.
The maintainability senior and the inherited architect of PR #257's scoring run named bin/loop_claude/loop/reload.py:38 fingerprint and :50 exec_if_edited: the fingerprint covers every bin/**/*.py, so editing statusline.py or rank_todo.py re-execs a running loop. Make fingerprint easier to read.
270. Compare each review pass's share of chunks that go to a second round, and move each check stability repeats across chunks into the worker's pre-review gate.
Weak evidence: the review folders under ~/.local/state/larv-review/, review replays left out, hold a stability round 2 beside 137 of 211 round-1 reports (65%), product-owner 73 of 173 (42%) and prose 8 of 277 (3%), on 2026-11-08. The architecture and principles passes run up to five rounds in a release review, so they don't compare. First check whether stability's round 2s repeat the same kinds of finding, such as a missing test or an unhandled error. Move only a check a program or the worker can run before review; leave judgement with the reviewer.
5. Key a project on its checkout's full path, and refuse to start while another session uses that key.
The key names the project's session, archive folder and models entry. project_key drops the leading /, so ~/code/a and /a share all three, and one loop can resume or delete the other's session.
bin/reading-cost keys the archive by the main checkout, the parent of git's common dir, since it runs in a chunk's worktree; loop-claude keys it by --show-toplevel, so a loop started inside a worktree archives where reading-cost never looks. Derive both keys one way.
156. Drop rank_todo.py's ranking of an old-shape table once no project under ~/code holds one.
While rank_todo.py ranks two shapes, projects follow two sets of scoring rules, against goal 1.3. Once every project has reshaped, drop OLD_SHAPE's weighing from bin/release_plan/ranking.py, RESHAPE, and the old shape's lines in rank_todo.py --help and rules/todo-items.md. Keep todo_table reading old tables: score_outcomes reads every past merge's todo.md. PR #283's paired ruling found the old-shape path made ranking.py's score, Scoring.weights, unknown_columns and todo_table.py's Moves harder to read; leave each easier, with one shape and moves that are always moves, and the Moves cell's rules in one place. Release 0.3.0's seats named ranking.py's score, weigh and rank and todo_table.py's OLD_SHAPE for the same fork: "every rule exists twice for the old table shape" (Mid A). Make weigh and rank easier to read too.
196. Replay the architecture, principles, overrides and product-owner passes with review-replay on Opus 5.5 projects, record each run in the review-replay routing decision, and, where a run qualifies a rung, add back the code that lowers a pass to it.
Item 159's chunk, PR #322, replayed only the prose and stability passes. Sonnet 5.5 at high missed 44 to 55% of what the project's rung found again in each, so the agent judged a replay of the other four unlikely to qualify a rung. The product-owner review of PR #322 asked that the plan keep them, so the ranking decides when they run. Each run costs hours of reviewer replays (goal 1.4), so the score keeps it low until a cheaper model joins the ladder. docs/decisions.md → routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds… records the runs so far. The release review of 0.6.0 deleted the code that lowered a pass, since no rung had qualified; the decision says where it goes back.
260. Compare the first 20 chunks whose worker got the reading rule with the 20 before them, on review rounds past each pass's first, unfinished verdicts, 🔴, 🟡 and misread findings, dollars and hours per chunk, and drop the rule where any rose.
Start once 20 chunks have run under the rule. The rule is READING_RULE in bin/loop_claude/cycle/prompts.py, which every fresh worker prompt holds; git log --first-parent -S READING_RULE on the default branch finds the merge that added it. A chunk got it when its first worker started after that merge. Measure as docs/decisions.md → loop-claude: the worker's claude compacts its context before it fills a 200k window… measured the window: rounds past each pass's first and the 🔴, 🟡 and misread counts from the round commit titles, unfinished verdicts from the outcomes files, hours from the worker's first message to its last, and dollars from the chunk's cost-state entry. Compare means, since the decision weighs the rule by all spend.
269. Measure the judge's cost per rung from the Judge cost: lines once 20 chunks hold them, and file an item where an xhigh verdict costs more than the interrupts it avoids.
The Judge cost: line is new under CHANGELOG.md → "Unreleased", and no outcomes file in the archive holds one yet (2026-11-08). Of the 460 verdicts that name their judge, 48 ran on opus at xhigh, 78 at high and 310 at medium. In this repo the judge reads about 175k characters of records each time: AGENTS.md, docs/decisions.md, CLAUDE.md, rules/ and larv-review's standards. bin/routing_figures/dollars.py already adds the lines to the dollars by rung.
51. Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints.
"Functional core, imperative shell", in principles/technical-principles.md: retried, the worker, first_answer, model_choice and say print to stderr themselves, and the tests swap the global stream.
268. End a run whose release is ready to cut with a notice on loop-claude's terminal, in place of an interrupt that waits for an answer.
On 2026-11-08 the human answered notice only: the loop prints that the release is ready to cut and ends; no interrupt, and nothing waits in loop-inbox.
Releases are the human's, and CLAUDE.md → "Working loop" already ends the loop once a release is ready to cut, so the ask settles nothing. 15 interrupts in the archive ask whether the human will cut a release. On 2026-11-08 one of the 4 questions waiting in ~/.local/state/loop-claude/ is such an ask, plainpages-plugin-fastjira's 0.3.0 since 2026-11-03. Today a release ready to cut opens claude's session at once (bin/loop_claude/cycle/run.py ask), and the loop stops if the human leaves it with nothing typed. The change also touches what an interrupt covers in CLAUDE.md → "Vocabulary". Where the change touches ask, make it easier to read, as the seats named it.
201. Save the human's tmux setups and restore them after a reboot, so work is up and running again quickly.
The human asked on 2026-10-21 for tmux support, above all for saving tmux setups. Don't build anything until the human has said exactly what they need. Ask them at least:
- What a setup holds, such as sessions, windows, panes, each pane's working directory and the command running in it, like one
loop-claudeper project. - When a setup is saved: on demand or on its own.
- How a setup comes back after a reboot.
- Whether the code that saves and restores setups must work on every machine that takes this checkout (goal 2), which decides whether that code lives in this repo.
- What else tmux support should cover.
Weigh the widely used tmux-resurrect and tmux-continuum plugins against a script of our own. Then re-score this item from the answers and that choice, and split it into items where it is more than one chunk.
267. Keep waiting when an interrupt gets an empty answer, at the terminal or in the claude opened for it, in place of stopping the loop.
11 runs in the archive ended with "Stopped: the human is needed to answer the question above, but the interrupt ended with nothing typed", each leaving its chunk open until the human restarted the loop. That stop comes from human_typed in bin/loop_claude/cycle/interrupt.py, after the claude opened for /talk or for a release ready to cut ends with no human turn. An empty line at the terminal stops the loop the same way, in first_answer. Item 268 removes one of the cases that opens claude.
4. Name each chunk's archive folder by the chunk's start time, so <key>/ lists chunks in the order they ran.
Today the folder is the SHA-256 of the transcript's first timestamped line, which tells the human nothing; that line carries the start time. Drop the fallback to the first line, which every chunk of a project shares, and give the tests' fake transcripts ISO timestamps. bin/reading-cost takes a chunk's id from the folder's first 12 characters and skips a chunk whose id docs/reading-cost.md holds, so give it a way to match a figure written under the old name.
187. Count the tokens a worker spends reading reviewers' reports, so goal 1.4's signal shows whether report files lowered what a review costs the worker's context.
Before PR #310, reports reached the worker as the Agent call's result, a median 19k tokens per chunk (2026-10-08, 68 archived transcripts). Now a reviewer returns its verdict and one line per finding, and the worker reads a 🔴 or 🟡 finding's entry from the report file. reading-cost counts only tracked files (bin/reading_cost/measure.py), and routing-figures counts only the Agent call's result (bin/routing_figures/chunk.py), so neither sees the report reads. Found by the architecture review of PR #310.
264. Report the comprehension seats and the escalated subagents as subagent types of their own in routing-figures.md, with their dollars.
Seats launch as general-purpose, so they hide inside that type: $301.24 of the $962.17 the figures of 2026-11-04 count, from 221 subagents. Escalations are counted by trigger only, never in dollars. bin/routing_figures/chunk.py groups subagents by their subagent_type; a seat is told apart by its prompt, and an escalated subagent by the escalation marker its prompt opens with.
266. Move the chunks archived under the key home/lilleman/.claude to the key claude-config, so the routing figures and score-outcomes count this repo's chunks as one project.
The archive holds 98 chunk folders under home/lilleman/.claude, from 2026-10-03 to 2026-10-29, and 4 under claude-config, from 2026-10-30 on. The routing figures of 2026-11-04 list no project home/lilleman/.claude. They find a chunk's later fixes on the default branch of the checkout its key names (bin/routing_figures/archive.py measure_project), and /home/lilleman/.claude is a release checkout on a detached HEAD. Item 5 fixes how keys are derived, not the chunks already archived. ~/.claude/loop-claude-models.json still holds a home/lilleman/.claude entry; it is machine state outside the repo, so removing it is a one-off step on each machine, not part of the change.
52. Move this checkout out of Claude Code's own directory, or record why it stays, since Claude Code loads the checkout's AGENTS.md as the user's instructions into every project's sessions.
"One owner per value" and "Prefer the vendor-neutral option", in principles/technical-principles.md: git, Claude Code and loop-claude all write ~/.claude, kept apart only by the .gitignore allowlist, and the repo's records live in a vendor directory. Claude Code also loads ~/.claude/AGENTS.md, this checkout's decision titles, as the user's instructions into every session of every project, so a worker on another project reads this repo's decisions as if they held there. That cost about $25 of cache writes and reads at list prices since 2026-10-21: 10.7k characters, about 2.7k tokens, in each of 1,084 threads.
181. Have the test of a quoted answer's step-up assert the stepped rung's facts as literals, not through prompts.rungs_held.
bin/loop_claude/test_loop_claude.py's test_a_quoted_answer_that_cites_an_escalation_trigger_steps_the_worker_up_before_it_resumes_and_archives_the_step builds its expected prompt from prompts.rungs_held, the code under test, so a bug there passes it. The technical principles' test bullet bans that shape. Assert the new rung and one review launch as literals, as the typed answer's step-up test does. Raised by the principles audit of PR #308.
192. Sort .gitignore's allowlist block, from !.claude/ to !workflow-bench/test_review_git.py, by code point, so rule-hook's ignorefile-sorted checks later edits to it again.
Found by review of PR #317. .gitignore:56-58 holds !bin/rule-hook above !bin/routing_figures/ and !bin/routing_figures/*.py, but o sorts before u. ignore_breaches in bin/rule_hook/rules.py skips a block that was already unsorted before an edit, so while this one stays unsorted no later edit to it is checked. The unsorted block breaches the technical principle "Deterministic order wherever order carries no meaning".
193. Narrow readme_goals's mark to Mark by a typed lookup in place of typing.cast, so a mark added to the regex but not the lookup fails at run time, and one added to the lookup but not Mark fails mypy.
Found by review of PR #317. bin/release_plan/ranking.py's readme_goals writes typing.cast(Mark, mark or None), which asserts the type goal_weights branches on without checking it: a mark added to the regex reaches goal_weights as a value Mark lacks. mypy does not read the regex, so no lookup lets it catch that. Replace the cast with a lookup typed dict[str, Mark], from each mark the regex captures, and "" for none, to its Mark value. A mark added to the regex but not the lookup then raises KeyError at run time, and a value added to the lookup but not Mark fails mypy.
194. Make test_waits_out_a_usage_limit_which_ends_a_run_of_failures expect a usage-limit wait independent of the wall-clock time that passes during the run.
Found by review of PR #317. bin/loop_claude/test_black_box.py:569-575 sets resets_at from the test's time.time() and expects the wait, rounded to tens, to be 3720. When more than about 5 s pass between that read and the loop's, the wait rounds to 3710 and the test fails with no bug. Item 102 covers only replacing the time.sleep swap; this item changes the expected value, for example by asserting the wait against resets_at + 120 minus the time the loop read, or a bound on it.
202. Add CLAUDE.md → "Prose"'s rule "Write negations as people say them", with its examples, to skills/larv-review/standards/prose.md, so the prose reviewer holds prose to it.
Found by the prose review of PR #329. prose.md opens by calling itself the full rules that CLAUDE.md → "Prose" keeps a core of, yet it lacks that rule, so a prose reviewer whose bar is the standards files never sees it.
265. Archive each wait that costs a chunk wall time, for a usage limit, low memory, a failed worker start or a heavy slot, with its start and end stamps, so goal 1.5's hours split waiting from work.
No record holds these waits today. An outcomes file holds verdicts, stops, step-ups and the judge's cost (bin/transcript_archive/outcomes.py), and ~/.local/state/run-heavy/ holds only the slots' lock files. loop-claude waits out a usage limit and low memory in bin/loop_claude/cycle/retry.py; run-heavy waits for a slot in bin/run_heavy/. A heavy slot's wait happens inside a worker's Bash call, so loop-claude doesn't see it; run-heavy would write it where the archive can find it.
169. Have rule-hook's todo-rank-check catch a todo.md that the same Bash call edits and then commits.
The hook judges a call before it runs, and changed in bin/rule_hook/todo_check.py compares todo.md with the index and HEAD as they stand then. A call that edits todo.md and commits it, such as a heredoc script followed by git commit, passes unchecked: PR #296's commit 149108e did, and the override marker it carried was never logged. larv-review's gate still runs rank_todo.py --check, so an unranked plan stops at merge rather than at commit. A git pre-commit hook would see the staged file.
174. In a chunk no item routed, ask the worker to name each todo.md item it takes, with its L and E + I, so loop-claude can route on an item it takes in.
The principles pass of PR #301 found that only routed_by in bin/loop_claude/cycle/prompts.py asks the worker to name each item it takes with its L and E + I, and fresh uses it only where an item routed the chunk. In an unrouted chunk the judge has no levels to report, so neither the taken-in L nor the taken-in E + I lifts the worker or the judge.
197. Let the human edit what they type at loop-claude's terminal questions with the arrow keys, so Left moves the cursor in place of printing ^[[D.
bin/loop_claude/effects.py hands loop-claude Python's builtin input, and nothing under bin/ imports readline, so the terminal gets no line editing: fixing a typo means erasing back to it. loop-claude asks two questions at its terminal. effects.input reads the pick of the project's model, in project/model_choice.py's ask_for_model. effects.input_or reads the interrupt's answer, in cycle/interrupt.py's first_answer, straight from stdin, since it also watches loop-inbox. Importing readline in effects.py gives input line editing, at two costs; input_or would also need to read through input while it watches. It moves the prompt from stderr to stdout, while the question above it stays on stderr. test_black_box.py waits for the prompt on stderr. readline's bracketed paste takes a pasted block whole, so an answer can hold several lines. prompts.typed_answer ends the answer at its first blank line, so the judge would read the rest of such a paste as loop-claude's words. The product-owner review of PR #308 raised the interrupt's case. lilleman, 2026-10-18: "Not super-important, but a bit irritating."