75 KiB
todo
Scoring
Score = 100 * V / (2**S + 32)
Bar = 0.1
Next ID = 332
Items
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 331 | 0.9.0 | question | Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder. | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.9 | |
| 330 | 0.9.0 | Replay the judge on each Fable 5.2 rung an Opus 5.5 project's E + I climbs it to, and keep it off any rung that misses an interrupt. | 3 | 3 | 2 | 6 | 3 | 1.2 by 4, 1.1 by 3 | 2.9 | ||
| 320 | 0.9.0 | Bench a worker below the project's rung with the advisor tool against the same worker alone and the project's rung, and file the switch if it wins. | 2 | 2 | 2 | 4 | 3 | 1.4 by 3, 1.1 by 2 | 2.0 | ||
| 315 | 0.9.0 | Start each reviewer under agents/ without CLAUDE.md, and hand it the vocabulary, the prose rules and the project's decision titles. |
3 | 2 | 3 | 6 | 4 | 1.4 by 6 | 2.0 | ||
| 312 | 0.9.0 | Research the tooling agents read, search and run commands with, and file an item for each tool that cuts tokens a replay shows keep quality. | 1 | 2 | 1 | 2 | 3 | 1.4 by 3, 1.5 by 2 | 2.0 | ||
| 325 | 0.9.0 | Give this repo's test suite one command that runs it as README → Test says, prints the failing tests and pytest's summary line, and saves the whole log to a file whose path it prints. | 3 | 2 | 2 | 5 | 3 | 1.4 by 3, 1.5 by 2 | 2.0 | ||
| 305 | 0.9.0 | Record with each archived verdict the claude-config commit that ran it, the raw tokens each model spent by kind, and each choice of the worker's that the human reversed, so a change to the setup can be judged before and after. | 3 | 2 | 3 | 6 | 5 | 1.4 by 4, 1.2 by 3 | 1.8 | ||
| 318 | 0.9.0 | Let review-replay qualify a rung whose recall is no lower than the project's rung's own replay, and replay Opus 5.5 at low effort on the prose pass. |
3 | 2 | 3 | 6 | 4 | 1.4 by 5 | 1.7 | ||
| 308 | 0.9.0 | Skip a stability round 2 whose every must-fix finding named a surviving mutant or a failing test, both of which the worker now checks before review. | 3 | 2 | 3 | 6 | 4 | 1.4 by 3, 1.5 by 2 | 1.6 | ||
| 295 | 0.9.0 | Have chunk-pr merge wait for the forge's required checks itself, and give workers one command that waits for a background log or report file. |
2 | 2 | 3 | 5 | 3 | 1.4 by 2, 1.5 by 2 | 1.6 | ||
| 319 | 0.9.0 | Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session. | 3 | 2 | 3 | 6 | 4 | 1.1 by 2, 1.4 by 2 | 1.4 | ||
| 309 | 0.9.0 | Run a scoring run's second pass on one head with two seats in place of four, once larv-review agreement shows the seats agree with each other more than a run agrees with its rerun. |
2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.3 | ||
| 299 | 0.9.0 | Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt. | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.2 | ||
| 311 | 0.9.0 | Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under ~/.claude/projects. |
2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.2 | ||
| 316 | 0.9.0 | Launch comprehension seats as an agent type of their own under agents/, without CLAUDE.md and the skill and agent listings a general-purpose subagent carries. |
2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.2 | ||
| 310 | 0.9.0 | Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then. | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 1.0 | ||
| 298 | 0.9.0 | Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where judge-replay changes no verdict kind. |
3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 1.0 | ||
| 326 | 0.9.0 | Let chunk-pr ready take the pull request's final title and set it as it marks the pull request ready. |
2 | 1 | 3 | 4 | 2 | 1.4 by 1, 1.5 by 1 | 0.9 | ||
| 327 | 0.9.0 | Have larv-review print a round commit's title prefix, counted from the round's report, and have rule-hook refuse a round title whose counts differ from it. |
3 | 2 | 3 | 6 | 4 | 1.1 by 1, 1.4 by 1 | 0.7 | ||
| 303 | 0.10.0 | defect | Archive each judge run's prompt and json reply beside the verdict it gave, so judge cost, replays and backups cover the judge as they cover the worker. | 2 | 2 | 3 | 5 | 3 | 1.4 by 3, 3 by 2 | 4.8 | |
| 304 | 0.10.0 | 303 | defect | Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later. | 2 | 2 | 2 | 4 | 3 | 1.4 by 2, 3 by 2 | 4.4 |
| 306 | 0.10.0 | Set a strategy that keeps loop-claude's archive and larv-review's review folders backed up and small over the years, compressing older entries without losing any. | 3 | 2 | 4 | 7 | 5 | 3 by 5 | 3.9 | ||
| 302 | 0.10.0 | Call a scoring run for an item whose tagline makes the comprehension floor its target, and not for one that only names the floor, such as in a quoted decision title. | 2 | 2 | 3 | 5 | 3 | code health by 2 | 3.6 | ||
| 287 | 0.10.0 | Move the forge helpers Forge, forge_of, runner, open_pull, merged_pull, pull_at and gitea into one module that chunk_preflight, chunk_pr, deploy_release and larv_review all import from, so no tool imports another tool's internals for them. |
3 | 1 | 3 | 5 | 5 | code health by 3 | 3.4 | ||
| 328 | 0.10.0 | owner | Script each step that 0.8.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release. | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.6 | |
| 288 | 0.10.0 | Point the two reading-cost decisions at code health and goal 1.4's counter-signal, in place of the goal 1.1 they cite. | 1 | 2 | 1 | 2 | 1 | code health by 1, 1.4 by 1 | 2.6 | ||
| 77 | 0.10.0 | Teach bin/metric-bench Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel. |
4 | 2 | 1 | 5 | 6 | 1.1 by 7, 1.4 by 7 | 2.4 | ||
| 321 | 0.10.0 | 318 | Evaluate the review passes that run on a rung item 318 qualified, against the chunks before them, and revert any pass whose findings or later fixes worsened. | 1 | 2 | 1 | 2 | 2 | 1.1 by 3, 1.4 by 2 | 2.3 | |
| 273 | 0.10.0 | decision | Re-tune the E + I steps of REVIEW_STEPS for 6 to 8 once 10 finished chunks have scored an E + I of 6 or more, from how well E + I orders their review rounds and 🔴 findings. |
1 | 2 | 2 | 3 | 2 | 1.1 by 2, 1.4 by 3 | 2.3 | |
| 268 | 0.10.0 | End a run whose release is ready to cut with a notice on loop-claude's terminal, in place of an interrupt that waits for an answer. | 2 | 2 | 3 | 5 | 3 | 1 by 5 | 2.2 | ||
| 171 | 0.10.0 | decision | Group the routing figures' review rounds by the reviewers' rung as well as the worker's, so a call blames a rung only for rounds its own reviews raised. | 3 | 2 | 2 | 5 | 4 | 1.2 by 3, 1.4 by 3 | 2.0 | |
| 300 | 0.10.0 | owner | Restructure the places the comprehension seats found hardest to follow, so the project reads at 7.0 or above on every dimension, Locality first: today the comprehension floor run at 52da15e scored Locality 6.00, Shape 6.25, Self-sufficiency 6.38 and Overall 6.38. |
4 | 2 | 3 | 7 | 8 | code health by 8 | 2.0 | |
| 264 | 0.10.0 | Report the comprehension seats and the escalated subagents as subagent types of their own in routing-figures.md, with their dollars. |
2 | 2 | 1 | 3 | 3 | 1.4 by 5 | 2.0 | ||
| 323 | 0.10.0 | Count a release review as landed once its pull request has merged, whatever its commits are titled. | 3 | 2 | 3 | 6 | 3 | 1.4 by 3, 1.5 by 2 | 2.0 | ||
| 149 | 0.10.0 | decision | Fill each release by priority up to a capacity it states, in place of the bar, and move what does not fit to a later release. | 4 | 2 | 3 | 7 | 5 | 1.1 by 7 | 1.8 | |
| 165 | 0.10.0 | Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant. | 3 | 2 | 1 | 4 | 5 | 1.1 by 4, 1.4 by 3 | 1.8 | ||
| 151 | 0.10.0 | decision | Keep 10 to 20 settled decisions per project as its goals' test set, and rerun them after a goal changes. | 3 | 2 | 2 | 5 | 5 | 1.2 by 7 | 1.8 | |
| 68 | 0.10.0 | principle | Test workflow-bench/compare.py's blinding and bench.py's run plan. |
2 | 1 | 1 | 2 | 4 | 1.1 by 5 | 1.8 | |
| 102 | 0.10.0 | Give loop-claude's retry delay, and the file it reads the machine's memory from, environment overrides, so the black-box test stops replacing time.sleep and machine_load.memory.MEMINFO through sitecustomize. |
1 | 1 | 2 | 2 | 3 | 1.1 by 4 | 1.7 | ||
| 135 | 0.10.0 | Fingerprint only the modules loop-claude imports, so editing another tool's code no longer reloads a running loop. | 2 | 2 | 3 | 5 | 3 | 1.1 by 4 | 1.7 | ||
| 5 | 0.10.0 | decision | Key a project on its checkout's full path, and refuse to start while another session uses that key. | 4 | 2 | 3 | 7 | 5 | 1.1 by 6 | 1.6 | |
| 291 | 0.10.0 | owner | Let principles/technical-principles.md → "Sane defaults" allow a library whose README goals ask that it work with no config to default the clock and randomness in its public constructor, where a documented option overrides each. |
1 | 2 | 3 | 4 | 1 | 1.2 by 3 | 1.4 | |
| 156 | 0.10.0 | decision | Drop rank_todo.py's ranking of an old-shape table once no project under ~/code holds one. |
2 | 1 | 3 | 4 | 4 | 1.1 by 4 | 1.4 | |
| 196 | 0.10.0 | decision | Replay the architecture, principles, overrides and product-owner passes with review-replay on Opus 5.5 projects, record each run in the review-replay routing decision, and, where a run qualifies a rung, add back the code that lowers a pass to it. |
2 | 2 | 3 | 5 | 2 | 1.4 by 3 | 1.3 | |
| 260 | 0.10.0 | decision | Compare the first 20 chunks whose worker got the reading rule with the 20 before them, on review rounds past each pass's first, unfinished verdicts, 🔴, 🟡 and misread findings, dollars and hours per chunk, and drop the rule where any rose. | 1 | 2 | 1 | 2 | 2 | 1.4 by 3 | 1.3 | |
| 51 | 0.10.0 | principle | Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints. | 2 | 1 | 3 | 4 | 5 | 1.1 by 5 | 1.3 | |
| 201 | 0.10.0 | owner | Save the human's tmux setups and restore them after a reboot, so work is up and running again quickly. | 3 | 2 | 1 | 4 | 4 | 1.5 by 4 | 1.3 | |
| 267 | 0.10.0 | Keep waiting when an interrupt gets an empty answer, at the terminal or in the claude opened for it, in place of stopping the loop. | 2 | 1 | 3 | 4 | 2 | 1.5 by 3 | 1.3 | ||
| 4 | 0.10.0 | Name each chunk's archive folder by the chunk's start time, so <key>/ lists chunks in the order they ran. |
3 | 2 | 3 | 6 | 3 | 1.1 by 3 | 1.3 | ||
| 324 | 0.10.0 | Count in the routing figures the finished chunks whose worker wrote no Chunk items: line. |
2 | 2 | 1 | 3 | 3 | 1.1 by 2, 1.4 by 1 | 1.2 | ||
| 187 | 0.10.0 | Count the tokens a worker spends reading reviewers' reports, so goal 1.4's signal shows whether report files lowered what a review costs the worker's context. | 2 | 2 | 1 | 3 | 3 | 1.4 by 3 | 1.2 | ||
| 266 | 0.10.0 | Move the chunks archived under the key home/lilleman/.claude to the key claude-config, so the routing figures and score-outcomes count this repo's chunks as one project. |
3 | 2 | 2 | 5 | 3 | 1.4 by 3 | 1.2 | ||
| 181 | 0.10.0 | principle | Have the test of a quoted answer's step-up assert the stepped rung's facts as literals, not through prompts.rungs_held. |
1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 1.0 | |
| 193 | 0.10.0 | Narrow readme_goals's mark to Mark by a typed lookup in place of typing.cast, so a mark added to the regex but not the lookup fails at run time, and one added to the lookup but not Mark fails mypy. |
1 | 1 | 3 | 3 | 1 | 1.1 by 2 | 1.0 | ||
| 194 | 0.10.0 | Make test_waits_out_a_usage_limit_which_ends_a_run_of_failures expect a usage-limit wait independent of the wall-clock time that passes during the run. |
1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 1.0 | ||
| 265 | 0.10.0 | Archive each wait that costs a chunk wall time, for a usage limit, low memory, a failed worker start or a heavy slot, with its start and end stamps, so goal 1.5's hours split waiting from work. | 3 | 2 | 2 | 5 | 4 | 1.5 by 3 | 1.0 | ||
| 169 | 0.10.0 | Have rule-hook's todo-rank-check catch a todo.md that the same Bash call edits and then commits. |
3 | 1 | 3 | 5 | 3 | 1.1 by 2 | 0.8 | ||
| 197 | 0.10.0 | Let the human edit what they type at loop-claude's terminal questions with the arrow keys, so Left moves the cursor in place of printing ^[[D. |
2 | 2 | 3 | 5 | 2 | 1.5 by 1 | 0.4 | ||
| 1 | 0.11.0 | question | Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume. | 1 | 2 | 1 | 2 | 1 | 0.0 |
Details
331. Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.
The architect's 🔀 answer on PR #493 (item 292): four decisions expire "when a new model joins the ladder". Fable 5.2 joined above every project model's rung, and none of these replays tested a rung above the project's, so the event fired with nothing new to test. Every later ladder rebuild would fire all four again.
review: each pass keeps a reviewer of its own: the merge trials ran on Opus 5.5 at medium. Reword the "Valid" line to name that rung.review: the principles audit runs only in the release review…: its trial ran on the project's rung. Reword the line to name that rung.routing: a review pass runs below the project's rung only on a rung that a replay…:review-replaytests only rungs below the project's rung. Have the line expire when those rungs change, the event item 196 names ("until a cheaper model joins the ladder").routing: on a model judge-replay covered, the judge runs…: the replay tested the project's rung and below, while the climb above rests on the ladder's order alone. Have the line name both, and point the climb at item 330.
Recommendation: reword all four as above. They are decisions the repo held before item 292, so the rewording is lilleman's call.
330. Replay the judge on each Fable 5.2 rung an Opus 5.5 project's E + I climbs it to, and keep it off any rung that misses an interrupt.
Item 292 put Fable 5.2 on the routing ladder. On an Opus 5.5 project the judge now climbs with its chunk's E + I onto Fable 5.2 at low, medium and high, at E + I 5, 6 and 8. Fable 5.2 has never returned a judge verdict, and the 2026-10-15 replay showed that another model can miss an interrupt in kind, not only in strength. routing: on a model judge-replay covered, the judge runs … rests on that replay.
bin/judge-replay replays only the project's rung and the rungs below it (down_from in judge_replay/main.py), so as it stands it never runs a Fable 5.2 rung. Give it the rungs the judge climbs to as well, run bin/judge-replay claude-opus-5-5, and record the result in that decision. Where a Fable 5.2 rung misses an interrupt the reference judge returns, keep the judge off it until a decision says otherwise.
The review passes need no replay: review-replay tests only rungs below the project's rung, which item 292 left as they were.
320. Bench a worker below the project's rung with the advisor tool against the same worker alone and the project's rung, and file the switch if it wins.
Anthropic reports Sonnet 4.6 with an Opus 4.6 advisor scored 2.7 points more on SWE-bench Multilingual at 11.9% less cost than Sonnet alone (the vendor's own run; https://claude.com/blog/the-advisor-strategy). The docs say /advisor works with -p, but not outright that the advisor is called during a -p run, so probe that first. Each advisor call hands the advisor the whole conversation, so count its cost. Bench on workflow-bench with archived L 1 and L 2 items: Sonnet with the advisor, Sonnet alone, Opus at medium. lilleman approved on 2026-11-11.
315. Start each reviewer under agents/ without CLAUDE.md, and hand it the vocabulary, the prose rules and the project's decision titles.
omitClaudeMd: true in an agent's frontmatter (Claude Code 2.1.271 or later) starts it without the user, project and local instruction files. About 77% of CLAUDE.md is the worker's: "Working loop", "Working with the codebase", "Git", "Handoffs", "Compact instructions". Over the archive, each 1,000 prefix tokens cost $31 across the six reviewers, so this saves about 1.9% of spend. A reviewer still needs "Vocabulary", "Prose" and the project's decision titles: put the first two in the agent bodies or a SubagentStart hook's additionalContext, and have larv-review reviewer-header print the titles.
Gate it per pass: keep it only where review-replay with the changed definition (through --agents) on 10 archived rounds finds what two replays of the unchanged reviewer find; the unchanged reviewer re-finds only 69 to 73% of its own findings, so one baseline doesn't show the noise. Revise the decision rules/: CLAUDE.md holds the worker's rules that no file path can scope. lilleman approved on 2026-11-11.
312. Research the tooling agents read, search and run commands with, and file an item for each tool that cuts tokens a replay shows keep quality.
lilleman, 2026-11-11: besides an output ceiling, keep evaluating better tooling. The token research of that day found tool results carried in context cost about 21% of spend, reads by subagents the most, and that the tools themselves move fast: deferred tool loading, output saved to files, code-execution over tools, LSP and repo-map navigation, structured test output. Item 231 tried a scout, an outline script and LSP in October and none kept every finding.
Run a research round like that of 2026-11-11: a lead on the project's rung plans the searches, cheaper subagents search and distil sourced findings, and the lead checks every claim it relies on against this archive's measured spend. File an item for each tool that a replay could test. lilleman, 2026-11-11: repeat the round once per claude-config release, not per project. When this item's chunk finishes, file the next round into the next claude-config release with items left, so the check comes back periodically.
The 0.8.0 audit (item 284) measured: over the 35 chunks that shipped 0.7.0, workers made 36 Edit calls but about 250 python3 heredocs that replace a string in a file and write it back, about 120k command tokens of the 135k that all 375 python3 calls took. They read files with sed in 493 Bash calls and cat in 188, for 351k and 328k result tokens, beside 691 Read calls. Edit refuses a file not read with the Read tool, so a worker that read a file with sed or cat probably edits it through python. Check that cause, and whether reading through Read, or an Edit that accepts such a file, cuts tokens and keeps quality.
325. Give this repo's test suite one command that runs it as README → Test says, prints the failing tests and pytest's summary line, and saves the whole log to a file whose path it prints.
The 0.8.0 audit (item 284) measured, over the 35 chunks that shipped 0.7.0: 163 to 176 Bash calls ran pytest, mutation runs excluded, in 29 chunks, for about 32k command tokens and 3,000 to 3,500 s of wall time. Each worker typed README → Test's incantation, with its pins, and cut the output with its own | tail -N: 14 spellings (tail sizes 1, 2, 3, 4, 5, 6, 8, 12, 15, 20, 25, 30 and 70, or no tail in 44 calls). A short tail cuts a failing run's evidence, and a long run without one floods the context.
Add one command, beside the README's Test section, that runs the suite as README → Test says, passes on any test paths or pytest options it is given, prints each failing test with its error and pytest's summary line, and saves the whole log to a file whose path it prints. Point README → Test at it, so the pins live in one place. Write its tests first: a passing run, a failing one, and a run that cannot start.
Left with the model: which tests to run, reading the failures and fixing them.
305. Record with each archived verdict the claude-config commit that ran it, the raw tokens each model spent by kind, and each choice of the worker's that the human reversed, so a change to the setup can be judged before and after.
Three records the archive lacks, asked for by lilleman on 2026-11-11 after the token research of that day.
- The setup that ran. A verdict names the worker's model, effort and item, but not the
~/.claudetag and commit the chunk ran under. Judging any change to the setup means comparing chunks before and after a deploy, and deploys lag behind cuts, so timestamps can't tell. Add a line such asConfig: v0.7.0 (<sha>)to each verdict, interrupt and answer. - Spend in raw tokens. Record what each cycle spent as raw token counts per model and per kind: input, cache write 5-minute, cache write 1-hour, cache read, output, thinking where known. Do it for the worker's main thread, each subagent (by agent type and description) and the judge. Dollars are derived later from model and tokens; as a secondary bonus, record the price table in force when the spend was made (from
bin/routing_figures/prices.py), so a later price change doesn't rewrite history. The token research measured this by hand on 2026-11-11 and had to deduplicate cumulative worker snapshots bymessage.id: count each message once. - Reversals. The decision of 2026-11-08 (item 262) lets a worker take its own recommendation on an unreserved choice "while the human reverses few of the choices workers take", and no tool counts those reversals. Have the judge flag an answer the human typed that overrides a choice the worker took and recorded, and count them per chunk in
score-outcomesbeside later fixes and misread findings. This is the quality signal a cheaper setup could quietly worsen.
Build on item 303's archive layout for the judge's part.
318. Let review-replay qualify a rung whose recall is no lower than the project's rung's own replay, and replay Opus 5.5 at low effort on the prose pass.
Today a rung qualifies only if it re-finds every finding the project's rung found. The project's rung itself re-finds only 69 to 73% of its own findings on a replay (29 of 40, 27 of 39, 20 of 29), so even a reviewer exactly as good would fail, and "no pass has such a rung" says less than it seems. Qualify a rung when its share of the archived findings is no lower than the project's rung's own replay share, within a margin, over enough rounds, and it misses no 🔴 that both project-rung runs found. Then replay Opus 5.5 at low effort on the prose pass first, the pass whose misses cost least; the effort frontmatter key sets it per reviewer.
lilleman approved on 2026-11-11 as somewhat risky, with a close evaluation in the next release: item 321. Revise the decision routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds…, and its title at the head of docs/decisions.md.
308. Skip a stability round 2 whose every must-fix finding named a surviving mutant or a failing test, both of which the worker now checks before review.
Rounds past the first cost $846 of the $9,154 the archive measured from 2026-09-30 to 2026-11-11; stability's were $281 over 440 runs. Now that the worker runs mutate-changes before each stability round (skills/larv-review/SKILL.md §2 → "Mutation tests"), a 🔴 that named a surviving mutant or a failing test has a mechanical answer: a run of mutate-changes on the fix shows the mutant killed, or the test passes. A round 2 whose every 🔴 and 🟡 is of that kind needs no reviewer.
Round 2 still matters elsewhere: 53 of 241 archived stability reports of round 2 or later open with BLOCK. So first count, from the archived round 2s, the BLOCKs that a run of mutate-changes and the tests on the fix would not have raised, and ship the skip only if none falls in a round it would skip. Evidence: one review round scored F1 0.376 and a second 0.26 to 0.30 with more false positives (arXiv 2603.16244, claude -p with Opus 4.6, 30 artifacts). Revise larv-review §3 → "Re-entry" and the decision review: a fix for a 🔴 reruns the pass that found it….
295. Have chunk-pr merge wait for the forge's required checks itself, and give workers one command that waits for a background log or report file.
In 0.7.0's 35 chunks, workers made 65 tea api calls in 26 chunks, mostly polling commit status and trying merge methods in turn, and ran 25 until … sleep loops that took 1,744 seconds. chunk-pr merge takes --gate-passed on the worker's word; waiting for the required checks itself makes that flag a fact, not a claim. This is the kind of step item 284 files, and can count as its first.
319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
On SWE-bench Verified with Haiku 4.5 and Opus 4.7, handing the strong model the weak model's transcript solved 69.2% at $1.61 a task; starting the strong model fresh solved 79.2% at $0.90 (arXiv 2608.24358, The Handoff Tax). A model switch also rereads the whole context uncached (Claude Code prompt caching docs). A worker that steps from Sonnet 5.5 to Opus 5.5 by --resume is the raw handoff; an effort-only step within one model keeps the cache and isn't. The escalated subagent, which starts fresh with the trigger's evidence, already follows the pattern that won.
Bench it first on workflow-bench: resume against a fresh start with a brief. The brief is the chunk's item, its branch and diff, open findings and the trigger. lilleman approved on 2026-11-11. Revise the decision routing: a worker whose run ends asking for a higher rung….
309. Run a scoring run's second pass on one head with two seats in place of four, once larv-review agreement shows the seats agree with each other more than a run agrees with its rerun.
Seats cost $1,143 of the $9,154 measured from 2026-09-30 to 2026-11-11 (12.5%), scoring runs $470 of that. A scoring run today is four seats, run twice on one head: eight seat runs. Two seats on the second pass make it six, and two on both passes four.
Why it should hold quality: the seats are one model with different personas, so their errors are correlated, and a correlated extra seat adds little. larv-review agreement showed on 2026-11-11 that Mid A and the inherited architect voted alike in 58 of 75 paired rulings, while their scores moved alike between two runs on one commit in only 1 of 4 pairs: seats agree with each other more than a run agrees with its rerun, so the rerun carries more new information than the third and fourth seat. Nine judges from seven model families carried about two independent votes (arXiv 2605.29800), and ten with error correlation 0.21 carried 3.5 (arXiv 2609.22512).
Decide once agreement has its 10 commits scored twice. Keep four seats on any pass that files a floor item, since its places come from seat reports. Revise the decision review: the comprehension check records how hard a project is to read…, which names the four-seat run.
299. Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.
The judge's LLM still decides three things: the verdict kind, the escalation it quotes, and whether a release is ready to cut (docs/decisions.md → routing: loop-claude reads the items a chunk takes …). This item replays the verdict kind. No single rule found so far separates interrupts: a "?" in the worker's final message appears in 48 interrupts and misses 66. A smaller LLM already failed the replay (docs/decisions.md → routing: on a model judge-replay covered, …: Sonnet returned another kind on 3 interrupts, Haiku missed 7). Untested are combined code rules, a decision table, and a small classifier trained and run on this machine on the archived verdicts (372 finished, 234 interrupt, 42 unfinished, 20 quoted). Run each offline, nothing sent to an outside service (goal 4), and count disagreements by kind; the bar is zero missed interrupts. A hosted decision model is out of scope, since it would send transcripts to an outside service.
311. Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under ~/.claude/projects.
Workers took 7,870 Bash results over 3,000 characters, $305 of carried context, against the reading rule in their prompt; stability reviewers another $145. BASH_MAX_OUTPUT_LENGTH (or bashOutputMaxChars) makes Claude Code save a longer result to a file and hand the model its path and the first 2,000 characters. loop-claude can hand it as it hands CLAUDE_CODE_AUTO_COMPACT_WINDOW. A ceiling of 6,000 characters would have removed $228 of carried output, 10,000 characters $122. Reads of old session transcripts cost $46; a rule-hook rule can refuse them.
Subagents inherit the variable, and the decision of 2026-10-27 found a 12k cap cost stability reviewers a finding (3 of 6 against 4 of 6). That cap left no file to read, and the difference is inside the noise that decision reports. So keep the ceiling only where a replay of the archived stability rounds finds what two baseline replays find, and join item 260's comparison.
316. Launch comprehension seats as an agent type of their own under agents/, without CLAUDE.md and the skill and agent listings a general-purpose subagent carries.
Seats run as general-purpose subagents, so each starts with the skill and agent listings, about 7,500 tokens, at $9.00 per 1,000 prefix tokens over the archive measured from 2026-09-30 to 2026-11-11: about 1.3% of spend. An agent under agents/ with Read, Grep, Glob and Bash, omitClaudeMd: true, and the seat's persona in its prompt drops both; a seat reads the project as a newcomer does, which is what it simulates. Keep each seat's prompt from skills/comprehension-panel/seat-prompts.md. Check that a scoring run's scores on one head stay within the spread two runs of today's seats show. lilleman approved on 2026-11-11.
310. Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.
281 worker resumes after the 1-hour cache expired rewrote a whole context, $379 of the $9,154 measured from 2026-09-30 to 2026-11-11 (4.1%). A compaction while the cache is warm reads the context at a twentieth of the input price and leaves about 20k tokens to rewrite later.
lilleman, 2026-11-11: do it on a timer, so an answer that comes in time doesn't cost anything. When loop-claude interrupts, start a timer a margin short of the cache lifetime (about 50 minutes of a 1-hour cache; read the lifetime from the session's last cache_creation usage). If no answer has come when it fires, compact the session, then keep waiting. Skip it below about 60k tokens of context, where the rewrite costs less than the compaction. A wait for a usage limit can't be helped this way.
Probe first that claude -p "/compact" --resume <id> compacts a session: the Agent SDK documents /compact as a prompt, the CLI pages don't show it. Claude Code compacts idle interactive sessions itself ("Compacted while idle"), but not -p runs. Check: the cost entries of 20 chunks with an interrupt, before and after.
298. Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where judge-replay changes no verdict kind.
153 Judge cost: entries total $109.76, a median of about $0.76 a verdict at every rung, and the cost follows the size of the records, not the transcript: about 76k characters in log and 188k in claude-config. In 224 of 251 recent verdicts (89%) the evidence sits in the worker's final message. A missed interrupt is the costly error, so the bar is zero changed interrupts and zero new rejections across the replay. The judge is about 4.5% of dollars, so the saving is small; each cycle also gets faster.
326. Let chunk-pr ready take the pull request's final title and set it as it marks the pull request ready.
The 0.8.0 audit (item 284) measured, over the 35 chunks that shipped 0.7.0: 30 tea pr edit calls in 18 chunks, about 2.0k command tokens. 17 of them, in 15 chunks, retitled a draft pull request the worker had opened with a provisional title; the rest looked up tea pr edit --help, rewrote a description or toggled the draft. On Gitea, chunk-pr ready already rewrites the title, dropping WIP: , so setting the final title there is the same call.
Let chunk-pr ready take an optional final title and set it as it marks the pull request ready, on GitHub and on Gitea, and refuse a title of more than one line. bin/chunk_pr/main.py:42 forge is a place the comprehension seats found hard; where the change touches it, make it easier to read, or leave it to item 287.
Left with the model: wording the title.
327. Have larv-review print a round commit's title prefix, counted from the round's report, and have rule-hook refuse a round title whose counts differ from it.
The 0.8.0 audit (item 284) measured, over the 35 chunks that shipped 0.7.0: 72 commit calls in 33 chunks titled Answer <pass> round <n>: R 🔴, Y 🟡 (K misread); <what changed>, about 11k command tokens, most of them the summary. Each worker counts R, Y and K by hand. All three are in the round's report, <pass>-round-<n>.md in the review folder: R and Y are its finding headings, and K counts the findings the reviewer marked [misread], as bin/review_rules/round_titles.py says. That module parses the counts for score-outcomes, which reports them as goal 1.1's signal, and rule-hook's round_cap.py checks only the round number, so nothing checks today that the counts match the report. Of the 8 titles whose report the audit could match, none miscounted; one report wrote its markers at line start rather than in ### headings, so the reports' format varies by reviewer.
Add a larv-review subcommand that prints a round title's prefix, up to and including the ;, from the round's report, and have rule-hook refuse a round title whose counts differ from it. Pin the marker format in each reviewer's definition under agents/, or count every format the reports use today. Where the report is missing, say so and let the commit through.
Left with the model: the summary after the ;, and answering the findings.
303. Archive each judge run's prompt and json reply beside the verdict it gave, so judge cost, replays and backups cover the judge as they cover the worker.
Today no judge transcript reaches loop-claude's archive. bin/claude_code/reply.py runs every headless reply, the judge's included, with --no-session-persistence, so its runs stay out of the session picker; Claude Code then writes no session file for the archive to copy. No decision chose to leave the judge out: loop-claude: the archive keeps the raw transcript was written for the worker's .jsonl. Goal 3 asks for the transcripts of unattended runs to be backed up, and the judge is one. The gap also leaves the judge's spend measurable only from the Judge cost: lines, which older verdicts lack ($136 recorded over 2026-09-30 to 2026-11-11), and leaves judge-replay rebuilding each prompt from today's records: of the 39 verdicts it replayed on 2026-10-15, 8 may rest on records that had changed since.
The judge runs with no tools and one turn (judge_args in bin/loop_claude/cycle/judge.py), so its whole transcript is the prompt it read and the json reply. Recommended: keep --no-session-persistence, and have the archive write the prompt and the reply beside the outcomes entry of the verdict, interrupt or rejected reply they produced, deduplicated by content hash as the worker's copies are. A failed write stops the loop, as that decision says. Persisting the session and copying its file would also work, but fills the picker and needs a cleanup. The prompt repeats the worker's transcript, which the archive already holds: measure the size per cycle first, and if it outgrows "small enough to copy whole", keep the prompt without the transcript part and name the worker copy and entry range it held. Then let judge-replay read the archived prompt where one exists, and price the judge from the archived reply's usage. Update that decision entry to say the judge's runs are archived too.
304. Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.
Today review-replay and judge-replay leave no transcript: both run claude through claude_reply in bin/claude_code/reply.py, whose --no-session-persistence makes Claude Code write no session file, as item 303 describes for the judge. A replay decides which rung a review pass or the judge may run on (routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows…, and the judge's rung on JUDGE_REPLAYED), so the runs behind a routing decision can't be read again, re-priced or re-scored once the decision is questioned. Goal 3 asks for the transcripts of unattended runs, the bench's included, to be backed up; workflow-bench/bench.py already keeps its claude stream in logs/.
Unlike the judge, a replayed reviewer uses tools over several turns, so its transcript is more than its prompt and reply. Recommended: have claude_reply take --output-format stream-json --verbose in place of json, keep --no-session-persistence, write the stream where the replay keeps its result, and read the reply from the stream's last result event. That also gives item 303 the judge's stream for free, so build on 303's archive layout and do both callers through the one runner.
306. Set a strategy that keeps loop-claude's archive and larv-review's review folders backed up and small over the years, compressing older entries without losing any.
Measured on 2026-11-11: loop-claude's archive is 6.3 GB (12,721 files from October alone, 0.84 GB older than 30 days), and ~/.local/state/larv-review/ is 997 MB, 984 MB of it under code/, with no pruning: skills/larv-review/SKILL.md → "Reports" keeps every folder after the merge, because review-replay replays rounds from them. The largest archived worker transcript, 23.4 MB, gzips to 9.1 MB at level 6. The disk holds 65 GB free today.
Write the strategy as a decision, then build it:
- Backup. Item 1 waits on ai-sandbox's data volume backup; check that both
~/.claude/loop-claude-transcriptsand~/.local/state/larv-reviewsit on that volume, since the review folders live outside~/.claude. - Retention. Keep everything: the archive is the evidence every routing and replay decision rests on. Compress entries older than a threshold (such as 30 days, or once a chunk is finished and measured) with a format the tools read transparently, such as zstd. Compress, verify the round trip, then remove the original, never the reverse; a failure keeps the original and stops, as a failed archive does.
- Readers.
bin/transcript_archive,routing_figures,score-outcomes,review-replay,judge-replayandreading-costmust read compressed entries; test each on one. - Revise
loop-claude: the archive keeps the raw transcript, whose validity is "while transcripts stay small enough to copy whole".
302. Call a scoring run for an item whose tagline makes the comprehension floor its target, and not for one that only names the floor, such as in a quoted decision title.
bin/larv_review/scoring.py sets FLOOR_ITEM = "comprehension floor" (line 7) and tests it as a substring of the tagline at line 35 and in bin/larv_review/plan.py:28. So any item whose tagline holds those words counts as a floor item. Item 301's tagline quoted the title of the decision "comprehension floor: …", so its chunk was due a scoring run it did not need. The deployed chunk-pr (release 0.7.0) did not refuse the merge, so it merged without one, and the run was recorded late, in PR #476.
Recommendation: a floor item's tagline opens with one fixed phrase, Reach the comprehension floor:, and both checks test that the tagline, with its leading ** stripped, starts with it, in place of the substring test. Reason: a quoted title or a passing mention never sits at a tagline's start, so the check needs no parsing and keeps one constant. An Exempt reason such as floor was the alternative; it would change rank_todo.py's schema and every project's table for the same effect.
The same chunk names the phrase where a floor item is filed: skills/larv-review/SKILL.md → "Floor item", the Next: lines floor_step prints, and larv-review --help (bin/larv_review/main.py, lines 35 and 89). It rewords open floor item 300's tagline to open with the phrase, and updates the tests in bin/larv_review/test_larv_review.py that use floor taglines (lines 120, 361, 435, 537), adding one where a tagline quotes the floor mid-sentence and is due no run.
287. Move the forge helpers Forge, forge_of, runner, open_pull, merged_pull, pull_at and gitea into one module that chunk_preflight, chunk_pr, deploy_release and larv_review all import from, so no tool imports another tool's internals for them.
Today the helpers live in two tools. Forge is defined in bin/chunk_preflight/gathered.py and imported by bin/chunk_preflight/choices.py, bin/chunk_preflight/facts.py, bin/chunk_preflight/main.py and bin/chunk_pr/main.py, and again through choices by bin/deploy_release/forge.py. forge_of lives in choices.py, which bin/chunk_pr/main.py, bin/deploy_release/cut.py and bin/larv_review/reviewer_header.py import. open_pull, runner and gitea live in bin/deploy_release/forge.py, which bin/chunk_pr/main.py and bin/larv_review/reviewer_header.py import. So deploy_release imports chunk_preflight, and chunk_pr and larv_review import both.
Both comprehension seats of PR #453's paired ruling (2026-11-09) named this web as a place that cost them time, and neither saw a reason for it: a second forge would need edits in chunk_preflight, deploy_release, chunk_pr and larv_review.
Put the seven helpers in a folder named by what it holds, such as bin/forge/, and import them from there. Keep each helper's behaviour and its tests. chunk_preflight/choices.py keeps pulls_command, which is preflight's own. Check docs/decisions.md for a mention of where a helper lives, and update it.
328. Script each step that 0.8.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.
lilleman asked, on 2026-10-16, for a standing audit each release, so the loop keeps moving work a program does as well from the model to scripts (goals 1.4 and 1.5). This audit reads the archived transcripts of the chunks that shipped release 0.8.0 for every step that chunks repeated by hand, before, during or after review. It keeps the steps whose answer a program computes the same way each time, such as a lookup, a check, a prompt assembled for a reviewer, a cleanup. File one item per such step, with the time and tokens it cost, into the next release with items left. Leave a step that needs judgement with the model. Run command-patterns 0.8.0 first; its table names the patterns, and read_threads and calls in bin/command_patterns/tally.py open the calls of one pattern. The 0.7.0 audit found that most steps its chunks repeated were already scripted by that release's own tools, so a step a shipped item already covers is not filed.
The chunk that ships this item files a copy of this audit in release 0.11.0, the release after this item's own, to audit release 0.10.0, rewording "0.8.0" to "0.10.0", and marks the copy owner, so the audit recurs each release until lilleman drops it.
288. Point the two reading-cost decisions at code health and goal 1.4's counter-signal, in place of the goal 1.1 they cite.
Both docs/decisions.md entries under reading cost: a rise is … and reading cost: larv-review measures each finished chunk … open "Serves goal 1.1: comprehension …". No goal says that any more: code health (CLAUDE.md → "Shared goal") holds comprehension, and since PR #455 a rising reading cost is goal 1.4's first counter-signal. Raised by the stability review of PR #455 as a 🟢 outside that change.
77. Teach bin/metric-bench Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.
2026-10-08, lilleman: no metric enters review until a rerun on more data shows it tracking the panel.
Before the rerun, set the AUC and top-k lift over chance that count as tracking, from a published threshold, so the rerun ends in a decision. The rerun needs places from a project in Go or TypeScript, the languages most other projects use, so the bench first measures those languages, run through each project's own tooling. Function and file length stay warnings to look into, never limits: a plain long function can be simple, and a hard one may need five-line pieces.
On 2026-10-08 only this repo's history named places: five scoring runs, all that day. The medians:
| Metric | AUC | Top k |
|---|---|---|
| Cognitive complexity | 0.66 | 0.26 |
| Length | 0.74 | 0.18 |
| Shared state | constant | constant |
| Change spread | 0.67 | 0.18 |
AUC is 0.5 for a coin toss, and top k scores 0.10 by chance. Shared state cannot be judged here: every unit in bin/ scores 0 on it. Every run's first place was a section of skills/larv-review/SKILL.md, which no code metric sees.
On 2026-10-09 a question bench asked a headless claude, allowed only Read, Grep and Glob, three questions a script answers from the syntax tree: which units call a unit, which units it calls, and which test files refer to it. It asked them about every unit the history names whose name is unique, plus as many units the history does not name. Five of the history's six scoring runs named such a unit. The tool is in commit 083cc78; it was deleted after the run.
Claude's misses ranked the named units at a coin toss. Top k scores 0.50 by chance on these units:
| Metric | AUC | Top k |
|---|---|---|
| Question misses | 0.49 | 0.50 |
| Cognitive complexity | 0.66 | 0.59 |
| Length | 0.75 | 0.69 |
| Change spread | 0.57 | 0.54 |
Claude's answers overlapped the truth by a median Jaccard index of 0.76. A run took in 62,643 new input tokens and cost $0.70; it did not record which model answered. Grep answers these questions whatever the code's shape, so no question enters review; a new question bench would need questions that take understanding the code, not finding it.
321. Evaluate the review passes that run on a rung item 318 qualified, against the chunks before them, and revert any pass whose findings or later fixes worsened.
lilleman, 2026-11-11: item 318's new bar is a risk, so check it closely a release later. Compare the first 20 chunks reviewed on each newly qualified rung with the 20 before: review rounds past the first, 🔴 and 🟡 per chunk, misread findings, and later fixes that score-outcomes counts (defects that escaped review). Revert a pass to the project's rung where any of those worsened beyond the spread of the 20 before.
273. Re-tune the E + I steps of REVIEW_STEPS for 6 to 8 once 10 finished chunks have scored an E + I of 6 or more, from how well E + I orders their review rounds and 🔴 findings.
docs/decisions.md → routing: one ladder… records why lilleman kept the steps on 2026-11-08. Of the 36 chunks it measured, 1 scored an E + I of 6 or more and none 7 or more, so the steps for 6 to 8 are untested. Start once score-outcomes shows 10 finished chunks with an E + I of 6 or more. Then read its rank correlation for E + I, and the routed-chunks table of routing-figures.md in loop-claude's archive ($LOOP_CLAUDE_TRANSCRIPTS, default ~/.claude/loop-claude-transcripts). Propose new steps to lilleman, whose call they are.
268. End a run whose release is ready to cut with a notice on loop-claude's terminal, in place of an interrupt that waits for an answer.
On 2026-11-08 the human answered notice only: the loop prints that the release is ready to cut and ends; no interrupt, and nothing waits in loop-inbox.
Releases are the human's, and CLAUDE.md → "Working loop" already ends the loop once a release is ready to cut, so the ask settles nothing. 15 interrupts in the archive ask whether the human will cut a release, and about 6 of the 45 interrupts from 2026-11-02 to 2026-11-10 said a release was ready to cut; log's 2.5.0 was asked twice within an hour on 2026-11-10. On 2026-11-08 one of the 4 questions waiting in ~/.local/state/loop-claude/ is such an ask, plainpages-plugin-fastjira's 0.3.0 since 2026-11-03. Today a release ready to cut opens claude's session at once (bin/loop_claude/cycle/run.py ask), and the loop stops if the human leaves it with nothing typed. The change also touches what an interrupt covers in CLAUDE.md → "Vocabulary". Where the change touches ask, make it easier to read, as the seats named it.
171. Group the routing figures' review rounds by the reviewers' rung as well as the worker's, so a call blames a rung only for rounds its own reviews raised.
Raised by the goals audit of PR #296. bin/routing_figures/calls.py groups a chunk's review rounds by the rung its worker started on, and calls for a routing change when a lower rung took more of them. Since PR #296 the reviewers may run above the worker's rung, on the one their item's E + I picks, and no figure groups rounds by it, though the archived worker line names the E + I it follows from. A cheap worker on an item of high E + I gets stronger reviewers, who may raise more rounds, and a call could then blame the worker's rung for them: an interrupt of lilleman that settles nothing (goal 1.2).
300. Restructure the places the comprehension seats found hardest to follow, so the project reads at 7.0 or above on every dimension, Locality first: today the comprehension floor run at 52da15e scored Locality 6.00, Shape 6.25, Self-sufficiency 6.38 and Overall 6.38.
Filed by the comprehension check of PR #473 at 13d1550 (larv-review record-scoring), which printed Floor: and Next:, and refreshed from the run of PR #475 at 52da15e. The lowest mean is Locality, 6.00. Refresh this item's places and aim from the newest run before the chunk starts; do not file a second floor item.
Aim: one ambitious refactor of the areas below, sized so every mean reaches 7.0 on the chunk's own scoring run. Where a place is already easier to read when the chunk starts, drop it.
Places the seats named, hardest first (larv-review places):
- Locality, Shape:
bin/loop_claude/routing.py:93unfinished_route: the floor rung, the escalation cap, the escalation mark, the unfinished count and the taken-in E + I are held at once, across about six helpers in five files. The mark's meaning lives inescalation_mark.py, and the reason for the floor and the cap sits in the decision log. - Locality, Shape:
bin/release_plan/release_standing.py:145next_release: the tag, changelog, before-first-release and renamed branches meet in theis_earliestexpression, which took a senior three reads. Nothing comments theNoneandnothing_cutcase. - Locality:
bin/loop_claude/cycle/judge.py:211quotes_checked: the judge's contract is split over prose constants, schema descriptions and checks after the fact. A bad quote silently changes the verdict's type. - Locality, Shape:
bin/loop_claude/cycle/run.py:146rule, withloop/close.py:18closed_cycle: the order of the ruling, rejection and step-up archive writes carries meaning, and nothing enforces it. Four near-identical archive helpers,run.py:86archivedamong them, sit in the cycle runner thoughloop/close.pyimports them, andrun_cyclereplaces the route partway through its loop. - Locality, Self-sufficiency:
bin/larv_review/reviewer_header.py:49unanswered_roundandskills/larv-review/SKILL.md:76"Round cap": the cap counts answer commits, so a worker that relaunches without committing gets the same round number. The cap is enforced in four units, split over SKILL.md and three modules. - Locality, Shape:
bin/larv_review/plan.py:66plan_reasons: a merge-base commit enters the parameter that item 296's chunk renamedref, eight failure branches mix git reads, file reads and logic, andaftermeans the checkout's text and then its items. - Locality:
bin/larv_review/owed.py:37shortfall: the one-run-or-two rule is keyed on the wordingEMPTIED_WORDSthat plan.py builds and SKILL.md quotes, andscoring_runsdefines a scoring run differently fromEntry.scored. Give the scoring-run logic one module, and have the number of runs owed come from a value, not from a sentence. - Locality:
bin/chunk_pr/main.py:42forge: the forge helpers live inchunk_preflightanddeploy_release, so a second forge needs edits in four packages. - Locality:
bin/rule_hook/reviewer_rung.py:40skill_breaches: three cases rest on assumptions made in another function. - Locality:
bin/rule_hook/repeat_read.py:119shown_lines: it rebuilds from transcript entries which lines are in the agent's context. - Locality:
bin/model_routing/route.py:88route_of: a route has two serializations, a flat archive line and a nested handover. - Self-sufficiency, Locality:
bin/loop_claude/cycle/prompts.py:59review_launchesand:119escalation: the worker's prompt is built from about 20 conditional sentence builders, which restate the routing logic. - Shape:
bin/release_plan/ranking.py:56rank,:304scoreand:315weight_of: nearly every function forks onSHAPEversusOLD_SHAPE, which doubles the module; item 156 removes the old shape. The constants read as magic untilrules/todo-items.mdis open. - Shape:
bin/review_rules/comprehension_history.py:127entries: a line-by-line parser that tolerates two legacy formats. - Shape:
bin/records/__init__.py:1records: one name for three concepts, ascutandPlanalso name several. - Shape:
bin/rule_hook/rules.py:25RULES: rules sit inline in one module while five others have a module each. - Navigation, Shape:
README.md:116bin/: about 55 entries in one flat list, some 31 of them packages, the worst fan-out level. - Self-sufficiency:
skills/larv-review/SKILL.md:147"Scoring run": dense rule prose with many cross-references; three sections rebuild one rule.
Scored L 4 because the refactor touches several components that hold known defects, E 2 because review and a manual check see a regression, I 3 because every session runs these tools. The chunk runs a scoring run, since its tagline names the comprehension floor.
264. Report the comprehension seats and the escalated subagents as subagent types of their own in routing-figures.md, with their dollars.
Seats launch as general-purpose, so they hide inside that type: $301.24 of the $962.17 the figures of 2026-11-04 count, from 221 subagents. By 2026-11-10 general-purpose was the second-largest spend, $585 of about $2,385, and by transcript size about two thirds of it were seats (291 launches), then 177 security passes. Escalations are counted by trigger only, never in dollars. bin/routing_figures/chunk.py groups subagents by their subagent_type; a seat is told apart by its prompt, and an escalated subagent by the escalation marker its prompt opens with.
323. Count a release review as landed once its pull request has merged, whatever its commits are titled.
release_reviewed in bin/release_plan/release_standing.py counts a release review only by a commit titled Release review of <release> on the default branch. In a replay of archived interrupts (judge-field-trials/), in two projects whose release review had merged, the default branch held no such commit, so chunk-preflight and routing read ReviewDue, and the next loop would run the release review again. skills/larv-review/SKILL.md opens the review's branch with an empty commit of that title, but a worker may skip it, and a rebase or squash merge, which chunk-pr uses where the forge allows no merge commit, may drop it. Detect the review by something every merge method keeps, and check the fix against both projects' histories. Until then the judge keeps release_ready (docs/decisions.md → routing: loop-claude reads the items a chunk takes …).
149. Fill each release by priority up to a capacity it states, in place of the bar, and move what does not fit to a later release.
A release's capacity is a sum of its items' cost, as the formula counts it. Each project states its own; the human sets the first value per project. Nothing is deleted by score.
165. Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant.
A test audit across lilleman's projects on 2026-10-16 found 10 of 33 sampled tests in this repo pinning wording that is not the contract. Examples of these and other weak tests: many tests in bin/loop_claude/test_loop_claude.py pin whole sentences of the question, archive, prompt or help text, up to about 750 characters. test_every_exit_1_stop_has_its_clause_in_help and test_every_failed_judge_run_has_its_clause_in_help parse the source with ast and list each Stop(...) and Failure(...) call by function name. bin/model_routing/test_ladder.py asserts len(LADDER) == 6. Rewrite each to the technical principles' test bullet: assert the facts the text must carry, such as an item ID, a rung or a model argument, and delete a test no real bug could fail. Before and after rewriting a test, run the mutation tool on the code it covers: every mutant the old test killed, the new one kills.
151. Keep 10 to 20 settled decisions per project as its goals' test set, and rerun them after a goal changes.
Each case is a question the project settled and the answer it chose. A fresh-context agent given only the README's premises answers each case; a case it answers differently says the goals no longer decide it. Written guidance can fail to change behaviour, which is why the goals get a test.
68. Test workflow-bench/compare.py's blinding and bench.py's run plan.
"Most, if not all, code should have automated tests", in principles/technical-principles.md: compare.blind() keeps the comparison blind, and a version name it misses invalidates the bench silently.
The tests also cover workflow-bench/bench.py's permission_flags, with a probe of a headless claude given its flags. It is denied a read outside the granted trees, a bare git status, and wrapper calls with --output, --no-index, a glued -aF/abs, --fil=/abs or a path outside the clone. A cd <elsewhere> && python3 <wrapper> … acts on the clone only. It is allowed the wrapper's log, diff, show, status, ls-files, add and commit -m. workflow-bench/test_review_git.py tests the wrapper's refusals directly.
102. Give loop-claude's retry delay, and the file it reads the machine's memory from, environment overrides, so the black-box test stops replacing time.sleep and machine_load.memory.MEMINFO through sitecustomize.
Found by the principles audit of PR #242. The SITECUSTOMIZE block in bin/loop_claude/test_black_box.py swaps the global time.sleep, the shape "Functional core, imperative shell" bans, and the audit of PR #376 found the same block swapping machine_load.memory.MEMINFO, the constant in bin/machine_load/memory.py. An override of each lets the test run the code as production runs it.
135. Fingerprint only the modules loop-claude imports, so editing another tool's code no longer reloads a running loop.
The maintainability senior and the inherited architect of PR #257's scoring run named fingerprint and exec_if_edited in bin/loop_claude/loop/reload.py: the fingerprint covers every bin/**/*.py, so editing statusline.py or rank_todo.py re-execs a running loop. Make fingerprint easier to read.
5. Key a project on its checkout's full path, and refuse to start while another session uses that key.
The key names the project's session, archive folder and models entry. project_key, in bin/transcript_archive/project_key.py, strips ~/code/ and then any leading /, so ~/code/a and /a share all three, and one loop can resume or delete the other's session.
bin/reading-cost keys the archive by the main checkout, the parent of git's common dir, since it runs in a chunk's worktree; loop-claude keys it by --show-toplevel, so a loop started inside a worktree archives where reading-cost never looks. Derive both keys one way.
291. Let principles/technical-principles.md → "Sane defaults" allow a library whose README goals ask that it work with no config to default the clock and randomness in its public constructor, where a documented option overrides each.
"Sane defaults" has a library ask for every input that can change between runs, the clock and randomness included, and leaves defaulting them to the code that wires a program together. A logging library's principles audit found that its constructor defaults both, as its first goal, useful with no config, asks. Its decision log records the defaults as a departure from the principle, and names this item as what ends it. Until then, each principles audit there may raise the defaults again.
2026-11-10, the human: keep the library's defaults, and give the principle an exception for this case, worded about as the tagline says. The exception has to square with "Functional core, imperative shell", which the same bullet cites and no other principle overrides.
156. Drop rank_todo.py's ranking of an old-shape table once no project under ~/code holds one.
While rank_todo.py ranks two shapes, projects follow two sets of scoring rules, against goal 1.3. Once every project has reshaped, drop OLD_SHAPE's weighing from bin/release_plan/ranking.py, RESHAPE, and the old shape's lines in rank_todo.py --help and rules/todo-items.md. Keep todo_table reading old tables: score_outcomes reads every past merge's todo.md. PR #283's paired ruling found the old-shape path made ranking.py's score, Scoring.weights, unknown_columns and todo_table.py's Moves harder to read; leave each easier, with one shape and moves that are always moves, and the Moves cell's rules in one place. Release 0.3.0's seats named ranking.py's score, weigh and rank and todo_table.py's OLD_SHAPE for the same fork: "every rule exists twice for the old table shape" (Mid A). Make weigh and rank easier to read too.
196. Replay the architecture, principles, overrides and product-owner passes with review-replay on Opus 5.5 projects, record each run in the review-replay routing decision, and, where a run qualifies a rung, add back the code that lowers a pass to it.
Item 159's chunk, PR #322, replayed only the prose and stability passes. Sonnet 5.5 at high missed 44 to 55% of what the project's rung found again in each, so the agent judged a replay of the other four unlikely to qualify a rung. The product-owner review of PR #322 asked that the plan keep them, so the ranking decides when they run. Each run costs hours of reviewer replays (goal 1.4), so the score keeps it low until a cheaper model joins the ladder. docs/decisions.md → routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds… records the runs so far. The release review of 0.6.0 deleted the code that lowered a pass, since no rung had qualified; the decision says where it goes back.
260. Compare the first 20 chunks whose worker got the reading rule with the 20 before them, on review rounds past each pass's first, unfinished verdicts, 🔴, 🟡 and misread findings, dollars and hours per chunk, and drop the rule where any rose.
Start once 20 chunks have run under the rule. The rule is READING_RULE in bin/loop_claude/cycle/prompts.py, which every fresh worker prompt holds; git log --first-parent -S READING_RULE on the default branch finds the merge that added it. A chunk got it when its first worker started after that merge. Measure as docs/decisions.md → loop-claude: the worker's claude compacts its context before it fills a 200k window… measured the window: rounds past each pass's first and the 🔴, 🟡 and misread counts from the round commit titles, unfinished verdicts from the outcomes files, hours from the worker's first message to its last, and dollars from the chunk's cost-state entry. Compare means, since the decision weighs the rule by all spend.
51. Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints.
"Functional core, imperative shell", in principles/technical-principles.md: say and input_or in bin/loop_effects/effects.py take the effects but print to stderr or stdout themselves, and retried and the rest print through say. Two bare print(file=sys.stderr) calls remain outside them, in bin/loop_claude/cycle/interrupt.py and bin/loop_claude/project/model_choice.py. The tests swap the global stream.
201. Save the human's tmux setups and restore them after a reboot, so work is up and running again quickly.
The human asked on 2026-10-21 for tmux support, above all for saving tmux setups. Don't build anything until the human has said exactly what they need. Ask them at least:
- What a setup holds, such as sessions, windows, panes, each pane's working directory and the command running in it, like one
loop-claudeper project. - When a setup is saved: on demand or on its own.
- How a setup comes back after a reboot.
- Whether the code that saves and restores setups must work on every machine that takes this checkout (goal 2), which decides whether that code lives in this repo.
- What else tmux support should cover.
Weigh the widely used tmux-resurrect and tmux-continuum plugins against a script of our own. Then re-score this item from the answers and that choice, and split it into items where it is more than one chunk.
267. Keep waiting when an interrupt gets an empty answer, at the terminal or in the claude opened for it, in place of stopping the loop.
11 runs in the archive ended with "Stopped: the human is needed to answer the question above, but the interrupt ended with nothing typed", each leaving its chunk open until the human restarted the loop. That stop comes from human_typed in bin/loop_claude/cycle/interrupt.py, after the claude opened for /talk or for a release ready to cut ends with no human turn. An empty line at the terminal stops the loop the same way, in first_answer. Item 268 removes one of the cases that opens claude.
4. Name each chunk's archive folder by the chunk's start time, so <key>/ lists chunks in the order they ran.
Today the folder is the SHA-256 of the transcript's first timestamped line, which tells the human nothing; that line carries the start time. Drop the fallback to the first line, which every chunk of a project shares, and give the tests' fake transcripts ISO timestamps. bin/reading-cost takes a chunk's id from the folder's first 12 characters and skips a chunk whose id docs/reading-cost.md holds, so give it a way to match a figure written under the old name.
324. Count in the routing figures the finished chunks whose worker wrote no Chunk items: line.
docs/decisions.md → routing: loop-claude reads the items a chunk takes … holds only while workers write the line when asked, and nothing counts a chunk whose worker skipped it. There loop-claude scores no item beyond the one the chunk was routed on, so the route holds, and loop-inbox lists only that item. Count in bin/routing_figures/, beside the chunks counted, those whose worker's transcript holds no such line, so the human sees when the decision's premise stops holding.
187. Count the tokens a worker spends reading reviewers' reports, so goal 1.4's signal shows whether report files lowered what a review costs the worker's context.
Before PR #310, reports reached the worker as the Agent call's result, a median 19k tokens per chunk (2026-10-08, 68 archived transcripts). Now a reviewer returns its verdict and one line per finding, and the worker reads a 🔴 or 🟡 finding's entry from the report file. reading-cost counts only tracked files (bin/reading_cost/measure.py), and routing-figures counts only the Agent call's result (bin/routing_figures/chunk.py), so neither sees the report reads. Found by the architecture review of PR #310.
266. Move the chunks archived under the key home/lilleman/.claude to the key claude-config, so the routing figures and score-outcomes count this repo's chunks as one project.
The archive holds 277 chunk folders under home/lilleman/.claude, from 2026-09-24 on, and 39 under claude-config (2026-11-10). The routing figures of 2026-11-04 list no project home/lilleman/.claude. They find a chunk's later fixes on the default branch of the checkout its key names (bin/routing_figures/archive.py measure_project), and /home/lilleman/.claude is a release checkout on a detached HEAD. Item 5 fixes how keys are derived, not the chunks already archived. ~/.claude/loop-claude-models.json still holds a home/lilleman/.claude entry; it is machine state outside the repo, so removing it is a one-off step on each machine, not part of the change.
181. Have the test of a quoted answer's step-up assert the stepped rung's facts as literals, not through prompts.rungs_held.
bin/loop_claude/test_loop_claude.py's test_a_quoted_answer_that_cites_an_escalation_trigger_steps_the_worker_up_before_it_resumes_and_archives_the_step builds its expected prompt from prompts.rungs_held, the code under test, so a bug there passes it. The technical principles' test bullet bans that shape. Assert the new rung and one review launch as literals, as the typed answer's step-up test does. Raised by the principles audit of PR #308.
193. Narrow readme_goals's mark to Mark by a typed lookup in place of typing.cast, so a mark added to the regex but not the lookup fails at run time, and one added to the lookup but not Mark fails mypy.
Found by review of PR #317. bin/release_plan/ranking.py's readme_goals writes typing.cast(Mark, mark or None), which asserts the type goal_weights branches on without checking it: a mark added to the regex reaches goal_weights as a value Mark lacks. mypy does not read the regex, so no lookup lets it catch that. Replace the cast with a lookup typed dict[str, Mark], from each mark the regex captures, and "" for none, to its Mark value. A mark added to the regex but not the lookup then raises KeyError at run time, and a value added to the lookup but not Mark fails mypy.
194. Make test_waits_out_a_usage_limit_which_ends_a_run_of_failures expect a usage-limit wait independent of the wall-clock time that passes during the run.
Found by review of PR #317. The black-box test, bin/loop_claude/test_black_box.py:609, not the unit test of the same name in test_loop_claude.py, which runs on a fake clock, sets resets_at from the test's time.time() and expects the wait, rounded to tens, to be 3720. When more than about 5 s pass between that read and the loop's, the wait rounds to 3710 and the test fails with no bug. Item 102 covers only replacing the time.sleep swap; this item changes the expected value, for example by asserting the wait against resets_at + 120 minus the time the loop read, or a bound on it.
265. Archive each wait that costs a chunk wall time, for a usage limit, low memory, a failed worker start or a heavy slot, with its start and end stamps, so goal 1.5's hours split waiting from work.
No record holds these waits today. An outcomes file holds verdicts, stops, step-ups and the judge's cost (bin/transcript_archive/outcomes.py), and ~/.local/state/run-heavy/ holds only the slots' lock files. loop-claude waits out a usage limit and low memory in bin/loop_claude/cycle/retry.py; run-heavy waits for a slot in bin/run_heavy/. A heavy slot's wait happens inside a worker's Bash call, so loop-claude doesn't see it; run-heavy would write it where the archive can find it.
169. Have rule-hook's todo-rank-check catch a todo.md that the same Bash call edits and then commits.
The hook judges a call before it runs, and changed in bin/rule_hook/todo_check.py compares todo.md with the index and HEAD as they stand then. A call that edits todo.md and commits it, such as a heredoc script followed by git commit, passes unchecked: PR #296's commit 149108e did, and the override marker it carried was never logged. larv-review's gate still runs rank_todo.py --check, so an unranked plan stops at merge rather than at commit. A git pre-commit hook would see the staged file.
197. Let the human edit what they type at loop-claude's terminal questions with the arrow keys, so Left moves the cursor in place of printing ^[[D.
bin/loop_effects/effects.py hands loop-claude Python's builtin input, and nothing under bin/ imports readline, so the terminal gets no line editing: fixing a typo means erasing back to it. loop-claude asks two questions at its terminal. effects.input reads the pick of the project's model, in project/model_choice.py's ask_for_model. effects.input_or reads the interrupt's answer, in cycle/interrupt.py's first_answer, straight from stdin, since it also watches loop-inbox. Importing readline in effects.py gives input line editing, at two costs; input_or would also need to read through input while it watches. It moves the prompt from stderr to stdout, while the question above it stays on stderr. test_black_box.py waits for the prompt on stderr. readline's bracketed paste takes a pasted block whole, so an answer can hold several lines. prompts.typed_answer ends the answer at its first blank line, so the judge would read the rest of such a paste as loop-claude's words. The product-owner review of PR #308 raised the interrupt's case. lilleman, 2026-10-18: "Not super-important, but a bit irritating."
1. Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.
It lives under ~/.claude, which .gitignore keeps from the remote, so a lost disk loses every unattended run's record. Back it up only to a destination private to me and encrypted at rest: the worker runs with permissions bypassed, so the archived tool results can hold credentials it read.
2026-10-02, the human: ai-sandbox is the VM loop-claude will run on, hosted on the server bender. bender-infra, the infrastructure-as-code project that manages bender, will back up ai-sandbox's data volume once ai-sandbox is rebuilt. The item waits for the human to say the rebuild has landed; do not remind them. Before writing the README line, check that ~/.claude lives on that volume and that the backup is private and encrypted at rest.
The README line covers the bench's transcripts in workflow-bench/logs/ too (goal 3).