Files
claude-config/todo.md
T

20 KiB
Raw Blame History

todo

Scoring

Score = -R - S/4 + 2*A + 2*G*W

Bar = 9

Next ID = 86

Goal W
1 1.00
1.1 0.99
1.2 0.98
1.3 0.97
1.4 0.96
1.5 0.95
2 0.90
3 0.85
4 0.80

Items

ID Release Exempt Item R S A G Goals Score
78 0.1.0 Generate comprehension questions with deterministic scripts — from symbols, the call graph and the tests, each answer checkable by a tool — and bench the scores an agent gets on them against the panel, reading cost and static metrics. 3 7 9 8 1.1, 1.4 29.1
85 0.1.0 decision Move larv-review's mechanical rules — the paired ruling's base and vote count, each pass's round count, the scoring run's trigger and the comprehension history's format — into one tested tool under bin/, so the skill says only when to run it and what each answer means. 4 7 8 9 1.1 28.1
1 0.1.0 question Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume. 3 5 8 9 3, 4 27.0
62 0.1.0 defect Lift Locality, Self-sufficiency and the overall to the comprehension floor, at the places the seats name. 4 6 6 8 1.1 22.3
21 0.1.0 Give settings.json's model one writer, and sort its keys. 3 3 6 7 1.3 21.8
49 0.1.0 principle Check that AGENTS.md lists exactly the titles of docs/decisions.md. 1 2 5 6 1.3 20.1
59 0.1.0 Bound a run of unfinished verdicts on one chunk, so a worker and judge that disagree cannot cycle unattended without end. 3 3 6 6 1.4 19.8
12 0.1.0 Run the test suite in CI on every PR, or record in docs/decisions.md why the local run is enough. 2 4 4 6 1.1 16.9
80 0.1.0 State in README → Install that every machine taking this checkout must be a sandbox, as settings.json's auto mode assumes. 1 1 5 5 4 16.8
83 0.1.0 Group bin/ by tool, so each tool's modules and tests sit together. 3 4 4 6 1.1 15.9
5 0.1.0 decision Key a project on its checkout's full path, and refuse to start while another session uses that key. 5 5 5 6 1.1 15.6
84 0.1.0 Check, at each release's scoring run, that seats in a paired ruling agree more often than two scoring runs on near-identical code do, as the paired-ruling decision assumes. 2 2 4 5 1.4 15.1
67 0.1.0 principle Point stability-reviewer at the technical principles in place of its own restatement of them. 2 3 4 5 1.3 14.9
66 0.1.0 principle Hold systems-architect's method thresholds only while the comprehension panel finds the flagged code hard. 3 3 4 5 1.1 14.2
69 0.1.0 principle Measure test coverage, so a chunk that lowers it is seen. 3 4 4 5 1.1 13.9
29 0.1.0 Make the contributor a standing member of every project's audience, unless its ## Audience says maintainability does not matter. 2 2 3 5 1.1, 1.4 13.4
68 0.1.0 principle Test workflow-bench/compare.py's blinding and bench.py's run plan. 2 4 3 5 1.1 12.9
65 0.1.0 Archive each judge reply the loop rejects. 3 4 4 5 3 12.5
82 0.1.0 principle Sort AGENTS.md's decision titles and docs/decisions.md's entries by title. 1 2 3 4 1.3 12.3
73 0.1.0 Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries. 3 4 4 4 1.4 11.7
6 0.1.0 decision Move what loop-claude writes out of ~/.claude.json, and ask the human about each write that must stay. 5 5 4 5 1.1 11.7
51 0.1.0 principle Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints. 3 5 3 5 1.1 11.7
50 0.1.0 principle Grant the bench's runs only the tools a review needs, in place of skipping every permission. 2 2 3 4 4 9.9
81 0.1.0 principle Have workflow-bench/bench.py return each run's failure as a value, so one failed clone does not hide the later runs' results. 2 2 3 4 4 9.9
4 0.1.0 Name each chunk's archive folder by the chunk's start time, so <key>/ lists chunks in the order they ran. 3 2 4 3 3 9.6
52 0.1.0 principle Record why this checkout lives in Claude Code's own directory, or move it out and link it in. 6 6 3 5 2 7.5
77 0.2.0 Rerun bin/metric-bench once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel. 2 4 6 7 1.1, 1.4 22.9

Details

78. Generate comprehension questions with deterministic scripts — from symbols, the call graph and the tests, each answer checkable by a tool — and bench the scores an agent gets on them against the panel, reading cost and static metrics.

A hand-written question bank goes stale as the project grows and leaves everything it does not ask about unmeasured. A script that samples symbols at random each run ("what calls X?", "where is Y validated?", "which test fails if Z changes?") covers the whole project over time, and the call graph, grep and the tests hold the answers, so no judge is needed. Each answer names the code it rests on, so a question whose code changed is regenerated. Keep the method only where the bench shows it tracks the panel at a lower cost.

85. Move larv-review's mechanical rules — the paired ruling's base and vote count, each pass's round count, the scoring run's trigger and the comprehension history's format — into one tested tool under bin/, so the skill says only when to run it and what each answer means.

2026-10-08, lilleman: next after the items that lower token use (73, 77, 78). Every comprehension run since PR #224 named skills/larv-review/SKILL.md §3 "Architecture" the hardest place, and PR #228's paired ruling ruled worse on rules a program should hold: the base's git log search and fallbacks, the vote arithmetic, a trigger written out three times, and the round count PR #225's seats said only prose enforces. PR #228's reviewers found about twenty edge cases in that text, among them squash against merge commits, entry order, a vote with no place, and a trigger judged before items are removed; tests can hold each, prose could not. Keep the skill's judgement calls in prose, and give the tool the history's format, so the base search does not hang on the wording of a line.

1. Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.

It lives under ~/.claude, which .gitignore keeps from the remote, so a lost disk loses every unattended run's record. Back it up only to a destination private to me and encrypted at rest: the worker runs with permissions bypassed, so the archived tool results can hold credentials it read.

2026-10-02, the human: ai-sandbox is the VM loop-claude will run on, hosted on the server bender. bender-infra, the infrastructure-as-code project that manages bender, will back up ai-sandbox's data volume once ai-sandbox is rebuilt. The item waits for the human to say the rebuild has landed; do not remind them. Before writing the README line, check that ~/.claude lives on that volume and that the backup is private and encrypted at rest.

The README line covers the bench's transcripts in workflow-bench/logs/ too (goal 3).

62. Lift Locality, Self-sufficiency and the overall to the comprehension floor, at the places the seats name.

Raised by the principles audit of PR #215 against KISS, in principles/technical-principles.md, and goal 1.1: the comprehension floor: entry in docs/decisions.md lists seven decisions to ship under the floor, and 0.1.0 waits on the floor. PR #215's last run put Locality at 6.63 and Self-sufficiency and the overall at 6.88. Its seats named, under Locality: the judge's contract spread over cycle/judge_reply.py, cycle/worker.py and cycle/run.py; remaining, which is Unfinished.evidence under another name, threaded through loop/main.py, loop/reload.py's LoopRun, cycle/run.py and worker_args; the quoted_answers cap held by case order in cycle/states.py's transition and restated in run.py's question_for; the four outcome texts run.py builds inline for archive. Under Self-sufficiency: skills/larv-review/SKILL.md, which needs CLAUDE.md → "Vocabulary" open beside it; the archive layout, defined only in bin/loop_claude/loop/main.py's docstring and not at transcript_archive.py; JUDGE_RULES' precedence in cycle/judge_reply.py, restated by verdict_from's branch order and the Reply field descriptions; the sys.path insert in bin/loop-claude that the from failure import lines rely on, unstated at the import sites.

PR #225's last run, at 8c9bf09, put Locality at 6.75 and Self-sufficiency and the overall at 6.88. Its seats named: the judge's policy spread over loop/main.py's docstring, cycle/judge_reply.py's JUDGE_RULES, cycle/judge.py's judge_prompt and cycle/worker.py's worker_args; loop_claude/claude_code/ and the read side of transcript_archive.py, which bin/reading_cost/ also imports; two meanings of a checkout's root in project/checkout.py's checkout_root and reading_cost/main.py's located; and skills/larv-review/SKILL.md §3's round count, which only prose enforces.

21. Give settings.json's model one writer, and sort its keys.

Claude Code rewrites "model" in this tracked file whenever a model is picked, so one /model dirties the checkout and overrides the synced choice.

Every write Claude Code makes to the file rewrites "model", claude plugin install included: Claude Code 2.1.288 turned "opus" into "opus[1m]". Each write also puts the keys back in Claude Code's own order, so a sorted file stays sorted only until the next write.

49. Check that AGENTS.md lists exactly the titles of docs/decisions.md.

DRY, in principles/technical-principles.md: the list is copied by hand, and sessions work from the copy.

59. Bound a run of unfinished verdicts on one chunk, so a worker and judge that disagree cannot cycle unattended without end.

Raised by the product-owner review of PR #211. The resumed worker gets the judge's quote of the work remaining, but a worker that still thinks the chunk done can answer each cycle the same way, and each cycle costs a worker and a judge run (goal 1.4).

12. Run the test suite in CI on every PR, or record in docs/decisions.md why the local run is enough.

The repo has no CI, so larv-review's gate is the local run, and larv-review's Gate skips even that when the last stability round left the suite green: an unattended chunk can merge on the worker's own word.

80. State in README → Install that every machine taking this checkout must be a sandbox, as settings.json's auto mode assumes.

Raised by the principles audit of PR #224. The settings.json decision is valid while every machine taking this checkout is a sandbox, but Install asks for a sandbox only for loop-claude, so a new machine is never told.

83. Group bin/ by tool, so each tool's modules and tests sit together.

Raised by the architect on PR #225 and by the inherited architect of PR #224's scoring run: bin/ is one flat folder of tools, the modules they share and the repo's checks, with loop_claude's tests beside them instead of beside the package. loop_claude/claude_code/ parses Claude Code's own formats and bin/reading_cost/measure.py imports it too, so it is a package of its own.

5. Key a project on its checkout's full path, and refuse to start while another session uses that key.

The key names the project's session, archive folder and models entry. project_key drops the leading /, so ~/code/a and /a share all three, and one loop can resume or delete the other's session.

bin/reading-cost keys the archive by the main checkout, the parent of git's common dir, since it runs in a chunk's worktree; loop-claude keys it by --show-toplevel, so a loop started inside a worktree archives where reading-cost never looks. Derive both keys one way.

84. Check, at each release's scoring run, that seats in a paired ruling agree more often than two scoring runs on near-identical code do, as the paired-ruling decision assumes.

Raised by the principles audit of PR #228: the review: a paired ruling … entry in docs/decisions.md is valid only while that holds, and docs/comprehension-history.md keeps every vote, yet nothing reads the agreement rate. Scoring runs on near-identical code are rare now, so compare with those already in the history, such as PR #224's and PR #225's, or make a second run for the check.

67. Point stability-reviewer at the technical principles in place of its own restatement of them.

DRY, in principles/technical-principles.md: agents/stability-reviewer.md restates principles in its own words, and the copy has drifted: it asks for "properly randomised UUIDs" where the principle requires UUID v7.

66. Hold systems-architect's method thresholds only while the comprehension panel finds the flagged code hard.

KISS, in principles/technical-principles.md, which keeps a limit only while readers find the flagged code hard. agents/systems-architect.md makes cyclomatic 7 and 24 lines gate rules from the first commit, a 🔴 unless a reason is stated.

69. Measure test coverage, so a chunk that lowers it is seen.

"Test coverage should not decline", in principles/technical-principles.md: nothing reports coverage, so a chunk that deletes a test or adds untested code passes unseen.

29. Make the contributor a standing member of every project's audience, unless its ## Audience says maintainability does not matter.

State that in CLAUDE.md's goals-and-audience rule, and have larv-review's comprehension floor name the contributor as the reader it serves. The floor's block then applies to a project whose audience includes the contributor, and a project that drops them skips it.

68. Test workflow-bench/compare.py's blinding and bench.py's run plan.

"Most, if not all, code should have automated tests", in principles/technical-principles.md: compare.blind() keeps the comparison blind, and a version name it misses invalidates the bench silently.

65. Archive each judge reply the loop rejects.

Goal 3, raised by the principles audit of PR #215: the judge runs with --no-session-persistence, and the first two rejected replies reach only stderr, so the archive cannot answer the 3am question of why the loop rejected a verdict.

82. Sort AGENTS.md's decision titles and docs/decisions.md's entries by title.

"Deterministic order", in principles/technical-principles.md, raised by the principles audit of PR #224: their order carries no meaning, so two chunks adding a decision conflict at the end, and a reader cannot find a title by scanning.

73. Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.

2026-10-08, lilleman: rescored from A10/G8 to A4/G4 on the measurement below, so other items that lower token use go first.

Measured over the 68 transcripts in loop-claude's archive on 2026-10-08, matching each Agent call to its tool result and to any <task-notification> naming its tool-use id. No report reached the worker twice: 813 arrived once as the hand-back, and 363 once as a notification after Claude Code ran the agent in the background. Reports took a median 19k tokens per chunk, 8% of the worker's peak context (24% at most). Lines carrying a severity mark are 9–24% of a reviewer's report, and the worker would still open the file for each 🔴 or 🟡 it fixes. A third of the report volume is the comprehension seats', whose votes and places the worker copies into the history. File reads fill more of the worker's context: sed -n output is 30% of its Bash results and grep output 11%, which item 78 and reading-cost target.

The reports would live in the session's scratchpad, such as <scratchpad>/larv-review/prose-round-1.md, so no project's checkout sees them. A later round gets the path of that pass's earlier report in place of the findings pasted in. How a report reaches the file:

  • Write access. Grant each reviewer Write. No reviewer under agents/ holds it today, an agent's tool list cannot confine Write to one folder, and ~/.claude/CLAUDE.md → "Working with the codebase" gives a reviewer that only reads no edits. Check whether a PreToolUse hook can tell a subagent's call from the worker's and deny a Write outside the scratchpad.
  • A wrapper. A general-purpose subagent launches the reviewer, saves the report and returns only the summary lines. Claude Code's documentation says a subagent cannot start another; drop this route unless a test shows one can.

This risks goal 1.1: a worker fixing from a one-line summary can miss context it needs.

6. Move what loop-claude writes out of ~/.claude.json, and ask the human about each write that must stay.

Claude Code owns that file and writes it. trust reads it, changes it and writes it back, so a write that lands in between is lost.

51. Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints.

"Functional core, imperative shell", in principles/technical-principles.md: archive, trust, retried, the worker, interrupt and model_choice print to stderr themselves, and the tests swap the global stream.

50. Grant the bench's runs only the tools a review needs, in place of skipping every permission.

"Always grant explicit access to data", in principles/technical-principles.md: workflow-bench/bench.py starts claude with --dangerously-skip-permissions, and no rule confines it to a sandbox, so a run on the host reaches the host's credentials and other checkouts.

81. Have workflow-bench/bench.py return each run's failure as a value, so one failed clone does not hide the later runs' results.

"Errors as values", in principles/technical-principles.md, raised by the principles audit of PR #224: bench.py raises KeyError and CalledProcessError, and pool.map re-raises the first while the other threads run on unreported.

4. Name each chunk's archive folder by the chunk's start time, so <key>/ lists chunks in the order they ran.

Today the folder is the SHA-256 of the transcript's first timestamped line, which tells the human nothing; that line carries the start time. Drop the fallback to the first line, which every chunk of a project shares, and give the tests' fake transcripts ISO timestamps. bin/reading-cost takes a chunk's id from the folder's first 12 characters and skips a chunk whose id docs/reading-cost.md holds, so give it a way to match a figure written under the old name.

"One owner per value" and "Prefer the vendor-neutral option", in principles/technical-principles.md: git, Claude Code and loop-claude all write ~/.claude, kept apart only by the .gitignore allowlist, and the repo's records live in a vendor directory.

77. Rerun bin/metric-bench once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.

2026-10-08, lilleman: no metric enters review until a rerun on more data shows it tracking the panel.

bin/metric-bench measures each Python function and method at each scoring run's commit, and scores how well each metric ranks the units the seats named above the rest. On 2026-10-08 only this repo's history named places: five scoring runs, all that day. The medians:

Metric AUC Top k
Cognitive complexity 0.66 0.26
Length 0.74 0.18
Shared state 0.50 0.06
Change spread 0.64 0.17

AUC is 0.5 for a coin toss. Shared state cannot be judged here, since no unit in bin/ writes state another method shares. Every run's first place was a section of skills/larv-review/SKILL.md, which no code metric sees. The rerun needs places from a project in Go or TypeScript, the languages most other projects use, so the bench first measures those languages, run through each project's own tooling. Function and file length stay warnings to look into, never limits: a plain long function can be simple, and a hard one may need five-line pieces.