Audit 0.9.0's chunks: file 1 scripted step, and the audit again for 0.15.0
CI / tests (pull_request) Successful in 3m46s

This commit is contained in:
2026-11-17 09:05:11 +01:00
parent a25b603766
commit b3c95c2c3d
2 changed files with 66 additions and 1 deletions
+34
View File
@@ -2491,3 +2491,37 @@ Read most (opens and searches):
- `bin/larv_review/main.py` 1
- `skills/comprehension-panel/SKILL.md` 1
- `skills/larv-review/SKILL.md` 1
## Reading cost, chunk 79999c427d31, finished 2026-11-17T07:55:21Z
100630 new input tokens before the first commit.
Read most (opens and searches):
- `bin/rule_hook/test_rule_hook.py` 8
- `bin/larv_review/main.py` 6
- `README.md` 5
- `bin/rule_hook/rules.py` 5
- `bin/larv_review/reviewer_header.py` 4
- `bin/git_cli/git_repo.py` 3
- `bin/larv_review/test_larv_review.py` 3
- `bin/review_rules/test_round_titles.py` 3
- `bin/rule_hook/main.py` 3
- `skills/larv-review/SKILL.md` 3
## Reading cost, chunk a45b9acf4a98, finished 2026-11-17T07:59:39Z
211156 new input tokens before the first commit.
Read most (opens and searches):
- `bin/loop_claude/test_loop_claude.py` 25
- `bin/loop_claude/test_black_box.py` 13
- `bin/loop_claude/routing.py` 12
- `docs/decisions.md` 11
- `bin/model_routing/test_ladder.py` 8
- `bin/model_routing/ladder.py` 5
- `bin/judge_replay/main.py` 4
- `bin/loop_claude/cycle/run.py` 4
- `bin/loop_claude/loop/close.py` 3
- `bin/model_routing/test_review_choice.py` 3
+32 -1
View File
@@ -6,13 +6,14 @@
`Bar = 0.1`
`Next ID = 396`
`Next ID = 398`
## Items
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 338 | 0.12.0 | | owner | **Script each step that 0.9.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.3 |
| 396 | 0.12.0 | | | **Print one section of a markdown file, picked by a fragment of its heading, with `markdown-section <file> <fragment>`.** | 2 | 1 | 3 | 4 | 4 | 1.4 by 3, 1.5 by 2 | 1.4 |
| 299 | 0.12.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
| 311 | 0.12.0 | | | **Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.0 |
| 316 | 0.12.0 | | | **Launch comprehension seats as an agent type of their own under `agents/`, without `CLAUDE.md` and the skill and agent listings a general-purpose subagent carries.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
@@ -62,6 +63,7 @@
| 169 | 0.14.0 | | | **Have rule-hook's `todo-rank-check` catch a `todo.md` that the same Bash call edits and then commits.** | 3 | 1 | 3 | 5 | 3 | 1.1 by 2 | 0.7 |
| 197 | 0.14.0 | | | **Let the human edit what they type at loop-claude's terminal questions with the arrow keys, so Left moves the cursor in place of printing `^[[D`.** | 2 | 2 | 3 | 5 | 2 | 1.5 by 1 | 0.4 |
| 1 | 0.15.0 | | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 1 | 2 | 1 | 2 | 1 | | 0.0 |
| 397 | 0.15.0 | | owner | **Script each step that the chunks of releases 0.10.0 to 0.14.0 repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.3 |
## Details
@@ -73,6 +75,25 @@ The 0.8.0 audit filed four steps, items 334, 335, 336 and 337, and found most ot
The chunk that ships this item files a copy of this audit in release 0.15.0, the release after this item's own, to audit releases 0.10.0 to 0.14.0, rewording "0.9.0" to them, and marks the copy `owner`, so the audit recurs each release until lilleman drops it.
### 396. Print one section of a markdown file, picked by a fragment of its heading, with `markdown-section <file> <fragment>`.
Found by item 338's audit of release 0.9.0. All 11 of its chunks read a section of a markdown file by hand: `grep -n` for the heading, then `sed -n <from>,<to>p` with a guessed range. 115 `sed -n` ranges and 42 `grep -n` calls on README.md, SKILL.md, CHANGELOG.md, `docs/` and `~/.claude/rules/*.md` cost about 10k command tokens and 112k result tokens (characters / 4), 447k characters. `docs/decisions.md` alone took 74 such calls in 10 chunks and 310k result characters; each of its 80 decisions is a `## ` section.
The tool prints from the first heading whose text holds the fragment, case-insensitive, to the next heading of the same or a higher level. Where nothing matches it fails, saying so; where several headings match it lists them. A record kept outside the checkout is the file `project-record open <record>` prints, so the tool takes that path too. It saves the locating call and the misaimed ranges, not the reading of the section; how many of the 157 calls only located a section is not measured. Once it ships, `CLAUDE.md` → "Working loop" → "Read the plan and the decisions only as far as the chunk needs" names it for reading one entry of `docs/decisions.md`, and README → Tools lists it. That list, `README.md:197`, is a place the comprehension seats found hard to navigate; make it easier to read where the entry joins it.
Left with the model: reading the section and judging what it says; and reading code, which item 342 researches.
Scored L 2: a new tool, but a small one in the pattern `todo-item show` and `project-record` already set, a package beside its launcher. I 3, since `CLAUDE.md` puts it in every session.
Touches:
- `bin/markdown-section`
- `bin/markdown_section/__init__.py`
- `bin/markdown_section/main.py`
- `bin/markdown_section/test_markdown_section.py`
- `README.md`: `## Tools` (small)
- `CLAUDE.md`: `## Working loop` (small)
### 299. Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.
The judge's LLM still decides three things: the verdict kind, the escalation it quotes, and whether a release is ready to cut (`docs/decisions.md` → `routing: loop-claude reads the items a chunk takes …`). This item replays the verdict kind. No single rule found so far separates interrupts: a "?" in the worker's final message appears in 48 interrupts and misses 66. A smaller LLM already failed the replay (`docs/decisions.md` → `routing: on a model judge-replay covered, …`: Sonnet returned another kind on 3 interrupts, Haiku missed 7). Untested are combined code rules, a decision table, and a small classifier trained and run on this machine on the archived verdicts (372 finished, 234 interrupt, 42 unfinished, 20 quoted). Run each offline, nothing sent to an outside service (goal 4), and count disagreements by kind; the bar is zero missed interrupts. A hosted decision model is out of scope, since it would send transcripts to an outside service.
@@ -455,3 +476,13 @@ It lives under `~/.claude`, which `.gitignore` keeps from the remote, so a lost
2026-10-02, the human: ai-sandbox is the VM `loop-claude` will run on, hosted on the server bender. bender-infra, the infrastructure-as-code project that manages bender, will back up ai-sandbox's data volume once ai-sandbox is rebuilt. The item waits for the human to say the rebuild has landed; do not remind them. Before writing the README line, check that `~/.claude` lives on that volume and that the backup is private and encrypted at rest.
The README line covers the bench's transcripts in `workflow-bench/logs/` too (goal 3).
### 397. Script each step that the chunks of releases 0.10.0 to 0.14.0 repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.
lilleman asked, on 2026-10-16, for a standing audit each release, so the loop keeps moving work a program does as well from the model to scripts (goals 1.4 and 1.5). This audit reads the archived transcripts of the chunks that shipped releases 0.10.0 to 0.14.0 for every step that chunks repeated by hand, before, during or after review. It keeps the steps whose answer a program computes the same way each time, such as a lookup, a check, a prompt assembled for a reviewer, a cleanup. File one item per such step, with the time and tokens it cost, into the next release with items left. Leave a step that needs judgement with the model. Run `command-patterns <version>` first for each of 0.10.0 to 0.14.0; each run's table is the audit's starting point. `read_threads` and `calls` in `bin/command_patterns/tally.py` open the calls of one pattern, or `command-patterns <version> --calls <file>` dumps them all as JSON lines. The 0.7.0 audit found that most steps its chunks repeated were already scripted by that release's own tools, so a step a shipped item already covers is not filed.
A pattern's call count is a poor guide: most of the 0.9.0 calls were reading and hand edits, which stay with the model. Cluster the calls by their command text with numbers and paths normalised away, then check each cluster against the tools in `~/.claude/bin` before filing.
The 0.8.0 audit filed four steps, items 334, 335, 336 and 337. The 0.9.0 audit filed one, item 396, and found most other repeated steps already covered by tools that shipped inside the release: `chunk-pr merge`'s wait for the checks, `run-tests`, `wait-for`, `larv-review round-title` and `todo-item show`.
The chunk that ships this item files a copy of this audit in the release after this item's own, auditing the releases cut since, and marks the copy `owner`, so the audit recurs each release until lilleman drops it.