Calibrate task-bench on fejkdata and record how many tasks and runs a bench takes #689

Merged
lilleman merged 7 commits from item-503 into main 2026-11-22 16:49:17 +01:00
6 changed files with 154 additions and 29 deletions
+4
View File
@@ -1,5 +1,9 @@
# Changelog
## Unreleased
- `task-bench`'s `--help` now says how far a bench of a commit against itself swings, in 19 cases of 20, with the default 5 bugs and 5 runs, and that a Gremlins run on a large project needs 16 GB of memory.
## 0.15.0
- `task-bench` writes `digest.md`, with each bug's decoys and up to two traces of its runs, and `task-bench gather` starts a claude on the project's rung that files a plan item for each place the digest shows to be hard, then removes the bench's folder. Each bench removes the folders earlier benches left, and `compress-archives` leaves lock files alone.
+6 -1
View File
@@ -30,7 +30,7 @@ the copy's top, with no network, so the image must hold what the tests would oth
base, which $PWD names, with `gremlins unleash ...` as "$@". Without it, the bench runs the `gremlins` on the host's PATH. For
example:
docker run --rm --memory 8g --memory-swap 8g -v "$PWD:$PWD" -w "$PWD" golang:1.27.4 \
docker run --rm --memory 16g --memory-swap 16g -v "$PWD:$PWD" -w "$PWD" golang:1.27.4 \
sh -c 'shift; exec go run github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0 "$@"' sh
It prints the base and the head, then a table: a row per bug and side with how many fixes passed and the medians, over the runs
@@ -46,6 +46,11 @@ That folder holds each run's result and stream, and digest.md, which says why ru
passing run with the median tokens. A trace has a line per tool call, such as `Read pick.go:120-180` or `Grep "readName" → 4 hits`, and the first 3 lines of what Haiku wrote
between calls.
Read a change against the noise. With the defaults, a bench of a commit against itself shows, in 19 cases of 20, an All row change
under 31% in output tokens, 50% in cache-read tokens and 21% in wall time. The mean of the bugs' own changes, which you work out
from the bug rows, stays under 12%, 16% and 10%. On `fejkdata`'s bugs every run passed, so the pass rate's noise is unmeasured.
`docs/decisions.md` → `task-bench: a bench takes` has the figures.
`task-bench gather`, run in the same checkout, turns the digest into plan items. It starts a claude on the project's rung, the
cheapest rung of the model loop-claude runs the project on, with the digest and the prompt in gather-prompt.md beside this file.
That claude files an item with `todo-item add` for each place the runs show to be hard, and leaves the plan's file uncommitted.
+35
View File
@@ -4228,3 +4228,38 @@ Inherited architect, decided by:
- Locality: `bin/plan/todo_table.py:110` `Item.awaits_merge`: a field nothing in its own package sets; waiting reads it, so it depends on a caller three packages out having run the marking.
- Locality: `bin/loop_claude/loop/chunk_windows.py:161` `chunks_to_start`: the stop condition is split across two packages, and one half is a string's truthiness.
- Locality: `bin/do_epic/scheduler.py:96` `open_blockers`: four functions gained a defaulted `unblocking`, so a caller that forgets it silently gets the old behaviour.
## 2026-11-22T15:29:07Z, item-503 at 16d9b3a0
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|---|---|---|---|---|---|
| Junior A | 6 | 5 | 5.5 | 5 | 5.5 |
| Mid A | 7 | 5.5 | 6 | 5 | 6 |
| Maintainability senior | 7 | 5.5 | 6 | 6 | 6 |
| Inherited architect | 7 | 6.5 | 6 | 7.5 | 7 |
| Mean | 6.75 | 5.63 | 5.88 | 5.88 | 6.13 |
Junior A, hardest first:
- Navigation, Overall: `bin/agent_cycle/run.py:102` `archived`: cites a close.py that doesn't exist, which cost a search
- Locality, Shape, Overall: `bin/do_epic/main.py:538` `handed_on`: five-way branch over outcome, automation and merged-status config
- Locality: `bin/do_epic/scheduler.py:172` `start_chunks`: chunk state is rebuilt from tmux window names no module owns
- Self-sufficiency: `bin/agent_cycle/run.py:32` `Cycle`: the why of its 17 fields sits in docs/decisions.md
Mid A, hardest first:
- Locality, Overall: `bin/chunk_pr/main.py:587` `levelled_and_finished`: step order is load-bearing and unmarked
- Shape, Overall: `bin/do_epic/main.py:539` `handed_on`: five branches over three configuration facts
- Self-sufficiency: `bin/larv_review/main.py:1` module docstring: the rules for votes and owed runs live only in a 167-line help text
Maintainability senior, hardest first:
- Locality, Overall: `bin/do_epic/main.py:379` `parked_through`: starts other chunks from what reads like a wait
- Locality: `bin/agent_cycle/tmux.py:92` `as_listed`: chunk state shared across processes in window names
- Shape: `bin/plan/ranking.py:239` `score`: the OLD_SHAPE fork doubles the paths through ranking
Inherited architect, would restructure first:
- Navigation, Shape, Overall: `README.md:234` "Packages under bin/": about 85 entries at one level, six packages missing from the list
- Locality: `bin/rule_hook/main.py:94` `breaches_of`: the round cap binds at commit time, not at reviewer launch
- Shape: `bin/chunk_pr/main.py:167` `step`: seven subcommands and inline forge branches in one 764-line module
+28
View File
@@ -44,6 +44,7 @@ Each project principle states one rule that the decisions below apply; their ent
- routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, steps one rung up as the judge interrupts or quotes an answer, or next cycle, up to two rungs above the rung its item picks
- routing: loop-claude reads the items a chunk takes from the worker's latest `Chunk items:` line and scores them by `todo.md` as the default branch held it when the cycle started; the judge reports the escalation and whether a release is ready to cut
- routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds
- task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise
- tests: each test earns its keep as the technical principles' test bullet says, and the worker takes a test's expected values from its item, never from the code
- the repo may name bender, bender-infra and ai-sandbox
- todo.md: an item of the next release predicts the files and parts it will touch under `Touches:` in its detail; chunk-preflight takes, in the plan's order, the items that overlap none taken before them as the chunks that may run at once, and the optional `touch-predictions` record keeps each chunk's prediction beside what it touched
@@ -818,6 +819,33 @@ Valid while Claude Code loads `~/.claude/rules/*.md` in every project, a path-sc
2026-10-21, lilleman. Serves goal 2: Claude Code rewrites `"model"` in `settings.json` on every `/model` and plugin install, so a tracked model let one machine's pick override every other's. lilleman chose a model per machine over one synced in `env.ANTHROPIC_MODEL`, which would override a `/model` pick from the next session on. A model per machine is simpler, and loop-claude picks each project's model in `~/.claude/loop-claude-models.json` anyway. The filter also sorts the keys, since each write puts them back in Claude Code's order. The cost: a move or pull that changes `settings.json` drops the machine's model. The effort keys, `effortLevel` and `modelSettings`, stay synced: the routing decision on a chunk's worker rung relies on the effort `settings.json` gives every interactive session. Valid while Claude Code writes the model it picks into `settings.json`.
## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise
2026-11-22, lilleman, through todo item 503. Serves goal 1.4, since a bench of 50 runs costs hours and tens of millions of tokens, and goal 1.1, since a bench that can't tell a change from noise doesn't prove anything.
**Defaults stay 5 bugs and 5 runs.** A sixth run per side cuts the spread of the mean of the bugs' cache-read changes from 16% to 13% for 10 more runs, about 35 minutes.
**A chunk-sized change doesn't rise above the noise.** PR #490 moved the medians by 5% to 9%. The spread at 5 × 6, the largest count measured, is 11% to 13%. It falls with the square root of the runs, so seeing a 9% change takes about 13 runs per side and bug: 130 runs, 8 hours and 70M cache-read tokens. No chunk waits that long. The bench shows a change of more than about 15% in tokens, if you work out the mean of the bugs' own changes from its bug rows. The All row it prints needs more than 31% in output tokens and 50% in cache-read tokens. What to try is filed as items 514 (print the mean of the bugs' changes, which has a third to half of the All row's spread in tokens), 515 (plant bugs Haiku fails at the base, so the pass rate moves) and 516 (measure the spread of the reads and the turns, which are counts, not token sums).
**Pilot first.** A new setup, or a project the bench hasn't run on, runs 2 bugs × 2 runs before a full bench (lilleman, 2026-11-21). A pilot takes about 40 minutes. The first pilot of this calibration passed no run, because the API refused a token that held a line break. It showed that within 15 minutes, where a full bench would have planted for 25 minutes first and printed the same.
Measured on `fejkdata` (Go, 5 bugs planted in the files `larv-review places` names, seed 1, Haiku at low effort), 2026-11-21:
- **Same commit at both sides** (head `c7dd7e15`, 5 bugs × 6 runs, after a pilot of 2 bugs × 2 runs): all 68 runs passed, as did the 30 counted chunk runs below, 98 in all. So the pass rate's spread is zero and it can't show a change of any size, though it is unmeasured on harder bugs. The All row's change was -5% in output tokens, -12% in cache-read tokens and -15% in wall time.
- **Spread by bugs × runs.** Each bug's 12 runs were split at random into two sides 4,000 times. Each cell is the 95th percentile of the change's size, in percent, first for the All row's change as the bench prints it, then for the mean of the bugs' own changes:
| Bugs × runs | Output tokens | Cache-read tokens | Wall time |
|---|---|---|---|
| 5 × 3 | 41 / 17 | 56 / 22 | 27 / 86 |
| 5 × 4 | 33 / 14 | 48 / 18 | 23 / 20 |
| 5 × 5 | 31 / 12 | 50 / 16 | 21 / 10 |
| 5 × 6 | 28 / 11 | 43 / 13 | 19 / 8.5 |
| 3 × 6 | 28 / 14 | 40 / 17 | 16 / 12 |
The All row pools every run, so its median lands in whichever bug's runs sit in the middle of the pooled costs, and a bug's cost differs from another's by up to 8×. In tokens, the mean of the bugs' changes has a third to half of the All row's spread. In wall time it gains only from 5 × 5. The 86 at 5 × 3 wasn't traced. For the mean of the bugs' token changes, 5 × 4 and 3 × 6 give the same spread (14 and 14, 18 and 17) for 40 and 36 runs.
- **One finished chunk**: `fejkdata`'s PR #490, base `33b7fa80`, head `5b9c2a14`, which the comprehension panel ruled better on Navigation, Locality and Shape and the same on Self-sufficiency. The bench planted 5 bugs and ran 38 of 50 runs before the machine restarted. The first 3 runs per bug and side, 30 runs, are the ones counted, and all passed. The mean of the bugs' changes was +4.6% in output tokens, +8.9% in cache-read tokens and +3.8% in turns, with the bugs' own changes in cache-read tokens from -41% to +47%. The two token figures sit inside the spread at 5 × 3, 17% and 22%; the spread of the turns wasn't measured. The mean of the wall times' changes, +105%, lies above the 86% of that row, from one bug whose head runs took 6.8 times as long at the median. Nobody read its streams to find out why. The head cost slightly more tokens where the panel finds it easier to read, and the wall time can't be judged, so the bench neither confirmed nor contradicted the panel.
- **Cost**: 5 bugs × 6 runs took 3.9 hours and 34M cache-read, 1.2M cache-write and 0.3M output tokens; a run takes about 3.5 minutes, and planting 16 to 25 minutes. A 5 × 5 bench is about 3.3 hours and 28M cache-read tokens. Gremlins ran out of 8 GB at the base of PR #490, so the help's example takes 16 GB.
## tests: CI runs the suite through `ci.sh` on every pull request, in a pinned Python container, and `main`'s branch protection holds a merge until it passes
2026-11-09, larv-review, through todo item 12. Serves goal 1.1: a merge no longer rests on the worker's word. larv-review's gate was the local run alone, which it skips when the last stability round left the suite green, so an unattended chunk could merge on the worker's own report. gitea.larvit.se's runners labelled `docker-host` run `.gitea/workflows/ci.yml`, which runs `ci.sh` on the committed HEAD in a pinned Python container; the script's header says how. A run takes about 3 minutes.
+17
View File
@@ -3367,3 +3367,20 @@ Read most (opens and searches):
- `bin/mutate-changes` 2
- `bin/mutate_changes/main.py` 2
- `bin/mutate_changes/test_mutate_changes_runs.py` 2
## Reading cost, chunk d56c1f21bd44, finished 2026-11-21T12:07:42Z
136856 new input tokens before the first commit.
Read most (opens and searches):
- `bin/task_bench/main.py` 8
- `bin/task_bench/test_task_bench.py` 6
- `.gitignore` 3
- `README.md` 3
- `bin/claude_code/entries.py` 3
- `bin/chunk_driver/project/model_choice.py` 2
- `bin/git_cli/git_repo.py` 2
- `bin/reading_cost/measure.py` 2
- `bin/review_rules/review_folder.py` 2
- `bin/task_bench/figures.py` 2
+64 -28
View File
@@ -6,15 +6,18 @@
`Bar = 0.1`
`Next ID = 513`
`Next ID = 517`
## Items
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 503 | 0.15.1 | | owner | **Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3 | 1.2 |
| 509 | 0.16.0 | 503 | owner | **Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.** | 3 | 2 | 3 | 6 | 5 | code health by 5, 1.1 by 3 | 5.6 |
| 504 | 0.16.0 | 503 | owner | **Bench new features in `task-bench`, each a spec for Haiku with acceptance tests it never sees and a reference patch that proves it can be built, kept in a `bench-tasks` folder the project config names, starting with six for fejkdata.** | 3 | 2 | 1 | 4 | 6 | 1.1 by 4, code health by 3 | 2.6 |
| 514 | 0.15.1 | | | **Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 3 | 1.1 |
| 516 | 0.15.1 | | | **Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 2 | 0.8 |
| 513 | 0.15.1 | | | **Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.** | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.8 |
| 515 | 0.15.1 | | | **Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3 | 0.7 |
| 509 | 0.16.0 | | owner | **Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.** | 3 | 2 | 3 | 6 | 5 | code health by 5, 1.1 by 3 | 5.6 |
| 504 | 0.16.0 | | owner | **Bench new features in `task-bench`, each a spec for Haiku with acceptance tests it never sees and a reference patch that proves it can be built, kept in a `bench-tasks` folder the project config names, starting with six for fejkdata.** | 3 | 2 | 1 | 4 | 6 | 1.1 by 4, code health by 3 | 2.6 |
| 505 | 0.16.0 | 504 | owner | **Bench adding source data in `task-bench`: hand Haiku a fake register that a seeded script generates for a made-up country, and check the importer, the data and the licence entry it adds, starting with five tasks for fejkdata.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 |
| 508 | 0.16.0 | | owner | **Plant bugs for `task-bench` in Python, Rust, TypeScript and PHP, from the mutants `mutate-changes` runs for each, so this repo benches itself.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 |
| 506 | 0.16.0 | 504 | owner | **Go over the bench features in each release review: reapply each reference patch, drop a feature that no longer breaks a goal, and add features until they cover the places the seats find hardest.** | 2 | 2 | 2 | 4 | 3 | 1.1 by 3 | 1.1 |
@@ -52,7 +55,7 @@
| 499 | 0.23.0 | | defect | **Stop `todo-item add` and `edit` from waiting on a stdin that stays open without data, so a call without a heredoc ends at once.** | 2 | 2 | 2 | 4 | 2 | 1.5 by 3, 1.4 by 2 | 1.9 |
| 445 | 0.23.0 | | | **Refuse a plan that holds an item a `Finish item <ID>` commit already removed, so a merge resolution cannot bring a finished item back.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3, 1.1 by 2 | 1.8 |
| 171 | 0.23.0 | | decision | **Group the routing figures' review rounds by the reviewers' rung as well as the worker's, so a call blames a rung only for rounds its own reviews raised.** | 3 | 2 | 2 | 5 | 4 | 1.2 by 3, 1.4 by 3 | 1.8 |
| 300 | 0.23.0 | | owner | **Reach the comprehension floor: restructure the places the comprehension seats found hardest to follow, so the project reads at 7.0 or above on every dimension, Locality and Self-sufficiency first; today the scoring run at a357cc7b scored Navigation 7.00, Locality 6.00, Shape 6.13, Self-sufficiency 6.00 and Overall 6.38.** | 4 | 2 | 3 | 7 | 8 | code health by 8 | 1.7 |
| 300 | 0.23.0 | | owner | **Reach the comprehension floor: restructure the places the comprehension seats found hardest to follow, so the project reads at 7.0 or above on every dimension, Locality first, then Shape and Self-sufficiency; today the scoring run at 16d9b3a0 scored Navigation 6.75, Locality 5.63, Shape 5.88, Self-sufficiency 5.88 and Overall 6.13.** | 4 | 2 | 3 | 7 | 8 | code health by 8 | 1.7 |
| 264 | 0.23.0 | | | **Report the comprehension seats and the escalated subagents as subagent types of their own in `routing-figures.md`, with their dollars.** | 2 | 2 | 1 | 3 | 3 | 1.4 by 5 | 1.7 |
| 323 | 0.23.0 | | | **Count a release review as landed once its pull request has merged, whatever its commits are titled.** | 3 | 2 | 3 | 6 | 3 | 1.4 by 3, 1.5 by 2 | 1.7 |
| 342 | 0.23.0 | | | **Research the tooling agents read, search and run commands with once more, and file an item for each tool that cuts tokens a replay shows keep quality.** | 1 | 2 | 1 | 2 | 3 | 1.4 by 3, 1.5 by 2 | 1.7 |
@@ -104,14 +107,52 @@
## Details
### 503. Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.
### 514. Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.
The decision records the spread of the pass rate and of the median tokens when both sides are the same commit, the task and run counts at which a chunk's change rises above that spread, and what one bench costs in tokens and hours. Where no affordable count rises above the noise, it says so, and files what to try instead. The bench's defaults take the counts it picks.
**Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.**
The All row pools every run, so its median lands in whichever bug's runs sit in the middle of the pooled costs, and its change swings by bug. Item 503 measured the pooled change's spread at up to three times the mean of the bugs' own changes: at 5 bugs × 5 runs, 31% against 12% in output tokens and 50% against 16% in cache-read tokens (`docs/decisions.md` → `task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Keep the per-side All rows where that stays cheap. Then restate the help's paragraph "Read a change against the noise" for the one figure the table prints, and update the decision's figures and its plan-items sentence to match.
Touches:
- `docs/decisions.md`: `## task-bench: how many tasks and runs a bench takes`
- `bin/task_bench/main.py`: `main` (small)
- `bin/task_bench/figures.py`: `table`, `change`
- `bin/task_bench/test_figures.py`: `TableTest`
- `bin/task_bench/main.py`: `(module)` (small)
- `docs/decisions.md`: `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise` (small)
### 516. Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.
**Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.**
Item 503 found no affordable count that lifts a chunk-sized change above the tokens' noise (`docs/decisions.md` → `task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Reads and turns are counts, not token sums, so they may be steadier. Run `task-bench` with the head against itself (`--base` the head's own commit) on a Go project, 5 bugs × 5 runs after a 2 × 2 pilot. Split each bug's runs at random into two sides, as the decision describes, and record each figure's 95th-percentile spread, for the All change and for the mean of the bugs' changes. Say in the decision whether either count can see a change the tokens can't. It needs a Claude token, given as `task-bench --help` says, and about 4 hours of wall time.
Touches:
- `docs/decisions.md`: `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`
### 513. Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.
**Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.**
Today `figures()` checks only `api_error_status`. During item 503's calibration a token the API refused over a line break ended runs with `terminal_reason: "api_error"`, a message naming `CLAUDE_CODE_OAUTH_TOKEN`, and no status, so each counted as a finished, not-passed run with 0 tokens, and the bench went on for hours. `task-bench --help` promises that a run the API refused stops the bench. Write the test first: a stream whose result holds that terminal reason and no status gives a Failure that quotes claude's message.
Touches:
- `bin/task_bench/figures.py`: `figures` (small)
- `bin/task_bench/test_figures.py`: `FiguresTest`
### 515. Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.
**Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.**
Item 503's calibration on `fejkdata` passed 98 of 98 same-commit and chunk runs, so the pass rate can't show a change of any size (`docs/decisions.md` → `task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). A research item: try ways to plant harder bugs, such as keeping a mutant only where a pilot of base runs passes fewer than 5 of 5, or planting two mutants in one function. Measure the pass rate's spread again on the same project, splitting runs at random as the decision describes. Its output is a `plant.py` change that keeps the way that works, or a decision that none did, with the figures either way. Run a 2 bugs × 2 runs pilot first, as the decision asks.
Touches:
- `bin/task_bench/plant.py`: `drawn`, `mutated`
- `bin/task_bench/test_plant.py`
- `bin/task_bench/main.py`: `benched`, `planted`
- `docs/decisions.md`: `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`
### 509. Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.
@@ -325,30 +366,25 @@ Have the gate, or `rank_todo.py --check` where it runs in a checkout, refuse a p
Raised by the goals audit of PR #296. `bin/routing_figures/calls.py` groups a chunk's review rounds by the rung its worker started on, and calls for a routing change when a lower rung took more of them. Since PR #296 the reviewers may run above the worker's rung, on the one their item's E + I picks, and no figure groups rounds by it, though the archived worker line names the E + I it follows from. A cheap worker on an item of high E + I gets stronger reviewers, who may raise more rounds, and a call could then blame the worker's rung for them: an interrupt of lilleman that settles nothing (goal 1.2).
### 300. Reach the comprehension floor: restructure the places the comprehension seats found hardest to follow, so the project reads at 7.0 or above on every dimension, Locality and Self-sufficiency first; today the scoring run at a357cc7b scored Navigation 7.00, Locality 6.00, Shape 6.13, Self-sufficiency 6.00 and Overall 6.38.
### 300. Reach the comprehension floor: restructure the places the comprehension seats found hardest to follow, so the project reads at 7.0 or above on every dimension, Locality first, then Shape and Self-sufficiency; today the scoring run at 16d9b3a0 scored Navigation 6.75, Locality 5.63, Shape 5.88, Self-sufficiency 5.88 and Overall 6.13.
Filed by the comprehension check of PR #473 at 13d1550 (`larv-review record-scoring`), which printed `Floor:` and `Next:`, and refreshed from the run of PR #646 at a357cc7b. The lowest means are Locality and Self-sufficiency, both 6.00. Locality: do-epic's parking and scheduling carry order-dependent state across loop turns, state files and flocks, and no code enforces the order. Self-sufficiency: the why of thresholds and orderings sits behind `docs/decisions.md → …` pointers, 109 of them in 79 modules, and the review rules live in prose that several code sites repeat. Refresh this item's places and aim from the newest run before the chunk starts; do not file a second floor item.
Filed by the comprehension check of PR #473 at 13d1550 (`larv-review record-scoring`), which printed `Floor:` and `Next:`, and refreshed from the run of item 503's chunk at 16d9b3a0. The lowest mean is Locality, at 5.63; Shape and Self-sufficiency follow, both at 5.88. Locality: do-epic and chunk-pr run order-dependent steps and share chunk state through tmux window names (`handed_on`, `parked_through`, `start_chunks`, `levelled_and_finished`), and no code enforces the order. Shape: `bin/do_epic/main.py` and `bin/chunk_pr/main.py` are hubs of about 700 and 764 lines, and the OLD_SHAPE fork in `bin/plan/ranking.py` doubles the paths through ranking. Self-sufficiency: the why behind code sits behind `docs/decisions.md → …` pointers, 82 of them in 64 modules. Refresh this item's places and aim from the newest run before the chunk starts; do not file a second floor item.
Aim: one ambitious refactor of the areas below, sized so every mean reaches 7.0 on the chunk's own scoring run. Where a place is already easier to read when the chunk starts, drop it.
Places the seats named, hardest first (`larv-review places`), Locality and Self-sufficiency first:
Places the seats named, hardest first (`larv-review places`), Locality first:
- Locality, Shape: `bin/do_epic/main.py:287` `parked_through`, with `:354` `resumed_news` and `:241` `chunk_run`: a three-step park and resume order, the `told`, `finished` and `news` flags and a poll that carries five locals across turns. The docstring gives the order, but no code enforces it and no comment states the loop's invariant. Three seats named it hardest; all three would least want to change it.
- Locality, Self-sufficiency: `bin/agent_cycle/routing.py:115` `unfinished_route`, with `:131` `escalated_route`: five rungs, a count, the escalation mark and the cap held at once. The module table resolves the what, but the why of each threshold is only in the decision log. Three seats named it.
- Locality: `bin/do_epic/scheduler.py:144` `start_chunks`: four overlapping sets feed a five-branch chain, counted under nested locks shared with loop-claude. Two seats named it.
- Locality, Shape: `bin/rule_hook/round_cap.py:10` `round_cap_breaches`, with `bin/review_rules/round_titles.py:33` `round_counts`, `bin/larv_review/reviewer_header.py:75` `unanswered_round` and `skills/larv-review/SKILL.md:89` "Round cap": the cap lives in four places across three packages, none owns the whole invariant, and the rules exist a third time in the help at `bin/larv_review/main.py:1`. Three seats named it.
- Shape, Self-sufficiency: `bin/project_config/settings.py:187` `load`: about twenty comprehension clauses in one `faults` list encode the rules that link settings; named predicates would make each findable. Two seats named it.
- Locality, Shape: `bin/chunk_pr/main.py:314` `merged`, with `:447` `levelled_and_finished`: a queue loop whose variables carry a string or a Failure, and a scoring-run check the name doesn't suggest. Two seats named it.
- Locality: `bin/plan/ranking.py:239` `score`, with `:144` `weigh`: the old and new table shapes run through the whole module. Two seats named it.
- Locality: `bin/agent_cycle/judge.py:76` `TASK_HEADING`, with `bin/model_routing/route.py:47` and `bin/loop_claude/loop/reload.py:36` `decode_handed`: archive and handover formats that a reworded heading or a renamed field breaks, guarded only by comments. Three seats named one of them.
- Shape: `bin/agent_cycle/judge.py:237` `quotes_checked`, with `bin/agent_cycle/run.py:150` `rule`: checks that quietly turn one verdict type into another, and one function that runs the judge, steps up and orders the archive writes.
- Locality: `bin/larv_review/plan.py:21` `Emptied`: a `str` subclass whose type carries meaning, lost by any code that rebuilds the string.
- Shape: `bin/do_epic/gates.py:42` `gate`: a long ordered chain where a missing value means something different on each line.
- Shape: `bin/agent_cycle/stuck_workers.py:52` `Watch.check`: the condition at line 59 mixes `or` and `and` without parentheses over state mutated out of sight.
- Self-sufficiency: `bin/agent_cycle/verdicts.py:41` and `bin/agent_cycle/run.py:103`: comments that cite `cycle.judge.re_ask_checked` and `close.py`, neither of which exists; `bin/agent_cycle/judge.py:116` `Rejected` names loop-claude in a module do-epic shares.
- Navigation, Shape: `README.md:225` "Packages under bin/": 84 entries in alphabetical prose with no grouping by layer, two bullets run together at `:239`, and 40 packages whose `__init__.py` is empty.
- Shape: `bin/loop_claude/loop/reload.py:18` `LoopRun`: loop-claude's core run state in a module named for self-reloading; `Run`, `ChunkRun` and `LoopRun` are one concept in three types.
- Locality: `reviewer-merge-trials/driver.py:4`: a frozen trial imports live code under `bin/`, so a refactor there breaks a record.
- Locality, Shape: `bin/do_epic/main.py:538` `handed_on`: a five-way branch over the outcome, the automation and the merged-status config. Two seats named it.
- Locality: `bin/do_epic/main.py:379` `parked_through`: starts other chunks from what reads like a wait.
- Locality: `bin/chunk_pr/main.py:587` `levelled_and_finished`: its step order is load-bearing and nothing marks it.
- Locality: `bin/do_epic/scheduler.py:172` `start_chunks`, with `bin/agent_cycle/tmux.py:92` `as_listed`: chunk state is rebuilt from tmux window names, shared across processes, which no module owns. Two seats named it.
- Locality: `bin/rule_hook/main.py:94` `breaches_of`: the round cap binds at commit time, not when a reviewer is launched.
- Navigation, Shape: `README.md:234` "Packages under bin/": about 85 entries at one level, and six packages missing from the list.
- Shape: `bin/chunk_pr/main.py:167` `step`: seven subcommands and inline forge branches in one 764-line module.
- Shape: `bin/plan/ranking.py:239` `score`: the OLD_SHAPE fork doubles the paths through ranking.
- Self-sufficiency: `bin/agent_cycle/run.py:32` `Cycle`: the why of its 17 fields sits in `docs/decisions.md`.
- Self-sufficiency: `bin/larv_review/main.py:1` module docstring: the rules for votes and owed runs live only in a 167-line help text.
- Navigation: `bin/agent_cycle/run.py:102` `archived`: cites a `close.py` that doesn't exist, which cost a seat a search.
### 264. Report the comprehension seats and the escalated subagents as subagent types of their own in `routing-figures.md`, with their dollars.