Answer prose round 1: 0 🔴, 10 🟡 (0 misread); rewrite the decision conclusions first with corrected figures, and clarify the help's noise paragraph

This commit is contained in:
2026-11-22 16:24:07 +01:00
parent 5c3ac81ed7
commit 16d9b3a059
3 changed files with 26 additions and 26 deletions
+5 -5
View File
@@ -39,11 +39,6 @@ the run first opened the file the bug is in. The median of the reads leaves out
per side over every bug, and the head's change against the base. The last line names the bench's folder: `task-bench` in the per side over every bug, and the head's change against the base. The last line names the bench's folder: `task-bench` in the
chunk's review folder. chunk's review folder.
Read a change against the noise: with the defaults, two benches of one commit differ by up to 31% in the output tokens of the All
row's change, 50% in the cache-read tokens and 21% in the wall time. The mean of the bugs' own changes differs by up to 12%, 16% and
10%. The pass rate doesn't move: on the bugs `fejkdata` gave, every run passed. `docs/decisions.md` → `task-bench: how many tasks
and runs a bench takes` has the figures.
That folder holds each run's result and stream, and digest.md, which says why runs failed in a few kilobytes per bug: That folder holds each run's result and stream, and digest.md, which says why runs failed in a few kilobytes per bug:
- the table; - the table;
- per bug its decoys, the files more than half of its failing runs read that the bug isn't in; - per bug its decoys, the files more than half of its failing runs read that the bug isn't in;
@@ -51,6 +46,11 @@ That folder holds each run's result and stream, and digest.md, which says why ru
passing run with the median tokens. A trace has a line per tool call, such as `Read pick.go:120-180` or `Grep "readName" → 4 hits`, and the first 3 lines of what Haiku wrote passing run with the median tokens. A trace has a line per tool call, such as `Read pick.go:120-180` or `Grep "readName" → 4 hits`, and the first 3 lines of what Haiku wrote
between calls. between calls.
Read a change against the noise. With the defaults, a bench of a commit against itself shows, in 19 cases of 20, an All row change
under 31% in output tokens, 50% in cache-read tokens and 21% in wall time. The mean of the bugs' own changes, which you work out
from the bug rows, stays under 12%, 16% and 10%. On `fejkdata`'s bugs every run passed, so the pass rate's noise is unmeasured.
`docs/decisions.md` → `task-bench: a bench takes` has the figures.
`task-bench gather`, run in the same checkout, turns the digest into plan items. It starts a claude on the project's rung, the `task-bench gather`, run in the same checkout, turns the digest into plan items. It starts a claude on the project's rung, the
cheapest rung of the model loop-claude runs the project on, with the digest and the prompt in gather-prompt.md beside this file. cheapest rung of the model loop-claude runs the project on, with the digest and the prompt in gather-prompt.md beside this file.
That claude files an item with `todo-item add` for each place the runs show to be hard, and leaves the plan's file uncommitted. That claude files an item with `todo-item add` for each place the runs show to be hard, and leaves the plan's file uncommitted.
+15 -15
View File
@@ -44,7 +44,7 @@ Each project principle states one rule that the decisions below apply; their ent
- routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, steps one rung up as the judge interrupts or quotes an answer, or next cycle, up to two rungs above the rung its item picks - routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, steps one rung up as the judge interrupts or quotes an answer, or next cycle, up to two rungs above the rung its item picks
- routing: loop-claude reads the items a chunk takes from the worker's latest `Chunk items:` line and scores them by `todo.md` as the default branch held it when the cycle started; the judge reports the escalation and whether a release is ready to cut - routing: loop-claude reads the items a chunk takes from the worker's latest `Chunk items:` line and scores them by `todo.md` as the default branch held it when the cycle started; the judge reports the escalation and whether a release is ready to cut
- routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds - routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds
- task-bench: how many tasks and runs a bench takes - task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise
- tests: each test earns its keep as the technical principles' test bullet says, and the worker takes a test's expected values from its item, never from the code - tests: each test earns its keep as the technical principles' test bullet says, and the worker takes a test's expected values from its item, never from the code
- the repo may name bender, bender-infra and ai-sandbox - the repo may name bender, bender-infra and ai-sandbox
- todo.md: an item of the next release predicts the files and parts it will touch under `Touches:` in its detail; chunk-preflight takes, in the plan's order, the items that overlap none taken before them as the chunks that may run at once, and the optional `touch-predictions` record keeps each chunk's prediction beside what it touched - todo.md: an item of the next release predicts the files and parts it will touch under `Touches:` in its detail; chunk-preflight takes, in the plan's order, the items that overlap none taken before them as the chunks that may run at once, and the optional `touch-predictions` record keeps each chunk's prediction beside what it touched
@@ -819,14 +819,20 @@ Valid while Claude Code loads `~/.claude/rules/*.md` in every project, a path-sc
2026-10-21, lilleman. Serves goal 2: Claude Code rewrites `"model"` in `settings.json` on every `/model` and plugin install, so a tracked model let one machine's pick override every other's. lilleman chose a model per machine over one synced in `env.ANTHROPIC_MODEL`, which would override a `/model` pick from the next session on. A model per machine is simpler, and loop-claude picks each project's model in `~/.claude/loop-claude-models.json` anyway. The filter also sorts the keys, since each write puts them back in Claude Code's order. The cost: a move or pull that changes `settings.json` drops the machine's model. The effort keys, `effortLevel` and `modelSettings`, stay synced: the routing decision on a chunk's worker rung relies on the effort `settings.json` gives every interactive session. Valid while Claude Code writes the model it picks into `settings.json`. 2026-10-21, lilleman. Serves goal 2: Claude Code rewrites `"model"` in `settings.json` on every `/model` and plugin install, so a tracked model let one machine's pick override every other's. lilleman chose a model per machine over one synced in `env.ANTHROPIC_MODEL`, which would override a `/model` pick from the next session on. A model per machine is simpler, and loop-claude picks each project's model in `~/.claude/loop-claude-models.json` anyway. The filter also sorts the keys, since each write puts them back in Claude Code's order. The cost: a move or pull that changes `settings.json` drops the machine's model. The effort keys, `effortLevel` and `modelSettings`, stay synced: the routing decision on a chunk's worker rung relies on the effort `settings.json` gives every interactive session. Valid while Claude Code writes the model it picks into `settings.json`.
## task-bench: how many tasks and runs a bench takes ## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise
2026-11-22, lilleman, through todo item 503. Serves goal 1.4, since a bench of 50 runs costs hours and tens of millions of tokens, and goal 1.1, since a bench that can't tell a change from noise doesn't prove anything. The bench keeps 5 bugs and 5 runs per side as its defaults, and cannot yet show a chunk-sized change. It sees one that moves a bug's median tokens by more than about 15%. 2026-11-22, lilleman, through todo item 503. Serves goal 1.4, since a bench of 50 runs costs hours and tens of millions of tokens, and goal 1.1, since a bench that can't tell a change from noise doesn't prove anything.
Measured on `fejkdata` (Go, 5 bugs from `larv-review places`, seed 1, Haiku at low effort), 2026-11-21: **Defaults stay 5 bugs and 5 runs.** A sixth run per side cuts the spread of the mean of the bugs' cache-read changes from 16% to 13% for 10 more runs, about 35 minutes.
- **Same commit at both sides** (head `c7dd7e15`, 5 bugs × 6 runs, after a pilot of 2 bugs × 2 runs): all 68 runs passed, so the pass rate's spread is zero and it can't show a change of any size. The table's last row changed by -5% in output tokens, -12% in cache-read tokens and -15% in wall time. **A chunk-sized change doesn't rise above the noise.** PR #490 moved the medians by 5% to 9%. The spread at 5 × 6, the largest count measured, is 11% to 13%. It falls with the square root of the runs, so seeing a 9% change takes about 13 runs per side and bug: 130 runs, 8 hours and 70M cache-read tokens. No chunk waits that long. The bench shows a change of more than about 15% in tokens, if you work out the mean of the bugs' own changes from its bug rows. The All row it prints needs more than 31% in output tokens and 50% in cache-read tokens. What to try is filed as items 513 (print the mean of the bugs' changes, which has a third to half of the All row's spread in tokens), 514 (plant bugs Haiku fails at the base, so the pass rate moves) and 515 (measure the spread of the reads and the turns, which are counts, not token sums).
- **Spread by count.** Each bug's 12 runs were split at random into two sides 4,000 times; the cells are the 95th percentile of the change's size, in percent, for the All row's change as the bench prints it, then for the mean of the bugs' own changes:
**Pilot first.** A new setup, or a project the bench hasn't run on, runs 2 bugs × 2 runs before a full bench (lilleman, 2026-11-21). A pilot takes about 40 minutes. The first pilot of this calibration passed no run, because the API refused a token that held a line break. It showed that within 15 minutes, where a full bench would have planted for 25 minutes first and printed the same.
Measured on `fejkdata` (Go, 5 bugs planted in the files `larv-review places` names, seed 1, Haiku at low effort), 2026-11-21:
- **Same commit at both sides** (head `c7dd7e15`, 5 bugs × 6 runs, after a pilot of 2 bugs × 2 runs): all 68 runs passed, as did the 30 counted chunk runs below, 98 in all. So the pass rate's spread is zero and it can't show a change of any size, though it is unmeasured on harder bugs. The All row's change was -5% in output tokens, -12% in cache-read tokens and -15% in wall time.
- **Spread by bugs × runs.** Each bug's 12 runs were split at random into two sides 4,000 times. Each cell is the 95th percentile of the change's size, in percent, first for the All row's change as the bench prints it, then for the mean of the bugs' own changes:
| Bugs × runs | Output tokens | Cache-read tokens | Wall time | | Bugs × runs | Output tokens | Cache-read tokens | Wall time |
|---|---|---|---| |---|---|---|---|
@@ -836,15 +842,9 @@ Measured on `fejkdata` (Go, 5 bugs from `larv-review places`, seed 1, Haiku at l
| 5 × 6 | 28 / 11 | 43 / 13 | 19 / 8.5 | | 5 × 6 | 28 / 11 | 43 / 13 | 19 / 8.5 |
| 3 × 6 | 28 / 14 | 40 / 17 | 16 / 12 | | 3 × 6 | 28 / 14 | 40 / 17 | 16 / 12 |
The All row pools every run, so a median lands in whichever bug is dearest and the figure swings by bug: the mean of the bugs' changes has a third of the spread. A bug's cost differs from another's by up to 8×. Bugs and runs trade evenly: 5 × 4 and 3 × 6 are the same spread for the same 36 to 40 runs. The All row pools every run, so its median lands in whichever bug's runs sit in the middle of the pooled costs, and a bug's cost differs from another's by up to 8×. In tokens, the mean of the bugs' changes has a third to half of the All row's spread. In wall time it gains only from 5 × 5. The 86 at 5 × 3 wasn't traced. For the mean of the bugs' token changes, 5 × 4 and 3 × 6 give the same spread (14 and 14, 18 and 17) for 40 and 36 runs.
- **One finished chunk**: `fejkdata`'s PR #490, base `33b7fa80`, head `5b9c2a14`, which the comprehension panel ruled better on Navigation, Locality and Shape and the same on Self-sufficiency. The bench planted 5 bugs, and ran 38 of 50 runs before the machine restarted. Its first 3 runs per bug and side, 30 runs, all passed. The mean of the bugs' changes was +4.6% in output tokens, +8.9% in cache-read tokens and +3.8% in turns, with the bugs' own changes in cache-read tokens from -41% to +47%. Those three sit inside the spread at 5 × 3. The mean of the wall times' changes, +105%, lies above its 86%, from one bug whose head runs took 6.8 times as long at the median; the streams weren't read for why. The head reads slightly dearer where the panel reads it better, so the bench neither confirmed nor contradicted the panel. - **One finished chunk**: `fejkdata`'s PR #490, base `33b7fa80`, head `5b9c2a14`, which the comprehension panel ruled better on Navigation, Locality and Shape and the same on Self-sufficiency. The bench planted 5 bugs and ran 38 of 50 runs before the machine restarted. The first 3 runs per bug and side, 30 runs, are the ones counted, and all passed. The mean of the bugs' changes was +4.6% in output tokens, +8.9% in cache-read tokens and +3.8% in turns, with the bugs' own changes in cache-read tokens from -41% to +47%. The two token figures sit inside the spread at 5 × 3, 17% and 22%; the spread of the turns wasn't measured. The mean of the wall times' changes, +105%, lies above the 86% of that row, from one bug whose head runs took 6.8 times as long at the median. Nobody read its streams to find out why. The head cost slightly more tokens where the panel finds it easier to read, and the wall time can't be judged, so the bench neither confirmed nor contradicted the panel.
- **Cost**: 5 bugs × 6 runs took 3.9 hours and 34M cache-read, 1.2M cache-write and 0.3M output tokens; a run takes about 3.5 minutes, and planting 16 to 25 minutes. A 5 × 5 bench is about 3.7 hours and 28M cache-read tokens. Gremlins ran out of 8 GB at the base of PR #490, so the help's example takes 16 GB. - **Cost**: 5 bugs × 6 runs took 3.9 hours and 34M cache-read, 1.2M cache-write and 0.3M output tokens; a run takes about 3.5 minutes, and planting 16 to 25 minutes. A 5 × 5 bench is about 3.3 hours and 28M cache-read tokens. Gremlins ran out of 8 GB at the base of PR #490, so the help's example takes 16 GB.
**Defaults stay 5 and 5.** A sixth run per side cuts the spread by 1.7 points of cache-read tokens for 10 more runs, half an hour.
**No affordable count rises above the noise of a chunk-sized change.** PR #490 moved the medians by 5% to 9%. The spread at 5 × 6, the largest count measured, is 11% to 13%; it falls with the square root of the runs, so seeing a 9% change takes about 13 runs per side and bug, 130 runs, 9 hours and 70M cache-read tokens. No chunk waits that long. What to try is filed as plan items: print the mean of the bugs' changes, since that is a third of the spread; plant bugs that Haiku fails at the base, so the pass rate moves; and measure whether the reads before the first open of the bug's file, a count and not a token sum, have less noise.
**Pilot first.** A new setup, or a project the bench hasn't run on, runs 2 bugs × 2 runs before a full bench (lilleman, 2026-11-21): a pilot finds a bench that fails every run, as the first one did on a token that the API refused over a line break, for 40 minutes and a few tokens, not hours.
## tests: CI runs the suite through `ci.sh` on every pull request, in a pinned Python container, and `main`'s branch protection holds a merge until it passes ## tests: CI runs the suite through `ci.sh` on every pull request, in a pinned Python container, and `main`'s branch protection holds a merge until it passes
+6 -6
View File
@@ -120,24 +120,24 @@ Touches:
**Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.** **Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.**
The All row pools every run, so its median lands in whichever bug is dearest, and its change swings by bug. Item 503 measured the pooled change's spread at up to three times the mean of the bugs' own changes: at 5 bugs × 5 runs, 31% against 12% in output tokens and 50% against 16% in cache-read tokens (`docs/decisions.md` → `## task-bench: how many tasks and runs a bench takes`). Keep the per-side All rows where that stays cheap. Then restate the help's paragraph "Read a change against the noise" for the one figure the table prints, and update the decision's figures and its plan-items sentence to match. The All row pools every run, so its median lands in whichever bug's runs sit in the middle of the pooled costs, and its change swings by bug. Item 503 measured the pooled change's spread at up to three times the mean of the bugs' own changes: at 5 bugs × 5 runs, 31% against 12% in output tokens and 50% against 16% in cache-read tokens (`docs/decisions.md` → `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Keep the per-side All rows where that stays cheap. Then restate the help's paragraph "Read a change against the noise" for the one figure the table prints, and update the decision's figures and its plan-items sentence to match.
Touches: Touches:
- `bin/task_bench/figures.py`: `table`, `change` - `bin/task_bench/figures.py`: `table`, `change`
- `bin/task_bench/test_figures.py`: `TableTest` - `bin/task_bench/test_figures.py`: `TableTest`
- `bin/task_bench/main.py`: `(module)` (small) - `bin/task_bench/main.py`: `(module)` (small)
- `docs/decisions.md`: `## task-bench: how many tasks and runs a bench takes` (small) - `docs/decisions.md`: `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise` (small)
### 515. Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'. ### 515. Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.
**Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.** **Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.**
Item 503 found no affordable count that lifts a chunk-sized change above the tokens' noise (`docs/decisions.md` → `## task-bench: how many tasks and runs a bench takes`). Reads and turns are counts, not token sums, so they may be steadier. Run `task-bench` with the head against itself (`--base` the head's own commit) on a Go project, 5 bugs × 5 runs after a 2 × 2 pilot. Split each bug's runs at random into two sides, as the decision describes, and record each figure's 95th-percentile spread, for the All change and for the mean of the bugs' changes. Say in the decision whether either count can see a change the tokens can't. It needs a Claude token, given as `task-bench --help` says, and about 4 hours of wall time. Item 503 found no affordable count that lifts a chunk-sized change above the tokens' noise (`docs/decisions.md` → `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Reads and turns are counts, not token sums, so they may be steadier. Run `task-bench` with the head against itself (`--base` the head's own commit) on a Go project, 5 bugs × 5 runs after a 2 × 2 pilot. Split each bug's runs at random into two sides, as the decision describes, and record each figure's 95th-percentile spread, for the All change and for the mean of the bugs' changes. Say in the decision whether either count can see a change the tokens can't. It needs a Claude token, given as `task-bench --help` says, and about 4 hours of wall time.
Touches: Touches:
- `docs/decisions.md`: `## task-bench: how many tasks and runs a bench takes` - `docs/decisions.md`: `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`
### 512. Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty. ### 512. Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.
@@ -154,14 +154,14 @@ Touches:
**Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.** **Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.**
Item 503's calibration on `fejkdata` passed 98 of 98 same-commit and chunk runs, so the pass rate can't show a change of any size (`docs/decisions.md` → `## task-bench: how many tasks and runs a bench takes`). A research item: try ways to plant harder bugs, such as keeping a mutant only where a pilot of base runs passes fewer than 5 of 5, or planting two mutants in one function. Measure the pass rate's spread again on the same project, splitting runs at random as the decision describes. Its output is a `plant.py` change that keeps the way that works, or a decision that none did, with the figures either way. Run a 2 bugs × 2 runs pilot first, as the decision asks. Item 503's calibration on `fejkdata` passed 98 of 98 same-commit and chunk runs, so the pass rate can't show a change of any size (`docs/decisions.md` → `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). A research item: try ways to plant harder bugs, such as keeping a mutant only where a pilot of base runs passes fewer than 5 of 5, or planting two mutants in one function. Measure the pass rate's spread again on the same project, splitting runs at random as the decision describes. Its output is a `plant.py` change that keeps the way that works, or a decision that none did, with the figures either way. Run a 2 bugs × 2 runs pilot first, as the decision asks.
Touches: Touches:
- `bin/task_bench/plant.py`: `drawn`, `mutated` - `bin/task_bench/plant.py`: `drawn`, `mutated`
- `bin/task_bench/test_plant.py` - `bin/task_bench/test_plant.py`
- `bin/task_bench/main.py`: `benched`, `planted` - `bin/task_bench/main.py`: `benched`, `planted`
- `docs/decisions.md`: `## task-bench: how many tasks and runs a bench takes` - `docs/decisions.md`: `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`
### 509. Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores. ### 509. Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.