Record the spread of the reads and the turns beside the tokens' in the task-bench decision #693
+12
-2
@@ -825,7 +825,7 @@ Valid while Claude Code loads `~/.claude/rules/*.md` in every project, a path-sc
|
||||
|
||||
**Defaults stay 5 bugs and 5 runs.** A sixth run per side cuts the spread of the mean of the bugs' cache-read changes from 16% to 13% for 10 more runs, about 35 minutes.
|
||||
|
||||
**A chunk-sized change doesn't rise above the noise.** PR #490 moved the medians by 5% to 9%. The spread at 5 × 6, the largest count measured, is 11% to 13%. It falls with the square root of the runs, so seeing a 9% change takes about 13 runs per side and bug: 130 runs, 8 hours and 70M cache-read tokens. No chunk waits that long. The bench shows a change of more than about 15% in tokens, since its All change is the mean of the bugs' own changes (item 514). Before that, the change of the pooled medians needed more than 31% in output tokens and 50% in cache-read tokens. What to try is filed as items 515 (plant bugs Haiku fails at the base, so the pass rate moves) and 516 (measure the spread of the reads and the turns, which are counts, not token sums).
|
||||
**A chunk-sized change doesn't rise above the noise.** PR #490 moved the medians by 5% to 9%. The spread at 5 × 6, the largest count measured, is 11% to 13%. It falls with the square root of the runs, so seeing a 9% change takes about 13 runs per side and bug: 130 runs, 8 hours and 70M cache-read tokens. No chunk waits that long. The bench shows a change of more than about 15% in tokens, since its All change is the mean of the bugs' own changes (item 514). Before that, the change of the pooled medians needed more than 31% in output tokens and 50% in cache-read tokens. To see a chunk-sized change, what to try is filed as item 515 (plant bugs Haiku fails at the base, so the pass rate moves). Reads and turns, which are counts, not token sums, spread as wide as the tokens: see "Reads and turns" below.
|
||||
|
||||
**Pilot first.** A new setup, or a project the bench hasn't run on, runs 2 bugs × 2 runs before a full bench (lilleman, 2026-11-21). A pilot takes about 40 minutes. The first pilot of this calibration passed no run, because the API refused a token that held a line break. It showed that within 15 minutes, where a full bench would have planted for 25 minutes first and printed the same.
|
||||
|
||||
@@ -843,7 +843,17 @@ Measured on `fejkdata` (Go, 5 bugs planted in the files `larv-review places` nam
|
||||
| 3 × 6 | 28 / 14 | 40 / 17 | 16 / 12 |
|
||||
|
||||
The All rows pool every run, so their median lands in whichever bug's runs sit in the middle of the pooled costs, and a bug's cost differs from another's by up to 8×. In tokens, the mean of the bugs' changes has a third to half of the pooled change's spread. In wall time, the mean gains only from 5 × 5. The 86 at 5 × 3 wasn't traced. For the mean of the bugs' token changes, 5 × 4 and 3 × 6 give the same spread (14 and 14, 18 and 17) for 40 and 36 runs. The bench still prints the pooled All rows per side, and the mean as the All change.
|
||||
- **One finished chunk**: `fejkdata`'s PR #490, base `33b7fa80`, head `5b9c2a14`, which the comprehension panel ruled better on Navigation, Locality and Shape and the same on Self-sufficiency. The bench planted 5 bugs and ran 38 of 50 runs before the machine restarted. The first 3 runs per bug and side, 30 runs, are the ones counted, and all passed. The mean of the bugs' changes was +4.6% in output tokens, +8.9% in cache-read tokens and +3.8% in turns, with the bugs' own changes in cache-read tokens from -41% to +47%. The two token figures sit inside the spread at 5 × 3, 17% and 22%; the spread of the turns wasn't measured. The mean of the wall times' changes, +105%, lies above the 86% of that row, from one bug whose head runs took 6.8 times as long at the median. Nobody read its streams to find out why. The head cost slightly more tokens where the panel finds it easier to read, and the wall time can't be judged, so the bench neither confirmed nor contradicted the panel.
|
||||
- **Reads and turns** (2026-11-22) spread as wide as the tokens. `fejkdata` at `095d9bb3` against itself, 5 bugs × 5 runs after a 2 × 2 pilot: all 58 runs passed. Each bug's 10 runs were split at random into two sides 4,000 times. Each cell is the 95th percentile of the change's size. The pooled medians' change is what the All change was before item 514; the mean of the bugs' own changes is what it is now:
|
||||
|
||||
| Figure | Pooled medians' change | Mean of the bugs' changes |
|
||||
|---|---|---|
|
||||
| Output tokens | 25% | 12% |
|
||||
| Cache-read tokens | 41% | 17% |
|
||||
| Turns | 31%, or 3.0 turns | 14%, or 1.8 turns |
|
||||
| Reads and searches before the first open of the bug's file | 4.0 reads | 0.6 reads |
|
||||
|
||||
Neither count sees a change the tokens can't. The bugs' median turns average 14.5 and their median reads 3.9, so 1.8 turns is 12% of a bug's turns and 0.6 reads is 15% of its reads, in line with the tokens' 12% and 17%. The pooled change's spread, 4.0 reads, is more than a bug's typical 3.9 reads. Reads are small whole numbers, from 0 to 9, so the change in reads is given in reads, not as a percentage. The 9 of the 50 runs that never opened the bug's file are left out of the reads' medians. The tokens' own figures moved by up to a fifth against the first calibration's above (25 and 41 against 31 and 50 for the pooled change), so each 95th percentile may be off by about a fifth.
|
||||
- **One finished chunk**: `fejkdata`'s PR #490, base `33b7fa80`, head `5b9c2a14`, which the comprehension panel ruled better on Navigation, Locality and Shape and the same on Self-sufficiency. The bench planted 5 bugs and ran 38 of 50 runs before the machine restarted. The first 3 runs per bug and side, 30 runs, are the ones counted, and all passed. The mean of the bugs' changes was +4.6% in output tokens, +8.9% in cache-read tokens and +3.8% in turns, with the bugs' own changes in cache-read tokens from -41% to +47%. The two token figures sit inside the spread at 5 × 3, 17% and 22%. The turns' +3.8% sits inside their spread at 5 × 5, 14%, and the spread at 5 × 3 can only be wider. The mean of the wall times' changes, +105%, lies above the 86% of that row, from one bug whose head runs took 6.8 times as long at the median. Nobody read its streams to find out why. The head cost slightly more tokens where the panel finds it easier to read, and the wall time can't be judged, so the bench neither confirmed nor contradicted the panel.
|
||||
- **Cost**: 5 bugs × 6 runs took 3.9 hours and 34M cache-read, 1.2M cache-write and 0.3M output tokens; a run takes about 3.5 minutes, and planting 16 to 25 minutes. A 5 × 5 bench is about 3.3 hours and 28M cache-read tokens. Gremlins ran out of 8 GB at the base of PR #490, so the help's example takes 16 GB.
|
||||
|
||||
## tests: CI runs the suite through `ci.sh` on every pull request, in a pinned Python container, and `main`'s branch protection holds a merge until it passes
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 516 | 0.15.1 | | | **Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 2 | 0.8 |
|
||||
| 515 | 0.15.1 | | | **Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3 | 0.7 |
|
||||
| 509 | 0.16.0 | | owner | **Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.** | 3 | 2 | 3 | 6 | 5 | code health by 5, 1.1 by 3 | 5.6 |
|
||||
| 504 | 0.16.0 | | owner | **Bench new features in `task-bench`, each a spec for Haiku with acceptance tests it never sees and a reference patch that proves it can be built, kept in a `bench-tasks` folder the project config names, starting with six for fejkdata.** | 3 | 2 | 1 | 4 | 6 | 1.1 by 4, code health by 3 | 2.6 |
|
||||
@@ -105,16 +104,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 516. Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.
|
||||
|
||||
**Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.**
|
||||
|
||||
Item 503 found no affordable count that lifts a chunk-sized change above the tokens' noise (`docs/decisions.md` → `task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Reads and turns are counts, not token sums, so they may be steadier. Run `task-bench` with the head against itself (`--base` the head's own commit) on a Go project, 5 bugs × 5 runs after a 2 × 2 pilot. Split each bug's runs at random into two sides, as the decision describes, and record each figure's 95th-percentile spread, for the change of the pooled medians, which the bench printed before item 514, and for the mean of the bugs' own changes, which it prints now. Say in the decision whether either count can see a change the tokens can't. It needs a Claude token, given as `task-bench --help` says, and about 4 hours of wall time.
|
||||
|
||||
Touches:
|
||||
|
||||
- `docs/decisions.md`: `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`
|
||||
|
||||
### 515. Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.
|
||||
|
||||
**Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.**
|
||||
|
||||
Reference in New Issue
Block a user