Answer prose round 2: 0 🔴, 0 🟡 (0 misread); label the pooled change and the mean once, and say what the filed items are for
CI / tests (pull_request) Has been cancelled
CI / tests (pull_request) Has been cancelled
This commit is contained in:
+2
-2
@@ -825,7 +825,7 @@ Valid while Claude Code loads `~/.claude/rules/*.md` in every project, a path-sc
|
||||
|
||||
**Defaults stay 5 bugs and 5 runs.** A sixth run per side cuts the spread of the mean of the bugs' cache-read changes from 16% to 13% for 10 more runs, about 35 minutes.
|
||||
|
||||
**A chunk-sized change doesn't rise above the noise.** PR #490 moved the medians by 5% to 9%. The spread at 5 × 6, the largest count measured, is 11% to 13%. It falls with the square root of the runs, so seeing a 9% change takes about 13 runs per side and bug: 130 runs, 8 hours and 70M cache-read tokens. No chunk waits that long. The bench shows a change of more than about 15% in tokens, since its All change is the mean of the bugs' own changes (item 514). Before that, the change of the pooled medians needed more than 31% in output tokens and 50% in cache-read tokens. What to try is filed as item 515 (plant bugs Haiku fails at the base, so the pass rate moves). Reads and turns, which are counts, not token sums, spread as wide as the tokens: see "Reads and turns" below.
|
||||
**A chunk-sized change doesn't rise above the noise.** PR #490 moved the medians by 5% to 9%. The spread at 5 × 6, the largest count measured, is 11% to 13%. It falls with the square root of the runs, so seeing a 9% change takes about 13 runs per side and bug: 130 runs, 8 hours and 70M cache-read tokens. No chunk waits that long. The bench shows a change of more than about 15% in tokens, since its All change is the mean of the bugs' own changes (item 514). Before that, the change of the pooled medians needed more than 31% in output tokens and 50% in cache-read tokens. To see a chunk-sized change, what to try is filed as item 515 (plant bugs Haiku fails at the base, so the pass rate moves). Reads and turns, which are counts, not token sums, spread as wide as the tokens: see "Reads and turns" below.
|
||||
|
||||
**Pilot first.** A new setup, or a project the bench hasn't run on, runs 2 bugs × 2 runs before a full bench (lilleman, 2026-11-21). A pilot takes about 40 minutes. The first pilot of this calibration passed no run, because the API refused a token that held a line break. It showed that within 15 minutes, where a full bench would have planted for 25 minutes first and printed the same.
|
||||
|
||||
@@ -843,7 +843,7 @@ Measured on `fejkdata` (Go, 5 bugs planted in the files `larv-review places` nam
|
||||
| 3 × 6 | 28 / 14 | 40 / 17 | 16 / 12 |
|
||||
|
||||
The All rows pool every run, so their median lands in whichever bug's runs sit in the middle of the pooled costs, and a bug's cost differs from another's by up to 8×. In tokens, the mean of the bugs' changes has a third to half of the pooled change's spread. In wall time, the mean gains only from 5 × 5. The 86 at 5 × 3 wasn't traced. For the mean of the bugs' token changes, 5 × 4 and 3 × 6 give the same spread (14 and 14, 18 and 17) for 40 and 36 runs. The bench still prints the pooled All rows per side, and the mean as the All change.
|
||||
- **Reads and turns** (item 516, 2026-11-22) spread as wide as the tokens. `fejkdata` at `095d9bb3` against itself, 5 bugs × 5 runs after a 2 × 2 pilot: all 58 runs passed. Each bug's 10 runs were split at random into two sides 4,000 times. Each cell is the 95th percentile of the change's size, for the change of the pooled medians, which the All change was before item 514, then for the mean of the bugs' own changes, which it is now:
|
||||
- **Reads and turns** (item 516, 2026-11-22) spread as wide as the tokens. `fejkdata` at `095d9bb3` against itself, 5 bugs × 5 runs after a 2 × 2 pilot: all 58 runs passed. Each bug's 10 runs were split at random into two sides 4,000 times. Each cell is the 95th percentile of the change's size. The pooled medians' change is what the All change was before item 514; the mean of the bugs' own changes is what it is now:
|
||||
|
||||
| Figure | Pooled medians' change | Mean of the bugs' changes |
|
||||
|---|---|---|
|
||||
|
||||
Reference in New Issue
Block a user