Merge remote-tracking branch 'origin/main' into item-503
CI / tests (pull_request) Has been cancelled
CI / tests (pull_request) Has been cancelled
# Conflicts: # todo.md
This commit is contained in:
+1
-1
@@ -825,7 +825,7 @@ Valid while Claude Code loads `~/.claude/rules/*.md` in every project, a path-sc
|
||||
|
||||
**Defaults stay 5 bugs and 5 runs.** A sixth run per side cuts the spread of the mean of the bugs' cache-read changes from 16% to 13% for 10 more runs, about 35 minutes.
|
||||
|
||||
**A chunk-sized change doesn't rise above the noise.** PR #490 moved the medians by 5% to 9%. The spread at 5 × 6, the largest count measured, is 11% to 13%. It falls with the square root of the runs, so seeing a 9% change takes about 13 runs per side and bug: 130 runs, 8 hours and 70M cache-read tokens. No chunk waits that long. The bench shows a change of more than about 15% in tokens, if you work out the mean of the bugs' own changes from its bug rows. The All row it prints needs more than 31% in output tokens and 50% in cache-read tokens. What to try is filed as items 513 (print the mean of the bugs' changes, which has a third to half of the All row's spread in tokens), 514 (plant bugs Haiku fails at the base, so the pass rate moves) and 515 (measure the spread of the reads and the turns, which are counts, not token sums).
|
||||
**A chunk-sized change doesn't rise above the noise.** PR #490 moved the medians by 5% to 9%. The spread at 5 × 6, the largest count measured, is 11% to 13%. It falls with the square root of the runs, so seeing a 9% change takes about 13 runs per side and bug: 130 runs, 8 hours and 70M cache-read tokens. No chunk waits that long. The bench shows a change of more than about 15% in tokens, if you work out the mean of the bugs' own changes from its bug rows. The All row it prints needs more than 31% in output tokens and 50% in cache-read tokens. What to try is filed as items 514 (print the mean of the bugs' changes, which has a third to half of the All row's spread in tokens), 515 (plant bugs Haiku fails at the base, so the pass rate moves) and 516 (measure the spread of the reads and the turns, which are counts, not token sums).
|
||||
|
||||
**Pilot first.** A new setup, or a project the bench hasn't run on, runs 2 bugs × 2 runs before a full bench (lilleman, 2026-11-21). A pilot takes about 40 minutes. The first pilot of this calibration passed no run, because the API refused a token that held a line break. It showed that within 15 minutes, where a full bench would have planted for 25 minutes first and printed the same.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user