Name decision titles in the plan's citations without the heading marker
CI / tests (pull_request) Successful in 4m53s

This commit was merged in pull request #689.
This commit is contained in:
2026-11-22 16:40:22 +01:00
parent 82511b3d0f
commit e687860824
+3 -3
View File
@@ -111,7 +111,7 @@
**Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.**
The All row pools every run, so its median lands in whichever bug's runs sit in the middle of the pooled costs, and its change swings by bug. Item 503 measured the pooled change's spread at up to three times the mean of the bugs' own changes: at 5 bugs × 5 runs, 31% against 12% in output tokens and 50% against 16% in cache-read tokens (`docs/decisions.md` → `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Keep the per-side All rows where that stays cheap. Then restate the help's paragraph "Read a change against the noise" for the one figure the table prints, and update the decision's figures and its plan-items sentence to match.
The All row pools every run, so its median lands in whichever bug's runs sit in the middle of the pooled costs, and its change swings by bug. Item 503 measured the pooled change's spread at up to three times the mean of the bugs' own changes: at 5 bugs × 5 runs, 31% against 12% in output tokens and 50% against 16% in cache-read tokens (`docs/decisions.md` → `task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Keep the per-side All rows where that stays cheap. Then restate the help's paragraph "Read a change against the noise" for the one figure the table prints, and update the decision's figures and its plan-items sentence to match.
Touches:
@@ -124,7 +124,7 @@ Touches:
**Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.**
Item 503 found no affordable count that lifts a chunk-sized change above the tokens' noise (`docs/decisions.md` → `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Reads and turns are counts, not token sums, so they may be steadier. Run `task-bench` with the head against itself (`--base` the head's own commit) on a Go project, 5 bugs × 5 runs after a 2 × 2 pilot. Split each bug's runs at random into two sides, as the decision describes, and record each figure's 95th-percentile spread, for the All change and for the mean of the bugs' changes. Say in the decision whether either count can see a change the tokens can't. It needs a Claude token, given as `task-bench --help` says, and about 4 hours of wall time.
Item 503 found no affordable count that lifts a chunk-sized change above the tokens' noise (`docs/decisions.md` → `task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Reads and turns are counts, not token sums, so they may be steadier. Run `task-bench` with the head against itself (`--base` the head's own commit) on a Go project, 5 bugs × 5 runs after a 2 × 2 pilot. Split each bug's runs at random into two sides, as the decision describes, and record each figure's 95th-percentile spread, for the All change and for the mean of the bugs' changes. Say in the decision whether either count can see a change the tokens can't. It needs a Claude token, given as `task-bench --help` says, and about 4 hours of wall time.
Touches:
@@ -145,7 +145,7 @@ Touches:
**Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.**
Item 503's calibration on `fejkdata` passed 98 of 98 same-commit and chunk runs, so the pass rate can't show a change of any size (`docs/decisions.md` → `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). A research item: try ways to plant harder bugs, such as keeping a mutant only where a pilot of base runs passes fewer than 5 of 5, or planting two mutants in one function. Measure the pass rate's spread again on the same project, splitting runs at random as the decision describes. Its output is a `plant.py` change that keeps the way that works, or a decision that none did, with the figures either way. Run a 2 bugs × 2 runs pilot first, as the decision asks.
Item 503's calibration on `fejkdata` passed 98 of 98 same-commit and chunk runs, so the pass rate can't show a change of any size (`docs/decisions.md` → `task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). A research item: try ways to plant harder bugs, such as keeping a mutant only where a pilot of base runs passes fewer than 5 of 5, or planting two mutants in one function. Measure the pass rate's spread again on the same project, splitting runs at random as the decision describes. Its output is a `plant.py` change that keeps the way that works, or a decision that none did, with the figures either way. Run a 2 bugs × 2 runs pilot first, as the decision asks.
Touches: