From e687860824fdd7c657ca4bb02257834859d2afe0 Mon Sep 17 00:00:00 2001 From: Lilleman auf Larv Date: Sun, 22 Nov 2026 16:40:22 +0100 Subject: [PATCH] Name decision titles in the plan's citations without the heading marker --- todo.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/todo.md b/todo.md index e706a844..f1d78ffe 100644 --- a/todo.md +++ b/todo.md @@ -111,7 +111,7 @@ **Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.** -The All row pools every run, so its median lands in whichever bug's runs sit in the middle of the pooled costs, and its change swings by bug. Item 503 measured the pooled change's spread at up to three times the mean of the bugs' own changes: at 5 bugs × 5 runs, 31% against 12% in output tokens and 50% against 16% in cache-read tokens (`docs/decisions.md` → `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Keep the per-side All rows where that stays cheap. Then restate the help's paragraph "Read a change against the noise" for the one figure the table prints, and update the decision's figures and its plan-items sentence to match. +The All row pools every run, so its median lands in whichever bug's runs sit in the middle of the pooled costs, and its change swings by bug. Item 503 measured the pooled change's spread at up to three times the mean of the bugs' own changes: at 5 bugs × 5 runs, 31% against 12% in output tokens and 50% against 16% in cache-read tokens (`docs/decisions.md` → `task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Keep the per-side All rows where that stays cheap. Then restate the help's paragraph "Read a change against the noise" for the one figure the table prints, and update the decision's figures and its plan-items sentence to match. Touches: @@ -124,7 +124,7 @@ Touches: **Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.** -Item 503 found no affordable count that lifts a chunk-sized change above the tokens' noise (`docs/decisions.md` → `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Reads and turns are counts, not token sums, so they may be steadier. Run `task-bench` with the head against itself (`--base` the head's own commit) on a Go project, 5 bugs × 5 runs after a 2 × 2 pilot. Split each bug's runs at random into two sides, as the decision describes, and record each figure's 95th-percentile spread, for the All change and for the mean of the bugs' changes. Say in the decision whether either count can see a change the tokens can't. It needs a Claude token, given as `task-bench --help` says, and about 4 hours of wall time. +Item 503 found no affordable count that lifts a chunk-sized change above the tokens' noise (`docs/decisions.md` → `task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Reads and turns are counts, not token sums, so they may be steadier. Run `task-bench` with the head against itself (`--base` the head's own commit) on a Go project, 5 bugs × 5 runs after a 2 × 2 pilot. Split each bug's runs at random into two sides, as the decision describes, and record each figure's 95th-percentile spread, for the All change and for the mean of the bugs' changes. Say in the decision whether either count can see a change the tokens can't. It needs a Claude token, given as `task-bench --help` says, and about 4 hours of wall time. Touches: @@ -145,7 +145,7 @@ Touches: **Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.** -Item 503's calibration on `fejkdata` passed 98 of 98 same-commit and chunk runs, so the pass rate can't show a change of any size (`docs/decisions.md` → `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). A research item: try ways to plant harder bugs, such as keeping a mutant only where a pilot of base runs passes fewer than 5 of 5, or planting two mutants in one function. Measure the pass rate's spread again on the same project, splitting runs at random as the decision describes. Its output is a `plant.py` change that keeps the way that works, or a decision that none did, with the figures either way. Run a 2 bugs × 2 runs pilot first, as the decision asks. +Item 503's calibration on `fejkdata` passed 98 of 98 same-commit and chunk runs, so the pass rate can't show a change of any size (`docs/decisions.md` → `task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). A research item: try ways to plant harder bugs, such as keeping a mutant only where a pilot of base runs passes fewer than 5 of 5, or planting two mutants in one function. Measure the pass rate's spread again on the same project, splitting runs at random as the decision describes. Its output is a `plant.py` change that keeps the way that works, or a decision that none did, with the figures either way. Run a 2 bugs × 2 runs pilot first, as the decision asks. Touches: