Finish item 514 in the plan
CI / tests (pull_request) Successful in 5m24s

This commit was merged in pull request #692.
This commit is contained in:
2026-11-22 17:49:53 +01:00
parent a95dd360c8
commit ab08c3c238
2 changed files with 1 additions and 14 deletions
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- `task-bench`'s table prints the mean of the bugs' own head/base changes as its All change, in place of the change of the two All rows' medians, which swung up to three times as far in tokens. The All rows per side stay, so the change row is no longer their difference, and a change from before 0.15.1 can't be compared with one from after it.
- `task-bench` now stops with a Failure on a run whose final result says `terminal_reason: "api_error"`, even where it holds no `api_error_status`, so a token the API refuses no longer lets the bench run on for hours with empty runs.
- `task-bench`'s `--help` now says how far a bench of a commit against itself swings, in 19 cases of 20, with the default 5 bugs and 5 runs, and that a Gremlins run on a large project needs 16 GB of memory.
-14
View File
@@ -12,7 +12,6 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 514 | 0.15.1 | | | **Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 3 | 1.1 |
| 516 | 0.15.1 | | | **Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 2 | 0.8 |
| 515 | 0.15.1 | | | **Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3 | 0.7 |
| 509 | 0.16.0 | | owner | **Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.** | 3 | 2 | 3 | 6 | 5 | code health by 5, 1.1 by 3 | 5.6 |
@@ -106,19 +105,6 @@
## Details
### 514. Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.
**Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.**
The All row pools every run, so its median lands in whichever bug's runs sit in the middle of the pooled costs, and its change swings by bug. Item 503 measured the pooled change's spread at up to three times the mean of the bugs' own changes: at 5 bugs × 5 runs, 31% against 12% in output tokens and 50% against 16% in cache-read tokens (`docs/decisions.md` → `task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Keep the per-side All rows where that stays cheap. Then restate the help's paragraph "Read a change against the noise" for the one figure the table prints, and update the decision's figures and its plan-items sentence to match.
Touches:
- `bin/task_bench/figures.py`: `table`, `change`
- `bin/task_bench/test_figures.py`: `TableTest`
- `bin/task_bench/main.py`: `(module)` (small)
- `docs/decisions.md`: `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise` (small)
### 516. Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.
**Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.**