diff --git a/CHANGELOG.md b/CHANGELOG.md index 90d16d4f..abd905a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## Unreleased +- `task-bench` now stops with a Failure on a run whose final result says `terminal_reason: "api_error"`, even where it holds no `api_error_status`, so a token the API refuses no longer lets the bench run on for hours with empty runs. - `task-bench`'s `--help` now says how far a bench of a commit against itself swings, in 19 cases of 20, with the default 5 bugs and 5 runs, and that a Gremlins run on a large project needs 16 GB of memory. ## 0.15.0 diff --git a/todo.md b/todo.md index f1d78ffe..1228a6f6 100644 --- a/todo.md +++ b/todo.md @@ -14,7 +14,6 @@ |---|---|---|---|---|---|---|---|---|---|---|---| | 514 | 0.15.1 | | | **Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 3 | 1.1 | | 516 | 0.15.1 | | | **Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 2 | 0.8 | -| 513 | 0.15.1 | | | **Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.** | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.8 | | 515 | 0.15.1 | | | **Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3 | 0.7 | | 509 | 0.16.0 | | owner | **Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.** | 3 | 2 | 3 | 6 | 5 | code health by 5, 1.1 by 3 | 5.6 | | 504 | 0.16.0 | | owner | **Bench new features in `task-bench`, each a spec for Haiku with acceptance tests it never sees and a reference patch that proves it can be built, kept in a `bench-tasks` folder the project config names, starting with six for fejkdata.** | 3 | 2 | 1 | 4 | 6 | 1.1 by 4, code health by 3 | 2.6 | @@ -130,17 +129,6 @@ Touches: - `docs/decisions.md`: `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise` -### 513. Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty. - -**Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.** - -Today `figures()` checks only `api_error_status`. During item 503's calibration a token the API refused over a line break ended runs with `terminal_reason: "api_error"`, a message naming `CLAUDE_CODE_OAUTH_TOKEN`, and no status, so each counted as a finished, not-passed run with 0 tokens, and the bench went on for hours. `task-bench --help` promises that a run the API refused stops the bench. Write the test first: a stream whose result holds that terminal reason and no status gives a Failure that quotes claude's message. - -Touches: - -- `bin/task_bench/figures.py`: `figures` (small) -- `bin/task_bench/test_figures.py`: `FiguresTest` - ### 515. Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision. **Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.**