Merge remote-tracking branch 'origin/main' into item-503
CI / tests (pull_request) Has been cancelled

# Conflicts:
#	todo.md
This commit is contained in:
2026-11-22 16:31:57 +01:00
3 changed files with 20 additions and 12 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
# Changelog
## Unreleased
## 0.15.0
- `task-bench` writes `digest.md`, with each bug's decoys and up to two traces of its runs, and `task-bench gather` starts a claude on the project's rung that files a plan item for each place the digest shows to be hard, then removes the bench's folder. Each bench removes the folders earlier benches left, and `compress-archives` leaves lock files alone.
- `task-bench` is new: it plants bugs that a Go project's tests catch, at the places the comprehension seats found hardest, and has Haiku fix each in a container at a base commit and at a head. It prints for each side how often the fix passes, with the median tokens, turns, wall time and reads before Haiku opens the file the bug is in. `task-bench --help` says how to run it.
+1 -1
View File
@@ -825,7 +825,7 @@ Valid while Claude Code loads `~/.claude/rules/*.md` in every project, a path-sc
**Defaults stay 5 bugs and 5 runs.** A sixth run per side cuts the spread of the mean of the bugs' cache-read changes from 16% to 13% for 10 more runs, about 35 minutes.
**A chunk-sized change doesn't rise above the noise.** PR #490 moved the medians by 5% to 9%. The spread at 5 × 6, the largest count measured, is 11% to 13%. It falls with the square root of the runs, so seeing a 9% change takes about 13 runs per side and bug: 130 runs, 8 hours and 70M cache-read tokens. No chunk waits that long. The bench shows a change of more than about 15% in tokens, if you work out the mean of the bugs' own changes from its bug rows. The All row it prints needs more than 31% in output tokens and 50% in cache-read tokens. What to try is filed as items 513 (print the mean of the bugs' changes, which has a third to half of the All row's spread in tokens), 514 (plant bugs Haiku fails at the base, so the pass rate moves) and 515 (measure the spread of the reads and the turns, which are counts, not token sums).
**A chunk-sized change doesn't rise above the noise.** PR #490 moved the medians by 5% to 9%. The spread at 5 × 6, the largest count measured, is 11% to 13%. It falls with the square root of the runs, so seeing a 9% change takes about 13 runs per side and bug: 130 runs, 8 hours and 70M cache-read tokens. No chunk waits that long. The bench shows a change of more than about 15% in tokens, if you work out the mean of the bugs' own changes from its bug rows. The All row it prints needs more than 31% in output tokens and 50% in cache-read tokens. What to try is filed as items 514 (print the mean of the bugs' changes, which has a third to half of the All row's spread in tokens), 515 (plant bugs Haiku fails at the base, so the pass rate moves) and 516 (measure the spread of the reads and the turns, which are counts, not token sums).
**Pilot first.** A new setup, or a project the bench hasn't run on, runs 2 bugs × 2 runs before a full bench (lilleman, 2026-11-21). A pilot takes about 40 minutes. The first pilot of this calibration passed no run, because the API refused a token that held a line break. It showed that within 15 minutes, where a full bench would have planted for 25 minutes first and printed the same.
+18 -10
View File
@@ -6,22 +6,23 @@
`Bar = 0.1`
`Next ID = 516`
`Next ID = 517`
## Items
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 503 | 0.15.0 | | owner | **Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3 | 1.2 |
| 513 | 0.15.0 | | | **Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 3 | 1.1 |
| 515 | 0.15.0 | | | **Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 2 | 0.8 |
| 512 | 0.15.0 | | | **Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.** | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.8 |
| 514 | 0.15.0 | | | **Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3 | 0.7 |
| 503 | 0.15.1 | | owner | **Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3 | 1.2 |
| 514 | 0.15.1 | | | **Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 3 | 1.1 |
| 516 | 0.15.1 | | | **Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 2 | 0.8 |
| 513 | 0.15.1 | | | **Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.** | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.8 |
| 515 | 0.15.1 | | | **Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3 | 0.7 |
| 509 | 0.16.0 | 503 | owner | **Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.** | 3 | 2 | 3 | 6 | 5 | code health by 5, 1.1 by 3 | 5.6 |
| 504 | 0.16.0 | 503 | owner | **Bench new features in `task-bench`, each a spec for Haiku with acceptance tests it never sees and a reference patch that proves it can be built, kept in a `bench-tasks` folder the project config names, starting with six for fejkdata.** | 3 | 2 | 1 | 4 | 6 | 1.1 by 4, code health by 3 | 2.6 |
| 505 | 0.16.0 | 504 | owner | **Bench adding source data in `task-bench`: hand Haiku a fake register that a seeded script generates for a made-up country, and check the importer, the data and the licence entry it adds, starting with five tasks for fejkdata.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 |
| 508 | 0.16.0 | | owner | **Plant bugs for `task-bench` in Python, Rust, TypeScript and PHP, from the mutants `mutate-changes` runs for each, so this repo benches itself.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 |
| 506 | 0.16.0 | 504 | owner | **Go over the bench features in each release review: reapply each reference patch, drop a feature that no longer breaks a goal, and add features until they cover the places the seats find hardest.** | 2 | 2 | 2 | 4 | 3 | 1.1 by 3 | 1.1 |
| 512 | 0.16.0 | | | **Make stability-reviewer check that keys built from path segments, or matched by prefix, cannot collide.** | 1 | 2 | 2 | 3 | 1 | 1.1 by 2 | 0.9 |
| 507 | 0.16.0 | 504 | owner | **Bench rebuilding a feature the project shipped: revert its commit at the head, hand Haiku its item tagline, and check with the tests that commit added.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3 | 0.7 |
| 477 | 0.17.0 | | question | **Rewrite `## Goals` and `## Audience` for a setup others install and run apart from their own `~/.claude`.** | 1 | 1 | 2 | 2 | 2 | | 0.0 |
| 478 | 0.17.0 | | question | **Name the project, its repo and its two commands, today proposed as `do-release` and `do-epic`.** | 1 | 1 | 2 | 2 | 1 | | 0.0 |
@@ -116,7 +117,7 @@ Touches:
- `docs/decisions.md`: `## task-bench: how many tasks and runs a bench takes`
- `bin/task_bench/main.py`: `main` (small)
### 513. Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.
### 514. Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.
**Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.**
@@ -129,7 +130,7 @@ Touches:
- `bin/task_bench/main.py`: `(module)` (small)
- `docs/decisions.md`: `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise` (small)
### 515. Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.
### 516. Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.
**Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.**
@@ -139,7 +140,7 @@ Touches:
- `docs/decisions.md`: `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`
### 512. Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.
### 513. Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.
**Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.**
@@ -150,7 +151,7 @@ Touches:
- `bin/task_bench/figures.py`: `figures` (small)
- `bin/task_bench/test_figures.py`: `FiguresTest`
### 514. Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.
### 515. Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.
**Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.**
@@ -202,6 +203,13 @@ Each tool reports a killed mutant by file and line, and some by column; the plan
A reference patch that no longer applies is rewritten by the review's model. A feature whose goal changed may have become real work, so it goes. New features follow item 504's rule: plausible for another audience, against one of this project's goals.
### 512. Make stability-reviewer check that keys built from path segments, or matched by prefix, cannot collide.
An outside reviewer found two bugs in one chunk that our stability round passed. Both were lookups keyed by a path built from segments or matched by prefix, where two inputs give one key: a linked table that has a column of its own name took the column's row, and two modules whose namespaces nest (`ns` and `ns.sub`) were credited to the first prefix. Add a bullet beside the separator bullet in `agents/stability-reviewer.md` §5: where the change builds a lookup key by joining segments, or matches one by prefix, look for two inputs that give one key, such as a segment equal to its parent's name or a namespace that is a dot-prefix of another, and report the input the lookup credits to the wrong entry.
Touches:
- `agents/stability-reviewer.md`: `### 5. Edge Cases & Input Validation` (small)
### 507. Bench rebuilding a feature the project shipped: revert its commit at the head, hand Haiku its item tagline, and check with the tests that commit added.
Such tasks cost nothing to write and are realistic. A commit whose revert doesn't apply cleanly at both the base and the head is skipped.