diff --git a/bin/task_bench/main.py b/bin/task_bench/main.py index ae9cd39b..08a40680 100644 --- a/bin/task_bench/main.py +++ b/bin/task_bench/main.py @@ -30,7 +30,7 @@ the copy's top, with no network, so the image must hold what the tests would oth base, which $PWD names, with `gremlins unleash ...` as "$@". Without it, the bench runs the `gremlins` on the host's PATH. For example: - docker run --rm --memory 8g --memory-swap 8g -v "$PWD:$PWD" -w "$PWD" golang:1.27.4 \ + docker run --rm --memory 16g --memory-swap 16g -v "$PWD:$PWD" -w "$PWD" golang:1.27.4 \ sh -c 'shift; exec go run github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0 "$@"' sh It prints the base and the head, then a table: a row per bug and side with how many fixes passed and the medians, over the runs @@ -39,6 +39,11 @@ the run first opened the file the bug is in. The median of the reads leaves out per side over every bug, and the head's change against the base. The last line names the bench's folder: `task-bench` in the chunk's review folder. +Read a change against the noise: with the defaults, two benches of one commit differ by up to 31% in the output tokens of the All +row's change, 50% in the cache-read tokens and 21% in the wall time. The mean of the bugs' own changes differs by up to 12%, 16% and +10%. The pass rate doesn't move: on the bugs `fejkdata` gave, every run passed. `docs/decisions.md` → `task-bench: how many tasks +and runs a bench takes` has the figures. + That folder holds each run's result and stream, and digest.md, which says why runs failed in a few kilobytes per bug: - the table; - per bug its decoys, the files more than half of its failing runs read that the bug isn't in; diff --git a/docs/decisions.md b/docs/decisions.md index f62f941e..1e06c131 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -44,6 +44,7 @@ Each project principle states one rule that the decisions below apply; their ent - routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, steps one rung up as the judge interrupts or quotes an answer, or next cycle, up to two rungs above the rung its item picks - routing: loop-claude reads the items a chunk takes from the worker's latest `Chunk items:` line and scores them by `todo.md` as the default branch held it when the cycle started; the judge reports the escalation and whether a release is ready to cut - routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds +- task-bench: how many tasks and runs a bench takes - tests: each test earns its keep as the technical principles' test bullet says, and the worker takes a test's expected values from its item, never from the code - the repo may name bender, bender-infra and ai-sandbox - todo.md: an item of the next release predicts the files and parts it will touch under `Touches:` in its detail; chunk-preflight takes, in the plan's order, the items that overlap none taken before them as the chunks that may run at once, and the optional `touch-predictions` record keeps each chunk's prediction beside what it touched @@ -818,6 +819,33 @@ Valid while Claude Code loads `~/.claude/rules/*.md` in every project, a path-sc 2026-10-21, lilleman. Serves goal 2: Claude Code rewrites `"model"` in `settings.json` on every `/model` and plugin install, so a tracked model let one machine's pick override every other's. lilleman chose a model per machine over one synced in `env.ANTHROPIC_MODEL`, which would override a `/model` pick from the next session on. A model per machine is simpler, and loop-claude picks each project's model in `~/.claude/loop-claude-models.json` anyway. The filter also sorts the keys, since each write puts them back in Claude Code's order. The cost: a move or pull that changes `settings.json` drops the machine's model. The effort keys, `effortLevel` and `modelSettings`, stay synced: the routing decision on a chunk's worker rung relies on the effort `settings.json` gives every interactive session. Valid while Claude Code writes the model it picks into `settings.json`. +## task-bench: how many tasks and runs a bench takes + +2026-11-22, lilleman, through todo item 503. Serves goal 1.4, since a bench of 50 runs costs hours and tens of millions of tokens, and goal 1.1, since a bench that can't tell a change from noise doesn't prove anything. The bench keeps 5 bugs and 5 runs per side as its defaults, and cannot yet show a chunk-sized change. It sees one that moves a bug's median tokens by more than about 15%. + +Measured on `fejkdata` (Go, 5 bugs from `larv-review places`, seed 1, Haiku at low effort), 2026-11-21: + +- **Same commit at both sides** (head `c7dd7e15`, 5 bugs × 6 runs, after a pilot of 2 bugs × 2 runs): all 68 runs passed, so the pass rate's spread is zero and it can't show a change of any size. The table's last row changed by -5% in output tokens, -12% in cache-read tokens and -15% in wall time. +- **Spread by count.** Each bug's 12 runs were split at random into two sides 4,000 times; the cells are the 95th percentile of the change's size, in percent, for the All row's change as the bench prints it, then for the mean of the bugs' own changes: + + | Bugs × runs | Output tokens | Cache-read tokens | Wall time | + |---|---|---|---| + | 5 × 3 | 41 / 17 | 56 / 22 | 27 / 86 | + | 5 × 4 | 33 / 14 | 48 / 18 | 23 / 20 | + | 5 × 5 | 31 / 12 | 50 / 16 | 21 / 10 | + | 5 × 6 | 28 / 11 | 43 / 13 | 19 / 8.5 | + | 3 × 6 | 28 / 14 | 40 / 17 | 16 / 12 | + + The All row pools every run, so a median lands in whichever bug is dearest and the figure swings by bug: the mean of the bugs' changes has a third of the spread. A bug's cost differs from another's by up to 8×. Bugs and runs trade evenly: 5 × 4 and 3 × 6 are the same spread for the same 36 to 40 runs. +- **One finished chunk**: `fejkdata`'s PR #490, base `33b7fa80`, head `5b9c2a14`, which the comprehension panel ruled better on Navigation, Locality and Shape and the same on Self-sufficiency. The bench planted 5 bugs, and ran 38 of 50 runs before the machine restarted. Its first 3 runs per bug and side, 30 runs, all passed. The mean of the bugs' changes was +4.6% in output tokens, +8.9% in cache-read tokens and +3.8% in turns, with the bugs' own changes in cache-read tokens from -41% to +47%. Those three sit inside the spread at 5 × 3. The mean of the wall times' changes, +105%, lies above its 86%, from one bug whose head runs took 6.8 times as long at the median; the streams weren't read for why. The head reads slightly dearer where the panel reads it better, so the bench neither confirmed nor contradicted the panel. +- **Cost**: 5 bugs × 6 runs took 3.9 hours and 34M cache-read, 1.2M cache-write and 0.3M output tokens; a run takes about 3.5 minutes, and planting 16 to 25 minutes. A 5 × 5 bench is about 3.7 hours and 28M cache-read tokens. Gremlins ran out of 8 GB at the base of PR #490, so the help's example takes 16 GB. + +**Defaults stay 5 and 5.** A sixth run per side cuts the spread by 1.7 points of cache-read tokens for 10 more runs, half an hour. + +**No affordable count rises above the noise of a chunk-sized change.** PR #490 moved the medians by 5% to 9%. The spread at 5 × 6, the largest count measured, is 11% to 13%; it falls with the square root of the runs, so seeing a 9% change takes about 13 runs per side and bug, 130 runs, 9 hours and 70M cache-read tokens. No chunk waits that long. What to try is filed as plan items: print the mean of the bugs' changes, since that is a third of the spread; plant bugs that Haiku fails at the base, so the pass rate moves; and measure whether the reads before the first open of the bug's file, a count and not a token sum, have less noise. + +**Pilot first.** A new setup, or a project the bench hasn't run on, runs 2 bugs × 2 runs before a full bench (lilleman, 2026-11-21): a pilot finds a bench that fails every run, as the first one did on a token that the API refused over a line break, for 40 minutes and a few tokens, not hours. + ## tests: CI runs the suite through `ci.sh` on every pull request, in a pinned Python container, and `main`'s branch protection holds a merge until it passes 2026-11-09, larv-review, through todo item 12. Serves goal 1.1: a merge no longer rests on the worker's word. larv-review's gate was the local run alone, which it skips when the last stability round left the suite green, so an unattended chunk could merge on the worker's own report. gitea.larvit.se's runners labelled `docker-host` run `.gitea/workflows/ci.yml`, which runs `ci.sh` on the committed HEAD in a pinned Python container; the script's header says how. A run takes about 3 minutes. diff --git a/todo.md b/todo.md index 5ade30f5..b80c56e5 100644 --- a/todo.md +++ b/todo.md @@ -6,13 +6,17 @@ `Bar = 0.1` -`Next ID = 512` +`Next ID = 516` ## Items | ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score | |---|---|---|---|---|---|---|---|---|---|---|---| | 503 | 0.15.0 | | owner | **Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3 | 1.2 | +| 513 | 0.15.0 | | | **Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 3 | 1.1 | +| 515 | 0.15.0 | | | **Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 2 | 0.8 | +| 512 | 0.15.0 | | | **Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.** | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.8 | +| 514 | 0.15.0 | | | **Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3 | 0.7 | | 509 | 0.16.0 | 503 | owner | **Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.** | 3 | 2 | 3 | 6 | 5 | code health by 5, 1.1 by 3 | 5.6 | | 504 | 0.16.0 | 503 | owner | **Bench new features in `task-bench`, each a spec for Haiku with acceptance tests it never sees and a reference patch that proves it can be built, kept in a `bench-tasks` folder the project config names, starting with six for fejkdata.** | 3 | 2 | 1 | 4 | 6 | 1.1 by 4, code health by 3 | 2.6 | | 505 | 0.16.0 | 504 | owner | **Bench adding source data in `task-bench`: hand Haiku a fake register that a seeded script generates for a made-up country, and check the importer, the data and the licence entry it adds, starting with five tasks for fejkdata.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 | @@ -112,6 +116,53 @@ Touches: - `docs/decisions.md`: `## task-bench: how many tasks and runs a bench takes` - `bin/task_bench/main.py`: `main` (small) +### 513. Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run. + +**Print the mean of the bugs' own head/base changes as the table's All change, in place of the change of the medians pooled over every run.** + +The All row pools every run, so its median lands in whichever bug is dearest, and its change swings by bug. Item 503 measured the pooled change's spread at up to three times the mean of the bugs' own changes: at 5 bugs × 5 runs, 31% against 12% in output tokens and 50% against 16% in cache-read tokens (`docs/decisions.md` → `## task-bench: how many tasks and runs a bench takes`). Keep the per-side All rows where that stays cheap. Then restate the help's paragraph "Read a change against the noise" for the one figure the table prints, and update the decision's figures and its plan-items sentence to match. + +Touches: + +- `bin/task_bench/figures.py`: `table`, `change` +- `bin/task_bench/test_figures.py`: `TableTest` +- `bin/task_bench/main.py`: `(module)` (small) +- `docs/decisions.md`: `## task-bench: how many tasks and runs a bench takes` (small) + +### 515. Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'. + +**Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.** + +Item 503 found no affordable count that lifts a chunk-sized change above the tokens' noise (`docs/decisions.md` → `## task-bench: how many tasks and runs a bench takes`). Reads and turns are counts, not token sums, so they may be steadier. Run `task-bench` with the head against itself (`--base` the head's own commit) on a Go project, 5 bugs × 5 runs after a 2 × 2 pilot. Split each bug's runs at random into two sides, as the decision describes, and record each figure's 95th-percentile spread, for the All change and for the mean of the bugs' changes. Say in the decision whether either count can see a change the tokens can't. It needs a Claude token, given as `task-bench --help` says, and about 4 hours of wall time. + +Touches: + +- `docs/decisions.md`: `## task-bench: how many tasks and runs a bench takes` + +### 512. Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty. + +**Stop the bench with a Failure on a run whose final `result` event says `terminal_reason: "api_error"`, even where its `api_error_status` is empty.** + +Today `figures()` checks only `api_error_status`. During item 503's calibration a token the API refused over a line break ended runs with `terminal_reason: "api_error"`, a message naming `CLAUDE_CODE_OAUTH_TOKEN`, and no status, so each counted as a finished, not-passed run with 0 tokens, and the bench went on for hours. `task-bench --help` promises that a run the API refused stops the bench. Write the test first: a stream whose result holds that terminal reason and no status gives a Failure that quotes claude's message. + +Touches: + +- `bin/task_bench/figures.py`: `figures` (small) +- `bin/task_bench/test_figures.py`: `FiguresTest` + +### 514. Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision. + +**Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.** + +Item 503's calibration on `fejkdata` passed 98 of 98 same-commit and chunk runs, so the pass rate can't show a change of any size (`docs/decisions.md` → `## task-bench: how many tasks and runs a bench takes`). A research item: try ways to plant harder bugs, such as keeping a mutant only where a pilot of base runs passes fewer than 5 of 5, or planting two mutants in one function. Measure the pass rate's spread again on the same project, splitting runs at random as the decision describes. Its output is a `plant.py` change that keeps the way that works, or a decision that none did, with the figures either way. Run a 2 bugs × 2 runs pilot first, as the decision asks. + +Touches: + +- `bin/task_bench/plant.py`: `drawn`, `mutated` +- `bin/task_bench/test_plant.py` +- `bin/task_bench/main.py`: `benched`, `planted` +- `docs/decisions.md`: `## task-bench: how many tasks and runs a bench takes` + ### 509. Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores. It stays a report beside the panel until about 10 chunks show whether the two agree; whether it can replace the scoring run or gate a merge is the human's call then.