diff --git a/bin/task_bench/main.py b/bin/task_bench/main.py index c6e1c591..21d4c454 100644 --- a/bin/task_bench/main.py +++ b/bin/task_bench/main.py @@ -36,8 +36,8 @@ example: It prints the base and the head, then a table: a row per bug and side with how many fixes passed and the medians, over the runs that finished, of the input, cache-write, cache-read and output tokens, the turns, the wall time, and the reads and searches before the run first opened the file the bug is in. The median of the reads leaves out a run that never opened that file. Then come a row -per side over every bug, and the head's change against the base: the pass count's difference, and for each other column the mean over -the bugs of that bug's own change, which isn't the difference of the two rows above it. The last line names the bench's folder: `task-bench` in the +per side over every bug, and the head's change against the base. That last row holds the pass count's difference and, for each other +column, the mean over the bugs of each bug's own change, so it isn't the difference of the two rows above it. The last line names the bench's folder: `task-bench` in the chunk's review folder. That folder holds each run's result and stream, and digest.md, which says why runs failed in a few kilobytes per bug: @@ -47,9 +47,9 @@ That folder holds each run's result and stream, and digest.md, which says why ru passing run with the median tokens. A trace has a line per tool call, such as `Read pick.go:120-180` or `Grep "readName" → 4 hits`, and the first 3 lines of what Haiku wrote between calls. -Read a change against the noise. With the defaults, a bench of a commit against itself shows, in 19 cases of 20, a change under -12% in output tokens, 16% in cache-read tokens and 10% in wall time. On `fejkdata`'s bugs every run passed, so the pass rate's noise -is unmeasured. +Read a change against the noise. Measured on `fejkdata`, a Go project, with the defaults, a bench of a commit against itself shows, +in 19 cases of 20, an All change under 12% in output tokens, 16% in cache-read tokens and 10% in wall time. On its bugs every run +passed, so the pass rate's noise is unmeasured. `docs/decisions.md` → `task-bench: a bench takes` has the figures. `task-bench gather`, run in the same checkout, turns the digest into plan items. It starts a claude on the project's rung, the