Answer product-owner round 1: 0 🔴, 0 🟡 (0 misread); the help calls the noise figures an All change, and the changelog says the row's meaning changed

This commit is contained in:
2026-11-22 17:32:04 +01:00
parent f5a691ff07
commit 4612f71592
+5 -5
View File
@@ -36,8 +36,8 @@ example:
It prints the base and the head, then a table: a row per bug and side with how many fixes passed and the medians, over the runs
that finished, of the input, cache-write, cache-read and output tokens, the turns, the wall time, and the reads and searches before
the run first opened the file the bug is in. The median of the reads leaves out a run that never opened that file. Then come a row
per side over every bug, and the head's change against the base: the pass count's difference, and for each other column the mean over
the bugs of that bug's own change, which isn't the difference of the two rows above it. The last line names the bench's folder: `task-bench` in the
per side over every bug, and the head's change against the base. That last row holds the pass count's difference and, for each other
column, the mean over the bugs of each bug's own change, so it isn't the difference of the two rows above it. The last line names the bench's folder: `task-bench` in the
chunk's review folder.
That folder holds each run's result and stream, and digest.md, which says why runs failed in a few kilobytes per bug:
@@ -47,9 +47,9 @@ That folder holds each run's result and stream, and digest.md, which says why ru
passing run with the median tokens. A trace has a line per tool call, such as `Read pick.go:120-180` or `Grep "readName" → 4 hits`, and the first 3 lines of what Haiku wrote
between calls.
Read a change against the noise. With the defaults, a bench of a commit against itself shows, in 19 cases of 20, a change under
12% in output tokens, 16% in cache-read tokens and 10% in wall time. On `fejkdata`'s bugs every run passed, so the pass rate's noise
is unmeasured.
Read a change against the noise. Measured on `fejkdata`, a Go project, with the defaults, a bench of a commit against itself shows,
in 19 cases of 20, an All change under 12% in output tokens, 16% in cache-read tokens and 10% in wall time. On its bugs every run
passed, so the pass rate's noise is unmeasured.
`docs/decisions.md` → `task-bench: a bench takes` has the figures.
`task-bench gather`, run in the same checkout, turns the digest into plan items. It starts a claude on the project's rung, the