Answer product-owner round 1: 0 🔴, 0 🟡 (0 misread); the help calls the noise figures an All change, and the changelog says the row's meaning changed
This commit is contained in:
@@ -36,8 +36,8 @@ example:
|
||||
It prints the base and the head, then a table: a row per bug and side with how many fixes passed and the medians, over the runs
|
||||
that finished, of the input, cache-write, cache-read and output tokens, the turns, the wall time, and the reads and searches before
|
||||
the run first opened the file the bug is in. The median of the reads leaves out a run that never opened that file. Then come a row
|
||||
per side over every bug, and the head's change against the base: the pass count's difference, and for each other column the mean over
|
||||
the bugs of that bug's own change, which isn't the difference of the two rows above it. The last line names the bench's folder: `task-bench` in the
|
||||
per side over every bug, and the head's change against the base. That last row holds the pass count's difference and, for each other
|
||||
column, the mean over the bugs of each bug's own change, so it isn't the difference of the two rows above it. The last line names the bench's folder: `task-bench` in the
|
||||
chunk's review folder.
|
||||
|
||||
That folder holds each run's result and stream, and digest.md, which says why runs failed in a few kilobytes per bug:
|
||||
@@ -47,9 +47,9 @@ That folder holds each run's result and stream, and digest.md, which says why ru
|
||||
passing run with the median tokens. A trace has a line per tool call, such as `Read pick.go:120-180` or `Grep "readName" → 4 hits`, and the first 3 lines of what Haiku wrote
|
||||
between calls.
|
||||
|
||||
Read a change against the noise. With the defaults, a bench of a commit against itself shows, in 19 cases of 20, a change under
|
||||
12% in output tokens, 16% in cache-read tokens and 10% in wall time. On `fejkdata`'s bugs every run passed, so the pass rate's noise
|
||||
is unmeasured.
|
||||
Read a change against the noise. Measured on `fejkdata`, a Go project, with the defaults, a bench of a commit against itself shows,
|
||||
in 19 cases of 20, an All change under 12% in output tokens, 16% in cache-read tokens and 10% in wall time. On its bugs every run
|
||||
passed, so the pass rate's noise is unmeasured.
|
||||
`docs/decisions.md` → `task-bench: a bench takes` has the figures.
|
||||
|
||||
`task-bench gather`, run in the same checkout, turns the digest into plan items. It starts a claude on the project's rung, the
|
||||
|
||||
Reference in New Issue
Block a user