This commit was merged in pull request #693.
This commit is contained in:
@@ -12,7 +12,6 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 516 | 0.15.1 | | | **Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 2 | 0.8 |
|
||||
| 515 | 0.15.1 | | | **Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3 | 0.7 |
|
||||
| 509 | 0.16.0 | | owner | **Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.** | 3 | 2 | 3 | 6 | 5 | code health by 5, 1.1 by 3 | 5.6 |
|
||||
| 504 | 0.16.0 | | owner | **Bench new features in `task-bench`, each a spec for Haiku with acceptance tests it never sees and a reference patch that proves it can be built, kept in a `bench-tasks` folder the project config names, starting with six for fejkdata.** | 3 | 2 | 1 | 4 | 6 | 1.1 by 4, code health by 3 | 2.6 |
|
||||
@@ -105,16 +104,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 516. Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.
|
||||
|
||||
**Record in a decision the 95th-percentile spread, at 5 bugs × 5 runs, of the reads before the first open of the bug's file and of the turns, beside the tokens'.**
|
||||
|
||||
Item 503 found no affordable count that lifts a chunk-sized change above the tokens' noise (`docs/decisions.md` → `task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`). Reads and turns are counts, not token sums, so they may be steadier. Run `task-bench` with the head against itself (`--base` the head's own commit) on a Go project, 5 bugs × 5 runs after a 2 × 2 pilot. Split each bug's runs at random into two sides, as the decision describes, and record each figure's 95th-percentile spread, for the change of the pooled medians, which the bench printed before item 514, and for the mean of the bugs' own changes, which it prints now. Say in the decision whether either count can see a change the tokens can't. It needs a Claude token, given as `task-bench --help` says, and about 4 hours of wall time.
|
||||
|
||||
Touches:
|
||||
|
||||
- `docs/decisions.md`: `## task-bench: a bench takes 5 bugs and 5 runs per side, after a 2 × 2 pilot on a new setup, and at no affordable count does a chunk-sized change rise above the noise`
|
||||
|
||||
### 515. Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.
|
||||
|
||||
**Plant bugs that Haiku fails at the base, so the pass rate's spread over two benches of one commit is above zero, and record it in a decision.**
|
||||
|
||||
Reference in New Issue
Block a user