This commit was merged in pull request #684.
This commit is contained in:
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- `task-bench` writes `digest.md`, with each bug's decoys and up to two traces of its runs, and `task-bench gather` starts a claude on the project's rung that files a plan item for each place the digest shows to be hard, then removes the bench's folder. Each bench removes the folders earlier benches left, and `compress-archives` leaves lock files alone.
|
||||
- `task-bench` is new: it plants bugs that a Go project's tests catch, at the places the comprehension seats found hardest, and has Haiku fix each in a container at a base commit and at a head. It prints for each side how often the fix passes, with the median tokens, turns, wall time and reads before Haiku opens the file the bug is in. `task-bench --help` says how to run it.
|
||||
- larv-review now runs each command that a changed doc line says prints something and hands the real output to the product-owner and prose reviewers, so a claim about output is checked by running it.
|
||||
- `do-epic` moves a parent issue whose sub-tasks a person filed: to done by `jira-finish-transition` once every sub-task is done, and before that by the transition the new setting `jira-sub-tasks-landed-transition` names, once each sub-task is done or in one of `jira-unblocking-statuses`. The transition must lead to one of those statuses. `--dry-run` prints each such move.
|
||||
|
||||
@@ -12,8 +12,7 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 502 | 0.15.0 | | owner | **Distil each `task-bench` run into a digest of short traces and decoys, the files most failing runs read but no fix touches, and have a stronger model turn it into plan items before the transcripts are deleted.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 4, code health by 4 | 4.8 |
|
||||
| 503 | 0.15.0 | 502 | owner | **Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3 | 1.2 |
|
||||
| 503 | 0.15.0 | | owner | **Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3 | 1.2 |
|
||||
| 509 | 0.16.0 | 503 | owner | **Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.** | 3 | 2 | 3 | 6 | 5 | code health by 5, 1.1 by 3 | 5.6 |
|
||||
| 504 | 0.16.0 | 503 | owner | **Bench new features in `task-bench`, each a spec for Haiku with acceptance tests it never sees and a reference patch that proves it can be built, kept in a `bench-tasks` folder the project config names, starting with six for fejkdata.** | 3 | 2 | 1 | 4 | 6 | 1.1 by 4, code health by 3 | 2.6 |
|
||||
| 505 | 0.16.0 | 504 | owner | **Bench adding source data in `task-bench`: hand Haiku a fake register that a seeded script generates for a made-up country, and check the importer, the data and the licence entry it adds, starting with five tasks for fejkdata.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 |
|
||||
@@ -104,24 +103,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 502. Distil each `task-bench` run into a digest of short traces and decoys, the files most failing runs read but no fix touches, and have a stronger model turn it into plan items before the transcripts are deleted.
|
||||
|
||||
The thought process of the runs shows why Haiku failed, but whole transcripts would flood the model that reads them. A program distils them; only the last step uses a model.
|
||||
|
||||
- **Trace.** One line per tool call, such as `Read pick.go:120-180` or `Grep "readName" → 4 hits`, with the model's text between calls cut to 3 lines.
|
||||
- **Decoys.** Per task, the files more than half its failing runs read that the fix doesn't change. They name the code that misled the runs.
|
||||
- **Digest.** The figures, the decoys, and per task two traces picked by rule: the failing run that spent the most tokens and the passing run with the median tokens. A few kilobytes per task. The raw transcripts stay beside it, for the model to open one by its run id where the digest leaves a question open.
|
||||
- **Gather.** `task-bench gather` starts a claude on the project's rung with the digest and a prompt kept beside the code. It files a plan item with `todo-item add` for each place to make easier, citing the decoys and traces that show it.
|
||||
- **Delete.** Once gather ends, the bench run's folder goes, transcripts and traces with it. Each bench run also deletes the folders earlier runs left, so a session that crashed doesn't leave data growing. The figures alone outlive the run, in what item 509 records.
|
||||
|
||||
Touches:
|
||||
|
||||
- `bin/task_bench/digest.py`
|
||||
- `bin/task_bench/gather.py`
|
||||
- `bin/task_bench/gather-prompt.md`
|
||||
- `bin/task_bench/main.py`: `main` (small)
|
||||
- `bin/task_bench/test_task_bench.py`
|
||||
|
||||
### 503. Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.
|
||||
|
||||
The decision records the spread of the pass rate and of the median tokens when both sides are the same commit, the task and run counts at which a chunk's change rises above that spread, and what one bench costs in tokens and hours. Where no affordable count rises above the noise, it says so, and files what to try instead. The bench's defaults take the counts it picks.
|
||||
|
||||
Reference in New Issue
Block a user