Run task-bench's Haiku in a container with every tool, in place of a Bash allowlist
CI / tests (pull_request) Successful in 3m46s
CI / tests (pull_request) Successful in 3m46s
This commit is contained in:
@@ -13,7 +13,7 @@
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 500 | 0.15.0 | | owner | **Mutate the Go lines a branch changes with Gremlins in `mutate-changes`, as it mutates Python, Rust, TypeScript and PHP, and print each mutant the tests do not kill.** | 2 | 2 | 2 | 4 | 4 | 1.1 by 4 | 1.2 |
|
||||
| 501 | 0.15.0 | 500 | owner | **Ship `task-bench`, which plants a bug the tests catch at the places the comprehension seats found hardest, has Haiku fix it in a vanilla Claude Code at a base and a head, and prints for each side how often the fix passes, with its median tokens, turns, wall time and reads before it opens a file the bug is in.** | 4 | 2 | 1 | 5 | 8 | 1.1 by 5, code health by 4 | 1.1 |
|
||||
| 501 | 0.15.0 | 500 | owner | **Ship `task-bench`, which plants a bug the tests catch at the places the comprehension seats found hardest, has Haiku fix it in a vanilla Claude Code at a base and a head, and prints for each side how often the fix passes, with its median tokens, turns, wall time and reads before it opens a file the bug is in.** | 4 | 2 | 1 | 5 | 9 | 1.1 by 5, code health by 4 | 0.6 |
|
||||
| 502 | 0.15.0 | 501 | owner | **Distil each `task-bench` run into a digest of short traces and decoys, the files most failing runs read but no fix touches, and have a stronger model turn it into plan items before the transcripts are deleted.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 4, code health by 4 | 4.8 |
|
||||
| 469 | 0.15.0 | | | **Move a do-epic parent issue whose sub-tasks a person filed to Release Ready once every sub-task has landed, and to Done once every sub-task is done, by transitions its config names.** | 3 | 2 | 2 | 5 | 4 | 1.2 by 4, 1.5 by 4, 1.6 by 3 | 3.1 |
|
||||
| 503 | 0.15.0 | 502 | owner | **Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3 | 1.2 |
|
||||
@@ -127,13 +127,13 @@ The panel's scores barely move between chunks, and `reading-cost` measures chunk
|
||||
|
||||
- **Plant.** Run Gremlins, as item 500 runs it, without `--diff`, over the files that hold the places `larv-review places` prints. Draw 5 of its `KILLED` mutants by a seeded pick, and re-apply each at its file, line and column. The task's text is the failing assertion's message, without the test's name, file or line, so it says what goes wrong and not where. Go is the first language; item 508 adds the others.
|
||||
- **Copy.** The tree at the commit, without `.git`, committed once with the bug in it, so `git diff` and `git log` show Haiku nothing.
|
||||
- **Run.** `claude -p --bare --strict-mcp-config --disable-slash-commands --no-session-persistence --model claude-haiku-5-5 --effort low --output-format stream-json`, with Read, Grep, Glob, Edit, Write, and Bash only for the project's test command, through `--allowedTools`. `--bare` drops the human's instructions, memories, hooks, skills and plugins, which would skew the figures and change with claude-config. The repo's README, `AGENTS.md` and `docs/` stay, since they are part of what the bench measures; a project's `CLAUDE.md`, which `--bare` skips, is passed in, so every project is measured alike.
|
||||
- **Run.** Inside a container that holds only the copy: the project's development image with Claude Code added, such as `golang` for fejkdata, signed in by a long-lived token in an environment variable, with a memory limit and the network only for the API. In it, `claude -p --bare --strict-mcp-config --disable-slash-commands --no-session-persistence --model claude-haiku-5-5 --effort low --output-format stream-json` runs with every tool, Bash included, so Haiku greps, builds, runs the CLI and runs single tests as an agent normally does. A narrower tool list would raise the figures for reasons outside the code, and an allowlist can't fence Bash anyway: a rule that allows `cat` allows it on any path, such as the real checkout, whose diff would give away the bug. `--bare` drops the human's instructions, memories, hooks, skills and plugins, which would skew the figures and change with claude-config. The repo's README, `AGENTS.md` and `docs/` stay, since they are part of what the bench measures; a project's `CLAUDE.md`, which `--bare` skips, is passed in, so every project is measured alike.
|
||||
- **Pair.** 5 runs per task at the base and at the head, in one bench, so a model or CLI update moves both sides. Each result names the Claude Code version and the model id.
|
||||
- **Check.** The project's test command over Haiku's changes to files other than tests, with the tests restored from the commit, so deleting a test doesn't pass.
|
||||
- **Measure.** From the stream: passed or not, tokens split into input, cache write, cache read and output, turns, wall time, and the reads before the first open of the file the bug is in.
|
||||
- **Print.** One row per task and side, then the head's change against the base. Results and transcripts go under the chunk's review folder.
|
||||
|
||||
It runs under `run-heavy`, and each container it starts has a memory limit. The test command and the mutator's prefix are flags, as in `mutate-changes`. Item 503 sets the task and run counts.
|
||||
It runs under `run-heavy`. The image, the test command and the mutator's prefix are flags, as in `mutate-changes`. Item 503 sets the task and run counts.
|
||||
|
||||
Touches:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user