From d813fdad5a5bbcd583955d8f0becfdc9bb69885e Mon Sep 17 00:00:00 2001 From: Lilleman auf Larv Date: Sat, 21 Nov 2026 03:40:18 +0100 Subject: [PATCH] Finish item 501 in the plan --- CHANGELOG.md | 1 + todo.md | 28 ++-------------------------- 2 files changed, 3 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 07ca85bd..4bdddbe0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## Unreleased +- `task-bench` is new: it plants bugs that a Go project's tests catch, at the places the comprehension seats found hardest, and has Haiku fix each in a container at a base commit and at a head. It prints for each side how often the fix passes, with the median tokens, turns, wall time and reads before Haiku opens the file the bug is in. `task-bench --help` says how to run it. - larv-review now runs each command that a changed doc line says prints something and hands the real output to the product-owner and prose reviewers, so a claim about output is checked by running it. - `do-epic` moves a parent issue whose sub-tasks a person filed: to done by `jira-finish-transition` once every sub-task is done, and before that by the transition the new setting `jira-sub-tasks-landed-transition` names, once each sub-task is done or in one of `jira-unblocking-statuses`. The transition must lead to one of those statuses. `--dry-run` prints each such move. - mutate-changes now mutates the Go lines a branch changes with Gremlins, as it does Python, Rust, TypeScript and PHP, and prints each mutant the tests do not kill. diff --git a/todo.md b/todo.md index ac026004..af2dc67c 100644 --- a/todo.md +++ b/todo.md @@ -12,13 +12,12 @@ | ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score | |---|---|---|---|---|---|---|---|---|---|---|---| -| 501 | 0.15.0 | | owner | **Ship `task-bench`, which plants a bug the tests catch at the places the comprehension seats found hardest, has Haiku fix it in a vanilla Claude Code at a base and a head, and prints for each side how often the fix passes, with its median tokens, turns, wall time and reads before it opens a file the bug is in.** | 4 | 2 | 1 | 5 | 9 | 1.1 by 5, code health by 4 | 0.6 | -| 502 | 0.15.0 | 501 | owner | **Distil each `task-bench` run into a digest of short traces and decoys, the files most failing runs read but no fix touches, and have a stronger model turn it into plan items before the transcripts are deleted.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 4, code health by 4 | 4.8 | +| 502 | 0.15.0 | | owner | **Distil each `task-bench` run into a digest of short traces and decoys, the files most failing runs read but no fix touches, and have a stronger model turn it into plan items before the transcripts are deleted.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 4, code health by 4 | 4.8 | | 503 | 0.15.0 | 502 | owner | **Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3 | 1.2 | | 509 | 0.16.0 | 503 | owner | **Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.** | 3 | 2 | 3 | 6 | 5 | code health by 5, 1.1 by 3 | 5.6 | | 504 | 0.16.0 | 503 | owner | **Bench new features in `task-bench`, each a spec for Haiku with acceptance tests it never sees and a reference patch that proves it can be built, kept in a `bench-tasks` folder the project config names, starting with six for fejkdata.** | 3 | 2 | 1 | 4 | 6 | 1.1 by 4, code health by 3 | 2.6 | | 505 | 0.16.0 | 504 | owner | **Bench adding source data in `task-bench`: hand Haiku a fake register that a seeded script generates for a made-up country, and check the importer, the data and the licence entry it adds, starting with five tasks for fejkdata.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 | -| 508 | 0.16.0 | 501 | owner | **Plant bugs for `task-bench` in Python, Rust, TypeScript and PHP, from the mutants `mutate-changes` runs for each, so this repo benches itself.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 | +| 508 | 0.16.0 | | owner | **Plant bugs for `task-bench` in Python, Rust, TypeScript and PHP, from the mutants `mutate-changes` runs for each, so this repo benches itself.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 | | 506 | 0.16.0 | 504 | owner | **Go over the bench features in each release review: reapply each reference patch, drop a feature that no longer breaks a goal, and add features until they cover the places the seats find hardest.** | 2 | 2 | 2 | 4 | 3 | 1.1 by 3 | 1.1 | | 507 | 0.16.0 | 504 | owner | **Bench rebuilding a feature the project shipped: revert its commit at the head, hand Haiku its item tagline, and check with the tests that commit added.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3 | 0.7 | | 477 | 0.17.0 | | question | **Rewrite `## Goals` and `## Audience` for a setup others install and run apart from their own `~/.claude`.** | 1 | 1 | 2 | 2 | 2 | | 0.0 | @@ -105,29 +104,6 @@ ## Details -### 501. Ship `task-bench`, which plants a bug the tests catch at the places the comprehension seats found hardest, has Haiku fix it in a vanilla Claude Code at a base and a head, and prints for each side how often the fix passes, with its median tokens, turns, wall time and reads before it opens a file the bug is in. - -The panel's scores barely move between chunks, and `reading-cost` measures chunks whose tasks differ. A planted task stays fixed, so a change in its cost comes from the code. The bench runs only programs; no model judges anything in it. - -- **Plant.** Run Gremlins, as item 500 runs it, without `--diff`, over the files that hold the places `larv-review places` prints. Draw 5 of its `KILLED` mutants by a seeded pick, and re-apply each at its file, line and column. The task's text is the failing assertion's message, without the test's name, file or line, so it says what goes wrong and not where. Go is the first language; item 508 adds the others. -- **Copy.** The tree at the commit, without `.git`, committed once with the bug in it, so `git diff` and `git log` show Haiku nothing. -- **Run.** Inside a container that holds only the copy: the project's development image with Claude Code added, such as `golang` for fejkdata, signed in by a long-lived token in an environment variable, with a memory limit and the network only for the API. In it, `claude -p --strict-mcp-config --disable-slash-commands --no-session-persistence --model claude-haiku-5-5 --effort low --output-format stream-json` runs with every tool, Bash included, so Haiku greps, builds, runs the CLI and runs single tests as an agent normally does. A narrower tool list would raise the figures for reasons outside the code, and an allowlist can't fence Bash anyway: a rule that allows `cat` allows it on any path, such as the real checkout, whose diff would give away the bug. Run without `--bare`: it can't sign in by a long-lived token (the decision `Claude Code in a home of its own`). The container's fresh Claude home is empty, so none of the human's instructions, memories, hooks, skills or plugins load; they would skew the figures and change with claude-config. The repo's README and `docs/` stay in the copy, since they are part of what the bench measures. A project's `CLAUDE.md` loads from the working folder, or its `AGENTS.md` where it has no `CLAUDE.md`; the bench takes that as the project's own setup. -- **Pair.** 5 runs per task at the base and at the head, in one bench, so a model or CLI update moves both sides. Each result names the Claude Code version and the model id. -- **Check.** The project's test command over Haiku's changes to files other than tests, with the tests restored from the commit, so deleting a test doesn't pass. -- **Measure.** From the stream: passed or not, tokens split into input, cache write, cache read and output, turns, wall time, and the reads before the first open of the file the bug is in. -- **Print.** One row per task and side, then the head's change against the base. Results and transcripts go under the chunk's review folder. - -It runs under `run-heavy`. The image, the test command and the mutator's prefix are flags, as in `mutate-changes`. Item 503 sets the task and run counts. - -Touches: - -- `bin/task-bench` -- `bin/task_bench/main.py` -- `bin/task_bench/plant.py` -- `bin/task_bench/runs.py` -- `bin/task_bench/test_task_bench.py` -- `README.md`: `## Tools` (small), `` ### Packages under `bin/` `` (small) - ### 502. Distil each `task-bench` run into a digest of short traces and decoys, the files most failing runs read but no fix touches, and have a stronger model turn it into plan items before the transcripts are deleted. The thought process of the runs shows why Haiku failed, but whole transcripts would flood the model that reads them. A program distils them; only the last step uses a model.