Files
claude-config/bin/task_bench/main.py
T

377 lines
21 KiB
Python

r"""Measure how hard a project's code is to fix a bug in, at a base commit and at a head: plant bugs its tests catch, have Haiku fix
each several times at both, and print how often the fix passes and what it took.
usage: task-bench --image <image> --test <shell text> [--base <ref>] [--head <ref>] [--prefix <shell text>] [--seed <n>] [--bugs <n>] [--runs <n>]
task-bench gather
Run it in a checkout, under ~/.claude/bin/run-heavy, with a long-lived token in CLAUDE_CODE_OAUTH_TOKEN, which `claude setup-token`
prints. The head is --head, by default HEAD. The base is the merge base of the head and --base, by default the default branch:
origin's as last fetched, else main or master. Where --base names the head's own commit, the bench runs the head against itself.
No model judges anything: the bench runs programs, and reads Haiku's own stream.
- Plant. It copies the tree at the base and runs Gremlins there, as `gremlins unleash`, over the Go files that hold the places
`larv-review places` prints, the ones the comprehension seats found hardest. It takes the mutants the tests kill in the order
--seed sets, 1 by default, until --bugs of them, 5 by default, have become bugs. A mutant becomes a bug where its line stands
unchanged at the head, and where, planted at each side, a test fails and prints a message that says more than the test's name.
Any other mutant is passed over, so a change to the very line of a bug is not benched. Go is the one language so far.
- Run. Haiku fixes each bug --runs times at each side, 5 by default, the sides in turn. A fix starts from a copy of that side's
tree with the bug in it, committed once, so git doesn't show which line the bug is on. In the copy, in a container, `claude -p
--strict-mcp-config --disable-slash-commands --no-session-persistence --model claude-haiku-5-5 --effort low` works with every
tool. It is told the message the failing test printed at that side, without the test's name, file or line. The container's
Claude home is new and empty, so the project's own CLAUDE.md, or AGENTS.md, in the copy is all the setup that loads. Its network
reaches api.anthropic.com and nothing else. A fix ends after 30 minutes.
- Check. The test command runs over the planted tree with each file that is no test as Haiku left it; a fix passes where it exits 0.
A test Haiku changed, removed or added doesn't count, and neither does a file under a `testdata` folder.
--image is the project's development image, such as golang:1.27.4. The bench adds Claude Code to it, as its installer installs it,
so the image needs curl and bash; the result names the version. Haiku works in that image. So does --test: sh runs that text at
the copy's top, with no network, so the image must hold what the tests would otherwise download.
--prefix is shell text that runs Gremlins, on the host, as mutate-changes' --prefix does: sh runs `<prefix> "$@"` in the copy of the
base, which $PWD names, with `gremlins unleash ...` as "$@". Without it, the bench runs the `gremlins` on the host's PATH. For
example:
docker run --rm --memory 16g --memory-swap 16g -v "$PWD:$PWD" -w "$PWD" golang:1.27.4 \
sh -c 'shift; exec go run github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0 "$@"' sh
It prints the base and the head, then a table: a row per bug and side with how many fixes passed and the medians, over the runs
that finished, of the input, cache-write, cache-read and output tokens, the turns, the wall time, and the reads and searches before
the run first opened the file the bug is in. The median of the reads leaves out a run that never opened that file. Then come a row
per side over every bug, and the head's change against the base. The last line names the bench's folder: `task-bench` in the
chunk's review folder.
That folder holds each run's result and stream, and digest.md, which says why runs failed in a few kilobytes per bug:
- the table;
- per bug its decoys, the files more than half of its failing runs read that the bug isn't in;
- per bug up to two traces: of a failing run, one that didn't finish or else the one that spent the most tokens, and of the
passing run with the median tokens. A trace has a line per tool call, such as `Read pick.go:120-180` or `Grep "readName" → 4 hits`, and the first 3 lines of what Haiku wrote
between calls.
Read a change against the noise. With the defaults, a bench of a commit against itself shows, in 19 cases of 20, an All row change
under 31% in output tokens, 50% in cache-read tokens and 21% in wall time. The mean of the bugs' own changes, which you work out
from the bug rows, stays under 12%, 16% and 10%. On `fejkdata`'s bugs every run passed, so the pass rate's noise is unmeasured.
`docs/decisions.md` → `task-bench: a bench takes` has the figures.
`task-bench gather`, run in the same checkout, turns the digest into plan items. It starts a claude on the project's rung, the
cheapest rung of the model loop-claude runs the project on, with the digest and the prompt in gather-prompt.md beside this file.
That claude files an item with `todo-item add` for each place the runs show to be hard, and leaves the plan's file uncommitted.
gather prints that claude's answer and removes the bench's folder. Where claude fails, the folder stays, for another try.
Each bench empties its own folder first, and removes the folder every earlier bench left under any review folder, gathered or
not, unless a bench or a gather is using it. So gather a bench before the next one starts.
Exit 1, saying why, on an unknown or repeated argument, a missing --image or --test, a --seed, --bugs or --runs that is no whole
number from 1, or no token; and where git, docker or Gremlins fails, the seats' places name no Go file, the tests fail at a side
without a bug, or no mutant becomes a bug. Where the API refuses a run, the bench stops there and exits 1 too, after the table of
the runs that finished.
gather exits 1, saying why, outside a checkout, without a digest, where loop-claude has saved no model for the project or the
model is not on its routing ladder, and where its claude fails or hits a usage limit.
"""
import dataclasses
import functools
import itertools
import json
import os
import pathlib
import shutil
import sys
import tarfile
import typing
import uuid
from task_bench.digest import Bug, digest, Run
from task_bench.runs import checked_tree, closed, Containers, docker_cli, fix_attempted, MEMORY_MIB, opened, ran_tests, TimedOut, TOKEN
from claude_code.reply import Claude
from task_bench.bench_folder import clear_left, held
from machine_load.slots import CPUS_PER_SLOT
from git_cli.git_repo import default_ref, Git, top
from task_bench.gather import DIGEST, gather, run_claude
from task_bench.plant import drawn, killed_mutants, line_at_head, Mutant, mutated, place_files, task_text
from mutate_changes.tool_process import failed, Ran, run_tool
from failure import Failure
from task_bench.figures import figures, Result, SIDES, table, WORK
from git_cli import git_repo
from review_rules.review_folder import made_review_folder, REPORTS
from larv_review.history_commands import read_places
from run_heavy.capped_docker import SHARE_VARIABLE
USAGE = "usage: task-bench --image <image> --test <shell text> [--base <ref>] [--head <ref>] [--prefix <shell text>] [--seed <n>] [--bugs <n>] [--runs <n>], or task-bench gather"
# A panel of three readers, seated as the README's audience, picked these options on 2026-11-21, each 3 to 0.
OPTIONS = ("--base", "--bugs", "--head", "--image", "--prefix", "--runs", "--seed", "--test")
COUNTS = {"--bugs": "5", "--runs": "5", "--seed": "1"}
# Opens the report's last line, which names the bench's folder.
KEPT = "The digest, and each run's result and stream, until `task-bench gather` files plan items from them or the next bench starts: "
# The planted commit doesn't name anyone, and neither a hook nor a signing key of the caller's reaches it.
COMMIT = ["-c", "user.name=task-bench", "-c", "user.email=task-bench@localhost", "-c", "commit.gpgsign=false", "-c", "core.hooksPath=/dev/null", "commit", "--quiet", "--message", "Import"]
@dataclasses.dataclass(frozen=True)
class Task:
"""A bug to fix: its number from 1, the mutant at the base, and per side the planted tree and what Haiku is told there."""
number: int
mutant: Mutant
trees: dict[str, str]
messages: dict[str, str]
def named(self) -> str:
return f"{self.mutant.path}:{self.mutant.line}:{self.mutant.column} {self.mutant.mutator}"
def main(argv: list[str], env: typing.Mapping[str, str], read: typing.Callable[[pathlib.Path], str | Failure] = read_places, claude: Claude = run_claude) -> int:
"""Runs task-bench with the arguments argv; git, docker and Gremlins run in the environment env, `read` reads what
`larv-review places` prints for the checkout, and `claude` runs the claude that gathers."""
if argv in (["-h"], ["--help"]):
print(__doc__.strip())
return 0
home = pathlib.Path(env.get("HOME") or pathlib.Path.home())
if argv[:1] == ["gather"]:
report = gather(env, home, claude) if argv == ["gather"] else Failure(f"gather takes no arguments; {USAGE}")
else:
given = options(argv, env)
if isinstance(given, Failure):
print(f"{given.reason}; {USAGE}", file=sys.stderr)
return 1
places = read(home)
report = places if isinstance(places, Failure) else bench(given, env, places, home)
if isinstance(report, Failure):
print(report.reason, file=sys.stderr)
return 1
print("\n".join(report))
return 0
def options(argv: list[str], env: typing.Mapping[str, str]) -> dict[str, str] | Failure:
given: dict[str, str] = {}
rest = list(argv)
while rest:
name, equals, value = rest.pop(0).partition("=")
if name not in OPTIONS:
return Failure(f"unknown argument {name}")
if name in given:
return Failure(f"{name} is given twice")
if not equals and not rest:
return Failure(f"{name} needs a value")
given[name] = value if equals else rest.pop(0)
for name in ("--image", "--test"):
if not given.get(name):
return Failure(f"{name} is missing")
for name, default in COUNTS.items():
if not given.setdefault(name, default).isdecimal() or int(given[name]) < 1:
return Failure(f"{name} takes a whole number from 1, not {given[name]}")
if not env.get(TOKEN):
return Failure(f"{TOKEN} holds no token")
return given
def bench(given: dict[str, str], env: typing.Mapping[str, str], places: str, home: pathlib.Path) -> list[str] | Failure:
"""The report of one bench of the checkout the process runs in, with `places` what `larv-review places` prints there."""
git = functools.partial(git_repo.git, env=env)
checkout = top(git)
if isinstance(checkout, Failure):
return checkout
sides = commits(given, checkout, git)
if isinstance(sides, Failure):
return sides
print(f"task-bench: base {sides['base'][:12]}, head {sides['head'][:12]}.", file=sys.stderr)
review = made_review_folder(checkout, home)
if isinstance(review, Failure):
return review
with held(review) as folder:
if isinstance(folder, Failure):
return folder
clear_left(home / REPORTS)
return bench_in(folder, given, env, places, checkout, sides, git)
def bench_in(folder: pathlib.Path, given: dict[str, str], env: typing.Mapping[str, str], places: str, checkout: str, sides: dict[str, str], git: Git) -> list[str] | Failure:
"""The report of a bench of the commits sides names, which empties folder and keeps its runs there."""
containers = Containers(docker_cli(env), f"task-bench-{uuid.uuid4().hex[:12]}", given["--test"], int(env.get(SHARE_VARIABLE) or MEMORY_MIB), f"{os.getuid()}:{os.getgid()}")
try:
shutil.rmtree(folder, ignore_errors=True)
trees: dict[str, str] = {}
for side, commit in sides.items():
tree = archived(commit, folder / "trees" / side, checkout, git)
if isinstance(tree, Failure):
return tree
trees[side] = tree
files = [path for path in place_files(places) if pathlib.Path(trees["base"], path).is_file()]
if not files:
return Failure("The places the seats found hard name no Go file that the base holds, so there is nowhere to plant a bug.")
run = functools.partial(run_tool, prefix=given.get("--prefix"), env=env, killpg=os.killpg)
mutants = killed_mutants(trees["base"], files, run, CPUS_PER_SLOT)
if isinstance(mutants, Failure):
return mutants
try:
found = opened(containers, given["--image"]) or benched(given, checkout, sides, trees, mutants, containers, folder, git)
finally:
closed(containers)
except OSError as failure:
return Failure(f"could not write under {folder}: {failure}")
return found if isinstance(found, Failure) else [*found, f"{KEPT}{folder}"]
def commits(given: dict[str, str], checkout: str, git: Git) -> dict[str, str] | Failure:
"""The base's commit and the head's."""
head = git("rev-parse", "--verify", f"{given.get('--head', 'HEAD')}^{{commit}}", cwd=checkout)
against = given.get("--base") or default_ref(cwd=checkout, run=git)
if isinstance(head, Failure):
return head
if isinstance(against, Failure):
return against
base = git("merge-base", against, head.strip(), cwd=checkout)
return base if isinstance(base, Failure) else {"base": base.strip(), "head": head.strip()}
def archived(commit: str, into: pathlib.Path, checkout: str, git: Git) -> str | Failure:
"""Makes `into` the tree at commit, as `git archive` holds it, and returns its path: without git's own data, and without a
path the project marks `export-ignore`."""
into.mkdir(parents=True)
archive = into.with_suffix(".tar")
wrote = git("archive", "--format=tar", "--output", str(archive), commit, cwd=checkout)
if isinstance(wrote, Failure):
return wrote
try:
with tarfile.open(archive) as held:
held.extractall(into, filter="tar")
except tarfile.TarError as failure:
return Failure(f"could not unpack the tree at {commit}: {failure}")
archive.unlink()
return str(into)
def benched(given: dict[str, str], checkout: str, sides: dict[str, str], trees: dict[str, str], mutants: list[Mutant], containers: Containers, folder: pathlib.Path, git: Git) -> list[str] | Failure:
"""Plants the bugs, runs each fix, writes the results into folder and returns the report."""
for side, tree in trees.items():
ran = tests_in_copy(containers, tree, folder)
if isinstance(ran, Failure) or ran.code:
return ran if isinstance(ran, Failure) else failed(f"The tests fail at the {side} without a planted bug: the test command", ran)
tasks: list[Task] = []
for mutant in drawn(mutants, int(given["--seed"])):
if len(tasks) == int(given["--bugs"]):
break
task = planted(len(tasks) + 1, mutant, checkout, sides, trees, containers, folder, git)
if isinstance(task, Failure):
return task
tasks += [task] if task else []
if not tasks:
return Failure(f"None of the {len(mutants)} mutants the tests kill becomes a bug: each is on a line the head changed, or the test it fails doesn't print a message.")
results: list[Result] = []
stopped = None
total = int(given["--runs"]) * len(tasks) * len(SIDES)
for number, task, side in itertools.product(range(1, int(given["--runs"]) + 1), tasks, SIDES):
result = fixed(task, side, number, containers, folder)
if isinstance(result, Failure):
stopped = result
break
results.append(result)
print(f"task-bench: run {len(results)} of {total}, bug {task.number} at the {side}: {'passed' if result.passed else 'not passed'}.", file=sys.stderr)
if stopped and not results:
return stopped
names = {task.number: task.named() for task in tasks}
(folder / "results.json").write_text(json.dumps({"sides": sides, "seed": int(given["--seed"]), "bugs": names, "runs": [recorded(result) for result in results]}, indent="\t") + "\n", encoding="utf-8")
in_a_tree = lambda path: any(pathlib.Path(tree, path).is_file() for tree in trees.values())
(folder / DIGEST).write_text(digest(sides, table(results, names), [digested(task, results, folder) for task in tasks], in_a_tree), encoding="utf-8", errors="replace")
if stopped:
# The runs that finished took hours and tokens, so the bench shows them before it fails.
return Failure("\n".join([stopped.reason, f"task-bench stopped after {len(results)} of {total} runs, which results.json holds in {folder}:", "", *table(results, names)]))
ran_on = sorted({f"{found.model} in Claude Code {found.version}" for result in results if (found := result.figures)})
short = [] if len(tasks) == int(given["--bugs"]) else [f"Planted {len(tasks)} of the {given['--bugs']} bugs asked for: no other mutant the tests kill becomes one."]
return [
f"task-bench: base {sides['base'][:12]}, head {sides['head'][:12]}. {counted(len(tasks), 'bug')}, each run {counted(int(given['--runs']), 'time')} at each side by {' and '.join(ran_on) or 'no run that finished'}.",
*short,
"",
*table(results, names),
"",
]
def planted(number: int, mutant: Mutant, checkout: str, sides: dict[str, str], trees: dict[str, str], containers: Containers, folder: pathlib.Path, git: Git) -> Task | Failure | None:
"""The task of fixing mutant, with its planted tree at each side; None where the mutant doesn't become a bug at one side or the other.
The bug is the same change to the same line at both sides, so only the code around it differs between them."""
diff = git("diff", "-U0", "--no-color", "--no-ext-diff", "--no-textconv", sides["base"], sides["head"], "--", mutant.path, cwd=checkout, errors="replace")
if isinstance(diff, Failure):
return diff
at_head = line_at_head(diff, mutant.line)
if at_head is None:
return None
planted_trees, messages = {}, {}
for side, placed in (("base", mutant), ("head", mutant._replace(line=at_head))):
source = pathlib.Path(trees[side], mutant.path)
changed = mutated(source.read_bytes(), placed) if source.is_file() else None
if changed is None:
return None
tree = folder / "bugs" / str(number) / side
shutil.rmtree(tree, ignore_errors=True)
shutil.copytree(trees[side], tree, symlinks=True)
(tree / mutant.path).write_bytes(changed)
ran = tests_in_copy(containers, str(tree), folder)
if isinstance(ran, Failure) and not isinstance(ran, TimedOut):
return ran
# Tests that never end under the bug have no message to tell Haiku.
message = None if isinstance(ran, TimedOut) else task_text(ran.output)
if message is None:
return None
for args in (["init", "--quiet"], ["add", "--all", "--force"], COMMIT):
done = git(*args, cwd=str(tree))
if isinstance(done, Failure):
return done
planted_trees[side], messages[side] = str(tree), message
return Task(number, mutant, planted_trees, messages)
def tests_in_copy(containers: Containers, tree: str, folder: pathlib.Path) -> Ran | Failure:
"""The test command's run in a copy of tree, removed afterwards, so what the tests write stays out of tree."""
copy = folder / "tested"
shutil.copytree(tree, copy, symlinks=True)
try:
return ran_tests(containers, str(copy))
finally:
shutil.rmtree(copy, ignore_errors=True)
def fixed(task: Task, side: str, number: int, containers: Containers, folder: pathlib.Path) -> Result | Failure:
"""One run: Haiku's fix of task at side in a copy of its planted tree, then the tests over what it changed. The stream stays in
folder; the copy is removed. Tests that run out of time don't pass."""
stream = stream_of(Result(task.number, side, number, False, None), folder)
stream.parent.mkdir(exist_ok=True)
copy, check = folder / "copy", folder / "check"
try:
shutil.copytree(task.trees[side], copy, symlinks=True)
attempted = fix_attempted(containers, str(copy), task.messages[side], stream)
if isinstance(attempted, Failure):
return attempted
found = figures(stream.read_bytes(), os.path.join(WORK, task.mutant.path))
if isinstance(found, Failure):
return found
checked_tree(task.trees[side], str(copy), str(check))
ran = ran_tests(containers, str(check))
if isinstance(ran, Failure) and not isinstance(ran, TimedOut):
return ran
return Result(task.number, side, number, isinstance(ran, Ran) and ran.code == 0, found)
finally:
shutil.rmtree(copy, ignore_errors=True)
shutil.rmtree(check, ignore_errors=True)
def stream_of(result: Result, folder: pathlib.Path) -> pathlib.Path:
"""The file that holds the stream of the run result names. Its name without the suffix is the run's id in the digest."""
return folder / "runs" / f"bug-{result.task}-{result.side}-{result.run}.jsonl"
def digested(task: Task, results: list[Result], folder: pathlib.Path) -> Bug:
"""The bug as the digest tells it, with each of its runs in results. A run that didn't finish has no tokens."""
runs = [Run(stream_of(result, folder).stem, result.passed, found.tokens.context() + found.tokens.output if (found := result.figures) else None, stream_of(result, folder).read_bytes()) for result in results if result.task == task.number]
return Bug(f"{task.number}: {task.named()}", task.mutant.path, task.messages["base"], runs)
def recorded(result: Result) -> dict:
"""A run as results.json holds it."""
run = {"bug": result.task, "side": result.side, "run": result.run, "passed": result.passed, "finished": result.figures is not None}
if not (found := result.figures):
return run
return {**run, "claude_code": found.version, "model": found.model, "tokens": dataclasses.asdict(found.tokens), "turns": found.turns, "seconds": found.seconds, "reads": found.reads}
def counted(count: int, noun: str) -> str:
return f"{count} {noun}{'' if count == 1 else 's'}"