377 lines
21 KiB
Python
377 lines
21 KiB
Python
r"""Measure how hard a project's code is to fix a bug in, at a base commit and at a head: plant bugs its tests catch, have Haiku fix
|
|
each several times at both, and print how often the fix passes and what it took.
|
|
|
|
usage: task-bench --image <image> --test <shell text> [--base <ref>] [--head <ref>] [--prefix <shell text>] [--seed <n>] [--bugs <n>] [--runs <n>]
|
|
task-bench gather
|
|
|
|
Run it in a checkout, under ~/.claude/bin/run-heavy, with a long-lived token in CLAUDE_CODE_OAUTH_TOKEN, which `claude setup-token`
|
|
prints. The head is --head, by default HEAD. The base is the merge base of the head and --base, by default the default branch:
|
|
origin's as last fetched, else main or master. Where --base names the head's own commit, the bench runs the head against itself.
|
|
No model judges anything: the bench runs programs, and reads Haiku's own stream.
|
|
|
|
- Plant. It copies the tree at the base and runs Gremlins there, as `gremlins unleash`, over the Go files that hold the places
|
|
`larv-review places` prints, the ones the comprehension seats found hardest. It takes the mutants the tests kill in the order
|
|
--seed sets, 1 by default, until --bugs of them, 5 by default, have become bugs. A mutant becomes a bug where its line stands
|
|
unchanged at the head, and where, planted at each side, a test fails and prints a message that says more than the test's name.
|
|
Any other mutant is passed over, so a change to the very line of a bug is not benched. Go is the one language so far.
|
|
- Run. Haiku fixes each bug --runs times at each side, 5 by default, the sides in turn. A fix starts from a copy of that side's
|
|
tree with the bug in it, committed once, so git doesn't show which line the bug is on. In the copy, in a container, `claude -p
|
|
--strict-mcp-config --disable-slash-commands --no-session-persistence --model claude-haiku-5-5 --effort low` works with every
|
|
tool. It is told the message the failing test printed at that side, without the test's name, file or line. The container's
|
|
Claude home is new and empty, so the project's own CLAUDE.md, or AGENTS.md, in the copy is all the setup that loads. Its network
|
|
reaches api.anthropic.com and nothing else. A fix ends after 30 minutes.
|
|
- Check. The test command runs over the planted tree with each file that is no test as Haiku left it; a fix passes where it exits 0.
|
|
A test Haiku changed, removed or added doesn't count, and neither does a file under a `testdata` folder.
|
|
|
|
--image is the project's development image, such as golang:1.27.4. The bench adds Claude Code to it, as its installer installs it,
|
|
so the image needs curl and bash; the result names the version. Haiku works in that image. So does --test: sh runs that text at
|
|
the copy's top, with no network, so the image must hold what the tests would otherwise download.
|
|
--prefix is shell text that runs Gremlins, on the host, as mutate-changes' --prefix does: sh runs `<prefix> "$@"` in the copy of the
|
|
base, which $PWD names, with `gremlins unleash ...` as "$@". Without it, the bench runs the `gremlins` on the host's PATH. For
|
|
example:
|
|
|
|
docker run --rm --memory 16g --memory-swap 16g -v "$PWD:$PWD" -w "$PWD" golang:1.27.4 \
|
|
sh -c 'shift; exec go run github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0 "$@"' sh
|
|
|
|
It prints the base and the head, then a table: a row per bug and side with how many fixes passed and the medians, over the runs
|
|
that finished, of the input, cache-write, cache-read and output tokens, the turns, the wall time, and the reads and searches before
|
|
the run first opened the file the bug is in. The median of the reads leaves out a run that never opened that file. Then come a row
|
|
per side over every bug, and the head's change against the base. The last line names the bench's folder: `task-bench` in the
|
|
chunk's review folder.
|
|
|
|
That folder holds each run's result and stream, and digest.md, which says why runs failed in a few kilobytes per bug:
|
|
- the table;
|
|
- per bug its decoys, the files more than half of its failing runs read that the bug isn't in;
|
|
- per bug up to two traces: of a failing run, one that didn't finish or else the one that spent the most tokens, and of the
|
|
passing run with the median tokens. A trace has a line per tool call, such as `Read pick.go:120-180` or `Grep "readName" → 4 hits`, and the first 3 lines of what Haiku wrote
|
|
between calls.
|
|
|
|
Read a change against the noise. With the defaults, a bench of a commit against itself shows, in 19 cases of 20, an All row change
|
|
under 31% in output tokens, 50% in cache-read tokens and 21% in wall time. The mean of the bugs' own changes, which you work out
|
|
from the bug rows, stays under 12%, 16% and 10%. On `fejkdata`'s bugs every run passed, so the pass rate's noise is unmeasured.
|
|
`docs/decisions.md` → `task-bench: a bench takes` has the figures.
|
|
|
|
`task-bench gather`, run in the same checkout, turns the digest into plan items. It starts a claude on the project's rung, the
|
|
cheapest rung of the model loop-claude runs the project on, with the digest and the prompt in gather-prompt.md beside this file.
|
|
That claude files an item with `todo-item add` for each place the runs show to be hard, and leaves the plan's file uncommitted.
|
|
gather prints that claude's answer and removes the bench's folder. Where claude fails, the folder stays, for another try.
|
|
|
|
Each bench empties its own folder first, and removes the folder every earlier bench left under any review folder, gathered or
|
|
not, unless a bench or a gather is using it. So gather a bench before the next one starts.
|
|
|
|
Exit 1, saying why, on an unknown or repeated argument, a missing --image or --test, a --seed, --bugs or --runs that is no whole
|
|
number from 1, or no token; and where git, docker or Gremlins fails, the seats' places name no Go file, the tests fail at a side
|
|
without a bug, or no mutant becomes a bug. Where the API refuses a run, the bench stops there and exits 1 too, after the table of
|
|
the runs that finished.
|
|
gather exits 1, saying why, outside a checkout, without a digest, where loop-claude has saved no model for the project or the
|
|
model is not on its routing ladder, and where its claude fails or hits a usage limit.
|
|
"""
|
|
import dataclasses
|
|
import functools
|
|
import itertools
|
|
import json
|
|
import os
|
|
import pathlib
|
|
import shutil
|
|
import sys
|
|
import tarfile
|
|
import typing
|
|
import uuid
|
|
|
|
from task_bench.digest import Bug, digest, Run
|
|
from task_bench.runs import checked_tree, closed, Containers, docker_cli, fix_attempted, MEMORY_MIB, opened, ran_tests, TimedOut, TOKEN
|
|
from claude_code.reply import Claude
|
|
from task_bench.bench_folder import clear_left, held
|
|
from machine_load.slots import CPUS_PER_SLOT
|
|
from git_cli.git_repo import default_ref, Git, top
|
|
from task_bench.gather import DIGEST, gather, run_claude
|
|
from task_bench.plant import drawn, killed_mutants, line_at_head, Mutant, mutated, place_files, task_text
|
|
from mutate_changes.tool_process import failed, Ran, run_tool
|
|
from failure import Failure
|
|
from task_bench.figures import figures, Result, SIDES, table, WORK
|
|
from git_cli import git_repo
|
|
from review_rules.review_folder import made_review_folder, REPORTS
|
|
from larv_review.history_commands import read_places
|
|
from run_heavy.capped_docker import SHARE_VARIABLE
|
|
|
|
USAGE = "usage: task-bench --image <image> --test <shell text> [--base <ref>] [--head <ref>] [--prefix <shell text>] [--seed <n>] [--bugs <n>] [--runs <n>], or task-bench gather"
|
|
# A panel of three readers, seated as the README's audience, picked these options on 2026-11-21, each 3 to 0.
|
|
OPTIONS = ("--base", "--bugs", "--head", "--image", "--prefix", "--runs", "--seed", "--test")
|
|
COUNTS = {"--bugs": "5", "--runs": "5", "--seed": "1"}
|
|
# Opens the report's last line, which names the bench's folder.
|
|
KEPT = "The digest, and each run's result and stream, until `task-bench gather` files plan items from them or the next bench starts: "
|
|
# The planted commit doesn't name anyone, and neither a hook nor a signing key of the caller's reaches it.
|
|
COMMIT = ["-c", "user.name=task-bench", "-c", "user.email=task-bench@localhost", "-c", "commit.gpgsign=false", "-c", "core.hooksPath=/dev/null", "commit", "--quiet", "--message", "Import"]
|
|
|
|
|
|
@dataclasses.dataclass(frozen=True)
|
|
class Task:
|
|
"""A bug to fix: its number from 1, the mutant at the base, and per side the planted tree and what Haiku is told there."""
|
|
number: int
|
|
mutant: Mutant
|
|
trees: dict[str, str]
|
|
messages: dict[str, str]
|
|
|
|
def named(self) -> str:
|
|
return f"{self.mutant.path}:{self.mutant.line}:{self.mutant.column} {self.mutant.mutator}"
|
|
|
|
|
|
def main(argv: list[str], env: typing.Mapping[str, str], read: typing.Callable[[pathlib.Path], str | Failure] = read_places, claude: Claude = run_claude) -> int:
|
|
"""Runs task-bench with the arguments argv; git, docker and Gremlins run in the environment env, `read` reads what
|
|
`larv-review places` prints for the checkout, and `claude` runs the claude that gathers."""
|
|
if argv in (["-h"], ["--help"]):
|
|
print(__doc__.strip())
|
|
return 0
|
|
home = pathlib.Path(env.get("HOME") or pathlib.Path.home())
|
|
if argv[:1] == ["gather"]:
|
|
report = gather(env, home, claude) if argv == ["gather"] else Failure(f"gather takes no arguments; {USAGE}")
|
|
else:
|
|
given = options(argv, env)
|
|
if isinstance(given, Failure):
|
|
print(f"{given.reason}; {USAGE}", file=sys.stderr)
|
|
return 1
|
|
places = read(home)
|
|
report = places if isinstance(places, Failure) else bench(given, env, places, home)
|
|
if isinstance(report, Failure):
|
|
print(report.reason, file=sys.stderr)
|
|
return 1
|
|
print("\n".join(report))
|
|
return 0
|
|
|
|
|
|
def options(argv: list[str], env: typing.Mapping[str, str]) -> dict[str, str] | Failure:
|
|
given: dict[str, str] = {}
|
|
rest = list(argv)
|
|
while rest:
|
|
name, equals, value = rest.pop(0).partition("=")
|
|
if name not in OPTIONS:
|
|
return Failure(f"unknown argument {name}")
|
|
if name in given:
|
|
return Failure(f"{name} is given twice")
|
|
if not equals and not rest:
|
|
return Failure(f"{name} needs a value")
|
|
given[name] = value if equals else rest.pop(0)
|
|
for name in ("--image", "--test"):
|
|
if not given.get(name):
|
|
return Failure(f"{name} is missing")
|
|
for name, default in COUNTS.items():
|
|
if not given.setdefault(name, default).isdecimal() or int(given[name]) < 1:
|
|
return Failure(f"{name} takes a whole number from 1, not {given[name]}")
|
|
if not env.get(TOKEN):
|
|
return Failure(f"{TOKEN} holds no token")
|
|
return given
|
|
|
|
|
|
def bench(given: dict[str, str], env: typing.Mapping[str, str], places: str, home: pathlib.Path) -> list[str] | Failure:
|
|
"""The report of one bench of the checkout the process runs in, with `places` what `larv-review places` prints there."""
|
|
git = functools.partial(git_repo.git, env=env)
|
|
checkout = top(git)
|
|
if isinstance(checkout, Failure):
|
|
return checkout
|
|
sides = commits(given, checkout, git)
|
|
if isinstance(sides, Failure):
|
|
return sides
|
|
print(f"task-bench: base {sides['base'][:12]}, head {sides['head'][:12]}.", file=sys.stderr)
|
|
review = made_review_folder(checkout, home)
|
|
if isinstance(review, Failure):
|
|
return review
|
|
with held(review) as folder:
|
|
if isinstance(folder, Failure):
|
|
return folder
|
|
clear_left(home / REPORTS)
|
|
return bench_in(folder, given, env, places, checkout, sides, git)
|
|
|
|
|
|
def bench_in(folder: pathlib.Path, given: dict[str, str], env: typing.Mapping[str, str], places: str, checkout: str, sides: dict[str, str], git: Git) -> list[str] | Failure:
|
|
"""The report of a bench of the commits sides names, which empties folder and keeps its runs there."""
|
|
containers = Containers(docker_cli(env), f"task-bench-{uuid.uuid4().hex[:12]}", given["--test"], int(env.get(SHARE_VARIABLE) or MEMORY_MIB), f"{os.getuid()}:{os.getgid()}")
|
|
try:
|
|
shutil.rmtree(folder, ignore_errors=True)
|
|
trees: dict[str, str] = {}
|
|
for side, commit in sides.items():
|
|
tree = archived(commit, folder / "trees" / side, checkout, git)
|
|
if isinstance(tree, Failure):
|
|
return tree
|
|
trees[side] = tree
|
|
files = [path for path in place_files(places) if pathlib.Path(trees["base"], path).is_file()]
|
|
if not files:
|
|
return Failure("The places the seats found hard name no Go file that the base holds, so there is nowhere to plant a bug.")
|
|
run = functools.partial(run_tool, prefix=given.get("--prefix"), env=env, killpg=os.killpg)
|
|
mutants = killed_mutants(trees["base"], files, run, CPUS_PER_SLOT)
|
|
if isinstance(mutants, Failure):
|
|
return mutants
|
|
try:
|
|
found = opened(containers, given["--image"]) or benched(given, checkout, sides, trees, mutants, containers, folder, git)
|
|
finally:
|
|
closed(containers)
|
|
except OSError as failure:
|
|
return Failure(f"could not write under {folder}: {failure}")
|
|
return found if isinstance(found, Failure) else [*found, f"{KEPT}{folder}"]
|
|
|
|
|
|
def commits(given: dict[str, str], checkout: str, git: Git) -> dict[str, str] | Failure:
|
|
"""The base's commit and the head's."""
|
|
head = git("rev-parse", "--verify", f"{given.get('--head', 'HEAD')}^{{commit}}", cwd=checkout)
|
|
against = given.get("--base") or default_ref(cwd=checkout, run=git)
|
|
if isinstance(head, Failure):
|
|
return head
|
|
if isinstance(against, Failure):
|
|
return against
|
|
base = git("merge-base", against, head.strip(), cwd=checkout)
|
|
return base if isinstance(base, Failure) else {"base": base.strip(), "head": head.strip()}
|
|
|
|
|
|
def archived(commit: str, into: pathlib.Path, checkout: str, git: Git) -> str | Failure:
|
|
"""Makes `into` the tree at commit, as `git archive` holds it, and returns its path: without git's own data, and without a
|
|
path the project marks `export-ignore`."""
|
|
into.mkdir(parents=True)
|
|
archive = into.with_suffix(".tar")
|
|
wrote = git("archive", "--format=tar", "--output", str(archive), commit, cwd=checkout)
|
|
if isinstance(wrote, Failure):
|
|
return wrote
|
|
try:
|
|
with tarfile.open(archive) as held:
|
|
held.extractall(into, filter="tar")
|
|
except tarfile.TarError as failure:
|
|
return Failure(f"could not unpack the tree at {commit}: {failure}")
|
|
archive.unlink()
|
|
return str(into)
|
|
|
|
|
|
def benched(given: dict[str, str], checkout: str, sides: dict[str, str], trees: dict[str, str], mutants: list[Mutant], containers: Containers, folder: pathlib.Path, git: Git) -> list[str] | Failure:
|
|
"""Plants the bugs, runs each fix, writes the results into folder and returns the report."""
|
|
for side, tree in trees.items():
|
|
ran = tests_in_copy(containers, tree, folder)
|
|
if isinstance(ran, Failure) or ran.code:
|
|
return ran if isinstance(ran, Failure) else failed(f"The tests fail at the {side} without a planted bug: the test command", ran)
|
|
tasks: list[Task] = []
|
|
for mutant in drawn(mutants, int(given["--seed"])):
|
|
if len(tasks) == int(given["--bugs"]):
|
|
break
|
|
task = planted(len(tasks) + 1, mutant, checkout, sides, trees, containers, folder, git)
|
|
if isinstance(task, Failure):
|
|
return task
|
|
tasks += [task] if task else []
|
|
if not tasks:
|
|
return Failure(f"None of the {len(mutants)} mutants the tests kill becomes a bug: each is on a line the head changed, or the test it fails doesn't print a message.")
|
|
results: list[Result] = []
|
|
stopped = None
|
|
total = int(given["--runs"]) * len(tasks) * len(SIDES)
|
|
for number, task, side in itertools.product(range(1, int(given["--runs"]) + 1), tasks, SIDES):
|
|
result = fixed(task, side, number, containers, folder)
|
|
if isinstance(result, Failure):
|
|
stopped = result
|
|
break
|
|
results.append(result)
|
|
print(f"task-bench: run {len(results)} of {total}, bug {task.number} at the {side}: {'passed' if result.passed else 'not passed'}.", file=sys.stderr)
|
|
if stopped and not results:
|
|
return stopped
|
|
names = {task.number: task.named() for task in tasks}
|
|
(folder / "results.json").write_text(json.dumps({"sides": sides, "seed": int(given["--seed"]), "bugs": names, "runs": [recorded(result) for result in results]}, indent="\t") + "\n", encoding="utf-8")
|
|
in_a_tree = lambda path: any(pathlib.Path(tree, path).is_file() for tree in trees.values())
|
|
(folder / DIGEST).write_text(digest(sides, table(results, names), [digested(task, results, folder) for task in tasks], in_a_tree), encoding="utf-8", errors="replace")
|
|
if stopped:
|
|
# The runs that finished took hours and tokens, so the bench shows them before it fails.
|
|
return Failure("\n".join([stopped.reason, f"task-bench stopped after {len(results)} of {total} runs, which results.json holds in {folder}:", "", *table(results, names)]))
|
|
ran_on = sorted({f"{found.model} in Claude Code {found.version}" for result in results if (found := result.figures)})
|
|
short = [] if len(tasks) == int(given["--bugs"]) else [f"Planted {len(tasks)} of the {given['--bugs']} bugs asked for: no other mutant the tests kill becomes one."]
|
|
return [
|
|
f"task-bench: base {sides['base'][:12]}, head {sides['head'][:12]}. {counted(len(tasks), 'bug')}, each run {counted(int(given['--runs']), 'time')} at each side by {' and '.join(ran_on) or 'no run that finished'}.",
|
|
*short,
|
|
"",
|
|
*table(results, names),
|
|
"",
|
|
]
|
|
|
|
|
|
def planted(number: int, mutant: Mutant, checkout: str, sides: dict[str, str], trees: dict[str, str], containers: Containers, folder: pathlib.Path, git: Git) -> Task | Failure | None:
|
|
"""The task of fixing mutant, with its planted tree at each side; None where the mutant doesn't become a bug at one side or the other.
|
|
|
|
The bug is the same change to the same line at both sides, so only the code around it differs between them."""
|
|
diff = git("diff", "-U0", "--no-color", "--no-ext-diff", "--no-textconv", sides["base"], sides["head"], "--", mutant.path, cwd=checkout, errors="replace")
|
|
if isinstance(diff, Failure):
|
|
return diff
|
|
at_head = line_at_head(diff, mutant.line)
|
|
if at_head is None:
|
|
return None
|
|
planted_trees, messages = {}, {}
|
|
for side, placed in (("base", mutant), ("head", mutant._replace(line=at_head))):
|
|
source = pathlib.Path(trees[side], mutant.path)
|
|
changed = mutated(source.read_bytes(), placed) if source.is_file() else None
|
|
if changed is None:
|
|
return None
|
|
tree = folder / "bugs" / str(number) / side
|
|
shutil.rmtree(tree, ignore_errors=True)
|
|
shutil.copytree(trees[side], tree, symlinks=True)
|
|
(tree / mutant.path).write_bytes(changed)
|
|
ran = tests_in_copy(containers, str(tree), folder)
|
|
if isinstance(ran, Failure) and not isinstance(ran, TimedOut):
|
|
return ran
|
|
# Tests that never end under the bug have no message to tell Haiku.
|
|
message = None if isinstance(ran, TimedOut) else task_text(ran.output)
|
|
if message is None:
|
|
return None
|
|
for args in (["init", "--quiet"], ["add", "--all", "--force"], COMMIT):
|
|
done = git(*args, cwd=str(tree))
|
|
if isinstance(done, Failure):
|
|
return done
|
|
planted_trees[side], messages[side] = str(tree), message
|
|
return Task(number, mutant, planted_trees, messages)
|
|
|
|
|
|
def tests_in_copy(containers: Containers, tree: str, folder: pathlib.Path) -> Ran | Failure:
|
|
"""The test command's run in a copy of tree, removed afterwards, so what the tests write stays out of tree."""
|
|
copy = folder / "tested"
|
|
shutil.copytree(tree, copy, symlinks=True)
|
|
try:
|
|
return ran_tests(containers, str(copy))
|
|
finally:
|
|
shutil.rmtree(copy, ignore_errors=True)
|
|
|
|
|
|
def fixed(task: Task, side: str, number: int, containers: Containers, folder: pathlib.Path) -> Result | Failure:
|
|
"""One run: Haiku's fix of task at side in a copy of its planted tree, then the tests over what it changed. The stream stays in
|
|
folder; the copy is removed. Tests that run out of time don't pass."""
|
|
stream = stream_of(Result(task.number, side, number, False, None), folder)
|
|
stream.parent.mkdir(exist_ok=True)
|
|
copy, check = folder / "copy", folder / "check"
|
|
try:
|
|
shutil.copytree(task.trees[side], copy, symlinks=True)
|
|
attempted = fix_attempted(containers, str(copy), task.messages[side], stream)
|
|
if isinstance(attempted, Failure):
|
|
return attempted
|
|
found = figures(stream.read_bytes(), os.path.join(WORK, task.mutant.path))
|
|
if isinstance(found, Failure):
|
|
return found
|
|
checked_tree(task.trees[side], str(copy), str(check))
|
|
ran = ran_tests(containers, str(check))
|
|
if isinstance(ran, Failure) and not isinstance(ran, TimedOut):
|
|
return ran
|
|
return Result(task.number, side, number, isinstance(ran, Ran) and ran.code == 0, found)
|
|
finally:
|
|
shutil.rmtree(copy, ignore_errors=True)
|
|
shutil.rmtree(check, ignore_errors=True)
|
|
|
|
|
|
def stream_of(result: Result, folder: pathlib.Path) -> pathlib.Path:
|
|
"""The file that holds the stream of the run result names. Its name without the suffix is the run's id in the digest."""
|
|
return folder / "runs" / f"bug-{result.task}-{result.side}-{result.run}.jsonl"
|
|
|
|
|
|
def digested(task: Task, results: list[Result], folder: pathlib.Path) -> Bug:
|
|
"""The bug as the digest tells it, with each of its runs in results. A run that didn't finish has no tokens."""
|
|
runs = [Run(stream_of(result, folder).stem, result.passed, found.tokens.context() + found.tokens.output if (found := result.figures) else None, stream_of(result, folder).read_bytes()) for result in results if result.task == task.number]
|
|
return Bug(f"{task.number}: {task.named()}", task.mutant.path, task.messages["base"], runs)
|
|
|
|
|
|
def recorded(result: Result) -> dict:
|
|
"""A run as results.json holds it."""
|
|
run = {"bug": result.task, "side": result.side, "run": result.run, "passed": result.passed, "finished": result.figures is not None}
|
|
if not (found := result.figures):
|
|
return run
|
|
return {**run, "claude_code": found.version, "model": found.model, "tokens": dataclasses.asdict(found.tokens), "turns": found.turns, "seconds": found.seconds, "reads": found.reads}
|
|
|
|
|
|
def counted(count: int, noun: str) -> str:
|
|
return f"{count} {noun}{'' if count == 1 else 's'}"
|