Add task-bench, which has Haiku fix planted bugs at a base and a head and prints what each side took #683
@@ -155,6 +155,9 @@
|
||||
!bin/subagent-rung
|
||||
!bin/subagent_rung/
|
||||
!bin/subagent_rung/*.py
|
||||
!bin/task-bench
|
||||
!bin/task_bench/
|
||||
!bin/task_bench/*.py
|
||||
!bin/test_*.py
|
||||
!bin/text_files/
|
||||
!bin/text_files/*.py
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- `task-bench` is new: it plants bugs that a Go project's tests catch, at the places the comprehension seats found hardest, and has Haiku fix each in a container at a base commit and at a head. It prints for each side how often the fix passes, with the median tokens, turns, wall time and reads before Haiku opens the file the bug is in. `task-bench --help` says how to run it.
|
||||
- larv-review now runs each command that a changed doc line says prints something and hands the real output to the product-owner and prose reviewers, so a claim about output is checked by running it.
|
||||
- `do-epic` moves a parent issue whose sub-tasks a person filed: to done by `jira-finish-transition` once every sub-task is done, and before that by the transition the new setting `jira-sub-tasks-landed-transition` names, once each sub-task is done or in one of `jira-unblocking-statuses`. The transition must lead to one of those statuses. `--dry-run` prints each such move.
|
||||
- mutate-changes now mutates the Go lines a branch changes with Gremlins, as it does Python, Rust, TypeScript and PHP, and prints each mutant the tests do not kill.
|
||||
|
||||
@@ -226,6 +226,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `settings_filter.py` is git's clean filter for `settings.json`: it drops `"model"` and sorts the keys. `.gitattributes` sets the filter `settings` on `settings.json`, and Install's `git config` line makes this script that filter.
|
||||
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
|
||||
- `subagent-rung [--plan] <model> <L>` prints the Agent tool's model and effort for a subagent's task, picked by its L from the project's model on the project's routing ladder, and with `--plan` never below the project's rung; `subagent-rung --help` states the rule.
|
||||
- `task-bench --image <image> --test <shell text> [<options>]` plants bugs that the tests catch, at the places the comprehension seats found hardest, has Haiku fix each at the base and at the head, and prints how often the fix passes and what it took. It needs docker and a token from `claude setup-token`; `task-bench --help` says how it plants, runs and checks.
|
||||
- `todo-item` reads and changes a project's plan, whatever holds it: `list` prints the items in the plan's order and `show <ID>` one item's row and detail; `add …` files an item under the next free ID, `edit <ID> …` sets an item's cells or detail, and `move <ID> --release <version>` sets an item's release; `finish <ID>` removes a finished item and adds the bullet passed with `--changelog` to `CHANGELOG.md`, except for a plan kept in Jira. Each change ranks a markdown plan. `todo-item --help` states the flags.
|
||||
- `touch-predictions record <ID>...` appends what a chunk's items were predicted to touch, under `Touches:` in their details, beside the files and parts its branch touched, to the record the config names as `touch-predictions`; `touch-predictions --help` shows the spelling and how a part is named. `chunk-preflight` reads the same predictions to name the items that may start chunks at once.
|
||||
- `wait-for <file> [<pattern>] [--timeout <seconds>]` waits until a file a background job writes holds a line matching the pattern, and prints it.
|
||||
@@ -274,6 +275,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `review_rules/` holds the larv-review rules that several tools apply: its passes, its review folder, its round commit titles and the comprehension history's format.
|
||||
- `routing_figures/` measures loop-claude's archive, and each of its projects' default branches, for loop-claude's routing calls.
|
||||
- `rung_replay/` holds what `judge_replay` and `review_replay` share: the parallel replay runner, the argument reader, the candidate rungs and the exits, and `judge_replay`'s rung search. It reads `agent_cycle`'s retry count and `chunk_driver`'s models file, and finds that file through `loop_state`'s paths. Its `archive.py` keeps each replay's runs and its report in a `replays/` folder of loop-claude's archive.
|
||||
- `task_bench/` holds `task-bench`, which plants bugs in a Go project and has Haiku fix them in containers.
|
||||
- `test_env.py` serves the tests.
|
||||
- `text_files/` reads a Markdown file's `##` sections, quotes a passage and joins names into a list, reads the text piped into a command, stamps times in UTC, replaces a file in one step, private where it is new, and gzips a file beside its plain name, by which it reads it afterwards.
|
||||
- `transcript_archive/` stores loop-claude's transcripts. Its `project_key.py` names each project's archive folder, loop-claude session and models-file entry.
|
||||
|
||||
@@ -59,28 +59,37 @@ def run_gremlins(top: str, base: str, run: Run, jobs: int) -> list[str] | Failur
|
||||
"""A line with the number of mutants on the changed lines and how many of them Gremlins reports as lived or not covered, then a
|
||||
line per such mutant, as `<path>:<line>:<column>: <status>: <mutator>`.
|
||||
|
||||
Gremlins runs in top and mutates the Go lines the branch changes since base. It skips `*_test.go` itself. It reads the diff from
|
||||
git, so `run` must hold the environment that `with_plain_git_diff` makes. Gremlins exits nonzero on a mutation score below a
|
||||
threshold the project sets, which its exit code doesn't tell from a run that broke, so any nonzero exit fails the run."""
|
||||
Gremlins mutates the Go lines the branch changes since base. It reads the diff from git, so `run` must hold the environment that
|
||||
`with_plain_git_diff` makes."""
|
||||
mutants = gremlins_mutants(top, ["--diff", base], run, jobs)
|
||||
if isinstance(mutants, Failure):
|
||||
return mutants
|
||||
# Gremlins lists the mutants on the unchanged lines of a hunk too, as SKIPPED. It lists the files in no fixed order.
|
||||
unkilled = sorted(mutant for mutant in mutants if mutant.status in GREMLINS_UNKILLED)
|
||||
on_changed_lines = sum(mutant.status != "SKIPPED" for mutant in mutants)
|
||||
return [f"gremlins: {on_changed_lines} mutants on changed lines, {len(unkilled)} not killed.", *(f"{m.path}:{m.line}:{m.column}: {m.status}: {m.mutator}" for m in unkilled)]
|
||||
|
||||
|
||||
def gremlins_mutants(top: str, arguments: list[str], run: Run, jobs: int) -> list[Mutant] | Failure:
|
||||
"""Each mutant in the report of `gremlins unleash <arguments>`, which runs in top.
|
||||
|
||||
Gremlins skips `*_test.go` itself. It exits nonzero on a mutation score below a threshold the project sets, which its exit code
|
||||
doesn't tell from a run that broke, so any nonzero exit fails the run."""
|
||||
scratch = scratch_folder(top)
|
||||
if isinstance(scratch, Failure):
|
||||
return scratch
|
||||
try:
|
||||
report = pathlib.Path(scratch, "report.json")
|
||||
ran = run(["gremlins", "unleash", "--diff", base, "--output", os.path.join(os.path.relpath(scratch, top), report.name), "--workers", str(jobs)], top)
|
||||
ran = run(["gremlins", "unleash", *arguments, "--output", os.path.join(os.path.relpath(scratch, top), report.name), "--workers", str(jobs)], top)
|
||||
if isinstance(ran, Failure) or ran.code:
|
||||
return ran if isinstance(ran, Failure) else whole_output("Gremlins", ran)
|
||||
# Gremlins writes no report where it finds no mutant, and `null` for a list with no entry.
|
||||
files = (json.loads(report.read_text(encoding="utf-8"))["files"] or []) if report.is_file() else []
|
||||
mutants = [Mutant(file["file_name"], mutant["line"], mutant["column"], mutant["status"], mutant["type"]) for file in files for mutant in file["mutations"] or []]
|
||||
return [Mutant(file["file_name"], mutant["line"], mutant["column"], mutant["status"], mutant["type"]) for file in files for mutant in file["mutations"] or []]
|
||||
except (OSError, ValueError, KeyError, TypeError) as failure:
|
||||
return Failure(f"could not read Gremlins' report in {scratch}: {failure}")
|
||||
finally:
|
||||
remove(scratch)
|
||||
# Gremlins lists the mutants on the unchanged lines of a hunk too, as SKIPPED. It lists the files in no fixed order.
|
||||
unkilled = sorted(mutant for mutant in mutants if mutant.status in GREMLINS_UNKILLED)
|
||||
on_changed_lines = sum(mutant.status != "SKIPPED" for mutant in mutants)
|
||||
return [f"gremlins: {on_changed_lines} mutants on changed lines, {len(unkilled)} not killed.", *(f"{m.path}:{m.line}:{m.column}: {m.status}: {m.mutator}" for m in unkilled)]
|
||||
|
||||
|
||||
def with_plain_git_diff(env: typing.Mapping[str, str]) -> dict[str, str]:
|
||||
|
||||
Executable
+15
@@ -0,0 +1,15 @@
|
||||
#!/usr/bin/env python3
|
||||
import os
|
||||
import pathlib
|
||||
import signal
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
|
||||
from task_bench import main
|
||||
|
||||
# Python ends on SIGTERM or SIGHUP without running a finally, which would leave the bench's containers, network and image. As
|
||||
# SystemExit, the run unwinds as on Ctrl-C.
|
||||
for number in (signal.SIGHUP, signal.SIGTERM):
|
||||
signal.signal(number, lambda number, frame: sys.exit(128 + number))
|
||||
sys.exit(main.main(sys.argv[1:], os.environ))
|
||||
@@ -0,0 +1,50 @@
|
||||
"""An HTTP proxy that tunnels `CONNECT` to the one `host:port` its first argument names and answers 403 to every other request.
|
||||
|
||||
task-bench runs it in a container of its own, as `python api_proxy.py <host:port> [<listen port>]`: the containers Haiku works in sit
|
||||
on a network with no route out, and reach the API through it and nothing else. It imports only the standard library, since that
|
||||
container doesn't hold anything more."""
|
||||
import asyncio
|
||||
import contextlib
|
||||
import sys
|
||||
|
||||
PORT = 8080
|
||||
|
||||
|
||||
async def served(allowed: str, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
|
||||
"""Answers one client: a tunnel to `allowed` where its request is `CONNECT <allowed>`, else 403."""
|
||||
with contextlib.suppress(OSError, asyncio.IncompleteReadError, asyncio.LimitOverrunError):
|
||||
try:
|
||||
request = (await reader.readuntil(b"\r\n\r\n")).split(b"\r\n")[0].decode("latin-1").split()
|
||||
if len(request) != 3 or request[0] != "CONNECT" or request[1] != allowed:
|
||||
writer.write(b"HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\n\r\n")
|
||||
await writer.drain()
|
||||
return
|
||||
host, _, port = allowed.rpartition(":")
|
||||
there_reader, there_writer = await asyncio.open_connection(host, int(port))
|
||||
writer.write(b"HTTP/1.1 200 Connection established\r\n\r\n")
|
||||
await writer.drain()
|
||||
# Either side closing ends the tunnel, so the other copy is cancelled.
|
||||
copies = [asyncio.ensure_future(copied(reader, there_writer)), asyncio.ensure_future(copied(there_reader, writer))]
|
||||
await asyncio.wait(copies, return_when=asyncio.FIRST_COMPLETED)
|
||||
for copy in copies:
|
||||
copy.cancel()
|
||||
there_writer.close()
|
||||
finally:
|
||||
writer.close()
|
||||
|
||||
|
||||
async def copied(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
|
||||
with contextlib.suppress(OSError):
|
||||
while data := await reader.read(65536):
|
||||
writer.write(data)
|
||||
await writer.drain()
|
||||
|
||||
|
||||
async def serve(allowed: str, port: int) -> None:
|
||||
server = await asyncio.start_server(lambda reader, writer: served(allowed, reader, writer), "0.0.0.0", port)
|
||||
async with server:
|
||||
await server.serve_forever()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(serve(sys.argv[1], int(sys.argv[2]) if len(sys.argv) > 2 else PORT))
|
||||
@@ -0,0 +1,116 @@
|
||||
"""What task-bench measures: the figures of one run, read from the stream its claude printed, and the table of a bench's runs."""
|
||||
import dataclasses
|
||||
import statistics
|
||||
import typing
|
||||
|
||||
from failure import Failure
|
||||
from claude_code.entries import json_object, parse_entry, tool_calls, ToolCall
|
||||
from reading_cost.measure import reads_of
|
||||
from claude_code.tokens import Tokens, tokens_of
|
||||
|
||||
# Where each run's container mounts its copy of the code, and its home folder: the paths in a run's stream start there.
|
||||
WORK = "/work"
|
||||
HOME = "/tmp"
|
||||
# The tools that read or search without a shell. A Bash call counts where `reads_of` finds a file it reads.
|
||||
READING_TOOLS = frozenset({"Glob", "Grep", "Read"})
|
||||
SIDES = ("base", "head")
|
||||
HEADER = ["| Bug | Side | Passed | Input | Cache write | Cache read | Output | Turns | Wall, s | Reads |", "|---|---|---|---|---|---|---|---|---|---|"]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Figures:
|
||||
"""One finished run, as its stream tells it. reads counts the reads and searches before the first open of the file the bug is
|
||||
in, and is None where the run never opened it."""
|
||||
version: str
|
||||
model: str
|
||||
tokens: Tokens
|
||||
turns: int
|
||||
seconds: float
|
||||
reads: int | None
|
||||
|
||||
|
||||
class Result(typing.NamedTuple):
|
||||
"""One run of a task at a side. figures is None where its claude ended without a result, as when it ran out of time."""
|
||||
|
||||
task: int
|
||||
side: str
|
||||
run: int
|
||||
passed: bool
|
||||
figures: Figures | None
|
||||
|
||||
|
||||
def figures(stream: bytes, bug_file: str) -> Figures | None | Failure:
|
||||
"""The figures of the run whose claude printed stream, with bug_file the bug's file as the run's container names it.
|
||||
|
||||
None where the stream holds no result. A Failure where the API refused the run, as on a token it doesn't take: such a run
|
||||
doesn't measure anything, and every later run would end the same."""
|
||||
events = [event for line in stream.splitlines() if (event := json_object(line))]
|
||||
result = next((event for event in reversed(events) if event.get("type") == "result"), None)
|
||||
if result is None:
|
||||
return None
|
||||
if result.get("api_error_status"):
|
||||
return Failure(f"claude could not use the API: {result.get('result')}")
|
||||
# {"type": "system", "subtype": "init", "model": "claude-haiku-5-5", "claude_code_version": "2.1.330", ...}
|
||||
init = next((event for event in events if event.get("subtype") == "init"), {})
|
||||
entries = [entry for line in stream.splitlines() if (entry := parse_entry(line))]
|
||||
calls = [call for _, call in tool_calls(entries)]
|
||||
return Figures(str(init.get("claude_code_version", "")), str(init.get("model", "")), tokens_of(result.get("usage")), int(result.get("num_turns") or 0), (result.get("duration_ms") or 0) / 1000, reads_before(calls, bug_file))
|
||||
|
||||
|
||||
def reads_before(calls: list[ToolCall], bug_file: str) -> int | None:
|
||||
count = 0
|
||||
for call in calls:
|
||||
read = reads_of(call, WORK, HOME).read
|
||||
if bug_file in read:
|
||||
return count
|
||||
count += call.name in READING_TOOLS or bool(read)
|
||||
return None
|
||||
|
||||
|
||||
def table(results: list[Result], tasks: dict[int, str]) -> list[str]:
|
||||
"""A row per task and side, then one per side over every task, then the head's change against the base. Each figure is the
|
||||
median over the side's runs that have figures, and Reads over those that opened the bug's file."""
|
||||
rows = [row(f"{task}: {tasks[task]}", side, [result for result in results if (result.task, result.side) == (task, side)]) for task in sorted(tasks) for side in SIDES]
|
||||
base, head = ([result for result in results if result.side == side] for side in SIDES)
|
||||
return [*HEADER, *rows, row("All", "base", base), row("All", "head", head), change(base, head)]
|
||||
|
||||
|
||||
def medians(results: list[Result]) -> list[float | None]:
|
||||
"""Input, cache write, cache read, output, turns, wall time and reads."""
|
||||
measured = [result.figures for result in results if result.figures]
|
||||
columns: list[list[float]] = [
|
||||
[found.tokens.input for found in measured],
|
||||
[found.tokens.write_5m + found.tokens.write_1h for found in measured],
|
||||
[found.tokens.read for found in measured],
|
||||
[found.tokens.output for found in measured],
|
||||
[found.turns for found in measured],
|
||||
[found.seconds for found in measured],
|
||||
[found.reads for found in measured if found.reads is not None],
|
||||
]
|
||||
return [statistics.median(column) if column else None for column in columns]
|
||||
|
||||
|
||||
def row(task: str, side: str, results: list[Result]) -> str:
|
||||
passed = f"{sum(result.passed for result in results)}/{len(results)}"
|
||||
return cells([task, side, passed, *("-" if median is None else number(median) for median in medians(results))])
|
||||
|
||||
|
||||
def change(base: list[Result], head: list[Result]) -> str:
|
||||
"""The head's row less the base's: a count as the difference, and tokens and wall time as that difference's share of the base's."""
|
||||
passed = sum(result.passed for result in head) - sum(result.passed for result in base)
|
||||
pairs = list(zip(medians(base), medians(head)))
|
||||
shares = [None if was is None or now is None or not was else f"{number(round(100 * (now - was) / was), signed=True)} %" for was, now in pairs]
|
||||
counts = [None if was is None or now is None else number(now - was, signed=True) for was, now in pairs]
|
||||
# Turns and reads are small counts, where a share misleads.
|
||||
mixed = [*shares[:4], counts[4], shares[5], counts[6]]
|
||||
return cells(["All", "change", number(passed, signed=True), *(found or "-" for found in mixed)])
|
||||
|
||||
|
||||
def number(value: float, signed: bool = False) -> str:
|
||||
"""A whole number as it is, another with one decimal; signed, with a + before a rise."""
|
||||
text = str(int(value)) if value == int(value) else f"{value:.1f}"
|
||||
return f"+{text}" if signed and value > 0 else text
|
||||
|
||||
|
||||
def cells(found: list[str]) -> str:
|
||||
return f"| {' | '.join(found)} |"
|
||||
@@ -0,0 +1,323 @@
|
||||
r"""Measure how hard a project's code is to fix a bug in, at a base commit and at a head: plant bugs its tests catch, have Haiku fix
|
||||
each several times at both, and print how often the fix passes and what it took.
|
||||
|
||||
usage: task-bench --image <image> --test <shell text> [--base <ref>] [--head <ref>] [--prefix <shell text>] [--seed <n>] [--bugs <n>] [--runs <n>]
|
||||
|
||||
Run it in a checkout, under ~/.claude/bin/run-heavy, with a long-lived token in CLAUDE_CODE_OAUTH_TOKEN, which `claude setup-token`
|
||||
prints. The head is --head, by default HEAD. The base is the merge base of the head and --base, by default the default branch:
|
||||
origin's as last fetched, else main or master. Where --base names the head's own commit, the bench runs the head against itself.
|
||||
No model judges anything: the bench runs programs, and reads Haiku's own stream.
|
||||
|
||||
- Plant. It copies the tree at the base and runs Gremlins there, as `gremlins unleash`, over the Go files that hold the places
|
||||
`larv-review places` prints, the ones the comprehension seats found hardest. It takes the mutants the tests kill in the order
|
||||
--seed sets, 1 by default, until --bugs of them, 5 by default, have become bugs. A mutant becomes a bug where its line stands
|
||||
unchanged at the head, and where, planted at each side, a test fails and prints a message that says more than the test's name.
|
||||
Any other mutant is passed over, so a change to the very line of a bug is not benched. Go is the one language so far.
|
||||
- Run. Haiku fixes each bug --runs times at each side, 5 by default, the sides in turn. A fix starts from a copy of that side's
|
||||
tree with the bug in it, committed once, so git doesn't show which line the bug is on. In the copy, in a container, `claude -p
|
||||
--strict-mcp-config --disable-slash-commands --no-session-persistence --model claude-haiku-5-5 --effort low` works with every
|
||||
tool. It is told the message the failing test printed at that side, without the test's name, file or line. The container's
|
||||
Claude home is new and empty, so the project's own CLAUDE.md, or AGENTS.md, in the copy is all the setup that loads. Its network
|
||||
reaches api.anthropic.com and nothing else. A fix ends after 30 minutes.
|
||||
- Check. The test command runs over the planted tree with each file that is no test as Haiku left it; a fix passes where it exits 0.
|
||||
A test Haiku changed, removed or added doesn't count, and neither does a file under a `testdata` folder.
|
||||
|
||||
--image is the project's development image, such as golang:1.27.4. The bench adds Claude Code to it, as its installer installs it,
|
||||
so the image needs curl and bash; the result names the version. Haiku works in that image. So does --test: sh runs that text at
|
||||
the copy's top, with no network, so the image must hold what the tests would otherwise download.
|
||||
--prefix is shell text that runs Gremlins, on the host, as mutate-changes' --prefix does: sh runs `<prefix> "$@"` in the copy of the
|
||||
base, which $PWD names, with `gremlins unleash ...` as "$@". Without it, the bench runs the `gremlins` on the host's PATH. For
|
||||
example:
|
||||
|
||||
docker run --rm --memory 8g --memory-swap 8g -v "$PWD:$PWD" -w "$PWD" golang:1.27.4 \
|
||||
sh -c 'shift; exec go run github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0 "$@"' sh
|
||||
|
||||
It prints the base and the head, then a table: a row per bug and side with how many fixes passed and the medians, over the runs
|
||||
that finished, of the input, cache-write, cache-read and output tokens, the turns, the wall time, and the reads and searches before
|
||||
the run first opened the file the bug is in. The median of the reads leaves out a run that never opened that file. Then come a row
|
||||
per side over every bug, and the head's change against the base. The last line names the folder that holds each run's result and
|
||||
stream: `task-bench` in the chunk's review folder. Each bench empties that `task-bench` folder first.
|
||||
|
||||
Exit 1, saying why, on an unknown or repeated argument, a missing --image or --test, a --seed, --bugs or --runs that is no whole
|
||||
number from 1, or no token; and where git, docker or Gremlins fails, the seats' places name no Go file, the tests fail at a side
|
||||
without a bug, or no mutant becomes a bug. Where the API refuses a run, the bench stops there and exits 1 too, after the table of
|
||||
the runs that finished.
|
||||
"""
|
||||
import dataclasses
|
||||
import functools
|
||||
import itertools
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import shutil
|
||||
import sys
|
||||
import tarfile
|
||||
import typing
|
||||
import uuid
|
||||
|
||||
from task_bench.runs import checked_tree, closed, Containers, docker_cli, fix_attempted, MEMORY_MIB, opened, ran_tests, TimedOut, TOKEN
|
||||
from machine_load.slots import CPUS_PER_SLOT
|
||||
from git_cli.git_repo import default_ref, Git, top
|
||||
from task_bench.plant import drawn, killed_mutants, line_at_head, Mutant, mutated, place_files, task_text
|
||||
from mutate_changes.tool_process import failed, Ran, run_tool
|
||||
from failure import Failure
|
||||
from task_bench.figures import figures, Result, SIDES, table, WORK
|
||||
from git_cli import git_repo
|
||||
from review_rules.review_folder import made_review_folder
|
||||
from larv_review.history_commands import read_places
|
||||
from run_heavy.capped_docker import SHARE_VARIABLE
|
||||
|
||||
USAGE = "usage: task-bench --image <image> --test <shell text> [--base <ref>] [--head <ref>] [--prefix <shell text>] [--seed <n>] [--bugs <n>] [--runs <n>]"
|
||||
# A panel of three readers, seated as the README's audience, picked these options on 2026-11-21, each 3 to 0.
|
||||
OPTIONS = ("--base", "--bugs", "--head", "--image", "--prefix", "--runs", "--seed", "--test")
|
||||
COUNTS = {"--bugs": "5", "--runs": "5", "--seed": "1"}
|
||||
# The planted commit doesn't name anyone, and neither a hook nor a signing key of the caller's reaches it.
|
||||
COMMIT = ["-c", "user.name=task-bench", "-c", "user.email=task-bench@localhost", "-c", "commit.gpgsign=false", "-c", "core.hooksPath=/dev/null", "commit", "--quiet", "--message", "Import"]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Task:
|
||||
"""A bug to fix: its number from 1, the mutant at the base, and per side the planted tree and what Haiku is told there."""
|
||||
number: int
|
||||
mutant: Mutant
|
||||
trees: dict[str, str]
|
||||
messages: dict[str, str]
|
||||
|
||||
def named(self) -> str:
|
||||
return f"{self.mutant.path}:{self.mutant.line}:{self.mutant.column} {self.mutant.mutator}"
|
||||
|
||||
|
||||
def main(argv: list[str], env: typing.Mapping[str, str], read: typing.Callable[[pathlib.Path], str | Failure] = read_places) -> int:
|
||||
"""Runs task-bench with the arguments argv; git, docker and Gremlins run in the environment env, and `read` reads what
|
||||
`larv-review places` prints for the checkout."""
|
||||
if argv in (["-h"], ["--help"]):
|
||||
print(__doc__.strip())
|
||||
return 0
|
||||
given = options(argv, env)
|
||||
if isinstance(given, Failure):
|
||||
print(f"{given.reason}; {USAGE}", file=sys.stderr)
|
||||
return 1
|
||||
home = pathlib.Path(env.get("HOME") or pathlib.Path.home())
|
||||
places = read(home)
|
||||
report = places if isinstance(places, Failure) else bench(given, env, places, home)
|
||||
if isinstance(report, Failure):
|
||||
print(report.reason, file=sys.stderr)
|
||||
return 1
|
||||
print("\n".join(report))
|
||||
return 0
|
||||
|
||||
|
||||
def options(argv: list[str], env: typing.Mapping[str, str]) -> dict[str, str] | Failure:
|
||||
given: dict[str, str] = {}
|
||||
rest = list(argv)
|
||||
while rest:
|
||||
name, equals, value = rest.pop(0).partition("=")
|
||||
if name not in OPTIONS:
|
||||
return Failure(f"unknown argument {name}")
|
||||
if name in given:
|
||||
return Failure(f"{name} is given twice")
|
||||
if not equals and not rest:
|
||||
return Failure(f"{name} needs a value")
|
||||
given[name] = value if equals else rest.pop(0)
|
||||
for name in ("--image", "--test"):
|
||||
if not given.get(name):
|
||||
return Failure(f"{name} is missing")
|
||||
for name, default in COUNTS.items():
|
||||
if not given.setdefault(name, default).isdecimal() or int(given[name]) < 1:
|
||||
return Failure(f"{name} takes a whole number from 1, not {given[name]}")
|
||||
if not env.get(TOKEN):
|
||||
return Failure(f"{TOKEN} holds no token")
|
||||
return given
|
||||
|
||||
|
||||
def bench(given: dict[str, str], env: typing.Mapping[str, str], places: str, home: pathlib.Path) -> list[str] | Failure:
|
||||
"""The report of one bench of the checkout the process runs in, with `places` what `larv-review places` prints there."""
|
||||
git = functools.partial(git_repo.git, env=env)
|
||||
checkout = top(git)
|
||||
if isinstance(checkout, Failure):
|
||||
return checkout
|
||||
sides = commits(given, checkout, git)
|
||||
if isinstance(sides, Failure):
|
||||
return sides
|
||||
print(f"task-bench: base {sides['base'][:12]}, head {sides['head'][:12]}.", file=sys.stderr)
|
||||
review = made_review_folder(checkout, home)
|
||||
if isinstance(review, Failure):
|
||||
return review
|
||||
folder = review / "task-bench"
|
||||
containers = Containers(docker_cli(env), f"task-bench-{uuid.uuid4().hex[:12]}", given["--test"], int(env.get(SHARE_VARIABLE) or MEMORY_MIB), f"{os.getuid()}:{os.getgid()}")
|
||||
try:
|
||||
shutil.rmtree(folder, ignore_errors=True)
|
||||
trees: dict[str, str] = {}
|
||||
for side, commit in sides.items():
|
||||
tree = archived(commit, folder / "trees" / side, checkout, git)
|
||||
if isinstance(tree, Failure):
|
||||
return tree
|
||||
trees[side] = tree
|
||||
files = [path for path in place_files(places) if pathlib.Path(trees["base"], path).is_file()]
|
||||
if not files:
|
||||
return Failure("The places the seats found hard name no Go file that the base holds, so there is nowhere to plant a bug.")
|
||||
run = functools.partial(run_tool, prefix=given.get("--prefix"), env=env, killpg=os.killpg)
|
||||
mutants = killed_mutants(trees["base"], files, run, CPUS_PER_SLOT)
|
||||
if isinstance(mutants, Failure):
|
||||
return mutants
|
||||
try:
|
||||
found = opened(containers, given["--image"]) or benched(given, checkout, sides, trees, mutants, containers, folder, git)
|
||||
finally:
|
||||
closed(containers)
|
||||
except OSError as failure:
|
||||
return Failure(f"could not write under {folder}: {failure}")
|
||||
return found if isinstance(found, Failure) else [*found, f"Each run's result and stream: {folder}"]
|
||||
|
||||
|
||||
def commits(given: dict[str, str], checkout: str, git: Git) -> dict[str, str] | Failure:
|
||||
"""The base's commit and the head's."""
|
||||
head = git("rev-parse", "--verify", f"{given.get('--head', 'HEAD')}^{{commit}}", cwd=checkout)
|
||||
against = given.get("--base") or default_ref(cwd=checkout, run=git)
|
||||
if isinstance(head, Failure):
|
||||
return head
|
||||
if isinstance(against, Failure):
|
||||
return against
|
||||
base = git("merge-base", against, head.strip(), cwd=checkout)
|
||||
return base if isinstance(base, Failure) else {"base": base.strip(), "head": head.strip()}
|
||||
|
||||
|
||||
def archived(commit: str, into: pathlib.Path, checkout: str, git: Git) -> str | Failure:
|
||||
"""Makes `into` the tree at commit, as `git archive` holds it, and returns its path: without git's own data, and without a
|
||||
path the project marks `export-ignore`."""
|
||||
into.mkdir(parents=True)
|
||||
archive = into.with_suffix(".tar")
|
||||
wrote = git("archive", "--format=tar", "--output", str(archive), commit, cwd=checkout)
|
||||
if isinstance(wrote, Failure):
|
||||
return wrote
|
||||
try:
|
||||
with tarfile.open(archive) as held:
|
||||
held.extractall(into, filter="tar")
|
||||
except tarfile.TarError as failure:
|
||||
return Failure(f"could not unpack the tree at {commit}: {failure}")
|
||||
archive.unlink()
|
||||
return str(into)
|
||||
|
||||
|
||||
def benched(given: dict[str, str], checkout: str, sides: dict[str, str], trees: dict[str, str], mutants: list[Mutant], containers: Containers, folder: pathlib.Path, git: Git) -> list[str] | Failure:
|
||||
"""Plants the bugs, runs each fix, writes the results into folder and returns the report."""
|
||||
for side, tree in trees.items():
|
||||
ran = tests_in_copy(containers, tree, folder)
|
||||
if isinstance(ran, Failure) or ran.code:
|
||||
return ran if isinstance(ran, Failure) else failed(f"The tests fail at the {side} without a planted bug: the test command", ran)
|
||||
tasks: list[Task] = []
|
||||
for mutant in drawn(mutants, int(given["--seed"])):
|
||||
if len(tasks) == int(given["--bugs"]):
|
||||
break
|
||||
task = planted(len(tasks) + 1, mutant, checkout, sides, trees, containers, folder, git)
|
||||
if isinstance(task, Failure):
|
||||
return task
|
||||
tasks += [task] if task else []
|
||||
if not tasks:
|
||||
return Failure(f"None of the {len(mutants)} mutants the tests kill becomes a bug: each is on a line the head changed, or the test it fails doesn't print a message.")
|
||||
results: list[Result] = []
|
||||
stopped = None
|
||||
total = int(given["--runs"]) * len(tasks) * len(SIDES)
|
||||
for number, task, side in itertools.product(range(1, int(given["--runs"]) + 1), tasks, SIDES):
|
||||
result = fixed(task, side, number, containers, folder)
|
||||
if isinstance(result, Failure):
|
||||
stopped = result
|
||||
break
|
||||
results.append(result)
|
||||
print(f"task-bench: run {len(results)} of {total}, bug {task.number} at the {side}: {'passed' if result.passed else 'not passed'}.", file=sys.stderr)
|
||||
if stopped and not results:
|
||||
return stopped
|
||||
names = {task.number: task.named() for task in tasks}
|
||||
(folder / "results.json").write_text(json.dumps({"sides": sides, "seed": int(given["--seed"]), "bugs": names, "runs": [recorded(result) for result in results]}, indent="\t") + "\n", encoding="utf-8")
|
||||
if stopped:
|
||||
# The runs that finished took hours and tokens, so the bench shows them before it fails.
|
||||
return Failure("\n".join([stopped.reason, f"task-bench stopped after {len(results)} of {total} runs, which results.json holds in {folder}:", "", *table(results, names)]))
|
||||
ran_on = sorted({f"{found.model} in Claude Code {found.version}" for result in results if (found := result.figures)})
|
||||
short = [] if len(tasks) == int(given["--bugs"]) else [f"Planted {len(tasks)} of the {given['--bugs']} bugs asked for: no other mutant the tests kill becomes one."]
|
||||
return [
|
||||
f"task-bench: base {sides['base'][:12]}, head {sides['head'][:12]}. {counted(len(tasks), 'bug')}, each run {counted(int(given['--runs']), 'time')} at each side by {' and '.join(ran_on) or 'no run that finished'}.",
|
||||
*short,
|
||||
"",
|
||||
*table(results, names),
|
||||
"",
|
||||
]
|
||||
|
||||
|
||||
def planted(number: int, mutant: Mutant, checkout: str, sides: dict[str, str], trees: dict[str, str], containers: Containers, folder: pathlib.Path, git: Git) -> Task | Failure | None:
|
||||
"""The task of fixing mutant, with its planted tree at each side; None where the mutant doesn't become a bug at one side or the other.
|
||||
|
||||
The bug is the same change to the same line at both sides, so only the code around it differs between them."""
|
||||
diff = git("diff", "-U0", "--no-color", "--no-ext-diff", "--no-textconv", sides["base"], sides["head"], "--", mutant.path, cwd=checkout, errors="replace")
|
||||
if isinstance(diff, Failure):
|
||||
return diff
|
||||
at_head = line_at_head(diff, mutant.line)
|
||||
if at_head is None:
|
||||
return None
|
||||
planted_trees, messages = {}, {}
|
||||
for side, placed in (("base", mutant), ("head", mutant._replace(line=at_head))):
|
||||
source = pathlib.Path(trees[side], mutant.path)
|
||||
changed = mutated(source.read_bytes(), placed) if source.is_file() else None
|
||||
if changed is None:
|
||||
return None
|
||||
tree = folder / "bugs" / str(number) / side
|
||||
shutil.rmtree(tree, ignore_errors=True)
|
||||
shutil.copytree(trees[side], tree, symlinks=True)
|
||||
(tree / mutant.path).write_bytes(changed)
|
||||
ran = tests_in_copy(containers, str(tree), folder)
|
||||
if isinstance(ran, Failure) and not isinstance(ran, TimedOut):
|
||||
return ran
|
||||
# Tests that never end under the bug have no message to tell Haiku.
|
||||
message = None if isinstance(ran, TimedOut) else task_text(ran.output)
|
||||
if message is None:
|
||||
return None
|
||||
for args in (["init", "--quiet"], ["add", "--all", "--force"], COMMIT):
|
||||
done = git(*args, cwd=str(tree))
|
||||
if isinstance(done, Failure):
|
||||
return done
|
||||
planted_trees[side], messages[side] = str(tree), message
|
||||
return Task(number, mutant, planted_trees, messages)
|
||||
|
||||
|
||||
def tests_in_copy(containers: Containers, tree: str, folder: pathlib.Path) -> Ran | Failure:
|
||||
"""The test command's run in a copy of tree, removed afterwards, so what the tests write stays out of tree."""
|
||||
copy = folder / "tested"
|
||||
shutil.copytree(tree, copy, symlinks=True)
|
||||
try:
|
||||
return ran_tests(containers, str(copy))
|
||||
finally:
|
||||
shutil.rmtree(copy, ignore_errors=True)
|
||||
|
||||
|
||||
def fixed(task: Task, side: str, number: int, containers: Containers, folder: pathlib.Path) -> Result | Failure:
|
||||
"""One run: Haiku's fix of task at side in a copy of its planted tree, then the tests over what it changed. The stream stays in
|
||||
folder; the copy is removed. Tests that run out of time don't pass."""
|
||||
stream = folder / "runs" / f"bug-{task.number}-{side}-{number}.jsonl"
|
||||
stream.parent.mkdir(exist_ok=True)
|
||||
copy, check = folder / "copy", folder / "check"
|
||||
try:
|
||||
shutil.copytree(task.trees[side], copy, symlinks=True)
|
||||
attempted = fix_attempted(containers, str(copy), task.messages[side], stream)
|
||||
if isinstance(attempted, Failure):
|
||||
return attempted
|
||||
found = figures(stream.read_bytes(), os.path.join(WORK, task.mutant.path))
|
||||
if isinstance(found, Failure):
|
||||
return found
|
||||
checked_tree(task.trees[side], str(copy), str(check))
|
||||
ran = ran_tests(containers, str(check))
|
||||
if isinstance(ran, Failure) and not isinstance(ran, TimedOut):
|
||||
return ran
|
||||
return Result(task.number, side, number, isinstance(ran, Ran) and ran.code == 0, found)
|
||||
finally:
|
||||
shutil.rmtree(copy, ignore_errors=True)
|
||||
shutil.rmtree(check, ignore_errors=True)
|
||||
|
||||
|
||||
def recorded(result: Result) -> dict:
|
||||
"""A run as results.json holds it."""
|
||||
run = {"bug": result.task, "side": result.side, "run": result.run, "passed": result.passed, "finished": result.figures is not None}
|
||||
if not (found := result.figures):
|
||||
return run
|
||||
return {**run, "claude_code": found.version, "model": found.model, "tokens": dataclasses.asdict(found.tokens), "turns": found.turns, "seconds": found.seconds, "reads": found.reads}
|
||||
|
||||
|
||||
def counted(count: int, noun: str) -> str:
|
||||
return f"{count} {noun}{'' if count == 1 else 's'}"
|
||||
@@ -0,0 +1,133 @@
|
||||
"""Pick the bugs task-bench plants: Gremlins' killed mutants in the Go files that hold the places the comprehension seats found hard,
|
||||
re-applied from its report, each with the failing assertion's message that tells Haiku what goes wrong."""
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import typing
|
||||
|
||||
from failure import Failure
|
||||
from mutate_changes.in_diff_tools import gremlins_mutants
|
||||
from mutate_changes.changed_lines import is_test
|
||||
from mutate_changes.tool_process import Run
|
||||
|
||||
# A place as `larv-review places` prints it: `<path>:<line>` in backticks.
|
||||
PLACE = re.compile(r"`([^`\s:]+):\d+`")
|
||||
# What each of Gremlins v0.6.0's mutators writes in place of a token: `tokenMutations` in its internal/engine/mappings.go, under the
|
||||
# names its report gives the mutators, from internal/mutator/mutator.go.
|
||||
MUTATIONS: dict[str, dict[bytes, bytes]] = {
|
||||
"ARITHMETIC_BASE": {b"+": b"-", b"*": b"/", b"/": b"*", b"%": b"*", b"-": b"+"},
|
||||
"CONDITIONALS_BOUNDARY": {b">=": b">", b">": b">=", b"<=": b"<", b"<": b"<="},
|
||||
"CONDITIONALS_NEGATION": {b"==": b"!=", b">=": b"<", b">": b"<=", b"<=": b">", b"<": b">=", b"!=": b"=="},
|
||||
"INCREMENT_DECREMENT": {b"--": b"++", b"++": b"--"},
|
||||
"INVERT_ASSIGNMENTS": {b"+=": b"-=", b"*=": b"/=", b"/=": b"*=", b"-=": b"+="},
|
||||
"INVERT_BITWISE": {b"&": b"|", b"|": b"&", b"^": b"&", b"&^": b"&", b"<<": b">>", b">>": b"<<"},
|
||||
"INVERT_BWASSIGN": {b"&=": b"|=", b"|=": b"&=", b"^=": b"&=", b"&^=": b"&=", b"<<=": b">>=", b">>=": b"<<="},
|
||||
"INVERT_LOGICAL": {b"&&": b"||", b"||": b"&&"},
|
||||
"INVERT_LOOPCTRL": {b"break": b"continue", b"continue": b"break"},
|
||||
"INVERT_NEGATIVES": {b"-": b"+"},
|
||||
"REMOVE_SELF_ASSIGNMENTS": dict.fromkeys((b"+=", b"&=", b"&^=", b"*=", b"|=", b"/=", b"%=", b"<<=", b">>=", b"-=", b"^="), b"="),
|
||||
}
|
||||
HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@", re.MULTILINE)
|
||||
# What a test printed, as go test shows it: ` pick_test.go:42: got 3, want 4`. A failed assertion and a log line look the same.
|
||||
PRINTED = re.compile(r"^( *)\S*_test\.go:\d+:(.*)$")
|
||||
# The lines that say whose output follows: `=== RUN TestPick` with -v, and `--- FAIL: TestPick (0.00s)` with and without it.
|
||||
TEST_MARK = re.compile(r"^( *)(?:=== (?:RUN|CONT|NAME)|--- (?:FAIL|PASS|SKIP):)\s+(\S+)")
|
||||
FAILED_TEST = re.compile(r"^\s*--- FAIL: (\S+)", re.MULTILINE)
|
||||
|
||||
|
||||
class Mutant(typing.NamedTuple):
|
||||
"""A mutant the project's tests killed, as Gremlins reports it: the mutator and the token it changed, by its line and its column in bytes, both from 1."""
|
||||
|
||||
path: str
|
||||
line: int
|
||||
column: int
|
||||
mutator: str
|
||||
|
||||
|
||||
def place_files(places: str) -> list[str]:
|
||||
"""The Go files that hold a place in what `larv-review places` prints, test files left out, in the order of their paths."""
|
||||
return sorted({path for path in PLACE.findall(places) if path.endswith(".go") and not is_test(path)})
|
||||
|
||||
|
||||
def killed_mutants(top: str, files: list[str], run: Run, jobs: int) -> list[Mutant] | Failure:
|
||||
"""The mutants in `files` that the tests of the tree at top kill, as Gremlins finds them there.
|
||||
|
||||
Gremlins takes no list of files to mutate, only patterns to leave out, so it gets one for each other Go file of the tree."""
|
||||
others = sorted(path for path in go_files(top) if path not in files)
|
||||
found = gremlins_mutants(top, [argument for path in others for argument in ("--exclude-files", f"^{re.escape(path)}$")], run, jobs)
|
||||
if isinstance(found, Failure):
|
||||
return found
|
||||
return [Mutant(mutant.path, mutant.line, mutant.column, mutant.mutator) for mutant in found if mutant.status == "KILLED" and mutant.path in files]
|
||||
|
||||
|
||||
def go_files(top: str) -> list[str]:
|
||||
"""Each Go file under top that Gremlins would mutate, by its path from top."""
|
||||
paths = (os.path.relpath(os.path.join(folder, name), top) for folder, _, names in os.walk(top) for name in names)
|
||||
return [path for path in paths if path.endswith(".go") and not is_test(path)]
|
||||
|
||||
|
||||
def drawn(mutants: list[Mutant], seed: int) -> list[Mutant]:
|
||||
"""The mutants in the order the seed draws them, whatever order Gremlins listed them in."""
|
||||
return random.Random(seed).sample(sorted(set(mutants)), len(set(mutants)))
|
||||
|
||||
|
||||
def mutated(source: bytes, mutant: Mutant) -> bytes | None:
|
||||
"""A file's source with the mutant applied, or None where its line and column hold no token its mutator changes."""
|
||||
lines = source.split(b"\n")
|
||||
if not 1 <= mutant.line <= len(lines):
|
||||
return None
|
||||
line = lines[mutant.line - 1]
|
||||
rest = line[mutant.column - 1:]
|
||||
# The longest token first: `>=` holds `>`, and Gremlins names the mutant by the whole token.
|
||||
token = max((token for token in MUTATIONS.get(mutant.mutator, {}) if rest.startswith(token)), key=len, default=None)
|
||||
if token is None:
|
||||
return None
|
||||
lines[mutant.line - 1] = line[:mutant.column - 1] + MUTATIONS[mutant.mutator][token] + rest[len(token):]
|
||||
return b"\n".join(lines)
|
||||
|
||||
|
||||
def line_at_head(diff: str, line: int) -> int | None:
|
||||
"""The number at the head of a file's line `line` at the base, or None where the head changed or removed that line.
|
||||
|
||||
diff is `git diff -U0 <base> <head> -- <file>`, so each hunk holds changed lines only."""
|
||||
moved = 0
|
||||
for old_start, old_count, new_start, new_count in ((int(a), int(b or 1), int(c), int(d or 1)) for a, b, c, d in HUNK.findall(diff)):
|
||||
# A hunk that only adds lines names the line it adds them after. One that only removes lines names the line the gap follows.
|
||||
old_end = old_start + old_count if old_count else old_start + 1
|
||||
new_end = new_start + new_count if new_count else new_start + 1
|
||||
if old_count and old_start <= line < old_end:
|
||||
return None
|
||||
if line >= old_end:
|
||||
moved = new_end - old_end
|
||||
return line + moved
|
||||
|
||||
|
||||
def task_text(output: str) -> str | None:
|
||||
"""What Haiku is told of a planted bug: the failing assertion's message in go test's output, without the test's name, file or
|
||||
line, so it says what goes wrong and not where. That is the last message of the first failed test that printed one, since a
|
||||
test logs before it asserts; a test with a failed subtest is left out, since it fails with the subtest. None where no failed test printed a message, as on a panic, a build that broke or a passing run,
|
||||
or where the message only names the test."""
|
||||
lines = output.splitlines()
|
||||
failed = set(FAILED_TEST.findall(output))
|
||||
# A subtest is named `TestPick/empty_input`; a message may name the whole or the test it belongs to.
|
||||
tests = {part for name in failed for part in (name, name.partition("/")[0])}
|
||||
marks: list[tuple[int, str]] = []
|
||||
last: dict[str, tuple[int, re.Match[str]]] = {}
|
||||
for index, line in enumerate(lines):
|
||||
if mark := TEST_MARK.match(line):
|
||||
marks.append((len(mark[1]), mark[2]))
|
||||
elif printed := PRINTED.match(line):
|
||||
# A test's output is indented deeper than its mark, so a line after a subtest's may be its parent's.
|
||||
test = next((name for indent, name in reversed(marks) if indent < len(printed[1])), None)
|
||||
if test is not None and test in failed and not any(other.startswith(f"{test}/") for other in failed):
|
||||
last[test] = index, printed
|
||||
if not last:
|
||||
return None
|
||||
index, found = next(iter(last.values()))
|
||||
indent = len(found[1])
|
||||
# A message's later lines are indented deeper than its first.
|
||||
later = lines[index + 1:]
|
||||
ends = next((at for at, text in enumerate(later) if len(text) - len(text.lstrip(" ")) <= indent), len(later))
|
||||
message = [text.strip() for text in (found[2], *later[:ends])]
|
||||
kept = [text for text in message if text and "_test.go" not in text and not any(name in text for name in tests)]
|
||||
return "\n".join(kept) or None
|
||||
@@ -0,0 +1,194 @@
|
||||
"""Run task-bench's containers: the image that adds Claude Code to the project's, the network that reaches only the API, one fix
|
||||
by Haiku in a copy of the code, and the project's tests over what it changed."""
|
||||
import contextlib
|
||||
import dataclasses
|
||||
import os
|
||||
import pathlib
|
||||
import shutil
|
||||
import subprocess
|
||||
import typing
|
||||
|
||||
from mutate_changes.tool_process import failed, Ran
|
||||
from failure import Failure
|
||||
from task_bench.figures import HOME, WORK
|
||||
from mutate_changes.changed_lines import is_test
|
||||
from task_bench.api_proxy import PORT
|
||||
|
||||
MODEL = "claude-haiku-5-5"
|
||||
# No MCP server, skill, command or stored session, so a run starts from the code alone; -p needs --verbose with stream-json, and no
|
||||
# one answers a permission prompt. Not --bare, which can't sign in by a long-lived token: docs/decisions.md → `Claude Code in a home
|
||||
# of its own`.
|
||||
CLAUDE = ["claude", "-p", "--strict-mcp-config", "--disable-slash-commands", "--no-session-persistence", "--model", MODEL, "--effort", "low", "--output-format", "stream-json", "--verbose", "--permission-mode", "bypassPermissions"]
|
||||
PROMPT = "A test of this project fails with the message below. Find the bug in the code and fix it. Change no test.\n\n{message}"
|
||||
TOKEN = "CLAUDE_CODE_OAUTH_TOKEN"
|
||||
API = "api.anthropic.com:443"
|
||||
# Holds only the proxy, which needs Python's standard library and nothing else.
|
||||
PROXY_IMAGE = "python:3.15.2-slim"
|
||||
# The installer puts claude in root's home, which no other user reads; each run works as the caller's user, so files it writes
|
||||
# in the copy are the caller's.
|
||||
DOCKERFILE = 'FROM {image}\nUSER root\nRUN curl -fsSL https://claude.ai/install.sh | bash && cp -L /root/.local/bin/claude /usr/local/bin/claude\n'
|
||||
BUILD_SECONDS = 20 * 60
|
||||
# How long one fix, or one run of the tests, may take before its container is removed; the run then counts as not passed.
|
||||
RUN_SECONDS = 30 * 60
|
||||
# What `docker run` exits with where the container, or its command, never started.
|
||||
NOT_STARTED = (125, 126, 127)
|
||||
# The memory each container gets without run-heavy, which hands its slot's share in RUN_HEAVY_MEMORY, in MiB.
|
||||
MEMORY_MIB = 4096
|
||||
|
||||
|
||||
class TimedOut(Failure):
|
||||
"""docker still ran when its time was up."""
|
||||
|
||||
|
||||
class Docker(typing.Protocol):
|
||||
"""Runs `docker <args>` and waits for it, at most `seconds`, and is TimedOut after that. Its output goes to `stdout` and its
|
||||
errors to `stderr` where the call gives them, and into the Ran otherwise."""
|
||||
|
||||
def __call__(self, args: list[str], *, seconds: float, stdin: str | None = None, stdout: typing.IO[bytes] | None = None, stderr: typing.IO[bytes] | None = None) -> Ran | Failure: ...
|
||||
|
||||
|
||||
def docker_cli(env: typing.Mapping[str, str], run: typing.Callable[..., subprocess.CompletedProcess[bytes]] = subprocess.run) -> Docker:
|
||||
def docker(args: list[str], *, seconds: float, stdin: str | None = None, stdout: typing.IO[bytes] | None = None, stderr: typing.IO[bytes] | None = None) -> Ran | Failure:
|
||||
try:
|
||||
ran = run(["docker", *args], env=env, input=None if stdin is None else stdin.encode(), stdin=subprocess.DEVNULL if stdin is None else None, stdout=stdout or subprocess.PIPE, stderr=stderr or subprocess.STDOUT, timeout=seconds, check=False)
|
||||
except OSError as failure:
|
||||
return Failure(f"docker could not start: {failure.strerror}")
|
||||
except subprocess.TimeoutExpired:
|
||||
return TimedOut(f"docker {args[0]} still ran after {round(seconds / 60)} min")
|
||||
return Ran(ran.returncode, (ran.stdout or b"").decode("utf-8", errors="replace"))
|
||||
|
||||
return docker
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Containers:
|
||||
"""What one bench holds in docker, all named after `name`, which no other bench shares."""
|
||||
docker: Docker
|
||||
name: str
|
||||
test: str
|
||||
memory_mib: int
|
||||
user: str
|
||||
|
||||
@property
|
||||
def image(self) -> str:
|
||||
return f"{self.name}:throwaway"
|
||||
|
||||
@property
|
||||
def proxy(self) -> str:
|
||||
return f"{self.name}-proxy"
|
||||
|
||||
@property
|
||||
def running(self) -> str:
|
||||
"""The one container that fixes or tests at a time."""
|
||||
return f"{self.name}-run"
|
||||
|
||||
def limits(self) -> list[str]:
|
||||
return ["--memory", f"{self.memory_mib}m", "--memory-swap", f"{self.memory_mib}m"]
|
||||
|
||||
|
||||
def opened(containers: Containers, image: str) -> Failure | None:
|
||||
"""Builds the bench's image, the project's with Claude Code added, and starts the network and its proxy.
|
||||
|
||||
The network has no route out. The proxy sits on it and on docker's default network, and tunnels to the API alone, so a run
|
||||
reaches the API and nothing else: not a package index, and not the repository the code came from."""
|
||||
docker, name = containers.docker, containers.name
|
||||
steps: list[tuple[list[str], float, str | None]] = [
|
||||
(["build", "--quiet", *containers.limits(), "--tag", containers.image, "-"], BUILD_SECONDS, DOCKERFILE.format(image=image)),
|
||||
(["network", "create", "--internal", name], 60, None),
|
||||
(["run", "--detach", "--rm", "--name", containers.proxy, "--memory", "256m", "--memory-swap", "256m", "--volume", f"{pathlib.Path(__file__).with_name('api_proxy.py')}:/api_proxy.py:ro", PROXY_IMAGE, "python", "/api_proxy.py", API], 300, None),
|
||||
(["network", "connect", name, containers.proxy], 60, None),
|
||||
]
|
||||
for args, seconds, stdin in steps:
|
||||
ran = docker(args, seconds=seconds, stdin=stdin)
|
||||
if isinstance(ran, Failure) or ran.code:
|
||||
return ran if isinstance(ran, Failure) else failed(f"docker {args[0]}", ran)
|
||||
return None
|
||||
|
||||
|
||||
def closed(containers: Containers) -> None:
|
||||
"""Removes what `opened` made and any container still running. A bench that broke half-way made only some of them, and removing
|
||||
one that is missing doesn't stop the rest."""
|
||||
for args in (["rm", "--force", containers.running], ["rm", "--force", containers.proxy], ["network", "rm", containers.name], ["image", "rm", containers.image]):
|
||||
containers.docker(args, seconds=120)
|
||||
|
||||
|
||||
def ran_tests(containers: Containers, tree: str) -> Ran | Failure:
|
||||
"""The project's test command, run at the top of tree with no network; the tests pass where it exits 0. TimedOut where it still
|
||||
ran after RUN_SECONDS, and a Failure where docker could not run it."""
|
||||
return run_in_image(containers, ["--network", "none", *in_tree(containers, tree), containers.image, "sh", "-c", containers.test])
|
||||
|
||||
|
||||
def fix_attempted(containers: Containers, copy: str, message: str, stream: pathlib.Path) -> Failure | None:
|
||||
"""Has Haiku fix the bug in copy, told only the failing assertion's message, and writes what its claude prints to `stream`, and its
|
||||
errors beside it. A fix that still runs after RUN_SECONDS ends there, with the stream as far as it came.
|
||||
|
||||
The container's home is new and empty, so no instruction, memory, hook, skill or plugin of the caller loads; and docker hands the
|
||||
token on by its name, so its value is on no command line. A Failure where docker could not run claude."""
|
||||
args = ["--network", containers.name, *in_tree(containers, copy), "--env", TOKEN, "--env", f"HTTPS_PROXY=http://{containers.proxy}:{PORT}", "--env", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1", containers.image, *CLAUDE, PROMPT.format(message=message)]
|
||||
errors_file = stream.with_suffix(".stderr")
|
||||
try:
|
||||
with stream.open("wb") as out, errors_file.open("wb") as errors:
|
||||
ran = run_in_image(containers, args, stdout=out, stderr=errors)
|
||||
except OSError as failure:
|
||||
return Failure(f"could not write {stream}: {failure}")
|
||||
# Out of time is the run's outcome, not the bench's failure.
|
||||
if isinstance(ran, TimedOut) or not isinstance(ran, Failure):
|
||||
return None
|
||||
return Failure(f"{ran.reason}; its errors are in {errors_file}")
|
||||
|
||||
|
||||
def run_in_image(containers: Containers, args: list[str], stdout: typing.IO[bytes] | None = None, stderr: typing.IO[bytes] | None = None) -> Ran | Failure:
|
||||
"""One `docker run` as the bench's one running container, for at most RUN_SECONDS, after which the container is removed.
|
||||
|
||||
A run whose command never started is a Failure, since the next would end the same: counted as a fix that failed, it would fill
|
||||
the table with runs that measured docker."""
|
||||
ran = containers.docker(["run", "--rm", "--name", containers.running, *args], seconds=RUN_SECONDS, stdout=stdout, stderr=stderr)
|
||||
if isinstance(ran, Failure):
|
||||
containers.docker(["rm", "--force", containers.running], seconds=120)
|
||||
return ran
|
||||
if ran.code in NOT_STARTED:
|
||||
return Failure(f"docker run could not start its command, and exited {ran.code}: {ran.output.strip()}".removesuffix(": "))
|
||||
return ran
|
||||
|
||||
|
||||
def in_tree(containers: Containers, tree: str) -> list[str]:
|
||||
"""The options that run a container in tree, mounted at WORK on every run so no path tells the sides apart, as the caller's user."""
|
||||
return [*containers.limits(), "--user", containers.user, "--env", f"HOME={HOME}", "--volume", f"{tree}:{WORK}", "--workdir", WORK]
|
||||
|
||||
|
||||
def checked_tree(planted: str, copy: str, into: str) -> None:
|
||||
"""Makes at `into` the tree the tests check a fix in: `planted`, the code with the bug as Haiku got it, with each file that is no
|
||||
test as `copy` holds it after the fix. So a test Haiku changed, removed or added doesn't count, and neither does a file under
|
||||
a `testdata` folder, where Go's tests keep the output they expect.
|
||||
|
||||
Raises OSError where a file cannot be copied."""
|
||||
shutil.copytree(planted, into, symlinks=True)
|
||||
before, after = set(entries(planted)), set(entries(copy))
|
||||
for path in before - after:
|
||||
if not of_tests(path):
|
||||
with contextlib.suppress(FileNotFoundError):
|
||||
os.unlink(os.path.join(into, path))
|
||||
for path in sorted(after):
|
||||
target = os.path.join(into, path)
|
||||
# A link is copied as a link, never followed; and one that would stand in for a folder of the planted tree is left out.
|
||||
if of_tests(path) or (os.path.isdir(target) and not os.path.islink(target)):
|
||||
continue
|
||||
os.makedirs(os.path.dirname(target), exist_ok=True)
|
||||
with contextlib.suppress(FileNotFoundError):
|
||||
os.unlink(target)
|
||||
shutil.copy2(os.path.join(copy, path), target, follow_symlinks=False)
|
||||
|
||||
|
||||
def of_tests(path: str) -> bool:
|
||||
return is_test(path) or "testdata" in pathlib.PurePath(path).parts
|
||||
|
||||
|
||||
def entries(top: str) -> list[str]:
|
||||
"""Each file and link under top, by its path from top, without git's own under `.git`."""
|
||||
found = []
|
||||
for folder, folders, names in os.walk(top):
|
||||
if folder == top and ".git" in folders:
|
||||
folders.remove(".git")
|
||||
links = [name for name in folders if os.path.islink(os.path.join(folder, name))]
|
||||
found += [os.path.relpath(os.path.join(folder, name), top) for name in (*names, *links)]
|
||||
return found
|
||||
@@ -0,0 +1,117 @@
|
||||
"""task-bench's figures: what it reads from one run's stream, and the table it prints for a bench."""
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from failure import Failure
|
||||
from task_bench.figures import Figures, figures, Result, table
|
||||
from claude_code.tokens import Tokens
|
||||
|
||||
INIT = {"type": "system", "subtype": "init", "cwd": "/work", "model": "claude-haiku-5-5", "claude_code_version": "2.1.330"}
|
||||
USAGE = {"input_tokens": 40, "cache_creation_input_tokens": 9000, "cache_read_input_tokens": 52000, "output_tokens": 800, "cache_creation": {"ephemeral_1h_input_tokens": 9000, "ephemeral_5m_input_tokens": 0}}
|
||||
RESULT = {"type": "result", "subtype": "success", "is_error": False, "num_turns": 7, "duration_ms": 61500, "usage": USAGE, "result": "Fixed."}
|
||||
|
||||
|
||||
def call(name: str, **given: str) -> dict:
|
||||
return {"type": "assistant", "message": {"id": f"msg_{name}_{sorted(given.values())}", "model": "claude-haiku-5-5", "content": [{"type": "tool_use", "id": "toolu_1", "name": name, "input": given}], "usage": {}}}
|
||||
|
||||
|
||||
def stream(*events: dict) -> bytes:
|
||||
return "\n".join(map(json.dumps, events)).encode() + b"\n"
|
||||
|
||||
|
||||
class FiguresTest(unittest.TestCase):
|
||||
def test_reads_the_version_model_tokens_turns_and_wall_time_from_the_streams_first_and_last_events(self):
|
||||
found = figures(stream(INIT, RESULT), "/work/calc.go")
|
||||
self.assertEqual(found, Figures("2.1.330", "claude-haiku-5-5", Tokens(input=40, write_1h=9000, read=52000, output=800), 7, 61.5, None))
|
||||
|
||||
def test_counts_the_reads_and_searches_before_the_first_open_of_the_file_the_bug_is_in(self):
|
||||
calls = [
|
||||
call("Grep", pattern="Pick"),
|
||||
call("Read", file_path="/work/README.md"),
|
||||
call("Bash", command="go test ./..."),
|
||||
call("Bash", command="cat render/pick.go | head -40"),
|
||||
call("Glob", pattern="**/*.go"),
|
||||
call("Read", file_path="/work/calc.go"),
|
||||
call("Read", file_path="/work/other.go"),
|
||||
call("Edit", file_path="/work/calc.go", old_string="a", new_string="b"),
|
||||
]
|
||||
found = figures(stream(INIT, *calls, RESULT), "/work/calc.go")
|
||||
self.assertEqual(found.reads, 4)
|
||||
|
||||
def test_a_shell_command_that_reads_the_file_opens_it(self):
|
||||
found = figures(stream(INIT, call("Read", file_path="/work/README.md"), call("Bash", command="sed -n 1,40p calc.go"), RESULT), "/work/calc.go")
|
||||
self.assertEqual(found.reads, 1)
|
||||
|
||||
def test_has_no_reads_where_the_run_never_opened_the_file(self):
|
||||
self.assertIsNone(figures(stream(INIT, call("Read", file_path="/work/README.md"), RESULT), "/work/calc.go").reads)
|
||||
|
||||
def test_a_stream_that_ends_without_a_result_has_no_figures(self):
|
||||
self.assertIsNone(figures(stream(INIT, call("Read", file_path="/work/calc.go")), "/work/calc.go"))
|
||||
self.assertIsNone(figures(b"", "/work/calc.go"))
|
||||
|
||||
def test_a_result_without_turns_or_a_duration_counts_none(self):
|
||||
found = figures(stream(INIT, {"type": "result", "usage": USAGE}), "/work/calc.go")
|
||||
self.assertEqual((found.turns, found.seconds), (0, 0))
|
||||
|
||||
def test_a_stream_without_an_init_names_no_version_or_model(self):
|
||||
found = figures(stream(RESULT), "/work/calc.go")
|
||||
self.assertEqual((found.version, found.model), ("", ""))
|
||||
|
||||
def test_a_shell_command_that_reads_under_the_runs_home_reads_no_file_of_the_code(self):
|
||||
found = figures(stream(INIT, call("Bash", command="cat ~/calc.go"), call("Read", file_path="/work/calc.go"), RESULT), "/work/calc.go")
|
||||
self.assertEqual(found.reads, 1)
|
||||
|
||||
def test_a_run_the_api_refused_is_a_failure_that_quotes_claude(self):
|
||||
refused = {**RESULT, "is_error": True, "api_error_status": 401, "result": "Failed to authenticate. API Error: 401 OAuth access token is invalid."}
|
||||
self.assertEqual(figures(stream(INIT, refused), "/work/calc.go"), Failure("claude could not use the API: Failed to authenticate. API Error: 401 OAuth access token is invalid."))
|
||||
|
||||
|
||||
def run(task: int, side: str, number: int, passed: bool, input_: int = 100, turns: int = 10, seconds: float = 60, reads: int | None = 3) -> Result:
|
||||
return Result(task, side, number, passed, Figures("2.1.330", "claude-haiku-5-5", Tokens(input=input_, write_5m=20, write_1h=980, read=5000, output=400), turns, seconds, reads))
|
||||
|
||||
|
||||
class TableTest(unittest.TestCase):
|
||||
def test_prints_a_row_per_task_and_side_with_the_pass_count_and_each_median_then_all_tasks_and_the_heads_change(self):
|
||||
results = [
|
||||
run(1, "base", 1, True, input_=100, turns=10, seconds=60, reads=3),
|
||||
run(1, "base", 2, False, input_=300, turns=20, seconds=120, reads=None),
|
||||
run(1, "base", 3, True, input_=200, turns=12, seconds=90, reads=5),
|
||||
run(1, "head", 1, True, input_=100, turns=8, seconds=45, reads=1),
|
||||
run(1, "head", 2, True, input_=100, turns=8, seconds=45, reads=1),
|
||||
run(1, "head", 3, True, input_=160, turns=9, seconds=50, reads=2),
|
||||
run(2, "base", 1, False, input_=200, turns=12, seconds=90, reads=4),
|
||||
run(2, "head", 1, True, input_=100, turns=9, seconds=45, reads=2),
|
||||
]
|
||||
self.assertEqual(table(results, {1: "calc.go:4:7 CONDITIONALS_NEGATION", 2: "pick.go:9:3 ARITHMETIC_BASE"}), [
|
||||
"| Bug | Side | Passed | Input | Cache write | Cache read | Output | Turns | Wall, s | Reads |",
|
||||
"|---|---|---|---|---|---|---|---|---|---|",
|
||||
"| 1: calc.go:4:7 CONDITIONALS_NEGATION | base | 2/3 | 200 | 1000 | 5000 | 400 | 12 | 90 | 4 |",
|
||||
"| 1: calc.go:4:7 CONDITIONALS_NEGATION | head | 3/3 | 100 | 1000 | 5000 | 400 | 8 | 45 | 1 |",
|
||||
"| 2: pick.go:9:3 ARITHMETIC_BASE | base | 0/1 | 200 | 1000 | 5000 | 400 | 12 | 90 | 4 |",
|
||||
"| 2: pick.go:9:3 ARITHMETIC_BASE | head | 1/1 | 100 | 1000 | 5000 | 400 | 9 | 45 | 2 |",
|
||||
"| All | base | 2/4 | 200 | 1000 | 5000 | 400 | 12 | 90 | 4 |",
|
||||
"| All | head | 4/4 | 100 | 1000 | 5000 | 400 | 8.5 | 45 | 1.5 |",
|
||||
"| All | change | +2 | -50 % | 0 % | 0 % | 0 % | -3.5 | -50 % | -2.5 |",
|
||||
])
|
||||
|
||||
def test_the_change_is_a_dash_where_the_base_measured_zero_and_signed_where_it_rose(self):
|
||||
results = [run(1, "base", 1, True, input_=0, turns=10, seconds=60, reads=0), run(1, "head", 1, False, input_=50, turns=12, seconds=120, reads=2)]
|
||||
self.assertEqual(table(results, {1: "calc.go:4:7 ARITHMETIC_BASE"})[-1], "| All | change | -1 | - | 0 % | 0 % | 0 % | +2 | +100 % | +2 |")
|
||||
|
||||
def test_a_run_without_figures_counts_as_not_passed_and_a_side_without_any_shows_a_dash(self):
|
||||
results = [Result(1, "base", 1, False, None), run(1, "head", 1, True)]
|
||||
self.assertEqual(table(results, {1: "calc.go:4:7 ARITHMETIC_BASE"})[2:], [
|
||||
"| 1: calc.go:4:7 ARITHMETIC_BASE | base | 0/1 | - | - | - | - | - | - | - |",
|
||||
"| 1: calc.go:4:7 ARITHMETIC_BASE | head | 1/1 | 100 | 1000 | 5000 | 400 | 10 | 60 | 3 |",
|
||||
"| All | base | 0/1 | - | - | - | - | - | - | - |",
|
||||
"| All | head | 1/1 | 100 | 1000 | 5000 | 400 | 10 | 60 | 3 |",
|
||||
"| All | change | +1 | - | - | - | - | - | - | - |",
|
||||
])
|
||||
|
||||
def test_the_change_is_a_dash_where_only_the_head_has_no_figures(self):
|
||||
results = [run(1, "base", 1, True), Result(1, "head", 1, False, None)]
|
||||
self.assertEqual(table(results, {1: "calc.go:4:7 ARITHMETIC_BASE"})[-1], "| All | change | -1 | - | - | - | - | - | - | - |")
|
||||
@@ -0,0 +1,225 @@
|
||||
"""task-bench's planter: which files it mutates, which mutants it draws, how it re-applies one and what it tells Haiku."""
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from task_bench.plant import drawn, go_files, killed_mutants, line_at_head, Mutant, mutated, place_files, task_text
|
||||
from failure import Failure
|
||||
from mutate_changes.tool_process import Ran
|
||||
|
||||
PLACES = """## 2026-11-19T22:33:15Z, PR #646 at a357cc7b
|
||||
|
||||
Junior A, hardest first:
|
||||
|
||||
- Locality, Shape: `internal/render/pick.go:287` `parked`: three steps held at once
|
||||
- Shape: `internal/render/pick_test.go:12` `TestPick`: a long table
|
||||
- Self-sufficiency: `README.md:89` "Round cap": three sections
|
||||
- Locality: `cli/report.go:41` the comment: points to `cli/gone.go`
|
||||
|
||||
Mid A, hardest first:
|
||||
|
||||
- Locality: `internal/render/pick.go:354` `resumed`: a loop
|
||||
- Locality: `data-import/geo.py:3` `load`: a long chain
|
||||
"""
|
||||
# go test's report of an assertion that failed in a subtest, then of one whose message runs over two lines.
|
||||
GO_FAILURE = """ok \texample.com/calc/cli\t0.004s
|
||||
--- FAIL: TestPick (0.00s)
|
||||
--- FAIL: TestPick/empty_input (0.00s)
|
||||
pick_test.go:42: Pick() = 3,
|
||||
want 4
|
||||
pick_test.go:50: got 1 row, want 2
|
||||
FAIL
|
||||
FAIL\texample.com/calc/render\t0.012s
|
||||
FAIL
|
||||
"""
|
||||
# testify prints the test's file and name inside the message.
|
||||
TESTIFY_FAILURE = """--- FAIL: TestTotal (0.00s)
|
||||
total_test.go:12:
|
||||
\tError Trace:\t/work/total_test.go:12
|
||||
\tError: \tNot equal:
|
||||
\t \texpected: 1
|
||||
\t \tactual : 2
|
||||
\tTest: \tTestTotal
|
||||
FAIL
|
||||
"""
|
||||
PANIC = """--- FAIL: TestPick (0.00s)
|
||||
panic: runtime error: index out of range [3] with length 3 [recovered]
|
||||
FAIL\texample.com/calc/render\t0.012s
|
||||
"""
|
||||
|
||||
|
||||
class PlaceFilesTest(unittest.TestCase):
|
||||
def test_names_each_go_file_that_holds_a_place_once_and_no_test_file_or_other_language(self):
|
||||
self.assertEqual(place_files(PLACES), ["cli/report.go", "internal/render/pick.go"])
|
||||
|
||||
def test_names_none_where_the_seats_found_no_places(self):
|
||||
self.assertEqual(place_files("Seats found no places hard."), [])
|
||||
|
||||
|
||||
class DrawnTest(unittest.TestCase):
|
||||
MUTANTS = [Mutant("a.go", line, 3, "ARITHMETIC_BASE") for line in range(1, 40)]
|
||||
|
||||
def test_a_seed_draws_the_same_order_whatever_order_gremlins_listed_the_mutants_in(self):
|
||||
self.assertEqual(drawn(self.MUTANTS, 7), drawn(self.MUTANTS[::-1], 7))
|
||||
|
||||
def test_draws_every_mutant_once(self):
|
||||
self.assertEqual(sorted(drawn(self.MUTANTS, 7)), self.MUTANTS)
|
||||
|
||||
def test_another_seed_draws_another_order(self):
|
||||
self.assertNotEqual(drawn(self.MUTANTS, 7), drawn(self.MUTANTS, 8))
|
||||
|
||||
|
||||
class MutatedTest(unittest.TestCase):
|
||||
SOURCE = b"package calc\n\nfunc f(a, b int) int {\n\tif a >= b {\n\t\treturn a - b\n\t}\n\tn -= 1\n\treturn a\n}\n"
|
||||
|
||||
def planted(self, line: int, column: int, mutator: str, source: bytes = SOURCE) -> str | None:
|
||||
found = mutated(source, Mutant("calc.go", line, column, mutator))
|
||||
return None if found is None else found.decode().splitlines()[line - 1]
|
||||
|
||||
def test_swaps_the_token_at_the_line_and_column_as_gremlins_mutator_does(self):
|
||||
self.assertEqual(self.planted(4, 7, "CONDITIONALS_NEGATION"), "\tif a < b {")
|
||||
self.assertEqual(self.planted(4, 7, "CONDITIONALS_BOUNDARY"), "\tif a > b {")
|
||||
self.assertEqual(self.planted(5, 12, "ARITHMETIC_BASE"), "\t\treturn a + b")
|
||||
self.assertEqual(self.planted(7, 4, "INVERT_ASSIGNMENTS"), "\tn += 1")
|
||||
self.assertEqual(self.planted(7, 4, "REMOVE_SELF_ASSIGNMENTS"), "\tn = 1")
|
||||
|
||||
def test_changes_no_other_byte(self):
|
||||
found = mutated(self.SOURCE, Mutant("calc.go", 5, 12, "ARITHMETIC_BASE"))
|
||||
self.assertEqual(found, self.SOURCE.replace(b"a - b", b"a + b"))
|
||||
|
||||
def test_counts_the_column_in_bytes_as_go_does(self):
|
||||
self.assertEqual(self.planted(1, 15, "CONDITIONALS_NEGATION", 'x := "åäö" == y\n'.encode()), 'x := "åäö" != y')
|
||||
|
||||
def test_is_none_where_the_place_holds_no_token_of_the_mutator(self):
|
||||
self.assertIsNone(self.planted(4, 6, "CONDITIONALS_NEGATION"))
|
||||
self.assertIsNone(self.planted(5, 12, "CONDITIONALS_NEGATION"))
|
||||
self.assertIsNone(self.planted(40, 1, "ARITHMETIC_BASE"))
|
||||
self.assertIsNone(self.planted(4, 70, "ARITHMETIC_BASE"))
|
||||
self.assertIsNone(self.planted(4, 7, "A_MUTATOR_GREMLINS_ADDS_LATER"))
|
||||
|
||||
|
||||
class TaskTextTest(unittest.TestCase):
|
||||
def test_is_the_first_failed_tests_message_without_the_tests_name_file_or_line(self):
|
||||
self.assertEqual(task_text(GO_FAILURE), "Pick() = 3,\nwant 4")
|
||||
|
||||
def test_leaves_out_each_line_that_names_the_test_or_its_file(self):
|
||||
self.assertEqual(task_text(TESTIFY_FAILURE), "Error: \tNot equal:\nexpected: 1\nactual : 2")
|
||||
|
||||
def test_is_none_where_no_assertion_failed_as_on_a_panic_or_a_passing_run(self):
|
||||
self.assertIsNone(task_text(PANIC))
|
||||
self.assertIsNone(task_text("ok \texample.com/calc\t0.004s\n"))
|
||||
|
||||
def test_takes_the_message_of_a_test_that_failed_not_what_a_passing_test_logged(self):
|
||||
output = "=== RUN TestServer\n server_test.go:12: listening on 127.0.0.1:0\n--- PASS: TestServer (0.00s)\n=== RUN TestPick\n pick_test.go:42: got 3, want 4\n--- FAIL: TestPick (0.00s)\nFAIL\n"
|
||||
self.assertEqual(task_text(output), "got 3, want 4")
|
||||
|
||||
def test_takes_the_last_message_of_the_failed_test_since_a_test_logs_before_it_asserts(self):
|
||||
self.assertEqual(task_text("--- FAIL: TestPick (0.00s)\n pick_test.go:40: input [1 2 3]\n pick_test.go:42: got 3, want 4\nFAIL\n"), "got 3, want 4")
|
||||
|
||||
def test_takes_the_first_failed_test_that_has_a_message(self):
|
||||
output = "--- FAIL: TestPick (0.00s)\n --- FAIL: TestPick/empty (0.00s)\n pick_test.go:42: got 3\n--- FAIL: TestLast (0.00s)\n pick_test.go:50: got 5\nFAIL\n"
|
||||
self.assertEqual(task_text(output), "got 3")
|
||||
|
||||
def test_a_line_printed_before_any_tests_mark_is_no_tests_message(self):
|
||||
self.assertEqual(task_text(" setup_test.go:3: starting\n--- FAIL: TestPick (0.00s)\n pick_test.go:42: got 3\nFAIL\n"), "got 3")
|
||||
self.assertIsNone(task_text(" setup_test.go:3: starting\n--- FAIL: TestPick (0.00s)\nFAIL\n"))
|
||||
|
||||
def test_takes_the_failed_subtests_message_over_what_its_parent_logged_before_it(self):
|
||||
output = "--- FAIL: TestPick (0.00s)\n pick_test.go:10: three cases\n --- FAIL: TestPick/empty (0.00s)\n pick_test.go:42: got 3\nFAIL\n"
|
||||
self.assertEqual(task_text(output), "got 3")
|
||||
|
||||
def test_is_none_where_only_passing_tests_logged(self):
|
||||
self.assertIsNone(task_text("=== RUN TestServer\n server_test.go:12: listening\n--- PASS: TestServer (0.00s)\nPASS\n"))
|
||||
|
||||
def test_is_none_where_the_message_does_not_say_anything_beyond_the_test(self):
|
||||
self.assertIsNone(task_text("--- FAIL: TestPick (0.00s)\n pick_test.go:42: TestPick failed\nFAIL\n"))
|
||||
|
||||
|
||||
class LineAtHeadTest(unittest.TestCase):
|
||||
# Lines 3 and 4 became one, two lines went in after line 9, and line 20 went.
|
||||
DIFF = "diff --git a/calc.go b/calc.go\n--- a/calc.go\n+++ b/calc.go\n@@ -3,2 +3 @@\n-a\n-b\n+c\n@@ -9,0 +9,2 @@\n+d\n+e\n@@ -20 +20,0 @@\n-f\n"
|
||||
|
||||
def test_a_line_above_every_change_keeps_its_number(self):
|
||||
self.assertEqual(line_at_head(self.DIFF, 2), 2)
|
||||
|
||||
def test_a_line_below_a_change_moves_by_the_lines_it_added_and_removed(self):
|
||||
self.assertEqual([line_at_head(self.DIFF, line) for line in (5, 9, 10, 19, 21)], [4, 8, 11, 20, 21])
|
||||
|
||||
def test_a_line_the_head_changed_or_removed_has_none(self):
|
||||
self.assertEqual([line_at_head(self.DIFF, line) for line in (3, 4, 20)], [None, None, None])
|
||||
|
||||
def test_no_diff_keeps_every_line(self):
|
||||
self.assertEqual(line_at_head("", 12), 12)
|
||||
|
||||
|
||||
class KilledMutantsTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
folder = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(folder.cleanup)
|
||||
self.top = pathlib.Path(folder.name)
|
||||
for path in ("calc.go", "calc_test.go", "render/pick.go", "render/a+b.go", "render/notes.md", "cli/main.go"):
|
||||
(self.top / path).parent.mkdir(parents=True, exist_ok=True)
|
||||
(self.top / path).write_text("package x\n")
|
||||
self.ran: list[tuple[list[str], str]] = []
|
||||
|
||||
def gremlins(self, report: dict | None, code: int = 0):
|
||||
def run(argv: list[str], cwd: str) -> Ran:
|
||||
self.ran.append((argv, cwd))
|
||||
if report is not None:
|
||||
pathlib.Path(cwd, argv[argv.index("--output") + 1]).write_text(json.dumps(report))
|
||||
return Ran(code, "gremlins said so")
|
||||
|
||||
return run
|
||||
|
||||
def test_names_each_go_file_gremlins_would_mutate_by_its_path_from_the_top(self):
|
||||
self.assertEqual(sorted(go_files(str(self.top))), ["calc.go", "cli/main.go", "render/a+b.go", "render/pick.go"])
|
||||
|
||||
def test_has_gremlins_leave_out_every_other_go_file_by_its_whole_path(self):
|
||||
killed_mutants(str(self.top), ["render/pick.go"], self.gremlins({"files": []}), 3)
|
||||
argv, cwd = self.ran[0]
|
||||
self.assertEqual(cwd, str(self.top))
|
||||
self.assertEqual(argv[:8], ["gremlins", "unleash", "--exclude-files", r"^calc\.go$", "--exclude-files", r"^cli/main\.go$", "--exclude-files", r"^render/a\+b\.go$"])
|
||||
self.assertEqual((argv[8], argv[10:]), ("--output", ["--workers", "3"]))
|
||||
|
||||
def test_keeps_the_killed_mutants_of_the_files_asked_for(self):
|
||||
report = {"files": [
|
||||
{"file_name": "render/pick.go", "mutations": [{"line": 4, "column": 7, "status": "KILLED", "type": "ARITHMETIC_BASE"}, {"line": 5, "column": 2, "status": "LIVED", "type": "INVERT_LOGICAL"}, {"line": 6, "column": 2, "status": "NOT COVERED", "type": "INVERT_LOGICAL"}]},
|
||||
{"file_name": "calc.go", "mutations": [{"line": 1, "column": 1, "status": "KILLED", "type": "ARITHMETIC_BASE"}]},
|
||||
]}
|
||||
self.assertEqual(killed_mutants(str(self.top), ["render/pick.go"], self.gremlins(report), 3), [Mutant("render/pick.go", 4, 7, "ARITHMETIC_BASE")])
|
||||
|
||||
def test_a_gremlins_that_fails_is_a_failure_with_what_it_printed(self):
|
||||
found = killed_mutants(str(self.top), ["render/pick.go"], self.gremlins(None, code=1), 3)
|
||||
self.assertIsInstance(found, Failure)
|
||||
self.assertIn("gremlins said so", found.reason)
|
||||
|
||||
|
||||
class EdgesTest(unittest.TestCase):
|
||||
def test_mutates_the_last_line_of_a_file(self):
|
||||
self.assertEqual(mutated(b"a\nb + c", Mutant("calc.go", 2, 3, "ARITHMETIC_BASE")), b"a\nb - c")
|
||||
self.assertIsNone(mutated(b"a\nb + c", Mutant("calc.go", 3, 3, "ARITHMETIC_BASE")))
|
||||
self.assertIsNone(mutated(b"a\nb + c", Mutant("calc.go", 0, 3, "ARITHMETIC_BASE")))
|
||||
|
||||
def test_takes_the_longest_token_at_the_column(self):
|
||||
self.assertEqual(mutated(b"a >= b", Mutant("calc.go", 1, 3, "CONDITIONALS_BOUNDARY")), b"a > b")
|
||||
self.assertEqual(mutated(b"a &^ b", Mutant("calc.go", 1, 3, "INVERT_BITWISE")), b"a & b")
|
||||
|
||||
def test_a_message_ends_at_the_next_line_indented_as_its_first(self):
|
||||
output = "--- FAIL: TestPick/empty_input (0.00s)\n pick_test.go:42: got 3\n want 4\n--- FAIL: TestLast (0.00s)\n pick_test.go:50: got 5\n"
|
||||
self.assertEqual(task_text(output), "got 3\nwant 4")
|
||||
self.assertEqual(task_text("--- FAIL: TestPick (0.00s)\n pick_test.go:42: got 3\n a line the test logged\n"), "got 3")
|
||||
|
||||
def test_a_message_runs_to_the_end_of_the_output(self):
|
||||
self.assertEqual(task_text("--- FAIL: TestPick (0.00s)\n pick_test.go:42: got 3\n want 4"), "got 3\nwant 4")
|
||||
|
||||
def test_a_message_may_name_a_part_of_a_subtests_name_that_is_no_test(self):
|
||||
output = "--- FAIL: TestPick/empty (0.00s)\n pick_test.go:42: empty input: got 3\n in TestPick\n in TestPick/empty\n"
|
||||
self.assertEqual(task_text(output), "empty input: got 3")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,591 @@
|
||||
"""task-bench from its command line down: a whole bench over a small repository with docker, Gremlins and claude faked, the tree a
|
||||
fix is checked in, and the proxy that reaches only the API."""
|
||||
import asyncio
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import textwrap
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from task_bench import api_proxy, main, runs
|
||||
from task_bench.runs import checked_tree, closed, Containers, docker_cli, fix_attempted, opened, ran_tests, TimedOut
|
||||
from task_bench.plant import drawn, Mutant
|
||||
from failure import Failure
|
||||
from test_env import isolated_env
|
||||
from mutate_changes.tool_process import Ran
|
||||
from task_bench.figures import Result
|
||||
|
||||
BASE_CODE = "package calc\n\nfunc Add(a, b int) int {\n\treturn a + b\n}\n\nfunc Twice(a int) int {\n\treturn a * 2\n}\n"
|
||||
# The head adds a line right above Add, which moves it down, and rewrites Twice's line.
|
||||
HEAD_CODE = BASE_CODE.replace("func Add", "// Add adds.\nfunc Add").replace("a * 2", "2 * a")
|
||||
# Stands in for `go test`: a shell text in a file named as a Go test, which fails as go test reports a failed assertion.
|
||||
TESTS = textwrap.dedent("""\
|
||||
grep -q 'a + b' calc.go && exit 0
|
||||
printf -- '--- FAIL: TestAdd (0.00s)\\n calc_test.go:9: Add(1, 2) = -1, want 3\\nFAIL\\n'
|
||||
exit 1
|
||||
""")
|
||||
# Reports `+` in Add, which the head keeps, and the `*` in Twice, which the head rewrites, as killed; and a mutant no test caught.
|
||||
GREMLINS = textwrap.dedent("""\
|
||||
import json, os, sys
|
||||
lines = open("calc.go").read().split("\\n")
|
||||
found = [{"line": at, "column": 11, "status": "KILLED", "type": "ARITHMETIC_BASE"} for at, line in enumerate(lines, 1) if " + " in line or " * " in line]
|
||||
report = {"files": [{"file_name": "calc.go", "mutations": [*found, {**found[0], "status": "LIVED", "type": "INVERT_NEGATIVES"}]}]}
|
||||
json.dump(report, open(sys.argv[sys.argv.index("--output") + 1], "w"))
|
||||
if os.environ.get("FAKE_GREMLINS_LOG"):
|
||||
json.dump([os.getcwd(), *sys.argv[1:]], open(os.environ["FAKE_GREMLINS_LOG"], "w"))
|
||||
""")
|
||||
# Logs each call. A `run` of the bench's image runs its command in the mounted tree: sh as it is, and claude as the fake below.
|
||||
DOCKER = textwrap.dedent("""\
|
||||
#!/usr/bin/env python3
|
||||
import json, os, subprocess, sys
|
||||
args = sys.argv[1:]
|
||||
with open(os.environ["FAKE_LOG"], "a") as log:
|
||||
log.write(json.dumps(args) + "\\n")
|
||||
if args[0] == "build":
|
||||
held = sys.stdin.read()
|
||||
if os.environ.get("FAKE_STDIN"):
|
||||
open(os.environ["FAKE_STDIN"], "w").write(held)
|
||||
if args[0] != "run" or "--detach" in args:
|
||||
sys.exit(int(os.environ.get("FAKE_FAILS") == args[0]))
|
||||
tree = args[args.index("--volume") + 1].rpartition(":")[0]
|
||||
command = args[next(at for at, arg in enumerate(args) if arg.endswith(":throwaway")) + 1:]
|
||||
if command[0] == "claude":
|
||||
command = [sys.executable, os.environ["FAKE_CLAUDE"], command[-1]]
|
||||
sys.exit(subprocess.run(command, cwd=tree).returncode)
|
||||
""")
|
||||
# Reads the README, then the file with the bug. At the head it fixes the bug; at the base it rewrites the test to pass instead.
|
||||
# From its FAKE_REFUSED_FROM-th run on, the API refuses it.
|
||||
CLAUDE = textwrap.dedent("""\
|
||||
import json, os, pathlib, sys
|
||||
def call(name, path):
|
||||
return {"type": "assistant", "message": {"id": "msg_" + path, "content": [{"type": "tool_use", "id": "toolu_" + path, "name": name, "input": {"file_path": path}}], "usage": {}}}
|
||||
count = pathlib.Path(os.environ["FAKE_LOG"] + ".fixes")
|
||||
count.write_text(count.read_text() + "." if count.exists() else ".")
|
||||
if len(count.read_text()) >= int(os.environ.get("FAKE_REFUSED_FROM") or 10 ** 9):
|
||||
print(json.dumps({"type": "result", "is_error": True, "api_error_status": 401, "result": "API Error: 401"}))
|
||||
sys.exit(1)
|
||||
print(json.dumps({"type": "system", "subtype": "init", "model": "claude-haiku-5-5", "claude_code_version": "2.1.330"}))
|
||||
print(json.dumps(call("Read", "/work/README.md")))
|
||||
print(json.dumps(call("Read", "/work/calc.go")))
|
||||
code = pathlib.Path("calc.go")
|
||||
if "// Add adds." in code.read_text():
|
||||
code.write_text(code.read_text().replace("a - b", "a + b"))
|
||||
else:
|
||||
pathlib.Path("calc_test.go").write_text("exit 0\\n")
|
||||
pathlib.Path("prompt").write_text(sys.argv[1])
|
||||
usage = {"input_tokens": 40, "cache_creation_input_tokens": 9000, "cache_read_input_tokens": 52000, "output_tokens": 800}
|
||||
print(json.dumps({"type": "result", "subtype": "success", "is_error": False, "num_turns": 3, "duration_ms": 61500, "usage": usage, "result": "Fixed."}))
|
||||
""")
|
||||
PLACES = "## 2026-10-08T12:00:00Z, PR #1 at aaaaaaa\n\nJunior A, hardest first:\n\n- `calc.go:3` `Add`: long\n- `gone.go:1` `Gone`: removed since\n- `README.md:1` `Calc`: thin\n"
|
||||
|
||||
|
||||
class BenchTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
folder = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(folder.cleanup)
|
||||
self.root = pathlib.Path(folder.name).resolve()
|
||||
self.home, self.repo, fakes = self.root / "home", self.root / "repo", self.root / "fakes"
|
||||
for made in (self.home, self.repo, fakes):
|
||||
made.mkdir()
|
||||
(fakes / "docker").write_text(DOCKER)
|
||||
(fakes / "docker").chmod(0o755)
|
||||
(fakes / "claude.py").write_text(CLAUDE)
|
||||
(fakes / "gremlins.py").write_text(GREMLINS)
|
||||
self.log = self.root / "docker.log"
|
||||
self.env = {**isolated_env(), "HOME": str(self.home), "PATH": f"{fakes}{os.pathsep}{os.environ['PATH']}", "FAKE_LOG": str(self.log), "FAKE_CLAUDE": str(fakes / "claude.py"), "CLAUDE_CODE_OAUTH_TOKEN": "sk-secret-token", "RUN_HEAVY_MEMORY": "2048"}
|
||||
self.given = {"--image": "golang:1.27.4", "--test": "sh calc_test.go", "--prefix": f"{sys.executable} {fakes / 'gremlins.py'}", "--seed": "1", "--bugs": "2", "--runs": "2"}
|
||||
self.git("init", "--quiet", "--initial-branch", "main")
|
||||
for name, text in (("calc.go", BASE_CODE), ("calc_test.go", TESTS), ("README.md", "# Calc\n")):
|
||||
(self.repo / name).write_text(text)
|
||||
# A link to nothing, which a copy that followed links would fail on.
|
||||
os.symlink("missing", self.repo / "dangling")
|
||||
self.base = self.commit("Base")
|
||||
self.git("checkout", "--quiet", "-b", "item-1")
|
||||
(self.repo / "calc.go").write_text(HEAD_CODE)
|
||||
self.head = self.commit("Head")
|
||||
# The bench runs from a folder below the checkout's top.
|
||||
(self.repo / "below").mkdir()
|
||||
self.enterContext(contextlib.chdir(self.repo / "below"))
|
||||
|
||||
def git(self, *args: str) -> str:
|
||||
return subprocess.run(["git", "-c", "user.name=t", "-c", "user.email=t@localhost", "-c", "commit.gpgsign=false", *args], cwd=self.repo, env=self.env, check=True, capture_output=True, text=True).stdout.strip()
|
||||
|
||||
def commit(self, title: str) -> str:
|
||||
self.git("add", "--all")
|
||||
self.git("commit", "--quiet", "--message", title)
|
||||
return self.git("rev-parse", "HEAD")
|
||||
|
||||
def bench(self, **given: str):
|
||||
with contextlib.redirect_stderr(io.StringIO()) as self.progress:
|
||||
return main.bench({**self.given, "--base": "main", **given}, self.env, PLACES, self.home)
|
||||
|
||||
def calls(self) -> list[list[str]]:
|
||||
return [json.loads(line) for line in self.log.read_text().splitlines()]
|
||||
|
||||
def test_plants_the_killed_mutant_the_head_left_alone_and_prints_how_each_side_fixed_it(self):
|
||||
report = self.bench()
|
||||
self.assertNotIsInstance(report, Failure, report)
|
||||
folder = pathlib.Path(report[-1].removeprefix("Each run's result and stream: "))
|
||||
self.assertEqual(report[:-1], [
|
||||
f"task-bench: base {self.base[:12]}, head {self.head[:12]}. 1 bug, each run 2 times at each side by claude-haiku-5-5 in Claude Code 2.1.330.",
|
||||
"Planted 1 of the 2 bugs asked for: no other mutant the tests kill becomes one.",
|
||||
"",
|
||||
"| Bug | Side | Passed | Input | Cache write | Cache read | Output | Turns | Wall, s | Reads |",
|
||||
"|---|---|---|---|---|---|---|---|---|---|",
|
||||
"| 1: calc.go:4:11 ARITHMETIC_BASE | base | 0/2 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |",
|
||||
"| 1: calc.go:4:11 ARITHMETIC_BASE | head | 2/2 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |",
|
||||
"| All | base | 0/2 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |",
|
||||
"| All | head | 2/2 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |",
|
||||
"| All | change | +2 | 0 % | 0 % | 0 % | 0 % | 0 | 0 % | 0 |",
|
||||
"",
|
||||
])
|
||||
self.assertTrue(folder.is_relative_to(self.home / ".local/state/larv-review") and folder.name == "task-bench", folder)
|
||||
results = json.loads((folder / "results.json").read_text())
|
||||
self.assertEqual((results["sides"], results["bugs"], len(results["runs"])), ({"base": self.base, "head": self.head}, {"1": "calc.go:4:11 ARITHMETIC_BASE"}, 4))
|
||||
self.assertEqual(results["runs"][0], {"bug": 1, "side": "base", "run": 1, "passed": False, "finished": True, "claude_code": "2.1.330", "model": "claude-haiku-5-5", "tokens": {"input": 40, "write_5m": 9000, "write_1h": 0, "read": 52000, "output": 800}, "turns": 3, "seconds": 61.5, "reads": 1})
|
||||
self.assertEqual(sorted(path.name for path in (folder / "runs").iterdir() if path.suffix == ".jsonl"), ["bug-1-base-1.jsonl", "bug-1-base-2.jsonl", "bug-1-head-1.jsonl", "bug-1-head-2.jsonl"])
|
||||
self.assertEqual(results["seed"], 1)
|
||||
self.assertEqual(self.progress.getvalue().splitlines(), [
|
||||
f"task-bench: base {self.base[:12]}, head {self.head[:12]}.",
|
||||
"task-bench: run 1 of 4, bug 1 at the base: not passed.",
|
||||
"task-bench: run 2 of 4, bug 1 at the head: passed.",
|
||||
"task-bench: run 3 of 4, bug 1 at the base: not passed.",
|
||||
"task-bench: run 4 of 4, bug 1 at the head: passed.",
|
||||
])
|
||||
self.assertEqual(sorted(path.name for path in folder.iterdir()), ["bugs", "results.json", "runs", "trees"])
|
||||
self.assertEqual((folder / "runs/bug-1-base-1.stderr").read_bytes(), b"")
|
||||
self.assertEqual(os.readlink(folder / "bugs/1/head/dangling"), "missing")
|
||||
|
||||
def test_the_command_line_runs_the_bench_on_the_seats_places_and_prints_its_report(self):
|
||||
argv = [part for name, value in self.given.items() for part in (name, value)]
|
||||
homes = []
|
||||
with contextlib.redirect_stdout(io.StringIO()) as out, contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertEqual(main.main(argv, self.env, lambda home: homes.append(home) or PLACES), 0)
|
||||
self.assertEqual(homes, [self.home])
|
||||
# Without --base, the base is the merge base with the default branch.
|
||||
self.assertEqual(out.getvalue().splitlines()[0], f"task-bench: base {self.base[:12]}, head {self.head[:12]}. 1 bug, each run 2 times at each side by claude-haiku-5-5 in Claude Code 2.1.330.")
|
||||
self.assertEqual(out.getvalue().splitlines()[3], "| Bug | Side | Passed | Input | Cache write | Cache read | Output | Turns | Wall, s | Reads |")
|
||||
self.assertTrue(out.getvalue().endswith("task-bench\n"), out.getvalue()[-80:])
|
||||
|
||||
def test_the_command_line_fails_with_the_reason_where_the_places_or_the_bench_fail(self):
|
||||
argv = [part for name, value in self.given.items() for part in (name, value)]
|
||||
for places, said in ((Failure("no history"), "no history\n"), ("- `gone.go:1` `Gone`: removed\n", "The places the seats found hard name no Go file that the base holds, so there is nowhere to plant a bug.\n")):
|
||||
with contextlib.redirect_stdout(io.StringIO()) as out, contextlib.redirect_stderr(io.StringIO()) as errors:
|
||||
self.assertEqual(main.main(argv, self.env, lambda home: places), 1)
|
||||
self.assertEqual((out.getvalue(), errors.getvalue().splitlines(keepends=True)[-1]), ("", said))
|
||||
|
||||
def test_the_head_is_the_commit_asked_for(self):
|
||||
report = self.bench(**{"--head": "main", "--runs": "1"})
|
||||
self.assertEqual(report[0], f"task-bench: base {self.base[:12]}, head {self.base[:12]}. 1 bug, each run 1 time at each side by claude-haiku-5-5 in Claude Code 2.1.330.")
|
||||
|
||||
def test_fails_on_a_head_or_a_base_git_does_not_know(self):
|
||||
for name in ("--head", "--base"):
|
||||
found = self.bench(**{name: "no-such-ref"})
|
||||
self.assertIsInstance(found, Failure)
|
||||
self.assertIn("no-such-ref", found.reason)
|
||||
|
||||
def test_the_image_is_built_from_the_one_asked_for_with_the_caps_memory(self):
|
||||
self.env["FAKE_STDIN"] = str(self.root / "dockerfile")
|
||||
self.env.pop("RUN_HEAVY_MEMORY")
|
||||
self.bench(**{"--runs": "1"})
|
||||
self.assertEqual((self.root / "dockerfile").read_text().splitlines()[0], "FROM golang:1.27.4")
|
||||
self.assertEqual(self.calls()[0][1:6], ["--quiet", "--memory", "4096m", "--memory-swap", "4096m"])
|
||||
|
||||
def test_gremlins_runs_through_the_prefix_in_the_copy_of_the_base_with_a_slots_workers(self):
|
||||
self.env["FAKE_GREMLINS_LOG"] = str(self.root / "gremlins.log")
|
||||
folder = pathlib.Path(self.bench(**{"--runs": "1"})[-1].removeprefix("Each run's result and stream: "))
|
||||
cwd, *argv = json.loads((self.root / "gremlins.log").read_text())
|
||||
self.assertEqual((cwd, argv[:2], argv[-2:]), (str(folder / "trees/base"), ["gremlins", "unleash"], ["--workers", str(main.CPUS_PER_SLOT)]))
|
||||
|
||||
def test_each_side_gets_its_own_tree_with_the_bug_committed_once_and_no_trace_of_the_checkout(self):
|
||||
folder = pathlib.Path(self.bench()[-1].removeprefix("Each run's result and stream: "))
|
||||
for side, code in (("base", BASE_CODE), ("head", HEAD_CODE)):
|
||||
tree = folder / "bugs/1" / side
|
||||
self.assertEqual((tree / "calc.go").read_text(), code.replace("a + b", "a - b"))
|
||||
log = subprocess.run(["git", "log", "--format=%an %s"], cwd=tree, env=self.env, check=True, capture_output=True, text=True).stdout
|
||||
self.assertEqual(log, "task-bench Import\n")
|
||||
self.assertEqual(subprocess.run(["git", "status", "--porcelain"], cwd=tree, env=self.env, check=True, capture_output=True, text=True).stdout, "")
|
||||
|
||||
def test_claude_runs_in_the_benchs_image_on_its_network_with_the_message_alone_and_the_token_by_name(self):
|
||||
self.bench(**{"--runs": "1"})
|
||||
calls = self.calls()
|
||||
fixes = [call for call in calls if "claude" in call]
|
||||
self.assertEqual(len(fixes), 2)
|
||||
name = fixes[0][fixes[0].index("--network") + 1]
|
||||
self.assertRegex(name, r"^task-bench-[0-9a-f]{12}$")
|
||||
image = fixes[0].index(f"{name}:throwaway")
|
||||
self.assertEqual(fixes[0][image + 1:], ["claude", "-p", "--strict-mcp-config", "--disable-slash-commands", "--no-session-persistence", "--model", "claude-haiku-5-5", "--effort", "low", "--output-format", "stream-json", "--verbose", "--permission-mode", "bypassPermissions", "A test of this project fails with the message below. Find the bug in the code and fix it. Change no test.\n\nAdd(1, 2) = -1, want 3"])
|
||||
options = fixes[0][:image]
|
||||
for expected in (["--memory", "2048m"], ["--memory-swap", "2048m"], ["--user", f"{os.getuid()}:{os.getgid()}"], ["--env", "HOME=/tmp"], ["--env", "CLAUDE_CODE_OAUTH_TOKEN"], ["--env", f"HTTPS_PROXY=http://{name}-proxy:8080"], ["--workdir", "/work"]):
|
||||
self.assertIn(expected, [options[at:at + 2] for at in range(len(options))])
|
||||
self.assertNotIn("sk-secret-token", self.log.read_text())
|
||||
self.assertEqual(calls[0][:1] + calls[0][-3:], ["build", "--tag", f"{name}:throwaway", "-"])
|
||||
self.assertEqual(calls[1], ["network", "create", "--internal", name])
|
||||
self.assertEqual([call[:2] for call in calls[-4:]], [["rm", "--force"], ["rm", "--force"], ["network", "rm"], ["image", "rm"]])
|
||||
|
||||
def test_the_tests_run_without_a_network(self):
|
||||
self.bench(**{"--runs": "1"})
|
||||
tests = [call for call in self.calls() if call[-3:] == ["sh", "-c", "sh calc_test.go"]]
|
||||
# Each side without a bug, each side with it, and each side's fix.
|
||||
self.assertEqual(len(tests), 6)
|
||||
for call in tests:
|
||||
self.assertEqual(call[call.index("--network") + 1], "none")
|
||||
|
||||
def test_benches_the_head_against_itself_where_the_base_is_the_head(self):
|
||||
report = self.bench(**{"--base": "HEAD", "--runs": "1", "--bugs": "1"})
|
||||
self.assertEqual(report[0], f"task-bench: base {self.head[:12]}, head {self.head[:12]}. 1 bug, each run 1 time at each side by claude-haiku-5-5 in Claude Code 2.1.330.")
|
||||
self.assertEqual(report[4:6], ["| 1: calc.go:5:11 ARITHMETIC_BASE | base | 1/1 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |", "| 1: calc.go:5:11 ARITHMETIC_BASE | head | 1/1 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |"])
|
||||
|
||||
def test_fails_where_the_places_name_no_go_file_of_the_base(self):
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
found = main.bench({**self.given, "--base": "main"}, self.env, "- `gone.go:1` `Gone`: removed\n- `README.md:1` `Calc`: thin\n", self.home)
|
||||
self.assertEqual(found, Failure("The places the seats found hard name no Go file that the base holds, so there is nowhere to plant a bug."))
|
||||
self.assertFalse(self.log.exists())
|
||||
|
||||
def test_fails_where_the_tests_fail_without_a_bug(self):
|
||||
found = self.bench(**{"--test": "echo broken; exit 2"})
|
||||
self.assertIsInstance(found, Failure)
|
||||
self.assertIn("The tests fail at the base without a planted bug", found.reason)
|
||||
self.assertIn("broken", found.reason)
|
||||
|
||||
def test_fails_where_no_mutant_becomes_a_bug(self):
|
||||
# Passing tests with the bug planted: no message to tell Haiku.
|
||||
(self.repo / "calc_test.go").write_text("exit 0\n")
|
||||
self.commit("Tests that pass with the bug")
|
||||
self.assertEqual(self.bench(), Failure("None of the 2 mutants the tests kill becomes a bug: each is on a line the head changed, or the test it fails doesn't print a message."))
|
||||
|
||||
def test_a_run_the_api_refuses_fails_the_bench_and_still_removes_what_it_made(self):
|
||||
self.env["FAKE_REFUSED_FROM"] = "1"
|
||||
self.assertEqual(self.bench(), Failure("claude could not use the API: API Error: 401"))
|
||||
self.assertEqual(self.calls()[-1][:2], ["image", "rm"])
|
||||
|
||||
def test_a_run_the_api_refuses_later_fails_the_bench_after_the_table_of_the_runs_that_finished(self):
|
||||
self.env["FAKE_REFUSED_FROM"] = "3"
|
||||
found = self.bench()
|
||||
self.assertIsInstance(found, Failure)
|
||||
lines = found.reason.splitlines()
|
||||
folder = next(self.home.glob(".local/state/larv-review/**/task-bench"))
|
||||
self.assertEqual(lines[:3], ["claude could not use the API: API Error: 401", f"task-bench stopped after 2 of 4 runs, which results.json holds in {folder}:", ""])
|
||||
self.assertEqual(lines[-3:-1], ["| All | base | 0/1 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |", "| All | head | 1/1 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |"])
|
||||
self.assertEqual([(run["side"], run["run"]) for run in json.loads((folder / "results.json").read_text())["runs"]], [("base", 1), ("head", 1)])
|
||||
|
||||
def test_a_change_to_another_file_neither_moves_nor_drops_the_bug(self):
|
||||
(self.repo / "notes.txt").write_text("one\ntwo\nthree\nfour\nfive\nsix\n")
|
||||
self.commit("Notes")
|
||||
report = self.bench(**{"--bugs": "1", "--runs": "1"})
|
||||
self.assertNotIsInstance(report, Failure, report)
|
||||
self.assertEqual(report[4].partition(" | ")[0], "| 1: calc.go:4:11 ARITHMETIC_BASE")
|
||||
|
||||
def test_the_seed_sets_which_mutant_becomes_the_first_bug(self):
|
||||
# Tests that catch both mutants, at a head benched against itself, where each mutant's line stands.
|
||||
(self.repo / "calc_test.go").write_text(TESTS.replace("calc.go &&", "calc.go && grep -q '2 \\* a' calc.go &&"))
|
||||
self.commit("Tests that catch Twice")
|
||||
mutants = [Mutant("calc.go", 5, 11, "ARITHMETIC_BASE"), Mutant("calc.go", 9, 11, "ARITHMETIC_BASE")]
|
||||
for seed in (1, 2, 3, 4):
|
||||
report = self.bench(**{"--base": "HEAD", "--bugs": "1", "--runs": "1", "--seed": str(seed)})
|
||||
self.assertNotIsInstance(report, Failure, report)
|
||||
first = drawn(mutants, seed)[0]
|
||||
self.assertEqual(report[4].partition(" | ")[0], f"| 1: calc.go:{first.line}:11 ARITHMETIC_BASE", seed)
|
||||
|
||||
def test_a_docker_that_fails_to_build_fails_the_bench(self):
|
||||
self.env["FAKE_FAILS"] = "build"
|
||||
found = self.bench()
|
||||
self.assertIsInstance(found, Failure)
|
||||
self.assertIn("docker build", found.reason)
|
||||
|
||||
|
||||
class MainTest(unittest.TestCase):
|
||||
def run_main(self, *argv: str, token: str = "t") -> tuple[int, str]:
|
||||
with contextlib.redirect_stderr(io.StringIO()) as errors, contextlib.redirect_stdout(io.StringIO()):
|
||||
return main.main(list(argv), {"CLAUDE_CODE_OAUTH_TOKEN": token, "HOME": "/nonexistent"}), errors.getvalue()
|
||||
|
||||
def test_refuses_an_unknown_repeated_or_valueless_argument(self):
|
||||
for argv, said in ((["--imgae", "x"], "unknown argument --imgae"), (["--image", "x", "--image=y"], "--image is given twice"), (["--image"], "--image needs a value")):
|
||||
code, errors = self.run_main(*argv)
|
||||
self.assertEqual((code, errors.startswith(f"{said}; usage: task-bench --image")), (1, True), errors)
|
||||
|
||||
def test_takes_a_value_after_an_equals_sign_too(self):
|
||||
with contextlib.redirect_stderr(io.StringIO()) as errors:
|
||||
self.assertEqual(main.main(["--image=golang:1.27.4", "--test=go test ./...", "--bugs=2"], {"CLAUDE_CODE_OAUTH_TOKEN": "t", "HOME": "/nonexistent"}, lambda home: Failure("read the places")), 1)
|
||||
self.assertEqual(errors.getvalue(), "read the places\n")
|
||||
|
||||
def test_refuses_a_missing_image_or_test_command(self):
|
||||
self.assertEqual(self.run_main("--test", "go test ./...")[1].partition(";")[0], "--image is missing")
|
||||
self.assertEqual(self.run_main("--image", "golang:1.27.4")[1].partition(";")[0], "--test is missing")
|
||||
|
||||
def test_refuses_a_count_that_is_no_whole_number_from_1(self):
|
||||
for name, value in (("--bugs", "0"), ("--runs", "-1"), ("--seed", "x")):
|
||||
self.assertEqual(self.run_main("--image", "i", "--test", "t", name, value)[1].partition(";")[0], f"{name} takes a whole number from 1, not {value}")
|
||||
|
||||
def test_refuses_to_start_without_a_token(self):
|
||||
self.assertEqual(self.run_main("--image", "i", "--test", "t", token="")[1].partition(";")[0], "CLAUDE_CODE_OAUTH_TOKEN holds no token")
|
||||
|
||||
def test_prints_its_help(self):
|
||||
with contextlib.redirect_stdout(io.StringIO()) as out:
|
||||
self.assertEqual((main.main(["--help"], {}), main.main(["-h"], {})), (0, 0))
|
||||
self.assertEqual(out.getvalue().count("usage: task-bench --image <image> --test <shell text>"), 2)
|
||||
|
||||
|
||||
def test_records_a_run_that_did_not_finish_without_figures(self):
|
||||
self.assertEqual(main.recorded(Result(1, "base", 2, False, None)), {"bug": 1, "side": "base", "run": 2, "passed": False, "finished": False})
|
||||
|
||||
|
||||
class Recorded:
|
||||
"""A docker that records each call and answers by the call's first word, else that it ran."""
|
||||
|
||||
def __init__(self, **answers: Ran | Failure):
|
||||
self.calls: list[tuple[list[str], float, str | None]] = []
|
||||
self.answers = answers
|
||||
|
||||
def __call__(self, args, *, seconds, stdin=None, stdout=None, stderr=None):
|
||||
self.calls.append((args, seconds, stdin))
|
||||
if stdout:
|
||||
stdout.write(b"stream")
|
||||
stderr.write(b"errors")
|
||||
return self.answers.get(args[0], Ran(0, "ran"))
|
||||
|
||||
|
||||
class InTurn(Recorded):
|
||||
"""A docker that answers each `run` with the next of `runs`."""
|
||||
|
||||
def __init__(self, *runs: Ran | Failure):
|
||||
super().__init__()
|
||||
self.runs = list(runs)
|
||||
|
||||
def __call__(self, args, *, seconds, stdin=None, stdout=None, stderr=None):
|
||||
answer = super().__call__(args, seconds=seconds, stdin=stdin, stdout=stdout, stderr=stderr)
|
||||
return self.runs.pop(0) if args[0] == "run" else answer
|
||||
|
||||
|
||||
class RunOutcomesTest(unittest.TestCase):
|
||||
MUTANT = Mutant("calc.go", 4, 11, "ARITHMETIC_BASE")
|
||||
|
||||
def setUp(self):
|
||||
folder = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(folder.cleanup)
|
||||
self.folder = pathlib.Path(folder.name)
|
||||
self.tree = self.folder / "tree"
|
||||
self.tree.mkdir()
|
||||
(self.tree / "calc.go").write_text(BASE_CODE)
|
||||
self.trees = {"base": str(self.tree), "head": str(self.tree)}
|
||||
self.task = main.Task(1, self.MUTANT, self.trees, {"base": "got 3", "head": "got 3"})
|
||||
|
||||
def containers(self, *runs: Ran | Failure) -> Containers:
|
||||
return Containers(InTurn(*runs), "bench", "go test ./...", 512, "1000:1000")
|
||||
|
||||
def test_a_fix_whose_tests_run_out_of_time_does_not_pass(self):
|
||||
late = TimedOut("docker run still ran after 30 min")
|
||||
self.assertEqual(main.fixed(self.task, "base", 1, self.containers(Ran(0, ""), late), self.folder), Result(1, "base", 1, False, None))
|
||||
|
||||
def test_a_fix_whose_tests_docker_could_not_run_fails_the_bench(self):
|
||||
broke = Failure("docker could not start: No such file or directory")
|
||||
self.assertEqual(main.fixed(self.task, "base", 1, self.containers(Ran(0, ""), broke), self.folder), broke)
|
||||
|
||||
def test_tests_docker_could_not_run_without_a_bug_fail_the_bench(self):
|
||||
broke = Failure("docker could not start: No such file or directory")
|
||||
self.assertEqual(main.benched({}, "", {}, self.trees, [], self.containers(broke), self.folder, None), broke)
|
||||
|
||||
def test_a_mutant_whose_tests_run_out_of_time_is_passed_over(self):
|
||||
late = TimedOut("docker run still ran after 30 min")
|
||||
self.assertIsNone(main.planted(1, self.MUTANT, "", {"base": "b", "head": "h"}, self.trees, self.containers(late), self.folder, lambda *args, **given: ""))
|
||||
|
||||
def test_a_mutant_whose_tests_docker_could_not_run_fails_the_bench(self):
|
||||
broke = Failure("docker could not start: No such file or directory")
|
||||
self.assertEqual(main.planted(1, self.MUTANT, "", {"base": "b", "head": "h"}, self.trees, self.containers(broke), self.folder, lambda *args, **given: ""), broke)
|
||||
|
||||
|
||||
class ContainersTest(unittest.TestCase):
|
||||
IN_TREE = ["--memory", "512m", "--memory-swap", "512m", "--user", "1000:1000", "--env", "HOME=/tmp", "--volume", "/tree:/work", "--workdir", "/work"]
|
||||
REMOVED = (["rm", "--force", "bench-run"], 120, None)
|
||||
|
||||
def setUp(self):
|
||||
folder = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(folder.cleanup)
|
||||
self.stream = pathlib.Path(folder.name, "run.jsonl")
|
||||
|
||||
def containers(self, **answers: Ran | Failure) -> Containers:
|
||||
self.docker = Recorded(**answers)
|
||||
return Containers(self.docker, "bench", "go test ./...", 512, "1000:1000")
|
||||
|
||||
def test_opens_the_image_the_network_without_a_route_out_and_the_proxy_on_both_networks(self):
|
||||
self.assertIsNone(opened(self.containers(), "golang:1.27.4"))
|
||||
proxy = str(pathlib.Path(runs.__file__).with_name("api_proxy.py"))
|
||||
self.assertEqual(self.docker.calls, [
|
||||
(["build", "--quiet", "--memory", "512m", "--memory-swap", "512m", "--tag", "bench:throwaway", "-"], 1200, "FROM golang:1.27.4\nUSER root\nRUN curl -fsSL https://claude.ai/install.sh | bash && cp -L /root/.local/bin/claude /usr/local/bin/claude\n"),
|
||||
(["network", "create", "--internal", "bench"], 60, None),
|
||||
(["run", "--detach", "--rm", "--name", "bench-proxy", "--memory", "256m", "--memory-swap", "256m", "--volume", f"{proxy}:/api_proxy.py:ro", "python:3.15.2-slim", "python", "/api_proxy.py", "api.anthropic.com:443"], 300, None),
|
||||
(["network", "connect", "bench", "bench-proxy"], 60, None),
|
||||
])
|
||||
|
||||
def test_opening_stops_at_the_first_step_that_fails_and_says_which(self):
|
||||
found = opened(self.containers(network=Ran(1, "pool overlaps")), "golang:1.27.4")
|
||||
self.assertEqual((found.reason.startswith("docker network"), "pool overlaps" in found.reason, len(self.docker.calls)), (True, True, 2))
|
||||
self.assertEqual(opened(self.containers(build=Failure("docker could not start")), "golang:1.27.4"), Failure("docker could not start"))
|
||||
|
||||
def test_closing_removes_the_running_container_the_proxy_the_network_and_the_image(self):
|
||||
closed(self.containers())
|
||||
self.assertEqual(self.docker.calls, [self.REMOVED, (["rm", "--force", "bench-proxy"], 120, None), (["network", "rm", "bench"], 120, None), (["image", "rm", "bench:throwaway"], 120, None)])
|
||||
|
||||
def test_the_tests_run_in_the_tree_without_a_network_for_at_most_half_an_hour(self):
|
||||
self.assertEqual(ran_tests(self.containers(), "/tree"), Ran(0, "ran"))
|
||||
self.assertEqual(self.docker.calls, [(["run", "--rm", "--name", "bench-run", "--network", "none", *self.IN_TREE, "bench:throwaway", "sh", "-c", "go test ./..."], 1800, None)])
|
||||
|
||||
def test_tests_that_run_out_of_time_are_a_failure_and_their_container_goes(self):
|
||||
late = TimedOut("docker run still ran after 30 min")
|
||||
self.assertEqual(ran_tests(self.containers(run=late), "/tree"), late)
|
||||
self.assertEqual(self.docker.calls[1:], [self.REMOVED])
|
||||
|
||||
def test_a_fix_writes_claudes_stream_and_its_errors_beside_it(self):
|
||||
self.assertIsNone(fix_attempted(self.containers(), "/tree", "got 3, want 4", self.stream))
|
||||
self.assertEqual((self.stream.read_bytes(), self.stream.with_suffix(".stderr").read_bytes()), (b"stream", b"errors"))
|
||||
(args, seconds, stdin), = self.docker.calls
|
||||
self.assertEqual((args[:6], seconds, stdin), (["run", "--rm", "--name", "bench-run", "--network", "bench"], 1800, None))
|
||||
self.assertEqual(args[6:18], self.IN_TREE)
|
||||
self.assertEqual(args[18:25], ["--env", "CLAUDE_CODE_OAUTH_TOKEN", "--env", "HTTPS_PROXY=http://bench-proxy:8080", "--env", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1", "bench:throwaway"])
|
||||
|
||||
def test_a_fix_that_runs_out_of_time_is_no_failure_and_its_container_goes(self):
|
||||
self.assertIsNone(fix_attempted(self.containers(run=TimedOut("docker run still ran after 30 min")), "/tree", "got 3", self.stream))
|
||||
self.assertEqual(self.docker.calls[1:], [self.REMOVED])
|
||||
|
||||
def test_a_run_whose_command_never_started_is_a_failure_that_says_what_docker_printed(self):
|
||||
for code in (125, 126, 127):
|
||||
self.assertEqual(ran_tests(self.containers(run=Ran(code, "sh: gotestsum: not found\n")), "/tree"), Failure(f"docker run could not start its command, and exited {code}: sh: gotestsum: not found"))
|
||||
self.assertEqual(ran_tests(self.containers(run=Ran(124, "late")), "/tree"), Ran(124, "late"))
|
||||
self.assertEqual(ran_tests(self.containers(run=Ran(128, "killed")), "/tree"), Ran(128, "killed"))
|
||||
self.assertEqual(fix_attempted(self.containers(run=Ran(125, "")), "/tree", "got 3", self.stream), Failure(f"docker run could not start its command, and exited 125; its errors are in {self.stream.with_suffix('.stderr')}"))
|
||||
|
||||
def test_a_fix_docker_could_not_start_is_a_failure(self):
|
||||
broke = Failure("docker could not start: No such file or directory")
|
||||
self.assertEqual(fix_attempted(self.containers(run=broke), "/tree", "got 3", self.stream), Failure(f"{broke.reason}; its errors are in {self.stream.with_suffix('.stderr')}"))
|
||||
self.assertEqual(self.docker.calls[1:], [self.REMOVED])
|
||||
|
||||
def test_a_stream_that_cannot_be_written_is_a_failure(self):
|
||||
found = fix_attempted(self.containers(), "/tree", "got 3", self.stream / "below" / "run.jsonl")
|
||||
self.assertTrue(found.reason.startswith(f"could not write {self.stream}/below/run.jsonl: "), found.reason)
|
||||
self.assertEqual(self.docker.calls, [])
|
||||
|
||||
|
||||
class DockerCliTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
folder = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(folder.cleanup)
|
||||
self.root = pathlib.Path(folder.name)
|
||||
(self.root / "docker").write_text('#!/bin/sh\necho "$@ $SAID"\ncat\necho broke >&2\nexit 3\n')
|
||||
(self.root / "docker").chmod(0o755)
|
||||
self.docker = docker_cli({"PATH": f"{self.root}{os.pathsep}{os.environ['PATH']}", "SAID": "caf\xe9"})
|
||||
|
||||
def test_runs_docker_in_the_environment_with_the_text_on_its_input_and_returns_its_code_and_all_it_printed(self):
|
||||
self.assertEqual(self.docker(["build", "-"], seconds=30, stdin="FROM x\n"), Ran(3, "build - caf\xe9\nFROM x\nbroke\n"))
|
||||
|
||||
def test_gives_docker_no_input_where_the_call_has_none(self):
|
||||
self.assertEqual(self.docker(["ps"], seconds=30), Ran(3, "ps caf\xe9\nbroke\n"))
|
||||
|
||||
def test_writes_the_output_and_the_errors_to_the_files_the_call_gives(self):
|
||||
with (self.root / "out").open("wb") as out, (self.root / "errors").open("wb") as errors:
|
||||
self.assertEqual(self.docker(["ps"], seconds=30, stdout=out, stderr=errors), Ran(3, ""))
|
||||
self.assertEqual(((self.root / "out").read_text(), (self.root / "errors").read_text()), ("ps caf\xe9\n", "broke\n"))
|
||||
|
||||
def test_output_that_is_no_utf_8_still_reads(self):
|
||||
(self.root / "docker").write_text("#!/bin/sh\nprintf 'a\\377b'\n")
|
||||
self.assertEqual(self.docker(["ps"], seconds=30), Ran(0, "a\ufffdb"))
|
||||
|
||||
def test_a_docker_that_is_not_there_is_a_failure(self):
|
||||
self.assertEqual(docker_cli({"PATH": str(self.root / "none")})(["ps"], seconds=30), Failure("docker could not start: No such file or directory"))
|
||||
|
||||
def test_a_docker_that_runs_too_long_is_a_failure_that_names_the_command_and_the_minutes(self):
|
||||
waited = []
|
||||
|
||||
def late(argv, timeout, **given):
|
||||
waited.append(timeout)
|
||||
raise subprocess.TimeoutExpired(argv, timeout)
|
||||
|
||||
self.assertEqual(docker_cli({}, late)(["run", "--rm"], seconds=90), TimedOut("docker run still ran after 2 min"))
|
||||
self.assertEqual(waited, [90])
|
||||
|
||||
|
||||
class CheckedTreeTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
folder = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(folder.cleanup)
|
||||
self.root = pathlib.Path(folder.name)
|
||||
|
||||
def tree(self, name: str, files: dict[str, str]) -> str:
|
||||
for path, text in files.items():
|
||||
(self.root / name / path).parent.mkdir(parents=True, exist_ok=True)
|
||||
(self.root / name / path).write_text(text)
|
||||
return str(self.root / name)
|
||||
|
||||
def held(self, name: str) -> dict[str, str]:
|
||||
top = self.root / name
|
||||
return {str(path.relative_to(top)): path.read_text() for path in sorted(top.rglob("*")) if path.is_file()}
|
||||
|
||||
def test_takes_the_code_as_fixed_and_the_tests_as_planted(self):
|
||||
planted = self.tree("planted", {"calc.go": "bug", "calc_test.go": "tests", "sub/old.go": "old", "sub/old_test.go": "old tests", ".git/HEAD": "ref"})
|
||||
copy = self.tree("copy", {"calc.go": "fixed", "calc_test.go": "rewritten", "extra_test.go": "added", "sub/new.go": "new", ".git/HEAD": "moved"})
|
||||
checked_tree(planted, copy, str(self.root / "check"))
|
||||
self.assertEqual(self.held("check"), {".git/HEAD": "ref", "calc.go": "fixed", "calc_test.go": "tests", "sub/new.go": "new", "sub/old_test.go": "old tests"})
|
||||
|
||||
def test_takes_the_files_under_testdata_as_planted_since_a_test_reads_what_it_expects_there(self):
|
||||
planted = self.tree("planted", {"calc.go": "bug", "testdata/want.golden": "3", "sub/testdata/gone.golden": "4", "testdata.go": "code"})
|
||||
copy = self.tree("copy", {"calc.go": "fixed", "testdata/want.golden": "-1", "testdata/added.golden": "5", "testdata.go": "fixed"})
|
||||
checked_tree(planted, copy, str(self.root / "check"))
|
||||
self.assertEqual(self.held("check"), {"calc.go": "fixed", "sub/testdata/gone.golden": "4", "testdata.go": "fixed", "testdata/want.golden": "3"})
|
||||
|
||||
def test_copies_a_link_as_a_link(self):
|
||||
planted = self.tree("planted", {"calc.go": "bug"})
|
||||
copy = self.tree("copy", {"calc.go": "fixed"})
|
||||
os.symlink("/etc/hostname", os.path.join(copy, "host.go"))
|
||||
checked_tree(planted, copy, str(self.root / "check"))
|
||||
self.assertEqual(os.readlink(self.root / "check/host.go"), "/etc/hostname")
|
||||
|
||||
def test_a_link_never_stands_in_for_a_folder_of_the_planted_tree_and_a_linked_folder_is_copied_as_a_link(self):
|
||||
planted = self.tree("planted", {"calc.go": "bug", "sub/old.go": "old", "gone/a.go": "a"})
|
||||
copy = self.tree("copy", {"calc.go": "fixed"})
|
||||
os.symlink("/etc", os.path.join(copy, "sub"))
|
||||
os.symlink("/etc", os.path.join(copy, "etc"))
|
||||
checked_tree(planted, copy, str(self.root / "check"))
|
||||
check = self.root / "check"
|
||||
self.assertEqual((check.joinpath("sub").is_symlink(), os.readlink(check / "etc")), (False, "/etc"))
|
||||
self.assertEqual(self.held("check"), {"calc.go": "fixed"})
|
||||
|
||||
|
||||
class ApiProxyTest(unittest.IsolatedAsyncioTestCase):
|
||||
async def asyncSetUp(self):
|
||||
async def echoed(reader, writer):
|
||||
writer.write(b"api:" + await reader.read(5))
|
||||
await writer.drain()
|
||||
writer.close()
|
||||
|
||||
# On an address of its own, so a tunnel to any other host doesn't find anything listening.
|
||||
api = await asyncio.start_server(echoed, "127.0.0.2", 0)
|
||||
self.allowed = f"127.0.0.2:{api.sockets[0].getsockname()[1]}"
|
||||
proxy = await asyncio.start_server(lambda reader, writer: api_proxy.served(self.allowed, reader, writer), "127.0.0.1", 0)
|
||||
self.port = proxy.sockets[0].getsockname()[1]
|
||||
for server in (api, proxy):
|
||||
self.addAsyncCleanup(server.wait_closed)
|
||||
self.addCleanup(server.close)
|
||||
|
||||
async def answered(self, request: bytes) -> bytes:
|
||||
reader, writer = await asyncio.open_connection("127.0.0.1", self.port)
|
||||
writer.write(request)
|
||||
answer = await asyncio.wait_for(reader.read(), 5)
|
||||
writer.close()
|
||||
return answer
|
||||
|
||||
async def test_tunnels_a_connect_to_the_one_allowed_target(self):
|
||||
self.assertEqual(await self.answered(f"CONNECT {self.allowed} HTTP/1.1\r\n\r\nhello".encode()), b"HTTP/1.1 200 Connection established\r\n\r\napi:hello")
|
||||
|
||||
async def test_answers_403_to_any_other_target_or_method(self):
|
||||
for request in (b"CONNECT example.com:443 HTTP/1.1\r\n\r\n", f"GET http://{self.allowed}/ HTTP/1.1\r\n\r\n".encode(), f"GET {self.allowed} HTTP/1.1\r\n\r\n".encode(), f"CONNECT {self.allowed}\r\n\r\n".encode(), b"junk\r\n\r\n"):
|
||||
self.assertEqual(await self.answered(request), b"HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\n\r\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -12,13 +12,12 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 501 | 0.15.0 | | owner | **Ship `task-bench`, which plants a bug the tests catch at the places the comprehension seats found hardest, has Haiku fix it in a vanilla Claude Code at a base and a head, and prints for each side how often the fix passes, with its median tokens, turns, wall time and reads before it opens a file the bug is in.** | 4 | 2 | 1 | 5 | 9 | 1.1 by 5, code health by 4 | 0.6 |
|
||||
| 502 | 0.15.0 | 501 | owner | **Distil each `task-bench` run into a digest of short traces and decoys, the files most failing runs read but no fix touches, and have a stronger model turn it into plan items before the transcripts are deleted.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 4, code health by 4 | 4.8 |
|
||||
| 502 | 0.15.0 | | owner | **Distil each `task-bench` run into a digest of short traces and decoys, the files most failing runs read but no fix touches, and have a stronger model turn it into plan items before the transcripts are deleted.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 4, code health by 4 | 4.8 |
|
||||
| 503 | 0.15.0 | 502 | owner | **Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3 | 1.2 |
|
||||
| 509 | 0.16.0 | 503 | owner | **Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.** | 3 | 2 | 3 | 6 | 5 | code health by 5, 1.1 by 3 | 5.6 |
|
||||
| 504 | 0.16.0 | 503 | owner | **Bench new features in `task-bench`, each a spec for Haiku with acceptance tests it never sees and a reference patch that proves it can be built, kept in a `bench-tasks` folder the project config names, starting with six for fejkdata.** | 3 | 2 | 1 | 4 | 6 | 1.1 by 4, code health by 3 | 2.6 |
|
||||
| 505 | 0.16.0 | 504 | owner | **Bench adding source data in `task-bench`: hand Haiku a fake register that a seeded script generates for a made-up country, and check the importer, the data and the licence entry it adds, starting with five tasks for fejkdata.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 |
|
||||
| 508 | 0.16.0 | 501 | owner | **Plant bugs for `task-bench` in Python, Rust, TypeScript and PHP, from the mutants `mutate-changes` runs for each, so this repo benches itself.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 |
|
||||
| 508 | 0.16.0 | | owner | **Plant bugs for `task-bench` in Python, Rust, TypeScript and PHP, from the mutants `mutate-changes` runs for each, so this repo benches itself.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 |
|
||||
| 506 | 0.16.0 | 504 | owner | **Go over the bench features in each release review: reapply each reference patch, drop a feature that no longer breaks a goal, and add features until they cover the places the seats find hardest.** | 2 | 2 | 2 | 4 | 3 | 1.1 by 3 | 1.1 |
|
||||
| 507 | 0.16.0 | 504 | owner | **Bench rebuilding a feature the project shipped: revert its commit at the head, hand Haiku its item tagline, and check with the tests that commit added.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3 | 0.7 |
|
||||
| 477 | 0.17.0 | | question | **Rewrite `## Goals` and `## Audience` for a setup others install and run apart from their own `~/.claude`.** | 1 | 1 | 2 | 2 | 2 | | 0.0 |
|
||||
@@ -105,29 +104,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 501. Ship `task-bench`, which plants a bug the tests catch at the places the comprehension seats found hardest, has Haiku fix it in a vanilla Claude Code at a base and a head, and prints for each side how often the fix passes, with its median tokens, turns, wall time and reads before it opens a file the bug is in.
|
||||
|
||||
The panel's scores barely move between chunks, and `reading-cost` measures chunks whose tasks differ. A planted task stays fixed, so a change in its cost comes from the code. The bench runs only programs; no model judges anything in it.
|
||||
|
||||
- **Plant.** Run Gremlins, as item 500 runs it, without `--diff`, over the files that hold the places `larv-review places` prints. Draw 5 of its `KILLED` mutants by a seeded pick, and re-apply each at its file, line and column. The task's text is the failing assertion's message, without the test's name, file or line, so it says what goes wrong and not where. Go is the first language; item 508 adds the others.
|
||||
- **Copy.** The tree at the commit, without `.git`, committed once with the bug in it, so `git diff` and `git log` show Haiku nothing.
|
||||
- **Run.** Inside a container that holds only the copy: the project's development image with Claude Code added, such as `golang` for fejkdata, signed in by a long-lived token in an environment variable, with a memory limit and the network only for the API. In it, `claude -p --strict-mcp-config --disable-slash-commands --no-session-persistence --model claude-haiku-5-5 --effort low --output-format stream-json` runs with every tool, Bash included, so Haiku greps, builds, runs the CLI and runs single tests as an agent normally does. A narrower tool list would raise the figures for reasons outside the code, and an allowlist can't fence Bash anyway: a rule that allows `cat` allows it on any path, such as the real checkout, whose diff would give away the bug. Run without `--bare`: it can't sign in by a long-lived token (the decision `Claude Code in a home of its own`). The container's fresh Claude home is empty, so none of the human's instructions, memories, hooks, skills or plugins load; they would skew the figures and change with claude-config. The repo's README and `docs/` stay in the copy, since they are part of what the bench measures. A project's `CLAUDE.md` loads from the working folder, or its `AGENTS.md` where it has no `CLAUDE.md`; the bench takes that as the project's own setup.
|
||||
- **Pair.** 5 runs per task at the base and at the head, in one bench, so a model or CLI update moves both sides. Each result names the Claude Code version and the model id.
|
||||
- **Check.** The project's test command over Haiku's changes to files other than tests, with the tests restored from the commit, so deleting a test doesn't pass.
|
||||
- **Measure.** From the stream: passed or not, tokens split into input, cache write, cache read and output, turns, wall time, and the reads before the first open of the file the bug is in.
|
||||
- **Print.** One row per task and side, then the head's change against the base. Results and transcripts go under the chunk's review folder.
|
||||
|
||||
It runs under `run-heavy`. The image, the test command and the mutator's prefix are flags, as in `mutate-changes`. Item 503 sets the task and run counts.
|
||||
|
||||
Touches:
|
||||
|
||||
- `bin/task-bench`
|
||||
- `bin/task_bench/main.py`
|
||||
- `bin/task_bench/plant.py`
|
||||
- `bin/task_bench/runs.py`
|
||||
- `bin/task_bench/test_task_bench.py`
|
||||
- `README.md`: `## Tools` (small), `` ### Packages under `bin/` `` (small)
|
||||
|
||||
### 502. Distil each `task-bench` run into a digest of short traces and decoys, the files most failing runs read but no fix touches, and have a stronger model turn it into plan items before the transcripts are deleted.
|
||||
|
||||
The thought process of the runs shows why Haiku failed, but whole transcripts would flood the model that reads them. A program distils them; only the last step uses a model.
|
||||
|
||||
Reference in New Issue
Block a user