Answer stability round 1: 0 🔴, 10 🟡 (0 misread); take the failed test's last message, fail on a docker run that never started, keep testdata as planted, show finished runs when the API refuses one, and test five survivors
This commit is contained in:
@@ -225,7 +225,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `settings_filter.py` is git's clean filter for `settings.json`: it drops `"model"` and sorts the keys. `.gitattributes` sets the filter `settings` on `settings.json`, and Install's `git config` line makes this script that filter.
|
||||
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
|
||||
- `subagent-rung [--plan] <model> <L>` prints the Agent tool's model and effort for a subagent's task, picked by its L from the project's model on the project's routing ladder, and with `--plan` never below the project's rung; `subagent-rung --help` states the rule.
|
||||
- `task-bench --image <image> --test <shell text> [--base <ref>] [--head <ref>]` plants bugs the tests catch where the comprehension seats found the code hardest, has Haiku fix each at the base and at the head, and prints how often the fix passes and what it took; `task-bench --help` says how it plants, runs and checks.
|
||||
- `task-bench --image <image> --test <shell text> [<options>]` plants bugs that the tests catch, at the places the comprehension seats found hardest, has Haiku fix each at the base and at the head, and prints how often the fix passes and what it took. It needs docker and a token from `claude setup-token`; `task-bench --help` says how it plants, runs and checks.
|
||||
- `todo-item` reads and changes a project's plan, whatever holds it: `list` prints the items in the plan's order and `show <ID>` one item's row and detail; `add …` files an item under the next free ID, `edit <ID> …` sets an item's cells or detail, and `move <ID> --release <version>` sets an item's release; `finish <ID>` removes a finished item and adds the bullet passed with `--changelog` to `CHANGELOG.md`, except for a plan kept in Jira. Each change ranks a markdown plan. `todo-item --help` states the flags.
|
||||
- `touch-predictions record <ID>...` appends what a chunk's items were predicted to touch, under `Touches:` in their details, beside the files and parts its branch touched, to the record the config names as `touch-predictions`; `touch-predictions --help` shows the spelling and how a part is named. `chunk-preflight` reads the same predictions to name the items that may start chunks at once.
|
||||
- `wait-for <file> [<pattern>] [--timeout <seconds>]` waits until a file a background job writes holds a line matching the pattern, and prints it.
|
||||
@@ -260,7 +260,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `review_rules/` holds the larv-review rules that several tools apply: its passes, its review folder, its round commit titles and the comprehension history's format.
|
||||
- `routing_figures/` measures loop-claude's archive, and each of its projects' default branches, for loop-claude's routing calls.
|
||||
- `rung_replay/` holds what `judge_replay` and `review_replay` share: the parallel replay runner, the argument reader, the candidate rungs and the exits, and `judge_replay`'s rung search. It reads `agent_cycle`'s retry count and `chunk_driver`'s models file, and finds that file through `loop_state`'s paths. Its `archive.py` keeps each replay's runs and its report in a `replays/` folder of loop-claude's archive.
|
||||
- `task_bench/` holds `task-bench`: `plant.py` picks the bugs from Gremlins' killed mutants, `runs.py` runs docker for each fix and each check, behind `api_proxy.py`, the one way out of a fix's network, `figures.py` reads a run's stream and prints the table, and `main.py` runs the bench.
|
||||
- `task_bench/` holds `task-bench`, which plants bugs in a Go project and has Haiku fix them in containers.
|
||||
- `test_env.py` serves the tests.
|
||||
- `text_files/` reads a Markdown file's `##` sections, quotes a passage and joins names into a list, reads the text piped into a command, stamps times in UTC, replaces a file in one step, private where it is new, and gzips a file beside its plain name, by which it reads it afterwards.
|
||||
- `transcript_archive/` stores loop-claude's transcripts. Its `project_key.py` names each project's archive folder, loop-claude session and models-file entry.
|
||||
@@ -275,8 +275,6 @@ CI runs `ci.sh` on every pull request, and `main` doesn't merge anything until i
|
||||
|
||||
To mutation-test the lines a branch changes, run `~/.claude/bin/run-heavy ~/.claude/bin/mutate-changes --prefix 'uvx --python "$(command -v python3)" --with pytest==9.2.0 --from "mutmut>=3,<4"'` in its checkout.
|
||||
|
||||
To bench how hard a Go project's code is to fix a bug in, at a branch's base and at its head, run `~/.claude/bin/run-heavy ~/.claude/bin/task-bench --image <image> --test '<test command>' --prefix '<how to run Gremlins>'` in its checkout, with a token from `claude setup-token` in `CLAUDE_CODE_OAUTH_TOKEN`. It needs docker, and a comprehension history that names hard places; `task-bench --help` gives a prefix.
|
||||
|
||||
## Tracking a new file
|
||||
|
||||
`.gitignore` is an allowlist. Track a file by negating it and its parent directories, then confirm with `git check-ignore -v <path>`:
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
task-bench runs it in a container of its own, as `python api_proxy.py <host:port> [<listen port>]`: the containers Haiku works in sit
|
||||
on a network with no route out, and reach the API through it and nothing else. It imports only the standard library, since that
|
||||
container holds nothing more."""
|
||||
container doesn't hold anything more."""
|
||||
import asyncio
|
||||
import contextlib
|
||||
import sys
|
||||
@@ -23,7 +23,7 @@ async def served(allowed: str, reader: asyncio.StreamReader, writer: asyncio.Str
|
||||
there_reader, there_writer = await asyncio.open_connection(host, int(port))
|
||||
writer.write(b"HTTP/1.1 200 Connection established\r\n\r\n")
|
||||
await writer.drain()
|
||||
# Either side closing ends the tunnel, so the other copy is cancelled rather than left waiting.
|
||||
# Either side closing ends the tunnel, so the other copy is cancelled.
|
||||
copies = [asyncio.ensure_future(copied(reader, there_writer)), asyncio.ensure_future(copied(there_reader, writer))]
|
||||
await asyncio.wait(copies, return_when=asyncio.FIRST_COMPLETED)
|
||||
for copy in copies:
|
||||
|
||||
@@ -14,7 +14,7 @@ HOME = "/tmp"
|
||||
# The tools that read or search without a shell. A Bash call counts where `reads_of` finds a file it reads.
|
||||
READING_TOOLS = frozenset({"Glob", "Grep", "Read"})
|
||||
SIDES = ("base", "head")
|
||||
HEADER = ["| Task | Side | Passed | Input | Cache write | Cache read | Output | Turns | Wall, s | Reads |", "|---|---|---|---|---|---|---|---|---|---|"]
|
||||
HEADER = ["| Bug | Side | Passed | Input | Cache write | Cache read | Output | Turns | Wall, s | Reads |", "|---|---|---|---|---|---|---|---|---|---|"]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -43,7 +43,7 @@ def figures(stream: bytes, bug_file: str) -> Figures | None | Failure:
|
||||
"""The figures of the run whose claude printed stream, with bug_file the bug's file as the run's container names it.
|
||||
|
||||
None where the stream holds no result. A Failure where the API refused the run, as on a token it doesn't take: such a run
|
||||
measures nothing, and every later run would end the same."""
|
||||
doesn't measure anything, and every later run would end the same."""
|
||||
events = [event for line in stream.splitlines() if (event := json_object(line))]
|
||||
result = next((event for event in reversed(events) if event.get("type") == "result"), None)
|
||||
if result is None:
|
||||
@@ -96,7 +96,7 @@ def row(task: str, side: str, results: list[Result]) -> str:
|
||||
|
||||
|
||||
def change(base: list[Result], head: list[Result]) -> str:
|
||||
"""The head's row less the base's: counts as they differ, and tokens and wall time as a share of the base's."""
|
||||
"""The head's row less the base's: a count as the difference, and tokens and wall time as that difference's share of the base's."""
|
||||
passed = sum(result.passed for result in head) - sum(result.passed for result in base)
|
||||
pairs = list(zip(medians(base), medians(head)))
|
||||
shares = [None if was is None or now is None or not was else f"{number(round(100 * (now - was) / was), signed=True)} %" for was, now in pairs]
|
||||
|
||||
+60
-45
@@ -5,44 +5,47 @@ usage: task-bench --image <image> --test <shell text> [--base <ref>] [--head <re
|
||||
|
||||
Run it in a checkout, under ~/.claude/bin/run-heavy, with a long-lived token in CLAUDE_CODE_OAUTH_TOKEN, which `claude setup-token`
|
||||
prints. The head is --head, by default HEAD. The base is the merge base of the head and --base, by default the default branch:
|
||||
origin's as last fetched, else main or master. `--base HEAD` benches the head against itself. No model judges anything: the bench
|
||||
runs programs, and reads Haiku's own stream.
|
||||
origin's as last fetched, else main or master. Where --base names the head's own commit, the bench runs the head against itself.
|
||||
No model judges anything: the bench runs programs, and reads Haiku's own stream.
|
||||
|
||||
- Plant. It copies the tree at the base and runs Gremlins there, as `gremlins unleash`, over the Go files that hold the places
|
||||
`larv-review places` prints, the ones the comprehension seats found hardest. Of the mutants the tests kill, it draws --bugs, 5 by
|
||||
default, in the order --seed sets, 1 by default. A drawn mutant becomes a bug where its line stands unchanged at the head, and
|
||||
where, planted at each side, the tests fail on an assertion whose message says something. Others are passed over, so a change to
|
||||
the very line of a bug is not benched. Go is the one language so far.
|
||||
- Run. Per bug and side, --runs times, 5 by default, sides in turn: a copy of that side's tree with the bug in it, committed once,
|
||||
so git shows nothing of the bug. In it, in a container, `claude -p --strict-mcp-config --disable-slash-commands
|
||||
--no-session-persistence --model claude-haiku-5-5 --effort low` works with every tool, told the failing assertion's message at that
|
||||
side, without the test's name, file or line. The container's Claude home is new and empty, so the project's own CLAUDE.md, or
|
||||
AGENTS.md, in the copy is all the setup that loads. Its network reaches api.anthropic.com and nothing else. A fix ends after 30
|
||||
minutes.
|
||||
`larv-review places` prints, the ones the comprehension seats found hardest. It takes the mutants the tests kill in the order
|
||||
--seed sets, 1 by default, until --bugs of them, 5 by default, have become bugs. A mutant becomes a bug where its line stands
|
||||
unchanged at the head, and where, planted at each side, a test fails and prints a message that says more than the test's name.
|
||||
Any other mutant is passed over, so a change to the very line of a bug is not benched. Go is the one language so far.
|
||||
- Run. Haiku fixes each bug --runs times at each side, 5 by default, the sides in turn. A fix starts from a copy of that side's
|
||||
tree with the bug in it, committed once, so git doesn't show which line the bug is on. In the copy, in a container, `claude -p
|
||||
--strict-mcp-config --disable-slash-commands --no-session-persistence --model claude-haiku-5-5 --effort low` works with every
|
||||
tool. It is told the message the failing test printed at that side, without the test's name, file or line. The container's
|
||||
Claude home is new and empty, so the project's own CLAUDE.md, or AGENTS.md, in the copy is all the setup that loads. Its network
|
||||
reaches api.anthropic.com and nothing else. A fix ends after 30 minutes.
|
||||
- Check. The test command runs over the planted tree with each file that is no test as Haiku left it; a fix passes where it exits 0.
|
||||
Tests Haiku changed, removed or added count for nothing.
|
||||
A test Haiku changed, removed or added doesn't count, and neither does a file under a `testdata` folder.
|
||||
|
||||
--image is the project's development image, such as golang:1.27.4. The bench adds Claude Code to it, as its installer installs it,
|
||||
so the image needs curl and bash; the result names the version. Haiku works in that image, and --test runs in it, as sh runs it at
|
||||
the copy's top, with no network: the image must hold what the tests download otherwise.
|
||||
so the image needs curl and bash; the result names the version. Haiku works in that image. So does --test: sh runs that text at
|
||||
the copy's top, with no network, so the image must hold what the tests would otherwise download.
|
||||
--prefix is shell text that runs Gremlins, on the host, as mutate-changes' --prefix does: sh runs `<prefix> "$@"` in the copy of the
|
||||
base, which $PWD names, with `gremlins unleash ...` as "$@". For example:
|
||||
base, which $PWD names, with `gremlins unleash ...` as "$@". Without it, the bench runs the `gremlins` on the host's PATH. For
|
||||
example:
|
||||
|
||||
docker run --rm --memory 8g --memory-swap 8g -v "$PWD:$PWD" -w "$PWD" golang:1.27.4 \
|
||||
sh -c 'shift; exec go run github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0 "$@"' sh
|
||||
|
||||
It prints the base and the head, then a table: a row per bug and side with how many fixes passed and the medians, over the runs
|
||||
that finished, of the input, cache-write, cache-read and output tokens, the turns, the wall time, and the reads and searches before
|
||||
the run first opened the file the bug is in; then a row per side over every bug, and the head's change against the base. Each
|
||||
run's result and stream are in the folder its last line names, `task-bench` in the chunk's review folder, which each bench empties
|
||||
first.
|
||||
the run first opened the file the bug is in. The median of the reads leaves out a run that never opened that file. Then come a row
|
||||
per side over every bug, and the head's change against the base. The last line names the folder that holds each run's result and
|
||||
stream: `task-bench` in the chunk's review folder. Each bench empties that `task-bench` folder first.
|
||||
|
||||
Exit 1, saying why, on an unknown or repeated argument, a missing --image or --test, a count that is no whole number from 1, or
|
||||
no token; and where git, docker or Gremlins fails, the seats' places name no Go file, the tests fail at a side without a bug, no
|
||||
mutant becomes a bug, or the API refuses a run.
|
||||
Exit 1, saying why, on an unknown or repeated argument, a missing --image or --test, a --seed, --bugs or --runs that is no whole
|
||||
number from 1, or no token; and where git, docker or Gremlins fails, the seats' places name no Go file, the tests fail at a side
|
||||
without a bug, or no mutant becomes a bug. Where the API refuses a run, the bench stops there and exits 1 too, after the table of
|
||||
the runs that finished.
|
||||
"""
|
||||
import dataclasses
|
||||
import functools
|
||||
import itertools
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
@@ -52,7 +55,7 @@ import tarfile
|
||||
import typing
|
||||
import uuid
|
||||
|
||||
from task_bench.runs import checked_tree, closed, Containers, docker_cli, fix_attempted, MEMORY_MIB, opened, ran_tests, TOKEN
|
||||
from task_bench.runs import checked_tree, closed, Containers, docker_cli, fix_attempted, MEMORY_MIB, opened, ran_tests, TimedOut, TOKEN
|
||||
from machine_load.slots import CPUS_PER_SLOT
|
||||
from git_cli.git_repo import default_ref, Git, top
|
||||
from task_bench.plant import drawn, killed_mutants, line_at_head, Mutant, mutated, place_files, task_text
|
||||
@@ -68,7 +71,7 @@ USAGE = "usage: task-bench --image <image> --test <shell text> [--base <ref>] [-
|
||||
# A panel of three readers, seated as the README's audience, picked these options on 2026-11-21, each 3 to 0.
|
||||
OPTIONS = ("--base", "--bugs", "--head", "--image", "--prefix", "--runs", "--seed", "--test")
|
||||
COUNTS = {"--bugs": "5", "--runs": "5", "--seed": "1"}
|
||||
# The planted commit names no one, and no hook or signing key of the caller's reaches it.
|
||||
# The planted commit doesn't name anyone, and neither a hook nor a signing key of the caller's reaches it.
|
||||
COMMIT = ["-c", "user.name=task-bench", "-c", "user.email=task-bench@localhost", "-c", "commit.gpgsign=false", "-c", "core.hooksPath=/dev/null", "commit", "--quiet", "--message", "Import"]
|
||||
|
||||
|
||||
@@ -109,8 +112,10 @@ def options(argv: list[str], env: typing.Mapping[str, str]) -> dict[str, str] |
|
||||
rest = list(argv)
|
||||
while rest:
|
||||
name, equals, value = rest.pop(0).partition("=")
|
||||
if name not in OPTIONS or name in given:
|
||||
return Failure(f"unknown or repeated argument {name}")
|
||||
if name not in OPTIONS:
|
||||
return Failure(f"unknown argument {name}")
|
||||
if name in given:
|
||||
return Failure(f"{name} is given twice")
|
||||
if not equals and not rest:
|
||||
return Failure(f"{name} needs a value")
|
||||
given[name] = value if equals else rest.pop(0)
|
||||
@@ -177,7 +182,8 @@ def commits(given: dict[str, str], checkout: str, git: Git) -> dict[str, str] |
|
||||
|
||||
|
||||
def archived(commit: str, into: pathlib.Path, checkout: str, git: Git) -> str | Failure:
|
||||
"""Makes `into` the tree at commit, without git's own data, and returns its path."""
|
||||
"""Makes `into` the tree at commit, as `git archive` holds it, and returns its path: without git's own data, and without a
|
||||
path the project marks `export-ignore`."""
|
||||
into.mkdir(parents=True)
|
||||
archive = into.with_suffix(".tar")
|
||||
wrote = git("archive", "--format=tar", "--output", str(archive), commit, cwd=checkout)
|
||||
@@ -207,31 +213,37 @@ def benched(given: dict[str, str], checkout: str, sides: dict[str, str], trees:
|
||||
return task
|
||||
tasks += [task] if task else []
|
||||
if not tasks:
|
||||
return Failure(f"None of the {len(mutants)} mutants the tests kill becomes a bug: each is on a line the head changed, or fails no assertion with a message.")
|
||||
return Failure(f"None of the {len(mutants)} mutants the tests kill becomes a bug: each is on a line the head changed, or the test it fails doesn't print a message.")
|
||||
results: list[Result] = []
|
||||
stopped = None
|
||||
total = int(given["--runs"]) * len(tasks) * len(SIDES)
|
||||
for number in range(1, int(given["--runs"]) + 1):
|
||||
for task in tasks:
|
||||
for side in SIDES:
|
||||
result = fixed(task, side, number, containers, folder)
|
||||
if isinstance(result, Failure):
|
||||
return result
|
||||
results.append(result)
|
||||
print(f"task-bench: run {len(results)} of {total}, bug {task.number} at the {side}: {'passed' if result.passed else 'not passed'}.", file=sys.stderr)
|
||||
(folder / "results.json").write_text(json.dumps({"sides": sides, "seed": int(given["--seed"]), "bugs": {task.number: task.named() for task in tasks}, "runs": [recorded(result) for result in results]}, indent="\t") + "\n", encoding="utf-8")
|
||||
for number, task, side in itertools.product(range(1, int(given["--runs"]) + 1), tasks, SIDES):
|
||||
result = fixed(task, side, number, containers, folder)
|
||||
if isinstance(result, Failure):
|
||||
stopped = result
|
||||
break
|
||||
results.append(result)
|
||||
print(f"task-bench: run {len(results)} of {total}, bug {task.number} at the {side}: {'passed' if result.passed else 'not passed'}.", file=sys.stderr)
|
||||
if stopped and not results:
|
||||
return stopped
|
||||
names = {task.number: task.named() for task in tasks}
|
||||
(folder / "results.json").write_text(json.dumps({"sides": sides, "seed": int(given["--seed"]), "bugs": names, "runs": [recorded(result) for result in results]}, indent="\t") + "\n", encoding="utf-8")
|
||||
if stopped:
|
||||
# The runs that finished took hours and tokens, so the bench shows them before it fails.
|
||||
return Failure("\n".join([stopped.reason, f"task-bench stopped after {len(results)} of {total} runs, which results.json holds in {folder}:", "", *table(results, names)]))
|
||||
ran_on = sorted({f"{found.model} in Claude Code {found.version}" for result in results if (found := result.figures)})
|
||||
short = [] if len(tasks) == int(given["--bugs"]) else [f"Planted {len(tasks)} of the {given['--bugs']} bugs asked for: no other mutant the tests kill becomes one."]
|
||||
return [
|
||||
f"task-bench: base {sides['base'][:12]}, head {sides['head'][:12]}. {counted(len(tasks), 'bug')}, fixed {counted(int(given['--runs']), 'time')} at each side by {' and '.join(ran_on) or 'no run that finished'}.",
|
||||
f"task-bench: base {sides['base'][:12]}, head {sides['head'][:12]}. {counted(len(tasks), 'bug')}, each run {counted(int(given['--runs']), 'time')} at each side by {' and '.join(ran_on) or 'no run that finished'}.",
|
||||
*short,
|
||||
"",
|
||||
*table(results, {task.number: task.named() for task in tasks}),
|
||||
*table(results, names),
|
||||
"",
|
||||
]
|
||||
|
||||
|
||||
def planted(number: int, mutant: Mutant, checkout: str, sides: dict[str, str], trees: dict[str, str], containers: Containers, folder: pathlib.Path, git: Git) -> Task | Failure | None:
|
||||
"""The task of fixing mutant, with its planted tree at each side; None where the mutant is no bug at both sides.
|
||||
"""The task of fixing mutant, with its planted tree at each side; None where the mutant doesn't become a bug at one side or the other.
|
||||
|
||||
The bug is the same change to the same line at both sides, so only the code around it differs between them."""
|
||||
diff = git("diff", "-U0", "--no-color", "--no-ext-diff", "--no-textconv", sides["base"], sides["head"], "--", mutant.path, cwd=checkout, errors="replace")
|
||||
@@ -251,9 +263,10 @@ def planted(number: int, mutant: Mutant, checkout: str, sides: dict[str, str], t
|
||||
shutil.copytree(trees[side], tree, symlinks=True)
|
||||
(tree / mutant.path).write_bytes(changed)
|
||||
ran = tests_in_copy(containers, str(tree), folder)
|
||||
if isinstance(ran, Failure):
|
||||
if isinstance(ran, Failure) and not isinstance(ran, TimedOut):
|
||||
return ran
|
||||
message = task_text(ran.output) if ran.code else None
|
||||
# Tests that never end under the bug have no message to tell Haiku.
|
||||
message = None if isinstance(ran, TimedOut) else task_text(ran.output)
|
||||
if message is None:
|
||||
return None
|
||||
for args in (["init", "--quiet"], ["add", "--all", "--force"], COMMIT):
|
||||
@@ -265,7 +278,7 @@ def planted(number: int, mutant: Mutant, checkout: str, sides: dict[str, str], t
|
||||
|
||||
|
||||
def tests_in_copy(containers: Containers, tree: str, folder: pathlib.Path) -> Ran | Failure:
|
||||
"""The test command's run in a copy of tree that goes afterwards, so what the tests write stays out of tree."""
|
||||
"""The test command's run in a copy of tree, removed afterwards, so what the tests write stays out of tree."""
|
||||
copy = folder / "tested"
|
||||
shutil.copytree(tree, copy, symlinks=True)
|
||||
try:
|
||||
@@ -276,7 +289,7 @@ def tests_in_copy(containers: Containers, tree: str, folder: pathlib.Path) -> Ra
|
||||
|
||||
def fixed(task: Task, side: str, number: int, containers: Containers, folder: pathlib.Path) -> Result | Failure:
|
||||
"""One run: Haiku's fix of task at side in a copy of its planted tree, then the tests over what it changed. The stream stays in
|
||||
folder; the copy goes."""
|
||||
folder; the copy is removed. Tests that run out of time don't pass."""
|
||||
stream = folder / "runs" / f"bug-{task.number}-{side}-{number}.jsonl"
|
||||
stream.parent.mkdir(exist_ok=True)
|
||||
copy, check = folder / "copy", folder / "check"
|
||||
@@ -290,7 +303,9 @@ def fixed(task: Task, side: str, number: int, containers: Containers, folder: pa
|
||||
return found
|
||||
checked_tree(task.trees[side], str(copy), str(check))
|
||||
ran = ran_tests(containers, str(check))
|
||||
return Result(task.number, side, number, not isinstance(ran, Failure) and ran.code == 0, found)
|
||||
if isinstance(ran, Failure) and not isinstance(ran, TimedOut):
|
||||
return ran
|
||||
return Result(task.number, side, number, isinstance(ran, Ran) and ran.code == 0, found)
|
||||
finally:
|
||||
shutil.rmtree(copy, ignore_errors=True)
|
||||
shutil.rmtree(check, ignore_errors=True)
|
||||
|
||||
+31
-17
@@ -28,13 +28,15 @@ MUTATIONS: dict[str, dict[bytes, bytes]] = {
|
||||
"REMOVE_SELF_ASSIGNMENTS": dict.fromkeys((b"+=", b"&=", b"&^=", b"*=", b"|=", b"/=", b"%=", b"<<=", b">>=", b"-=", b"^="), b"="),
|
||||
}
|
||||
HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@", re.MULTILINE)
|
||||
# go test's report of a failed assertion: ` pick_test.go:42: got 3, want 4`, under the `--- FAIL: TestPick (0.00s)` of its test.
|
||||
ASSERTION = re.compile(r"^( *)\S*_test\.go:\d+:(.*)$")
|
||||
# What a test printed, as go test shows it: ` pick_test.go:42: got 3, want 4`. A failed assertion and a log line look the same.
|
||||
PRINTED = re.compile(r"^( *)\S*_test\.go:\d+:(.*)$")
|
||||
# The lines that say whose output follows: `=== RUN TestPick` with -v, and `--- FAIL: TestPick (0.00s)` with and without it.
|
||||
TEST_MARK = re.compile(r"^( *)(?:=== (?:RUN|CONT|NAME)|--- (?:FAIL|PASS|SKIP):)\s+(\S+)")
|
||||
FAILED_TEST = re.compile(r"^\s*--- FAIL: (\S+)", re.MULTILINE)
|
||||
|
||||
|
||||
class Mutant(typing.NamedTuple):
|
||||
"""A mutant Gremlins' tests killed: the mutator and the token it changed, by its line and its column in bytes, both from 1."""
|
||||
"""A mutant the project's tests killed, as Gremlins reports it: the mutator and the token it changed, by its line and its column in bytes, both from 1."""
|
||||
|
||||
path: str
|
||||
line: int
|
||||
@@ -90,7 +92,7 @@ def line_at_head(diff: str, line: int) -> int | None:
|
||||
diff is `git diff -U0 <base> <head> -- <file>`, so each hunk holds changed lines only."""
|
||||
moved = 0
|
||||
for old_start, old_count, new_start, new_count in ((int(a), int(b or 1), int(c), int(d or 1)) for a, b, c, d in HUNK.findall(diff)):
|
||||
# A hunk that only adds names the line it adds after, and one that only removes the line the gap is after.
|
||||
# A hunk that only adds lines names the line it adds them after. One that only removes lines names the line the gap follows.
|
||||
old_end = old_start + old_count if old_count else old_start + 1
|
||||
new_end = new_start + new_count if new_count else new_start + 1
|
||||
if old_count and old_start <= line < old_end:
|
||||
@@ -101,19 +103,31 @@ def line_at_head(diff: str, line: int) -> int | None:
|
||||
|
||||
|
||||
def task_text(output: str) -> str | None:
|
||||
"""What Haiku is told of a planted bug: the message of the first assertion that failed in go test's output, without the test's
|
||||
name, file or line, so it says what goes wrong and not where. None where no assertion failed, as on a panic, a build that
|
||||
broke or a passing run, or where its message says nothing beyond the test."""
|
||||
"""What Haiku is told of a planted bug: the failing assertion's message in go test's output, without the test's name, file or
|
||||
line, so it says what goes wrong and not where. That is the last message of the first failed test that printed one, since a
|
||||
test logs before it asserts. None where no failed test printed a message, as on a panic, a build that broke or a passing run,
|
||||
or where the message only names the test."""
|
||||
lines = output.splitlines()
|
||||
failed = set(FAILED_TEST.findall(output))
|
||||
# A subtest is named `TestPick/empty_input`; a message may name the whole or the test it belongs to.
|
||||
tests = {part for name in FAILED_TEST.findall(output) for part in (name, name.partition("/")[0])}
|
||||
tests = {part for name in failed for part in (name, name.partition("/")[0])}
|
||||
marks: list[tuple[int, str]] = []
|
||||
last: dict[str, tuple[int, re.Match[str]]] = {}
|
||||
for index, line in enumerate(lines):
|
||||
if found := ASSERTION.match(line):
|
||||
indent = len(found[1])
|
||||
# A message's later lines are indented deeper than its first.
|
||||
later = lines[index + 1:]
|
||||
ends = next((at for at, text in enumerate(later) if len(text) - len(text.lstrip(" ")) <= indent), len(later))
|
||||
message = [text.strip() for text in (found[2], *later[:ends])]
|
||||
kept = [text for text in message if text and "_test.go" not in text and not any(test in text for test in tests)]
|
||||
return "\n".join(kept) or None
|
||||
return None
|
||||
if mark := TEST_MARK.match(line):
|
||||
marks.append((len(mark[1]), mark[2]))
|
||||
elif printed := PRINTED.match(line):
|
||||
# A test's output is indented deeper than its mark, so a line after a subtest's may be its parent's.
|
||||
test = next((name for indent, name in reversed(marks) if indent < len(printed[1])), None)
|
||||
if test is not None and test in failed:
|
||||
last[test] = index, printed
|
||||
if not last:
|
||||
return None
|
||||
index, found = next(iter(last.values()))
|
||||
indent = len(found[1])
|
||||
# A message's later lines are indented deeper than its first.
|
||||
later = lines[index + 1:]
|
||||
ends = next((at for at, text in enumerate(later) if len(text) - len(text.lstrip(" ")) <= indent), len(later))
|
||||
message = [text.strip() for text in (found[2], *later[:ends])]
|
||||
kept = [text for text in message if text and "_test.go" not in text and not any(name in text for name in tests)]
|
||||
return "\n".join(kept) or None
|
||||
|
||||
+43
-20
@@ -30,13 +30,19 @@ DOCKERFILE = 'FROM {image}\nUSER root\nRUN curl -fsSL https://claude.ai/install.
|
||||
BUILD_SECONDS = 20 * 60
|
||||
# How long one fix, or one run of the tests, may take before its container is removed; the run then counts as not passed.
|
||||
RUN_SECONDS = 30 * 60
|
||||
# What `docker run` exits with where the container, or its command, never started.
|
||||
NOT_STARTED = (125, 126, 127)
|
||||
# The memory each container gets without run-heavy, which hands its slot's share in RUN_HEAVY_MEMORY, in MiB.
|
||||
MEMORY_MIB = 4096
|
||||
|
||||
|
||||
class TimedOut(Failure):
|
||||
"""docker still ran when its time was up."""
|
||||
|
||||
|
||||
class Docker(typing.Protocol):
|
||||
"""Runs `docker <args>` and waits for it, at most `seconds`. Its output goes to `stdout` and its errors to `stderr` where the
|
||||
call gives them, and into the Ran otherwise."""
|
||||
"""Runs `docker <args>` and waits for it, at most `seconds`, and is TimedOut after that. Its output goes to `stdout` and its
|
||||
errors to `stderr` where the call gives them, and into the Ran otherwise."""
|
||||
|
||||
def __call__(self, args: list[str], *, seconds: float, stdin: str | None = None, stdout: typing.IO[bytes] | None = None, stderr: typing.IO[bytes] | None = None) -> Ran | Failure: ...
|
||||
|
||||
@@ -48,7 +54,7 @@ def docker_cli(env: typing.Mapping[str, str], run: typing.Callable[..., subproce
|
||||
except OSError as failure:
|
||||
return Failure(f"docker could not start: {failure.strerror}")
|
||||
except subprocess.TimeoutExpired:
|
||||
return Failure(f"docker {args[0]} still ran after {round(seconds / 60)} min")
|
||||
return TimedOut(f"docker {args[0]} still ran after {round(seconds / 60)} min")
|
||||
return Ran(ran.returncode, (ran.stdout or b"").decode("utf-8", errors="replace"))
|
||||
|
||||
return docker
|
||||
@@ -100,18 +106,16 @@ def opened(containers: Containers, image: str) -> Failure | None:
|
||||
|
||||
|
||||
def closed(containers: Containers) -> None:
|
||||
"""Removes what `opened` made and any container still running, where each exists; a bench that broke half-way leaves the rest."""
|
||||
"""Removes what `opened` made and any container still running. A bench that broke half-way made only some of them, and removing
|
||||
one that is missing doesn't stop the rest."""
|
||||
for args in (["rm", "--force", containers.running], ["rm", "--force", containers.proxy], ["network", "rm", containers.name], ["image", "rm", containers.image]):
|
||||
containers.docker(args, seconds=120)
|
||||
|
||||
|
||||
def ran_tests(containers: Containers, tree: str) -> Ran | Failure:
|
||||
"""The project's test command, run at the top of tree with no network; the tests pass where it exits 0. A Failure where it still
|
||||
ran after RUN_SECONDS."""
|
||||
ran = containers.docker(["run", "--rm", "--name", containers.running, "--network", "none", *in_tree(containers, tree), containers.image, "sh", "-c", containers.test], seconds=RUN_SECONDS)
|
||||
if isinstance(ran, Failure):
|
||||
containers.docker(["rm", "--force", containers.running], seconds=120)
|
||||
return ran
|
||||
"""The project's test command, run at the top of tree with no network; the tests pass where it exits 0. TimedOut where it still
|
||||
ran after RUN_SECONDS, and a Failure where docker could not run it."""
|
||||
return run_in_image(containers, ["--network", "none", *in_tree(containers, tree), containers.image, "sh", "-c", containers.test])
|
||||
|
||||
|
||||
def fix_attempted(containers: Containers, copy: str, message: str, stream: pathlib.Path) -> Failure | None:
|
||||
@@ -119,18 +123,32 @@ def fix_attempted(containers: Containers, copy: str, message: str, stream: pathl
|
||||
errors beside it. A fix that still runs after RUN_SECONDS ends there, with the stream as far as it came.
|
||||
|
||||
The container's home is new and empty, so no instruction, memory, hook, skill or plugin of the caller loads; and docker hands the
|
||||
token on by its name, so its value is on no command line. A Failure where docker itself could not run."""
|
||||
args = ["run", "--rm", "--name", containers.running, "--network", containers.name, *in_tree(containers, copy), "--env", TOKEN, "--env", f"HTTPS_PROXY=http://{containers.proxy}:{PORT}", "--env", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1", containers.image, *CLAUDE, PROMPT.format(message=message)]
|
||||
token on by its name, so its value is on no command line. A Failure where docker could not run claude."""
|
||||
args = ["--network", containers.name, *in_tree(containers, copy), "--env", TOKEN, "--env", f"HTTPS_PROXY=http://{containers.proxy}:{PORT}", "--env", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1", containers.image, *CLAUDE, PROMPT.format(message=message)]
|
||||
errors_file = stream.with_suffix(".stderr")
|
||||
try:
|
||||
with stream.open("wb") as out, stream.with_suffix(".stderr").open("wb") as errors:
|
||||
ran = containers.docker(args, seconds=RUN_SECONDS, stdout=out, stderr=errors)
|
||||
with stream.open("wb") as out, errors_file.open("wb") as errors:
|
||||
ran = run_in_image(containers, args, stdout=out, stderr=errors)
|
||||
except OSError as failure:
|
||||
return Failure(f"could not write {stream}: {failure}")
|
||||
# Out of time is the run's outcome, not the bench's failure.
|
||||
if isinstance(ran, TimedOut) or not isinstance(ran, Failure):
|
||||
return None
|
||||
return Failure(f"{ran.reason}; its errors are in {errors_file}")
|
||||
|
||||
|
||||
def run_in_image(containers: Containers, args: list[str], stdout: typing.IO[bytes] | None = None, stderr: typing.IO[bytes] | None = None) -> Ran | Failure:
|
||||
"""One `docker run` as the bench's one running container, for at most RUN_SECONDS, after which the container is removed.
|
||||
|
||||
A run whose command never started is a Failure, since the next would end the same: counted as a fix that failed, it would fill
|
||||
the table with runs that measured docker."""
|
||||
ran = containers.docker(["run", "--rm", "--name", containers.running, *args], seconds=RUN_SECONDS, stdout=stdout, stderr=stderr)
|
||||
if isinstance(ran, Failure):
|
||||
containers.docker(["rm", "--force", containers.running], seconds=120)
|
||||
# Out of time is the run's outcome, not the bench's failure.
|
||||
return None if "still ran" in ran.reason else ran
|
||||
return None
|
||||
return ran
|
||||
if ran.code in NOT_STARTED:
|
||||
return Failure(f"docker run could not start its command, and exited {ran.code}: {ran.output.strip()}".removesuffix(": "))
|
||||
return ran
|
||||
|
||||
|
||||
def in_tree(containers: Containers, tree: str) -> list[str]:
|
||||
@@ -140,19 +158,20 @@ def in_tree(containers: Containers, tree: str) -> list[str]:
|
||||
|
||||
def checked_tree(planted: str, copy: str, into: str) -> None:
|
||||
"""Makes at `into` the tree the tests check a fix in: `planted`, the code with the bug as Haiku got it, with each file that is no
|
||||
test as `copy` holds it after the fix. So a test Haiku changed, removed or added counts for nothing.
|
||||
test as `copy` holds it after the fix. So a test Haiku changed, removed or added doesn't count, and neither does a file under
|
||||
a `testdata` folder, where Go's tests keep the output they expect.
|
||||
|
||||
Raises OSError where a file cannot be copied."""
|
||||
shutil.copytree(planted, into, symlinks=True)
|
||||
before, after = set(entries(planted)), set(entries(copy))
|
||||
for path in before - after:
|
||||
if not is_test(path):
|
||||
if not of_tests(path):
|
||||
with contextlib.suppress(FileNotFoundError):
|
||||
os.unlink(os.path.join(into, path))
|
||||
for path in sorted(after):
|
||||
target = os.path.join(into, path)
|
||||
# A link is copied as a link, never followed; and one that would stand in for a folder of the planted tree is left out.
|
||||
if is_test(path) or (os.path.isdir(target) and not os.path.islink(target)):
|
||||
if of_tests(path) or (os.path.isdir(target) and not os.path.islink(target)):
|
||||
continue
|
||||
os.makedirs(os.path.dirname(target), exist_ok=True)
|
||||
with contextlib.suppress(FileNotFoundError):
|
||||
@@ -160,6 +179,10 @@ def checked_tree(planted: str, copy: str, into: str) -> None:
|
||||
shutil.copy2(os.path.join(copy, path), target, follow_symlinks=False)
|
||||
|
||||
|
||||
def of_tests(path: str) -> bool:
|
||||
return is_test(path) or "testdata" in pathlib.PurePath(path).parts
|
||||
|
||||
|
||||
def entries(top: str) -> list[str]:
|
||||
"""Each file and link under top, by its path from top, without git's own under `.git`."""
|
||||
found = []
|
||||
|
||||
@@ -87,7 +87,7 @@ class TableTest(unittest.TestCase):
|
||||
run(2, "head", 1, True, input_=100, turns=9, seconds=45, reads=2),
|
||||
]
|
||||
self.assertEqual(table(results, {1: "calc.go:4:7 CONDITIONALS_NEGATION", 2: "pick.go:9:3 ARITHMETIC_BASE"}), [
|
||||
"| Task | Side | Passed | Input | Cache write | Cache read | Output | Turns | Wall, s | Reads |",
|
||||
"| Bug | Side | Passed | Input | Cache write | Cache read | Output | Turns | Wall, s | Reads |",
|
||||
"|---|---|---|---|---|---|---|---|---|---|",
|
||||
"| 1: calc.go:4:7 CONDITIONALS_NEGATION | base | 2/3 | 200 | 1000 | 5000 | 400 | 12 | 90 | 4 |",
|
||||
"| 1: calc.go:4:7 CONDITIONALS_NEGATION | head | 3/3 | 100 | 1000 | 5000 | 400 | 8 | 45 | 1 |",
|
||||
@@ -111,3 +111,7 @@ class TableTest(unittest.TestCase):
|
||||
"| All | head | 1/1 | 100 | 1000 | 5000 | 400 | 10 | 60 | 3 |",
|
||||
"| All | change | +1 | - | - | - | - | - | - | - |",
|
||||
])
|
||||
|
||||
def test_the_change_is_a_dash_where_only_the_head_has_no_figures(self):
|
||||
results = [run(1, "base", 1, True), Result(1, "head", 1, False, None)]
|
||||
self.assertEqual(table(results, {1: "calc.go:4:7 ARITHMETIC_BASE"})[-1], "| All | change | -1 | - | - | - | - | - | - | - |")
|
||||
|
||||
@@ -103,7 +103,7 @@ class MutatedTest(unittest.TestCase):
|
||||
|
||||
|
||||
class TaskTextTest(unittest.TestCase):
|
||||
def test_is_the_first_failing_assertions_message_without_the_tests_name_file_or_line(self):
|
||||
def test_is_the_first_failed_tests_message_without_the_tests_name_file_or_line(self):
|
||||
self.assertEqual(task_text(GO_FAILURE), "Pick() = 3,\nwant 4")
|
||||
|
||||
def test_leaves_out_each_line_that_names_the_test_or_its_file(self):
|
||||
@@ -113,7 +113,21 @@ class TaskTextTest(unittest.TestCase):
|
||||
self.assertIsNone(task_text(PANIC))
|
||||
self.assertIsNone(task_text("ok \texample.com/calc\t0.004s\n"))
|
||||
|
||||
def test_is_none_where_the_message_says_nothing_beyond_the_test(self):
|
||||
def test_takes_the_message_of_a_test_that_failed_not_what_a_passing_test_logged(self):
|
||||
output = "=== RUN TestServer\n server_test.go:12: listening on 127.0.0.1:0\n--- PASS: TestServer (0.00s)\n=== RUN TestPick\n pick_test.go:42: got 3, want 4\n--- FAIL: TestPick (0.00s)\nFAIL\n"
|
||||
self.assertEqual(task_text(output), "got 3, want 4")
|
||||
|
||||
def test_takes_the_last_message_of_the_failed_test_since_a_test_logs_before_it_asserts(self):
|
||||
self.assertEqual(task_text("--- FAIL: TestPick (0.00s)\n pick_test.go:40: input [1 2 3]\n pick_test.go:42: got 3, want 4\nFAIL\n"), "got 3, want 4")
|
||||
|
||||
def test_takes_the_first_failed_test_that_has_a_message(self):
|
||||
output = "--- FAIL: TestPick (0.00s)\n --- FAIL: TestPick/empty (0.00s)\n pick_test.go:42: got 3\n--- FAIL: TestLast (0.00s)\n pick_test.go:50: got 5\nFAIL\n"
|
||||
self.assertEqual(task_text(output), "got 3")
|
||||
|
||||
def test_is_none_where_only_passing_tests_logged(self):
|
||||
self.assertIsNone(task_text("=== RUN TestServer\n server_test.go:12: listening\n--- PASS: TestServer (0.00s)\nPASS\n"))
|
||||
|
||||
def test_is_none_where_the_message_does_not_say_anything_beyond_the_test(self):
|
||||
self.assertIsNone(task_text("--- FAIL: TestPick (0.00s)\n pick_test.go:42: TestPick failed\nFAIL\n"))
|
||||
|
||||
|
||||
@@ -187,7 +201,7 @@ class EdgesTest(unittest.TestCase):
|
||||
self.assertEqual(mutated(b"a &^ b", Mutant("calc.go", 1, 3, "INVERT_BITWISE")), b"a & b")
|
||||
|
||||
def test_a_message_ends_at_the_next_line_indented_as_its_first(self):
|
||||
output = "--- FAIL: TestPick/empty_input (0.00s)\n pick_test.go:42: got 3\n want 4\n pick_test.go:50: got 5\n"
|
||||
output = "--- FAIL: TestPick/empty_input (0.00s)\n pick_test.go:42: got 3\n want 4\n--- FAIL: TestLast (0.00s)\n pick_test.go:50: got 5\n"
|
||||
self.assertEqual(task_text(output), "got 3\nwant 4")
|
||||
self.assertEqual(task_text("--- FAIL: TestPick (0.00s)\n pick_test.go:42: got 3\n a line the test logged\n"), "got 3")
|
||||
|
||||
|
||||
@@ -15,7 +15,8 @@ import unittest
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from task_bench import api_proxy, main, runs
|
||||
from task_bench.runs import checked_tree, closed, Containers, docker_cli, fix_attempted, opened, ran_tests
|
||||
from task_bench.runs import checked_tree, closed, Containers, docker_cli, fix_attempted, opened, ran_tests, TimedOut
|
||||
from task_bench.plant import drawn, Mutant
|
||||
from failure import Failure
|
||||
from test_env import isolated_env
|
||||
from mutate_changes.tool_process import Ran
|
||||
@@ -60,11 +61,14 @@ DOCKER = textwrap.dedent("""\
|
||||
sys.exit(subprocess.run(command, cwd=tree).returncode)
|
||||
""")
|
||||
# Reads the README, then the file with the bug. At the head it fixes the bug; at the base it rewrites the test to pass instead.
|
||||
# From its FAKE_REFUSED_FROM-th run on, the API refuses it.
|
||||
CLAUDE = textwrap.dedent("""\
|
||||
import json, os, pathlib, sys
|
||||
def call(name, path):
|
||||
return {"type": "assistant", "message": {"id": "msg_" + path, "content": [{"type": "tool_use", "id": "toolu_" + path, "name": name, "input": {"file_path": path}}], "usage": {}}}
|
||||
if os.environ.get("FAKE_REFUSED"):
|
||||
count = pathlib.Path(os.environ["FAKE_LOG"] + ".fixes")
|
||||
count.write_text(count.read_text() + "." if count.exists() else ".")
|
||||
if len(count.read_text()) >= int(os.environ.get("FAKE_REFUSED_FROM") or 10 ** 9):
|
||||
print(json.dumps({"type": "result", "is_error": True, "api_error_status": 401, "result": "API Error: 401"}))
|
||||
sys.exit(1)
|
||||
print(json.dumps({"type": "system", "subtype": "init", "model": "claude-haiku-5-5", "claude_code_version": "2.1.330"}))
|
||||
@@ -130,10 +134,10 @@ class BenchTest(unittest.TestCase):
|
||||
self.assertNotIsInstance(report, Failure, report)
|
||||
folder = pathlib.Path(report[-1].removeprefix("Each run's result and stream: "))
|
||||
self.assertEqual(report[:-1], [
|
||||
f"task-bench: base {self.base[:12]}, head {self.head[:12]}. 1 bug, fixed 2 times at each side by claude-haiku-5-5 in Claude Code 2.1.330.",
|
||||
f"task-bench: base {self.base[:12]}, head {self.head[:12]}. 1 bug, each run 2 times at each side by claude-haiku-5-5 in Claude Code 2.1.330.",
|
||||
"Planted 1 of the 2 bugs asked for: no other mutant the tests kill becomes one.",
|
||||
"",
|
||||
"| Task | Side | Passed | Input | Cache write | Cache read | Output | Turns | Wall, s | Reads |",
|
||||
"| Bug | Side | Passed | Input | Cache write | Cache read | Output | Turns | Wall, s | Reads |",
|
||||
"|---|---|---|---|---|---|---|---|---|---|",
|
||||
"| 1: calc.go:4:11 ARITHMETIC_BASE | base | 0/2 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |",
|
||||
"| 1: calc.go:4:11 ARITHMETIC_BASE | head | 2/2 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |",
|
||||
@@ -166,8 +170,8 @@ class BenchTest(unittest.TestCase):
|
||||
self.assertEqual(main.main(argv, self.env, lambda home: homes.append(home) or PLACES), 0)
|
||||
self.assertEqual(homes, [self.home])
|
||||
# Without --base, the base is the merge base with the default branch.
|
||||
self.assertEqual(out.getvalue().splitlines()[0], f"task-bench: base {self.base[:12]}, head {self.head[:12]}. 1 bug, fixed 2 times at each side by claude-haiku-5-5 in Claude Code 2.1.330.")
|
||||
self.assertEqual(out.getvalue().splitlines()[3], "| Task | Side | Passed | Input | Cache write | Cache read | Output | Turns | Wall, s | Reads |")
|
||||
self.assertEqual(out.getvalue().splitlines()[0], f"task-bench: base {self.base[:12]}, head {self.head[:12]}. 1 bug, each run 2 times at each side by claude-haiku-5-5 in Claude Code 2.1.330.")
|
||||
self.assertEqual(out.getvalue().splitlines()[3], "| Bug | Side | Passed | Input | Cache write | Cache read | Output | Turns | Wall, s | Reads |")
|
||||
self.assertTrue(out.getvalue().endswith("task-bench\n"), out.getvalue()[-80:])
|
||||
|
||||
def test_the_command_line_fails_with_the_reason_where_the_places_or_the_bench_fail(self):
|
||||
@@ -179,7 +183,7 @@ class BenchTest(unittest.TestCase):
|
||||
|
||||
def test_the_head_is_the_commit_asked_for(self):
|
||||
report = self.bench(**{"--head": "main", "--runs": "1"})
|
||||
self.assertEqual(report[0], f"task-bench: base {self.base[:12]}, head {self.base[:12]}. 1 bug, fixed 1 time at each side by claude-haiku-5-5 in Claude Code 2.1.330.")
|
||||
self.assertEqual(report[0], f"task-bench: base {self.base[:12]}, head {self.base[:12]}. 1 bug, each run 1 time at each side by claude-haiku-5-5 in Claude Code 2.1.330.")
|
||||
|
||||
def test_fails_on_a_head_or_a_base_git_does_not_know(self):
|
||||
for name in ("--head", "--base"):
|
||||
@@ -236,7 +240,7 @@ class BenchTest(unittest.TestCase):
|
||||
|
||||
def test_benches_the_head_against_itself_where_the_base_is_the_head(self):
|
||||
report = self.bench(**{"--base": "HEAD", "--runs": "1", "--bugs": "1"})
|
||||
self.assertEqual(report[0], f"task-bench: base {self.head[:12]}, head {self.head[:12]}. 1 bug, fixed 1 time at each side by claude-haiku-5-5 in Claude Code 2.1.330.")
|
||||
self.assertEqual(report[0], f"task-bench: base {self.head[:12]}, head {self.head[:12]}. 1 bug, each run 1 time at each side by claude-haiku-5-5 in Claude Code 2.1.330.")
|
||||
self.assertEqual(report[4:6], ["| 1: calc.go:5:11 ARITHMETIC_BASE | base | 1/1 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |", "| 1: calc.go:5:11 ARITHMETIC_BASE | head | 1/1 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |"])
|
||||
|
||||
def test_fails_where_the_places_name_no_go_file_of_the_base(self):
|
||||
@@ -255,13 +259,41 @@ class BenchTest(unittest.TestCase):
|
||||
# Passing tests with the bug planted: nothing to tell Haiku.
|
||||
(self.repo / "calc_test.go").write_text("exit 0\n")
|
||||
self.commit("Tests that catch nothing")
|
||||
self.assertEqual(self.bench(), Failure("None of the 2 mutants the tests kill becomes a bug: each is on a line the head changed, or fails no assertion with a message."))
|
||||
self.assertEqual(self.bench(), Failure("None of the 2 mutants the tests kill becomes a bug: each is on a line the head changed, or the test it fails doesn't print a message."))
|
||||
|
||||
def test_a_run_the_api_refuses_fails_the_bench_and_still_removes_what_it_made(self):
|
||||
self.env["FAKE_REFUSED"] = "1"
|
||||
self.env["FAKE_REFUSED_FROM"] = "1"
|
||||
self.assertEqual(self.bench(), Failure("claude could not use the API: API Error: 401"))
|
||||
self.assertEqual(self.calls()[-1][:2], ["image", "rm"])
|
||||
|
||||
def test_a_run_the_api_refuses_later_fails_the_bench_after_the_table_of_the_runs_that_finished(self):
|
||||
self.env["FAKE_REFUSED_FROM"] = "3"
|
||||
found = self.bench()
|
||||
self.assertIsInstance(found, Failure)
|
||||
lines = found.reason.splitlines()
|
||||
folder = next(self.home.glob(".local/state/larv-review/**/task-bench"))
|
||||
self.assertEqual(lines[:3], ["claude could not use the API: API Error: 401", f"task-bench stopped after 2 of 4 runs, which results.json holds in {folder}:", ""])
|
||||
self.assertEqual(lines[-3:-1], ["| All | base | 0/1 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |", "| All | head | 1/1 | 40 | 9000 | 52000 | 800 | 3 | 61.5 | 1 |"])
|
||||
self.assertEqual([(run["side"], run["run"]) for run in json.loads((folder / "results.json").read_text())["runs"]], [("base", 1), ("head", 1)])
|
||||
|
||||
def test_a_change_to_another_file_neither_moves_nor_drops_the_bug(self):
|
||||
(self.repo / "notes.txt").write_text("one\ntwo\nthree\nfour\nfive\nsix\n")
|
||||
self.commit("Notes")
|
||||
report = self.bench(**{"--bugs": "1", "--runs": "1"})
|
||||
self.assertNotIsInstance(report, Failure, report)
|
||||
self.assertEqual(report[4].partition(" | ")[0], "| 1: calc.go:4:11 ARITHMETIC_BASE")
|
||||
|
||||
def test_the_seed_sets_which_mutant_becomes_the_first_bug(self):
|
||||
# Tests that catch both mutants, at a head benched against itself, where each mutant's line stands.
|
||||
(self.repo / "calc_test.go").write_text(TESTS.replace("calc.go &&", "calc.go && grep -q '2 \\* a' calc.go &&"))
|
||||
self.commit("Tests that catch Twice")
|
||||
mutants = [Mutant("calc.go", 5, 11, "ARITHMETIC_BASE"), Mutant("calc.go", 9, 11, "ARITHMETIC_BASE")]
|
||||
for seed in (1, 2, 3, 4):
|
||||
report = self.bench(**{"--base": "HEAD", "--bugs": "1", "--runs": "1", "--seed": str(seed)})
|
||||
self.assertNotIsInstance(report, Failure, report)
|
||||
first = drawn(mutants, seed)[0]
|
||||
self.assertEqual(report[4].partition(" | ")[0], f"| 1: calc.go:{first.line}:11 ARITHMETIC_BASE", seed)
|
||||
|
||||
def test_a_docker_that_fails_to_build_fails_the_bench(self):
|
||||
self.env["FAKE_FAILS"] = "build"
|
||||
found = self.bench()
|
||||
@@ -275,7 +307,7 @@ class MainTest(unittest.TestCase):
|
||||
return main.main(list(argv), {"CLAUDE_CODE_OAUTH_TOKEN": token, "HOME": "/nonexistent"}), errors.getvalue()
|
||||
|
||||
def test_refuses_an_unknown_repeated_or_valueless_argument(self):
|
||||
for argv, said in ((["--imgae", "x"], "unknown or repeated argument --imgae"), (["--image", "x", "--image=y"], "unknown or repeated argument --image"), (["--image"], "--image needs a value")):
|
||||
for argv, said in ((["--imgae", "x"], "unknown argument --imgae"), (["--image", "x", "--image=y"], "--image is given twice"), (["--image"], "--image needs a value")):
|
||||
code, errors = self.run_main(*argv)
|
||||
self.assertEqual((code, errors.startswith(f"{said}; usage: task-bench --image")), (1, True), errors)
|
||||
|
||||
@@ -320,6 +352,55 @@ class Recorded:
|
||||
return self.answers.get(args[0], Ran(0, "ran"))
|
||||
|
||||
|
||||
class InTurn(Recorded):
|
||||
"""A docker that answers each `run` with the next of `runs`."""
|
||||
|
||||
def __init__(self, *runs: Ran | Failure):
|
||||
super().__init__()
|
||||
self.runs = list(runs)
|
||||
|
||||
def __call__(self, args, *, seconds, stdin=None, stdout=None, stderr=None):
|
||||
answer = super().__call__(args, seconds=seconds, stdin=stdin, stdout=stdout, stderr=stderr)
|
||||
return self.runs.pop(0) if args[0] == "run" else answer
|
||||
|
||||
|
||||
class RunOutcomesTest(unittest.TestCase):
|
||||
MUTANT = Mutant("calc.go", 4, 11, "ARITHMETIC_BASE")
|
||||
|
||||
def setUp(self):
|
||||
folder = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(folder.cleanup)
|
||||
self.folder = pathlib.Path(folder.name)
|
||||
self.tree = self.folder / "tree"
|
||||
self.tree.mkdir()
|
||||
(self.tree / "calc.go").write_text(BASE_CODE)
|
||||
self.trees = {"base": str(self.tree), "head": str(self.tree)}
|
||||
self.task = main.Task(1, self.MUTANT, self.trees, {"base": "got 3", "head": "got 3"})
|
||||
|
||||
def containers(self, *runs: Ran | Failure) -> Containers:
|
||||
return Containers(InTurn(*runs), "bench", "go test ./...", 512, "1000:1000")
|
||||
|
||||
def test_a_fix_whose_tests_run_out_of_time_does_not_pass(self):
|
||||
late = TimedOut("docker run still ran after 30 min")
|
||||
self.assertEqual(main.fixed(self.task, "base", 1, self.containers(Ran(0, ""), late), self.folder), Result(1, "base", 1, False, None))
|
||||
|
||||
def test_a_fix_whose_tests_docker_could_not_run_fails_the_bench(self):
|
||||
broke = Failure("docker could not start: No such file or directory")
|
||||
self.assertEqual(main.fixed(self.task, "base", 1, self.containers(Ran(0, ""), broke), self.folder), broke)
|
||||
|
||||
def test_tests_docker_could_not_run_without_a_bug_fail_the_bench(self):
|
||||
broke = Failure("docker could not start: No such file or directory")
|
||||
self.assertEqual(main.benched({}, "", {}, self.trees, [], self.containers(broke), self.folder, None), broke)
|
||||
|
||||
def test_a_mutant_whose_tests_run_out_of_time_is_passed_over(self):
|
||||
late = TimedOut("docker run still ran after 30 min")
|
||||
self.assertIsNone(main.planted(1, self.MUTANT, "", {"base": "b", "head": "h"}, self.trees, self.containers(late), self.folder, lambda *args, **given: ""))
|
||||
|
||||
def test_a_mutant_whose_tests_docker_could_not_run_fails_the_bench(self):
|
||||
broke = Failure("docker could not start: No such file or directory")
|
||||
self.assertEqual(main.planted(1, self.MUTANT, "", {"base": "b", "head": "h"}, self.trees, self.containers(broke), self.folder, lambda *args, **given: ""), broke)
|
||||
|
||||
|
||||
class ContainersTest(unittest.TestCase):
|
||||
IN_TREE = ["--memory", "512m", "--memory-swap", "512m", "--user", "1000:1000", "--env", "HOME=/tmp", "--volume", "/tree:/work", "--workdir", "/work"]
|
||||
REMOVED = (["rm", "--force", "bench-run"], 120, None)
|
||||
@@ -357,7 +438,7 @@ class ContainersTest(unittest.TestCase):
|
||||
self.assertEqual(self.docker.calls, [(["run", "--rm", "--name", "bench-run", "--network", "none", *self.IN_TREE, "bench:throwaway", "sh", "-c", "go test ./..."], 1800, None)])
|
||||
|
||||
def test_tests_that_run_out_of_time_are_a_failure_and_their_container_goes(self):
|
||||
late = Failure("docker run still ran after 30 min")
|
||||
late = TimedOut("docker run still ran after 30 min")
|
||||
self.assertEqual(ran_tests(self.containers(run=late), "/tree"), late)
|
||||
self.assertEqual(self.docker.calls[1:], [self.REMOVED])
|
||||
|
||||
@@ -370,12 +451,19 @@ class ContainersTest(unittest.TestCase):
|
||||
self.assertEqual(args[18:25], ["--env", "CLAUDE_CODE_OAUTH_TOKEN", "--env", "HTTPS_PROXY=http://bench-proxy:8080", "--env", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1", "bench:throwaway"])
|
||||
|
||||
def test_a_fix_that_runs_out_of_time_is_no_failure_and_its_container_goes(self):
|
||||
self.assertIsNone(fix_attempted(self.containers(run=Failure("docker run still ran after 30 min")), "/tree", "got 3", self.stream))
|
||||
self.assertIsNone(fix_attempted(self.containers(run=TimedOut("docker run still ran after 30 min")), "/tree", "got 3", self.stream))
|
||||
self.assertEqual(self.docker.calls[1:], [self.REMOVED])
|
||||
|
||||
def test_a_run_whose_command_never_started_is_a_failure_that_says_what_docker_printed(self):
|
||||
for code in (125, 126, 127):
|
||||
self.assertEqual(ran_tests(self.containers(run=Ran(code, "sh: gotestsum: not found\n")), "/tree"), Failure(f"docker run could not start its command, and exited {code}: sh: gotestsum: not found"))
|
||||
self.assertEqual(ran_tests(self.containers(run=Ran(124, "late")), "/tree"), Ran(124, "late"))
|
||||
self.assertEqual(ran_tests(self.containers(run=Ran(128, "killed")), "/tree"), Ran(128, "killed"))
|
||||
self.assertEqual(fix_attempted(self.containers(run=Ran(125, "")), "/tree", "got 3", self.stream), Failure(f"docker run could not start its command, and exited 125; its errors are in {self.stream.with_suffix('.stderr')}"))
|
||||
|
||||
def test_a_fix_docker_could_not_start_is_a_failure(self):
|
||||
broke = Failure("docker could not start: No such file or directory")
|
||||
self.assertEqual(fix_attempted(self.containers(run=broke), "/tree", "got 3", self.stream), broke)
|
||||
self.assertEqual(fix_attempted(self.containers(run=broke), "/tree", "got 3", self.stream), Failure(f"{broke.reason}; its errors are in {self.stream.with_suffix('.stderr')}"))
|
||||
self.assertEqual(self.docker.calls[1:], [self.REMOVED])
|
||||
|
||||
def test_a_stream_that_cannot_be_written_is_a_failure(self):
|
||||
@@ -418,7 +506,7 @@ class DockerCliTest(unittest.TestCase):
|
||||
waited.append(timeout)
|
||||
raise subprocess.TimeoutExpired(argv, timeout)
|
||||
|
||||
self.assertEqual(docker_cli({}, late)(["run", "--rm"], seconds=90), Failure("docker run still ran after 2 min"))
|
||||
self.assertEqual(docker_cli({}, late)(["run", "--rm"], seconds=90), TimedOut("docker run still ran after 2 min"))
|
||||
self.assertEqual(waited, [90])
|
||||
|
||||
|
||||
@@ -444,6 +532,12 @@ class CheckedTreeTest(unittest.TestCase):
|
||||
checked_tree(planted, copy, str(self.root / "check"))
|
||||
self.assertEqual(self.held("check"), {".git/HEAD": "ref", "calc.go": "fixed", "calc_test.go": "tests", "sub/new.go": "new", "sub/old_test.go": "old tests"})
|
||||
|
||||
def test_takes_the_files_under_testdata_as_planted_since_a_test_reads_what_it_expects_there(self):
|
||||
planted = self.tree("planted", {"calc.go": "bug", "testdata/want.golden": "3", "sub/testdata/gone.golden": "4", "testdata.go": "code"})
|
||||
copy = self.tree("copy", {"calc.go": "fixed", "testdata/want.golden": "-1", "testdata/added.golden": "5", "testdata.go": "fixed"})
|
||||
checked_tree(planted, copy, str(self.root / "check"))
|
||||
self.assertEqual(self.held("check"), {"calc.go": "fixed", "sub/testdata/gone.golden": "4", "testdata.go": "fixed", "testdata/want.golden": "3"})
|
||||
|
||||
def test_copies_a_link_as_a_link(self):
|
||||
planted = self.tree("planted", {"calc.go": "bug"})
|
||||
copy = self.tree("copy", {"calc.go": "fixed"})
|
||||
@@ -469,8 +563,9 @@ class ApiProxyTest(unittest.IsolatedAsyncioTestCase):
|
||||
await writer.drain()
|
||||
writer.close()
|
||||
|
||||
api = await asyncio.start_server(echoed, "127.0.0.1", 0)
|
||||
self.allowed = f"127.0.0.1:{api.sockets[0].getsockname()[1]}"
|
||||
# On an address of its own, so a tunnel to any other host finds nothing listening.
|
||||
api = await asyncio.start_server(echoed, "127.0.0.2", 0)
|
||||
self.allowed = f"127.0.0.2:{api.sockets[0].getsockname()[1]}"
|
||||
proxy = await asyncio.start_server(lambda reader, writer: api_proxy.served(self.allowed, reader, writer), "127.0.0.1", 0)
|
||||
self.port = proxy.sockets[0].getsockname()[1]
|
||||
for server in (api, proxy):
|
||||
|
||||
Reference in New Issue
Block a user