Mutate the Go lines a branch changes with Gremlins in mutate-changes #678

Merged
lilleman merged 11 commits from item-500 into main 2026-11-21 00:46:36 +01:00
7 changed files with 163 additions and 28 deletions
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- mutate-changes now mutates the Go lines a branch changes with Gremlins, as it does Python, Rust, TypeScript and PHP, and prints each mutant the tests do not kill.
- A project's config may set `epic-branch`, such as `epic-branch = "{epic}"`: do-epic then keeps one working branch per epic, level with the default branch by merge commits, and each chunk branches off it and merges into it without waiting for a person, even under `merge = "human"`. `chunk-pr open-epic` opens the finished epic's pull request into the default branch for a person to merge. `review-request-comment` names a comment `chunk-pr` posts after each push to ask a forge's reviewer for a review, and `jira-pull-request-open-status` and `jira-pull-request-merged-status` name the statuses do-epic moves an issue to as its pull request opens and merges.
- `mutate-changes` mutates a Python script from its own folder where a test beside it imports it by its file name, as in `import run_harness`, so mutmut finds the tests' calls to it. A crash inside mutmut now becomes one report line that names mutmut's error, not a traceback, and the other folders still report.
- do-epic starts an issue in Jira as its chunk starts: it assigns the issue to the account it works as, unless someone else holds it, and the new settings `jira-start-transition` and `jira-sprint-board` in its `config.toml` move the issue and add it to a board's active sprint.
+2 -2
View File
@@ -7,9 +7,9 @@ import typing
# A hunk header of `git diff -U0`. The new side's count is 0 where the hunk only deletes, and left out where it is 1.
HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@", re.MULTILINE)
STRYKER = (".cjs", ".cts", ".js", ".jsx", ".mjs", ".mts", ".ts", ".tsx")
TOOLS = {".php": "infection", ".py": "mutmut", ".rs": "cargo-mutants", **dict.fromkeys(STRYKER, "stryker")}
TOOLS = {".go": "gremlins", ".php": "infection", ".py": "mutmut", ".rs": "cargo-mutants", **dict.fromkeys(STRYKER, "stryker")}
TEST_FOLDERS = frozenset({"__tests__", "test", "tests"})
TEST_NAME = re.compile(r"test_.*\.py|.*_test\.py|conftest\.py|.*\.(?:spec|test)\.[^.]+")
TEST_NAME = re.compile(r"test_.*\.py|.*_test\.py|conftest\.py|.*_test\.go|.*\.(?:spec|test)\.[^.]+")
def added_lines(diff: str) -> frozenset[int]:
+54 -1
View File
@@ -1,6 +1,8 @@
"""Mutate a branch's changed lines with cargo-mutants, Stryker and Infection, each of which takes the changed lines itself."""
"""Mutate a branch's changed lines with cargo-mutants, Gremlins, Infection and Stryker, each of which takes the changed lines itself."""
import json
import os
import pathlib
import typing
from mutate_changes.changed_lines import by_folder, label, runs
from mutate_changes.tool_process import failed, Ran, remove, Run, scratch_folder
@@ -10,6 +12,11 @@ from git_cli.git_repo import Git
# cargo-mutants' exit codes for a run that finished: every mutant caught, some missed, and some timed out.
CARGO_FINISHED = frozenset({0, 2, 3})
CARGO_OUTCOMES = ("caught", "missed", "timeout", "unviable")
GREMLINS_UNKILLED = frozenset({"LIVED", "NOT COVERED"})
# What Gremlins v0.6.0 needs of the diff it reads from git: one run of lines per hunk is all it mutates, so a hunk holds the changed
# lines and none around them, nor a neighbouring change; it takes each file's path after a b/; and it reads no colour codes. A user's git
# settings change each.
GREMLINS_GIT_SETTINGS = {"diff.context": "0", "diff.noprefix": "false", "diff.mnemonicPrefix": "false", "diff.srcPrefix": "a/", "diff.dstPrefix": "b/", "color.diff": "never", "diff.interHunkContext": "0"}
def run_cargo_mutants(top: str, base: str, paths: list[str], run: Run, git: Git, jobs: int) -> list[str] | Failure:
@@ -38,6 +45,52 @@ def run_cargo_mutants(top: str, base: str, paths: list[str], run: Run, git: Git,
return [f"cargo-mutants: {sum(map(len, outcomes.values()))} mutants on changed lines, {len(uncaught)} not caught.", *uncaught]
class Mutant(typing.NamedTuple):
"""A mutant in Gremlins' report, in the order the report prints them."""
path: str
line: int
column: int
status: str
mutator: str
def run_gremlins(top: str, base: str, run: Run, jobs: int) -> list[str] | Failure:
"""A line with the number of mutants on the changed lines and how many of them Gremlins reports as lived or not covered, then a
line per such mutant, as `<path>:<line>:<column>: <status>: <mutator>`.
Gremlins runs in top and mutates the Go lines the branch changes since base. It skips `*_test.go` itself. It reads the diff from
git, so `run` must hold the environment that `with_plain_git_diff` makes. Gremlins exits nonzero on a mutation score below a
threshold the project sets, which its exit code doesn't tell from a run that broke, so any nonzero exit fails the run."""
scratch = scratch_folder(top)
if isinstance(scratch, Failure):
return scratch
try:
report = pathlib.Path(scratch, "report.json")
ran = run(["gremlins", "unleash", "--diff", base, "--output", os.path.join(os.path.relpath(scratch, top), report.name), "--workers", str(jobs)], top)
if isinstance(ran, Failure) or ran.code:
return ran if isinstance(ran, Failure) else whole_output("Gremlins", ran)
# Gremlins writes no report where it finds no mutant, and `null` for a list with no entry.
files = (json.loads(report.read_text(encoding="utf-8"))["files"] or []) if report.is_file() else []
mutants = [Mutant(file["file_name"], mutant["line"], mutant["column"], mutant["status"], mutant["type"]) for file in files for mutant in file["mutations"] or []]
except (OSError, ValueError, KeyError, TypeError) as failure:
return Failure(f"could not read Gremlins' report in {scratch}: {failure}")
finally:
remove(scratch)
# Gremlins lists the mutants on the unchanged lines of a hunk too, as SKIPPED. It lists the files in no fixed order.
unkilled = sorted(mutant for mutant in mutants if mutant.status in GREMLINS_UNKILLED)
on_changed_lines = sum(mutant.status != "SKIPPED" for mutant in mutants)
return [f"gremlins: {on_changed_lines} mutants on changed lines, {len(unkilled)} not killed.", *(f"{m.path}:{m.line}:{m.column}: {m.status}: {m.mutator}" for m in unkilled)]
def with_plain_git_diff(env: typing.Mapping[str, str]) -> dict[str, str]:
"""env with the settings of GREMLINS_GIT_SETTINGS added to the ones it holds, as git reads them from its GIT_CONFIG_* variables."""
held = env.get("GIT_CONFIG_COUNT", "")
first = int(held) if held.isdecimal() else 0
added = {f"GIT_CONFIG_{part}_{first + index}": text for index, (key, value) in enumerate(GREMLINS_GIT_SETTINGS.items()) for part, text in (("KEY", key), ("VALUE", value))}
return {**env, **added, "GIT_CONFIG_COUNT": str(first + len(GREMLINS_GIT_SETTINGS))}
def outcome_lines(path: pathlib.Path) -> list[str]:
return path.read_text(encoding="utf-8", errors="replace").splitlines() if path.is_file() else []
+20 -7
View File
@@ -1,10 +1,10 @@
"""Mutation-test the lines a branch adds or changes, and print each mutant its tests don't kill.
r"""Mutation-test the lines a branch adds or changes, and print each mutant its tests don't kill.
usage: mutate-changes [--base <ref>] [--prefix <shell text>] [--jobs <n>]
Run it in a checkout, under ~/.claude/bin/run-heavy. It diffs the working tree's tracked files against the merge base of HEAD and
--base, by default the default branch: origin's as last fetched, else main or master. It picks the tool by each changed file's
extension, and leaves out test files: test_*.py, *_test.py, conftest.py, *.test.* and *.spec.*, and any file under a tests/, test/
extension, and leaves out test files: test_*.py, *_test.py, *_test.go, conftest.py, *.test.* and *.spec.*, and any file under a tests/, test/
or __tests__/ folder.
- .py: mutmut 3, in a scratch copy of the nearest folder above the file that holds a setup.cfg or a pyproject.toml, else of the
@@ -16,6 +16,9 @@ or __tests__/ folder.
in its own process, so a mutant that only a child process reaches, such as a tool a test starts, shows as `no tests`. A
pyproject.toml with a [tool.mutmut] section fails the run. A crash inside mutmut, such as a KeyError in its worker pool,
becomes one line for its folder, naming the error and telling you to mutate that folder's changed lines by hand.
- .go: Gremlins, as `gremlins unleash --diff`, in the checkout's top, where its go.mod must be. Gremlins reads the diff from git, so
the run adds settings to the GIT_CONFIG_* variables, after any the environment holds: no context lines, since Gremlins mutates
every line of a hunk, so one hunk per change; a/ and b/ as the file prefixes; and no colour.
- .rs: cargo-mutants, as `cargo mutants --in-diff`, in the checkout's top.
- .cjs, .cts, .js, .jsx, .mjs, .mts, .ts, .tsx: Stryker, as `npx --no --package=@stryker-mutator/core stryker run --mutate` on the
changed lines, in the nearest folder above the file that holds a package.json. A file with no package.json above it isn't
@@ -25,8 +28,14 @@ or __tests__/ folder.
--prefix is shell text that runs each tool: sh runs `<prefix> "$@"` in the tool's folder, which $PWD names, with the tool's command
as "$@". For example, 'uvx --from "mutmut>=3,<4"'. Without --prefix, the command runs as it is. A container must mount $PWD and
run there, as 'docker compose -f "$(git rev-parse --show-toplevel)/compose.yaml" run --rm -v "$PWD:$PWD" -w "$PWD" app' does. "$1"
is the command's first word, mutmut, cargo, npx or vendor/bin/infection, so a prefix can pick what to run per tool:
is the command's first word, mutmut, cargo, gremlins, npx or vendor/bin/infection, so a prefix can pick what to run per tool:
'case "$1" in mutmut) set -- uvx --from "mutmut>=3,<4" "$@";; esac;'.
Gremlins' git runs in the container too, so a Go prefix mounts git's data and passes on the GIT_CONFIG_* variables, and allows git
to read a checkout that the container's root user doesn't own. In a git worktree, git's data is in the main checkout, outside $PWD:
common="$(git rev-parse --path-format=absolute --git-common-dir)"
docker run --rm -v "$PWD:$PWD" -v "$common:$common" -w "$PWD" $(printenv | sed -n 's/^\(GIT_CONFIG_[A-Z0-9_]*\)=.*/-e \1/p') \
<image holding gremlins> sh -c 'git config --global --add safe.directory "*" && exec "$@"' sh
--jobs sets how many tests each tool runs at once, 4 by default: one heavy slot's CPUs. Under run-heavy, a container that sets
neither memory limit gets the slot's share, in MiB in RUN_HEAVY_MEMORY; set --jobs so that that many test runs fit in it.
@@ -34,6 +43,8 @@ neither memory limit gets the slot's share, in MiB in RUN_HEAVY_MEMORY; set --jo
It prints:
- for mutmut, per folder, a line with the mutant count, then a line per mutant on a changed line that is not killed, as
`<path>:<line>: <status>: <the line as mutated>`, or `<path>: <status>: <name>, on a line mutate-changes cannot find`;
- for Gremlins, a line with the mutant count and how many it did not kill, then a line per mutant that lived or is not covered, as
`<path>:<line>:<column>: <status>: <mutator>`;
- for cargo-mutants, a line with the mutant count, then a line per mutant it missed or that timed out, as
`<path>:<line>:<column>: <outcome>: <mutation>`;
- for Stryker and Infection, a line naming the folder the tool ran in, then the tool's own report;
@@ -42,9 +53,9 @@ It prints:
- last, the wall time.
Exit 1, saying why, on an unknown or repeated argument, an option without its value, or a --jobs that is no whole number from 1.
Exit 1 too when git fails, a file cannot be read or copied, a tool cannot start, or a tool fails before it reports. Stryker and
Infection also exit nonzero on a mutation score below a threshold the project sets, which fails the run as well and prints their
whole report.
Exit 1 too when git fails, a file cannot be read or copied, a tool cannot start, or a tool fails before it reports. Stryker, Infection
and Gremlins also exit nonzero on a mutation score below a threshold the project sets, which fails the run as well and prints their
whole output.
"""
import functools
import os
@@ -57,7 +68,7 @@ from machine_load.slots import CPUS_PER_SLOT
from git_cli.git_repo import default_ref, Git, top
from failure import Failure
from git_cli import git_repo
from mutate_changes.in_diff_tools import run_cargo_mutants, run_infection, run_stryker
from mutate_changes.in_diff_tools import run_cargo_mutants, run_gremlins, run_infection, run_stryker, with_plain_git_diff
from mutate_changes.mutmut_tool import run_mutmut
from mutate_changes.tool_process import run_tool
@@ -125,6 +136,8 @@ def mutate(given: dict[str, str], env: typing.Mapping[str, str]) -> list[str] |
for tool, files in sorted(changed.items()):
if tool == "cargo-mutants":
lines = run_cargo_mutants(checkout, merge_base, sorted(files), run, git, jobs)
elif tool == "gremlins":
lines = run_gremlins(checkout, merge_base, functools.partial(run, env=with_plain_git_diff(env)), jobs)
elif tool == "infection":
lines = run_infection(checkout, merge_base, run, jobs)
elif tool == "stryker":
+21 -2
View File
@@ -22,6 +22,7 @@ from test_env import isolated_env
from mutate_changes.mutmut_setup import left_out, mutmut_config
from mutate_changes import main
from mutate_changes.tool_process import run_tool, scratch_folder
from mutate_changes.in_diff_tools import with_plain_git_diff
SOURCE = "def double(n):\n\treturn n * 2\n\n\nclass Pt:\n\tdef moved(self, by=1):\n\t\treturn self.x + by\n"
# The shape of the file mutmut 3 writes under mutants/: the original of each function and its mutants, with the original's lines.
@@ -66,12 +67,14 @@ class ChangedLines(unittest.TestCase):
"src/Order.php": "infection",
"web/app.tsx": "stryker",
"web/app.mjs": "stryker",
"cmd/serve/main.go": "gremlins",
"bin/x/test_main.py": None,
"bin/x/main_test.py": None,
"pkg/conftest.py": None,
"tests/helpers.py": None,
"web/app.test.ts": None,
"web/app.spec.js": None,
"cmd/serve/main_test.go": None,
"web/__tests__/a.js": None,
"README.md": None,
}
@@ -140,8 +143,24 @@ class ChangedLines(unittest.TestCase):
self.assertEqual(crash_in_mutmut(trace((f"{lib}/x.py", 1), error="")), None)
def test_not_mutated_names_the_files_no_tool_covers_then_the_test_files(self):
self.assertEqual(not_mutated(["README.md", "a.py", "b.go", "test_a.py", "web/a.test.ts"]), ["Not mutated, as no tool covers their language: README.md, b.go.", "Not mutated, as test files: test_a.py, web/a.test.ts."])
self.assertEqual((not_mutated(["a.py", "test_a.py"]), not_mutated(["a.py", "b.go"]), not_mutated([])), (["Not mutated, as test files: test_a.py."], ["Not mutated, as no tool covers their language: b.go."], []))
self.assertEqual(not_mutated(["README.md", "a.py", "b.rb", "test_a.py", "web/a.test.ts"]), ["Not mutated, as no tool covers their language: README.md, b.rb.", "Not mutated, as test files: test_a.py, web/a.test.ts."])
self.assertEqual((not_mutated(["a.py", "test_a.py"]), not_mutated(["a.py", "b.rb"]), not_mutated([])), (["Not mutated, as test files: test_a.py."], ["Not mutated, as no tool covers their language: b.rb."], []))
class InDiffTools(unittest.TestCase):
def test_the_git_settings_gremlins_needs_are_added_after_the_ones_the_environment_holds(self):
held = {"HOME": "/h", "GIT_CONFIG_COUNT": "1", "GIT_CONFIG_KEY_0": "safe.directory", "GIT_CONFIG_VALUE_0": "*"}
env = with_plain_git_diff(held)
self.assertEqual(env["GIT_CONFIG_COUNT"], "8")
self.assertEqual({name: value for name, value in env.items() if name in held}, held | {"GIT_CONFIG_COUNT": "8"})
settings = [(env[f"GIT_CONFIG_KEY_{index}"], env[f"GIT_CONFIG_VALUE_{index}"]) for index in range(1, 8)]
self.assertEqual(settings, [("diff.context", "0"), ("diff.noprefix", "false"), ("diff.mnemonicPrefix", "false"), ("diff.srcPrefix", "a/"), ("diff.dstPrefix", "b/"), ("color.diff", "never"), ("diff.interHunkContext", "0")])
def test_the_git_settings_start_at_the_first_for_an_environment_without_a_count_or_with_one_that_is_not_a_number(self):
for held in ({}, {"GIT_CONFIG_COUNT": "many"}, {"GIT_CONFIG_COUNT": ""}):
with self.subTest(held):
env = with_plain_git_diff(held)
self.assertEqual((env["GIT_CONFIG_COUNT"], env["GIT_CONFIG_KEY_0"], env["GIT_CONFIG_KEY_6"], "GIT_CONFIG_KEY_7" in env), ("7", "diff.context", "diff.interHunkContext", False))
class MutmutSetup(unittest.TestCase):
+64 -2
View File
@@ -62,10 +62,12 @@ else:
print("Traceback (most recent call last):\n File \"/uv/site-packages/mutmut/__main__.py\", line 1050, in _run\n File \"/uv/site-packages/mutmut/workers/isolation.py\", line 889, in wait_for_result\n worker = pop(pid)\nKeyError: 974598")
sys.exit(1)
"""
# cargo-mutants, Infection and Stryker, which log the folder they run in and how they were called, and exit with the code the test
# cargo-mutants, Gremlins, Infection and Stryker, which log the folder they run in and how they were called, and exit with the code the test
# gives. cargo-mutants also logs the diff it is given and writes its outcome files, all but unviable.txt, as a run without an
# unviable mutant does; with CAUGHT_UNREADABLE set, nobody may read caught.txt, and with OUTPUT_LOCKED, nobody may delete from mutants.out.
FAKE_TOOLS = r"""import os, pathlib, sys
# Gremlins also logs the git settings it finds in its environment and writes a JSON report in Gremlins' shape. With REPORT set, it
# writes that text in its place, or with REPORT=none, no report, as Gremlins does where it finds no mutant.
FAKE_TOOLS = r"""import json, os, pathlib, sys
log, code, args = pathlib.Path(sys.argv[1]), int(sys.argv[2]), sys.argv[3:]
with log.open("a") as written:
written.write(f"{os.getcwd()}\n{' '.join(args)}\n")
@@ -80,6 +82,14 @@ with log.open("a") as written:
(outcomes / "caught.txt").chmod(0)
if os.environ.get("OUTPUT_LOCKED"):
outcomes.chmod(0o555)
if args[0] == "gremlins":
git_config = (name for name in sorted(os.environ) if name.startswith(("GIT_CONFIG_COUNT", "GIT_CONFIG_KEY_", "GIT_CONFIG_VALUE_")))
written.write("".join(f"{name}={os.environ[name]}\n" for name in git_config))
killed, skipped, uncovered, lived = (dict(type=kind, status=status, line=line, column=column) for kind, status, line, column in (("ARITHMETIC_BASE", "KILLED", 4, 11), ("CONDITIONALS_BOUNDARY", "SKIPPED", 8, 11), ("CONDITIONALS_BOUNDARY", "NOT COVERED", 12, 11), ("ARITHMETIC_BASE", "LIVED", 4, 15)))
files = [dict(file_name="calc.go", mutations=[killed, skipped, uncovered, lived]), dict(file_name="sub/other.go", mutations=[dict(type="INVERT_NEGATIVES", status="LIVED", line=2, column=9)])]
report = os.environ.get("REPORT") or json.dumps(dict(go_module="x", files=files))
if report != "none":
pathlib.Path(args[args.index("--output") + 1]).write_text(report)
print(f"{args[0]} reports")
sys.exit(code)
"""
@@ -404,6 +414,53 @@ class MutateChangesRuns(unittest.TestCase):
self.assertRegex(ran.stderr, rf"^could not hand cargo-mutants the diff or read its outcomes in {re.escape(str(self.checkout.resolve()))}/\.mutate-changes-\w+: ")
self.assertEqual(sorted(path.name for path in self.checkout.iterdir()), [".git", "README.md", "package.json", "pkg", "src", "web"])
def change_go(self):
"""Commit a Go file and its test, then change both."""
(self.checkout / "calc.go").write_text("package calc\n\nfunc Add(a, b int) int {\n\treturn a + b\n}\n")
(self.checkout / "calc_test.go").write_text("package calc\n")
self.commit()
(self.checkout / "calc.go").write_text("package calc\n\nfunc Add(a, b int) int {\n\treturn a + b + 0\n}\n")
(self.checkout / "calc_test.go").write_text("package calc\n\nfunc helper() {}\n")
def test_go_goes_to_gremlins_at_the_top_and_prints_each_mutant_that_lived_or_is_not_covered(self):
self.change_go()
fake = f"python3 {self.temp / 'fake_tools.py'} {self.log} 0"
ran = self.run_tool("--base", "HEAD", "--jobs=3", "--prefix", fake, cwd=self.checkout / "pkg")
report = ["gremlins: 4 mutants on changed lines, 3 not killed.", "calc.go:4:15: LIVED: ARITHMETIC_BASE", "calc.go:12:11: NOT COVERED: CONDITIONALS_BOUNDARY", "sub/other.go:2:9: LIVED: INVERT_NEGATIVES"]
self.assertEqual((ran.returncode, ran.stdout.splitlines()[:-2], ran.stderr), (0, report, ""))
self.assertEqual(ran.stdout.splitlines()[-2], "Not mutated, as test files: calc_test.go.")
folder, command, config = self.log.read_text().split("\n", 2)
self.assertEqual(folder, str(self.checkout.resolve()))
base = self.git("rev-parse", "HEAD").strip()
self.assertRegex(command, rf"^gremlins unleash --diff {base} --output (\.mutate-changes-\w+)/report\.json --workers 3$")
keys = ["diff.context", "diff.noprefix", "diff.mnemonicPrefix", "diff.srcPrefix", "diff.dstPrefix", "color.diff", "diff.interHunkContext"]
values = ["0", "false", "false", "a/", "b/", "never", "0"]
self.assertEqual(config, "GIT_CONFIG_COUNT=7\n" + "".join(f"GIT_CONFIG_KEY_{n}={key}\n" for n, key in enumerate(keys)) + "".join(f"GIT_CONFIG_VALUE_{n}={value}\n" for n, value in enumerate(values)))
self.assertEqual(sorted(path.name for path in self.checkout.iterdir()), [".git", "README.md", "calc.go", "calc_test.go", "pkg"])
def test_a_gremlins_run_that_finds_no_mutant_reports_none(self):
self.change_go()
for report in ("none", '{"files": null}', '{"files": [{"file_name": "calc.go", "mutations": null}]}'):
with self.subTest(report):
ran = self.run_tool("--base", "HEAD", "--prefix", f"python3 {self.temp / 'fake_tools.py'} {self.log} 0", env={"REPORT": report})
self.assertEqual((ran.returncode, ran.stdout.splitlines()[0], ran.stderr), (0, "gremlins: 0 mutants on changed lines, 0 not killed.", ""))
def test_a_gremlins_run_that_fails_or_leaves_a_report_it_cannot_read_fails_the_run(self):
self.change_go()
fake = f"python3 {self.temp / 'fake_tools.py'} {self.log}"
# Gremlins also exits nonzero on a mutation score below the project's threshold, where its output lists what survived.
ran = self.run_tool("--base", "HEAD", "--prefix", f"{fake} 10")
self.assertEqual((ran.returncode, ran.stderr), (1, "Gremlins exited 10:\ngremlins reports\n"))
for report in ("not json", "[]", '{"files": 3}'):
with self.subTest(report):
ran = self.run_tool("--base", "HEAD", "--prefix", f"{fake} 0", env={"REPORT": report})
self.assertEqual(ran.returncode, 1)
self.assertRegex(ran.stderr, rf"^could not read Gremlins' report in {re.escape(str(self.checkout.resolve()))}/\.mutate-changes-\w+: ")
ran = self.run_tool("--base", "HEAD", env={"PATH": str(self.path_with_git())})
self.assertEqual(ran.returncode, 1)
self.assertRegex(ran.stderr, "^gremlins unleash .* could not start: ")
self.assertEqual(sorted(path.name for path in self.checkout.iterdir()), [".git", "README.md", "calc.go", "calc_test.go", "pkg"])
def test_a_branch_that_adds_no_line_a_tool_mutates_says_so_and_names_what_it_changes(self):
# A file the branch deletes is no longer there to mutate, so the report doesn't name it.
(self.checkout / "pkg" / "old.md").write_text("Gone.\n")
@@ -425,6 +482,11 @@ class MutateChangesRuns(unittest.TestCase):
ran = self.run_tool(flag)
self.assertEqual((ran.returncode, ran.stdout.splitlines()[0]), (0, "Mutation-test the lines a branch adds or changes, and print each mutant its tests don't kill."))
def test_help_shows_the_gremlins_prefix_that_mounts_gits_data_in_a_worktree(self):
text = self.run_tool("--help").stdout
self.assertIn("-v \"$common:$common\"", text)
self.assertIn("--git-common-dir", text)
def test_unknown_arguments_fail_saying_why(self):
usage = "usage: mutate-changes [--base <ref>] [--prefix <shell text>] [--jobs <n>]"
for args, reason in (
+1 -14
View File
@@ -12,8 +12,7 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 500 | 0.15.0 | | owner | **Mutate the Go lines a branch changes with Gremlins in `mutate-changes`, as it mutates Python, Rust, TypeScript and PHP, and print each mutant the tests do not kill.** | 2 | 2 | 2 | 4 | 4 | 1.1 by 4 | 1.2 |
| 501 | 0.15.0 | 500 | owner | **Ship `task-bench`, which plants a bug the tests catch at the places the comprehension seats found hardest, has Haiku fix it in a vanilla Claude Code at a base and a head, and prints for each side how often the fix passes, with its median tokens, turns, wall time and reads before it opens a file the bug is in.** | 4 | 2 | 1 | 5 | 9 | 1.1 by 5, code health by 4 | 0.6 |
| 501 | 0.15.0 | | owner | **Ship `task-bench`, which plants a bug the tests catch at the places the comprehension seats found hardest, has Haiku fix it in a vanilla Claude Code at a base and a head, and prints for each side how often the fix passes, with its median tokens, turns, wall time and reads before it opens a file the bug is in.** | 4 | 2 | 1 | 5 | 9 | 1.1 by 5, code health by 4 | 0.6 |
| 502 | 0.15.0 | 501 | owner | **Distil each `task-bench` run into a digest of short traces and decoys, the files most failing runs read but no fix touches, and have a stronger model turn it into plan items before the transcripts are deleted.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 4, code health by 4 | 4.8 |
| 469 | 0.15.0 | | | **Move a do-epic parent issue whose sub-tasks a person filed to Release Ready once every sub-task has landed, and to Done once every sub-task is done, by transitions its config names.** | 3 | 2 | 2 | 5 | 4 | 1.2 by 4, 1.5 by 4, 1.6 by 3 | 3.1 |
| 503 | 0.15.0 | 502 | owner | **Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3 | 1.2 |
@@ -108,18 +107,6 @@
## Details
### 500. Mutate the Go lines a branch changes with Gremlins in `mutate-changes`, as it mutates Python, Rust, TypeScript and PHP, and print each mutant the tests do not kill.
fejkdata runs Gremlins v0.6.0 from a compose service of its own today, so a Go project's review mutates outside the shared tool. Run `gremlins unleash --diff <merge base> --output <file>` through `--prefix`, as the other tools run, read its JSON report, and print each mutant that `LIVED` or is `NOT COVERED` as `<path>:<line>:<column>: <status>: <mutator>`. Leave out `*_test.go`. Gremlins v0.6.0 mutates one run of lines per hunk, so its git runs with `diff.context=0`, as fejkdata's compose file sets. A worktree keeps git's data outside the folder a container mounts: `--help` shows the prefix that mounts it, as fejkdata's README does with `GIT_COMMON_DIR`.
Touches:
- `bin/mutate_changes/main.py`: `(module)` (small), `mutate` (small)
- `bin/mutate_changes/changed_lines.py`: `TOOLS` (small), `TEST_NAME` (small)
- `bin/mutate_changes/in_diff_tools.py`: `run_gremlins`
- `bin/mutate_changes/test_mutate_changes.py`
- `README.md`: `## Tools` (small)
### 501. Ship `task-bench`, which plants a bug the tests catch at the places the comprehension seats found hardest, has Haiku fix it in a vanilla Claude Code at a base and a head, and prints for each side how often the fix passes, with its median tokens, turns, wall time and reads before it opens a file the bug is in.
The panel's scores barely move between chunks, and `reading-cost` measures chunks whose tasks differ. A planted task stays fixed, so a change in its cost comes from the code. The bench runs only programs; no model judges anything in it.