Merge remote-tracking branch 'origin/main' into plan-standalone

This commit is contained in:
Lilleman
2026-11-20 21:53:45 +01:00
9 changed files with 172 additions and 36 deletions
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- `mutate-changes` mutates a Python script from its own folder where a test beside it imports it by its file name, as in `import run_harness`, so mutmut finds the tests' calls to it. A crash inside mutmut now becomes one report line that names mutmut's error, not a traceback, and the other folders still report.
- do-epic starts an issue in Jira as its chunk starts: it assigns the issue to the account it works as, unless someone else holds it, and the new settings `jira-start-transition` and `jira-sprint-board` in its `config.toml` move the issue and add it to a board's active sprint.
- Load no MCP server in the bench's headless `claude -p` judge and in the reviewer-merge trials' new-findings call either, so text an outsider wrote cannot reach one. A new check fails any Python file that passes `--tools` more often than `--strict-mcp-config`.
- Load no MCP server in the six headless `claude -p` calls that deny the model tools, so text an outsider wrote cannot reach one.
+42
View File
@@ -0,0 +1,42 @@
"""Files that tests import by their file name, which mutmut finds only when it runs in the file's own folder.
A test in docker/scripts/ that runs `import run_harness` gets it because pytest, or unittest's discover, puts that folder on sys.path. mutmut run from the
checkout's top names the same file docker.scripts.run_harness, so it finds none of the tests' calls to it."""
import ast
import os
import pathlib
import posixpath
from mutate_changes.changed_lines import is_test
def imported_by_file_name(top: str, grouped: dict[str, dict[str, frozenset[int]]], configs: tuple[str, ...]) -> dict[str, dict[str, frozenset[int]]]:
"""grouped, with each file in its "." group moved to its own folder where a test beside it imports it by its file name and the folder has no __init__.py.
A file in a folder that holds one of configs stays, as does every file where the top holds one: the settings there decide how its tests import it."""
if "." not in grouped or any(os.path.isfile(os.path.join(top, name)) for name in configs):
return grouped
regrouped: dict[str, dict[str, frozenset[int]]] = {name: files for name, files in grouped.items() if name != "."}
for path, lines in grouped["."].items():
own = posixpath.dirname(path) or "."
moved = posixpath.basename(path)[:-3] in top_level_imports(os.path.join(top, own)) and not os.path.isfile(os.path.join(top, own, "__init__.py"))
regrouped.setdefault(own if moved else ".", {})[posixpath.basename(path) if moved else path] = lines
return regrouped
def top_level_imports(folder: str) -> set[str]:
"""The top-level module names that the test files in folder import, as `a` for `import a.b`. A test file that can't be read or parsed adds none."""
names: set[str] = set()
for test in sorted(pathlib.Path(folder).glob("*.py")):
if not is_test(test.name):
continue
try:
tree = ast.parse(test.read_text(encoding="utf-8"))
except (OSError, SyntaxError, UnicodeDecodeError, ValueError):
continue
for node in ast.walk(tree):
if isinstance(node, ast.Import):
names.update(alias.name.partition(".")[0] for alias in node.names)
elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module:
names.add(node.module.partition(".")[0])
return names
+8 -4
View File
@@ -8,10 +8,14 @@ extension, and leaves out test files: test_*.py, *_test.py, conftest.py, *.test.
or __tests__/ folder.
- .py: mutmut 3, in a scratch copy of the nearest folder above the file that holds a setup.cfg or a pyproject.toml, else of the
checkout's top. It mutates each function or method a changed line falls in, and counts the mutants on changed lines. The
folder's setup.cfg settings stay, but source_paths and also_copy are set, and pytest leaves out each test that fails without a
mutant; the report names their files. mutmut sees only the code a test runs in its own process, so a mutant that only a child
process reaches, such as a tool a test starts, shows as `no tests`. A pyproject.toml with a [tool.mutmut] section fails the run.
checkout's top. The exception is a file that a test beside it imports by its file name, as `import run_harness`, where the
folder has no __init__.py and neither it nor the checkout's top holds a setup.cfg or pyproject.toml: mutmut runs in that file's
own folder, as there the tests find the file under the same name. It mutates each function or method a changed line falls in,
and counts the mutants on changed lines. The folder's setup.cfg settings stay, but source_paths and also_copy are set, and
pytest leaves out each test that fails without a mutant; the report names their files. mutmut sees only the code a test runs
in its own process, so a mutant that only a child process reaches, such as a tool a test starts, shows as `no tests`. A
pyproject.toml with a [tool.mutmut] section fails the run. A crash inside mutmut, such as a KeyError in its worker pool,
becomes one line for its folder, naming the error and telling you to mutate that folder's changed lines by hand.
- .rs: cargo-mutants, as `cargo mutants --in-diff`, in the checkout's top.
- .cjs, .cts, .js, .jsx, .mjs, .mts, .ts, .tsx: Stryker, as `npx --no --package=@stryker-mutator/core stryker run --mutate` on the
changed lines, in the nearest folder above the file that holds a package.json. A file with no package.json above it isn't
+17
View File
@@ -4,6 +4,7 @@ mutmut names a mutant `<module>.<key>__mutmut_<number>`. The key is x_<function>
import ast
import functools
from typing import NamedTuple
import re
from collections.abc import Sequence
@@ -40,6 +41,22 @@ def named(name: str, paths: list[str]) -> tuple[str, str, int] | None:
return (found[0], key, int(number)) if found and number.isdigit() else None
TRACEBACK = "Traceback (most recent call last):"
FRAME = re.compile(r'^\s*File "(.*?)", line (\d+)', re.MULTILINE)
MUTMUT_FILE = re.compile(r"[/\\]site-packages[/\\](mutmut[/\\].*)")
def crash_in_mutmut(output: str) -> str | None:
"""The error and place of the last traceback in output, where its last frame is in the mutmut package, else None."""
block = output.rpartition(TRACEBACK)[2] if TRACEBACK in output else ""
frames = list(FRAME.finditer(block))
if not frames:
return None
installed = MUTMUT_FILE.search(frames[-1][1])
error = next((line for line in block[frames[-1].end():].splitlines()[1:] if line and not line[0].isspace()), "")
return f"{error}, in {installed[1].replace(chr(92), '/')} line {frames[-1][2]}" if installed and error else None
def statuses(results: str) -> dict[str, str]:
"""Each mutant's status by name, from the ` <name>: <status>` lines `mutmut results --all true` prints."""
return {name: status for name, _, status in (line.strip().partition(": ") for line in results.splitlines()) if "__mutmut_" in name and status}
+13 -9
View File
@@ -1,8 +1,8 @@
"""Mutate the changed lines of a branch's Python files with mutmut 3, in a scratch copy of the folder each file's tests run from.
mutmut reads its settings from the folder it runs in, writes beside them, and names each module by its path from there. So each file is
mutated from the nearest folder above it, up to the checkout's top, that holds a setup.cfg or a pyproject.toml: the folder its tests
import it from."""
mutated from the folder its tests import it from: the nearest folder above it, up to the checkout's top, that holds a setup.cfg or a
pyproject.toml, or the file's own folder where a test beside it imports it by its file name (bare_imports.py)."""
import dataclasses
import os
import pathlib
@@ -11,10 +11,11 @@ import shutil
import tomllib
from mutate_changes.changed_lines import by_folder, label
from mutate_changes.mutmut_mutants import crash_in_mutmut, function_keys, mutant_globs, mutated_line, named, statuses
from mutate_changes.tool_process import failed, Ran, remove, Run, scratch_folder
from failure import Failure
from mutate_changes.mutmut_mutants import function_keys, mutant_globs, mutated_line, named, statuses
from git_cli.git_repo import Git
from mutate_changes.bare_imports import imported_by_file_name
from mutate_changes.mutmut_setup import left_out, mutmut_config
CAUGHT = frozenset({"caught by type check", "killed"})
@@ -47,7 +48,7 @@ def run_mutmut(top: str, changed: dict[str, frozenset[int]], run: Run, git: Git,
changed maps each file, by its path from top, to its changed lines."""
report: list[str] = []
for path, files in sorted(by_folder(top, changed, CONFIGS).items()):
for path, files in sorted(imported_by_file_name(top, by_folder(top, changed, CONFIGS), CONFIGS).items()):
lines = mutate_folder(top, path, files, run, git, jobs)
if isinstance(lines, Failure):
return lines
@@ -126,7 +127,10 @@ def mutate_copy(copy: str, copied: list[str], folder: Folder, argv: list[str], r
ran = run_until_stats_pass(copy, copied, folder, argv, run)
if isinstance(ran, Failure):
return ran
output, left = ran
(output, code), left = ran
crash = crash_in_mutmut(output)
if code and crash:
return [f"mutmut in {folder.label} crashed with {crash}: mutate its changed lines by hand."]
if not os.path.isdir(os.path.join(copy, "mutants")):
return Failure(f"mutmut in {folder.label} wrote no mutants/ in {copy}; --prefix must run it in $PWD")
if NO_MUTANT in output:
@@ -141,8 +145,8 @@ def mutate_copy(copy: str, copied: list[str], folder: Folder, argv: list[str], r
return [f"mutmut in {folder.label}: {counted} mutants on changed lines, {len(shown)} not killed.{unreached}{left_note}", *shown]
def run_until_stats_pass(copy: str, copied: list[str], folder: Folder, argv: list[str], run: Run) -> tuple[str, list[str]] | Failure:
"""mutmut's output, and the options that left out each test that fails before any mutant: mutmut runs no mutant while one fails."""
def run_until_stats_pass(copy: str, copied: list[str], folder: Folder, argv: list[str], run: Run) -> tuple[tuple[str, int], list[str]] | Failure:
"""mutmut's output and exit code, and the options that left out each test that fails before any mutant: mutmut runs no mutant while one fails."""
ran = configured_run(copy, copied, folder, [], argv, run)
left: list[str] = []
if isinstance(ran, Ran) and STATS_FAILED in ran.output:
@@ -151,9 +155,9 @@ def run_until_stats_pass(copy: str, copied: list[str], folder: Folder, argv: lis
ran = configured_run(copy, copied, folder, left, argv, run)
if isinstance(ran, Failure):
return ran
if ran.code and NO_MUTANT not in ran.output and NO_TEST not in ran.output:
if ran.code and NO_MUTANT not in ran.output and NO_TEST not in ran.output and not crash_in_mutmut(ran.output):
return failed(f"mutmut run in {folder.label}", ran)
return ran.output, left
return (ran.output, ran.code), left
def configured_run(copy: str, copied: list[str], folder: Folder, options: list[str], argv: list[str], run: Run) -> Ran | Failure:
+52 -1
View File
@@ -15,8 +15,9 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from mutate_changes.changed_lines import added_lines, by_folder, not_mutated, runs, tool_for
from mutate_changes.mutmut_tool import copy_files
from mutate_changes.mutmut_mutants import crash_in_mutmut, function_keys, mutant_globs, mutated_line, named, statuses
from failure import Failure
from mutate_changes.mutmut_mutants import function_keys, mutant_globs, mutated_line, named, statuses
from mutate_changes.bare_imports import imported_by_file_name
from test_env import isolated_env
from mutate_changes.mutmut_setup import left_out, mutmut_config
from mutate_changes import main
@@ -88,6 +89,56 @@ class ChangedLines(unittest.TestCase):
self.assertEqual(grouped, {"pkg": {"inner/deep.py": lines}, "pkg/inner/lib": {"x.py": lines}, ".": {"top.py": lines}})
self.assertEqual(by_folder(top, {"web/src/a.ts": lines, "pkg/b.ts": lines}, ("package.json",)), {"web": {"src/a.ts": lines}, ".": {"pkg/b.ts": lines}})
def test_a_file_its_sibling_test_imports_by_file_name_moves_to_its_own_folder(self):
with tempfile.TemporaryDirectory() as top:
tests = {
"docker/scripts/a_helper.py": "import not_a_test\n",
"docker/scripts/test_a_bad.py": "def (:\n",
"docker/scripts/test_b_run.py": "import unittest\nimport run_harness as rh\nimport dotted.sub\nfrom other import thing\nfrom dotted2.sub import z\nfrom .rel import y\n",
"docker/scripts/test_c_odd.py": "import \u00e9\n",
"pkg/test_pkg.py": "from pkg.mod import x\nimport mod\n",
"pkg/__init__.py": "",
"lib/test_lib.py": "import lib_tool\n",
"lib/__init__.py": "",
"test_top.py": "import top_tool\n",
}
for path, text in tests.items():
pathlib.Path(top, path).parent.mkdir(parents=True, exist_ok=True)
pathlib.Path(top, path).write_text(text, encoding="utf-8")
lines = frozenset({1})
names = ("docker/scripts/run_harness.py", "docker/scripts/dotted.py", "docker/scripts/other.py", "docker/scripts/dotted2.py", "docker/scripts/rel.py",
"docker/scripts/solo.py", "docker/scripts/not_a_test.py", "pkg/mod.py", "lib/lib_tool.py", "top_tool.py", "solo_top.py")
grouped = {".": {name: lines for name in names}, "web": {"app.py": lines}}
moved = ("run_harness.py", "dotted.py", "other.py", "dotted2.py")
self.assertEqual(imported_by_file_name(top, grouped, ("setup.cfg",)), {
".": {"docker/scripts/rel.py": lines, "docker/scripts/solo.py": lines, "docker/scripts/not_a_test.py": lines, "pkg/mod.py": lines, "lib/lib_tool.py": lines, "top_tool.py": lines, "solo_top.py": lines},
"docker/scripts": {name: lines for name in moved},
"web": {"app.py": lines},
})
self.assertEqual(imported_by_file_name(top, {"web": {"app.py": lines}}, ("setup.cfg",)), {"web": {"app.py": lines}})
pathlib.Path(top, "setup.cfg").write_text("")
self.assertEqual(imported_by_file_name(top, grouped, ("setup.cfg",)), grouped)
def test_a_traceback_that_ends_inside_mutmut_is_a_crash_of_mutmut(self):
def trace(*frames, error="KeyError: 974598", after=""):
shown = "".join(f' File "{path}", line {number}, in f\n code()\n ~~^^\n' for path, number in frames)
return f"Traceback (most recent call last):\n{shown}{error}\n{after}"
lib = "/uv/lib/python3.14/site-packages/mutmut"
inside = trace((f"{lib}/__main__.py", 9), (f"{lib}/workers/isolation.py", 889), (f"{lib}/workers/isolation.py", 12))
outside = trace((f"{lib}/__main__.py", 9), ("/repo/calc.py", 3), error="KeyError: 1")
place = "KeyError: 974598, in mutmut/workers/isolation.py line 12"
self.assertEqual((crash_in_mutmut(inside), crash_in_mutmut("Running stats\n" + inside + "\n")), (place, place))
self.assertEqual(crash_in_mutmut(inside + "later output\n"), place)
self.assertEqual(crash_in_mutmut(outside + trace((f"{lib}/x.py", 1), error="E: 2")), "E: 2, in mutmut/x.py line 1")
self.assertEqual(crash_in_mutmut(trace((f"{lib}/x.py", 1), error="E: 2") + outside), None)
self.assertEqual(crash_in_mutmut(trace(("C:\\py\\Lib\\site-packages\\mutmut\\workers\\x.py", 5))), "KeyError: 974598, in mutmut/workers/x.py line 5")
for path in ("/mutmut/repo/calc.py", "mutmut_tool.py", "<string>", "/a/site-packages/mutmut_tool.py"):
with self.subTest(path):
self.assertEqual(crash_in_mutmut(trace((path, 1))), None)
self.assertEqual(crash_in_mutmut(trace((f"{lib}/x.py", 1)).replace("Traceback (most recent call last):", "")), None)
self.assertEqual((crash_in_mutmut("all fine"), crash_in_mutmut(""), crash_in_mutmut("Traceback (most recent call last):\nKeyError: 1")), (None,) * 3)
self.assertEqual(crash_in_mutmut(trace((f"{lib}/x.py", 1), error="")), None)
def test_not_mutated_names_the_files_no_tool_covers_then_the_test_files(self):
self.assertEqual(not_mutated(["README.md", "a.py", "b.go", "test_a.py", "web/a.test.ts"]), ["Not mutated, as no tool covers their language: README.md, b.go.", "Not mutated, as test files: test_a.py, web/a.test.ts."])
self.assertEqual((not_mutated(["a.py", "test_a.py"]), not_mutated(["a.py", "b.go"]), not_mutated([])), (["Not mutated, as test files: test_a.py."], ["Not mutated, as no tool covers their language: b.go."], []))
@@ -58,6 +58,9 @@ else:
if mode == "setup-fault":
print("Stopping early, because tests recorded trampoline hits but none match any mutant key. This usually means imports bypass the mutants.")
sys.exit(1)
if mode == "crash":
print("Traceback (most recent call last):\n File \"/uv/site-packages/mutmut/__main__.py\", line 1050, in _run\n File \"/uv/site-packages/mutmut/workers/isolation.py\", line 889, in wait_for_result\n worker = pop(pid)\nKeyError: 974598")
sys.exit(1)
"""
# cargo-mutants, Infection and Stryker, which log the folder they run in and how they were called, and exit with the code the test
# gives. cargo-mutants also logs the diff it is given and writes its outcome files, all but unviable.txt, as a run without an
@@ -205,6 +208,10 @@ class MutateChangesRuns(unittest.TestCase):
ran = self.run_tool(*self.faked("no-test"))
self.assertEqual(ran.stdout.splitlines()[:3], ["mutmut in pkg: 2 mutants on changed lines, 2 not killed. No test reaches them.", "pkg/calc.py:6: no tests: return n * 4", "pkg/calc.py:6: no tests: return n / 3"], ran.stderr)
def test_a_crash_inside_mutmut_is_one_line_naming_mutmut_and_the_error_and_the_run_goes_on(self):
ran = self.run_tool(*self.faked("crash"))
self.assertEqual((ran.returncode, ran.stderr, ran.stdout.splitlines()[0]), (0, "", "mutmut in pkg crashed with KeyError: 974598, in mutmut/workers/isolation.py line 889: mutate its changed lines by hand."))
def test_a_run_that_stops_early_for_another_reason_fails_the_run(self):
ran = self.run_tool(*self.faked("setup-fault"))
self.assertEqual((ran.returncode, ran.stderr.splitlines()[0]), (1, "mutmut run in pkg exited 1:"))
+31
View File
@@ -3236,3 +3236,34 @@ Read most (opens and searches):
- `bin/do_epic/test_issue_scores.py` 1
- `bin/do_epic/test_repo_pick.py` 1
- `bin/model_routing/ladder.py` 1
## Reading cost, chunk 4fb66d10f849, finished 2026-11-20T17:43:35Z
54949 new input tokens before the first commit.
Read most (opens and searches):
- `workflow-bench/test_advisor_bench.py` 3
- `bin/repo_checks/test_no_patched_globals.py` 2
- `bin/repo_checks/test_trials_imports.py` 2
- `bin/repo_checks/__init__.py` 1
- `ci.sh` 1
- `reviewer-merge-trials/analyze.py` 1
- `workflow-bench/advisor_bench.py` 1
## Reading cost, chunk 79a2a453c4b8, finished 2026-11-20T18:02:46Z
248834 new input tokens before the first commit.
Read most (opens and searches):
- `bin/do_epic/test_main.py` 14
- `bin/do_epic/main.py` 6
- `bin/do_epic/backend/jira.py` 5
- `bin/do_epic/backend/test_jira.py` 5
- `bin/do_epic/test_do_epic.py` 4
- `README.md` 3
- `bin/do_epic/backend/contract.py` 3
- `bin/do_epic/config.py` 3
- `bin/atlassian/jira.py` 2
- `bin/do_epic/backend/fake.py` 2
+1 -22
View File
@@ -12,9 +12,8 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 474 | 0.15.0 | | | **Run `mutate-changes` in a repo whose tests import a script by its file name, such as Timewave's `docs-and-tools`, and name a crash inside mutmut in one line.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 4 | 1.5 |
| 472 | 0.15.0 | | owner | **Let a project's config name a working branch per epic, such as `TW-12345`, that chunks branch off, merge into and keep level with `main`, so a person merges the finished epic into `main` in one pull request.** | 4 | 2 | 3 | 7 | 7 | 1.5 by 6, 1.6 by 7 | 1.1 |
| 467 | 0.15.0 | 472, 474 | question | **Cut 0.15.0 as soon as every other item in it has merged, before its release review, and run that review in 0.15.1.** | 1 | 1 | 1 | 1 | 1 | | 0.0 |
| 467 | 0.15.0 | 472 | question | **Cut 0.15.0 as soon as every other item in it has merged, before its release review, and run that review in 0.15.1.** | 1 | 1 | 1 | 1 | 1 | | 0.0 |
| 469 | 0.15.1 | | | **Move a do-epic parent issue whose sub-tasks a person filed to Release Ready once every sub-task has landed, and to Done once every sub-task is done, by transitions its config names.** | 3 | 2 | 2 | 5 | 4 | 1.2 by 4, 1.5 by 4, 1.6 by 3 | 3.1 |
| 477 | 0.16.0 | | question | **Rewrite `## Goals` and `## Audience` for a setup others install and run apart from their own `~/.claude`.** | 1 | 1 | 2 | 2 | 2 | | 0.0 |
| 478 | 0.16.0 | | question | **Name the project, its repo and its two commands, today proposed as `do-release` and `do-epic`.** | 1 | 1 | 2 | 2 | 1 | | 0.0 |
@@ -100,26 +99,6 @@
## Details
### 474. Run `mutate-changes` in a repo whose tests import a script by its file name, such as Timewave's `docs-and-tools`, and name a crash inside mutmut in one line.
Found on 2026-11-20 while shipping TW-13648 in Timewave's `docs-and-tools`: `mutate-changes` failed twice there, so its worker mutated 18 changed lines by hand instead. The first run failed on a module path: the tests in `docker/scripts/` import the script under test as a top-level module (`import run_harness`), which mutmut's copy of the tree doesn't resolve as the tests expect. The second run crashed inside mutmut, with a `KeyError` in its worker pool.
- Find why each run failed, from a run against `docs-and-tools` at the commit TW-13648 merged.
- Where the module path is the cause, make `mutate-changes` run tests that import a script by its file name, as `docs-and-tools` does, the way they run outside mutmut.
- Where mutmut itself crashes, catch that crash and say so in one line naming mutmut and the error, so the worker knows to mutate by hand, instead of a traceback.
- Item 462 covers tests that fail in mutmut's copy of claude-config itself; check whether one fix serves both.
Release: lilleman moved it to 0.15.0 on 2026-11-20, ahead of epic TW-13503, which changes `docs-and-tools` in six sub-tasks. mutmut costs no tokens and tries every mutation it knows on each changed line; a worker that mutates by hand pays tokens for each edit, test run and revert, and tried only 18 mutants on TW-13648.
Touches:
- `bin/mutate_changes/mutmut_tool.py`: `mutate_folder`, `read_folder`, `project_setup`, `copy_files`, `mutate_copy`, `configured_run`
- `bin/mutate_changes/mutmut_setup.py`: `mutmut_config`
- `bin/mutate_changes/mutmut_mutants.py`: `module`, `mutant_globs` (small)
- `bin/mutate_changes/tool_process.py`: `failed` (small)
- `bin/mutate_changes/test_mutate_changes.py`
- `bin/mutate_changes/test_mutate_changes_runs.py`
### 472. Let a project's config name a working branch per epic, such as `TW-12345`, that chunks branch off, merge into and keep level with `main`, so a person merges the finished epic into `main` in one pull request.
Today every chunk branches off the default branch and its pull request merges into it. A team whose rules keep merges into `main` with a person, such as Timewave's, then sets `merge = "human"`, and the loop stops at every chunk until someone merges. lilleman asked on 2026-11-20 for a working branch per epic instead: agents merge each chunk into a branch named as the epic, such as `TW-12345`, and when the whole epic is done and reviewed, a person merges that branch into `main` in one pull request.