Merge remote-tracking branch 'origin/main' into item-469
CI / tests (pull_request) Has been cancelled
CI / tests (pull_request) Has been cancelled
This commit is contained in:
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- mutate-changes now mutates the Go lines a branch changes with Gremlins, as it does Python, Rust, TypeScript and PHP, and prints each mutant the tests do not kill.
|
||||
- A project's config may set `epic-branch`, such as `epic-branch = "{epic}"`: do-epic then keeps one working branch per epic, level with the default branch by merge commits, and each chunk branches off it and merges into it without waiting for a person, even under `merge = "human"`. `chunk-pr open-epic` opens the finished epic's pull request into the default branch for a person to merge. `review-request-comment` names a comment `chunk-pr` posts after each push to ask a forge's reviewer for a review, and `jira-pull-request-open-status` and `jira-pull-request-merged-status` name the statuses do-epic moves an issue to as its pull request opens and merges.
|
||||
- `mutate-changes` mutates a Python script from its own folder where a test beside it imports it by its file name, as in `import run_harness`, so mutmut finds the tests' calls to it. A crash inside mutmut now becomes one report line that names mutmut's error, not a traceback, and the other folders still report.
|
||||
- do-epic starts an issue in Jira as its chunk starts: it assigns the issue to the account it works as, unless someone else holds it, and the new settings `jira-start-transition` and `jira-sprint-board` in its `config.toml` move the issue and add it to a board's active sprint.
|
||||
|
||||
@@ -7,9 +7,9 @@ import typing
|
||||
# A hunk header of `git diff -U0`. The new side's count is 0 where the hunk only deletes, and left out where it is 1.
|
||||
HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@", re.MULTILINE)
|
||||
STRYKER = (".cjs", ".cts", ".js", ".jsx", ".mjs", ".mts", ".ts", ".tsx")
|
||||
TOOLS = {".php": "infection", ".py": "mutmut", ".rs": "cargo-mutants", **dict.fromkeys(STRYKER, "stryker")}
|
||||
TOOLS = {".go": "gremlins", ".php": "infection", ".py": "mutmut", ".rs": "cargo-mutants", **dict.fromkeys(STRYKER, "stryker")}
|
||||
TEST_FOLDERS = frozenset({"__tests__", "test", "tests"})
|
||||
TEST_NAME = re.compile(r"test_.*\.py|.*_test\.py|conftest\.py|.*\.(?:spec|test)\.[^.]+")
|
||||
TEST_NAME = re.compile(r"test_.*\.py|.*_test\.py|conftest\.py|.*_test\.go|.*\.(?:spec|test)\.[^.]+")
|
||||
|
||||
|
||||
def added_lines(diff: str) -> frozenset[int]:
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
"""Mutate a branch's changed lines with cargo-mutants, Stryker and Infection, each of which takes the changed lines itself."""
|
||||
"""Mutate a branch's changed lines with cargo-mutants, Gremlins, Infection and Stryker, each of which takes the changed lines itself."""
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import typing
|
||||
|
||||
from mutate_changes.changed_lines import by_folder, label, runs
|
||||
from mutate_changes.tool_process import failed, Ran, remove, Run, scratch_folder
|
||||
@@ -10,6 +12,11 @@ from git_cli.git_repo import Git
|
||||
# cargo-mutants' exit codes for a run that finished: every mutant caught, some missed, and some timed out.
|
||||
CARGO_FINISHED = frozenset({0, 2, 3})
|
||||
CARGO_OUTCOMES = ("caught", "missed", "timeout", "unviable")
|
||||
GREMLINS_UNKILLED = frozenset({"LIVED", "NOT COVERED"})
|
||||
# What Gremlins v0.6.0 needs of the diff it reads from git: one run of lines per hunk is all it mutates, so a hunk holds the changed
|
||||
# lines and none around them, nor a neighbouring change; it takes each file's path after a b/; and it reads no colour codes. A user's git
|
||||
# settings change each.
|
||||
GREMLINS_GIT_SETTINGS = {"diff.context": "0", "diff.noprefix": "false", "diff.mnemonicPrefix": "false", "diff.srcPrefix": "a/", "diff.dstPrefix": "b/", "color.diff": "never", "diff.interHunkContext": "0"}
|
||||
|
||||
|
||||
def run_cargo_mutants(top: str, base: str, paths: list[str], run: Run, git: Git, jobs: int) -> list[str] | Failure:
|
||||
@@ -38,6 +45,52 @@ def run_cargo_mutants(top: str, base: str, paths: list[str], run: Run, git: Git,
|
||||
return [f"cargo-mutants: {sum(map(len, outcomes.values()))} mutants on changed lines, {len(uncaught)} not caught.", *uncaught]
|
||||
|
||||
|
||||
class Mutant(typing.NamedTuple):
|
||||
"""A mutant in Gremlins' report, in the order the report prints them."""
|
||||
|
||||
path: str
|
||||
line: int
|
||||
column: int
|
||||
status: str
|
||||
mutator: str
|
||||
|
||||
|
||||
def run_gremlins(top: str, base: str, run: Run, jobs: int) -> list[str] | Failure:
|
||||
"""A line with the number of mutants on the changed lines and how many of them Gremlins reports as lived or not covered, then a
|
||||
line per such mutant, as `<path>:<line>:<column>: <status>: <mutator>`.
|
||||
|
||||
Gremlins runs in top and mutates the Go lines the branch changes since base. It skips `*_test.go` itself. It reads the diff from
|
||||
git, so `run` must hold the environment that `with_plain_git_diff` makes. Gremlins exits nonzero on a mutation score below a
|
||||
threshold the project sets, which its exit code doesn't tell from a run that broke, so any nonzero exit fails the run."""
|
||||
scratch = scratch_folder(top)
|
||||
if isinstance(scratch, Failure):
|
||||
return scratch
|
||||
try:
|
||||
report = pathlib.Path(scratch, "report.json")
|
||||
ran = run(["gremlins", "unleash", "--diff", base, "--output", os.path.join(os.path.relpath(scratch, top), report.name), "--workers", str(jobs)], top)
|
||||
if isinstance(ran, Failure) or ran.code:
|
||||
return ran if isinstance(ran, Failure) else whole_output("Gremlins", ran)
|
||||
# Gremlins writes no report where it finds no mutant, and `null` for a list with no entry.
|
||||
files = (json.loads(report.read_text(encoding="utf-8"))["files"] or []) if report.is_file() else []
|
||||
mutants = [Mutant(file["file_name"], mutant["line"], mutant["column"], mutant["status"], mutant["type"]) for file in files for mutant in file["mutations"] or []]
|
||||
except (OSError, ValueError, KeyError, TypeError) as failure:
|
||||
return Failure(f"could not read Gremlins' report in {scratch}: {failure}")
|
||||
finally:
|
||||
remove(scratch)
|
||||
# Gremlins lists the mutants on the unchanged lines of a hunk too, as SKIPPED. It lists the files in no fixed order.
|
||||
unkilled = sorted(mutant for mutant in mutants if mutant.status in GREMLINS_UNKILLED)
|
||||
on_changed_lines = sum(mutant.status != "SKIPPED" for mutant in mutants)
|
||||
return [f"gremlins: {on_changed_lines} mutants on changed lines, {len(unkilled)} not killed.", *(f"{m.path}:{m.line}:{m.column}: {m.status}: {m.mutator}" for m in unkilled)]
|
||||
|
||||
|
||||
def with_plain_git_diff(env: typing.Mapping[str, str]) -> dict[str, str]:
|
||||
"""env with the settings of GREMLINS_GIT_SETTINGS added to the ones it holds, as git reads them from its GIT_CONFIG_* variables."""
|
||||
held = env.get("GIT_CONFIG_COUNT", "")
|
||||
first = int(held) if held.isdecimal() else 0
|
||||
added = {f"GIT_CONFIG_{part}_{first + index}": text for index, (key, value) in enumerate(GREMLINS_GIT_SETTINGS.items()) for part, text in (("KEY", key), ("VALUE", value))}
|
||||
return {**env, **added, "GIT_CONFIG_COUNT": str(first + len(GREMLINS_GIT_SETTINGS))}
|
||||
|
||||
|
||||
def outcome_lines(path: pathlib.Path) -> list[str]:
|
||||
return path.read_text(encoding="utf-8", errors="replace").splitlines() if path.is_file() else []
|
||||
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
"""Mutation-test the lines a branch adds or changes, and print each mutant its tests don't kill.
|
||||
r"""Mutation-test the lines a branch adds or changes, and print each mutant its tests don't kill.
|
||||
|
||||
usage: mutate-changes [--base <ref>] [--prefix <shell text>] [--jobs <n>]
|
||||
|
||||
Run it in a checkout, under ~/.claude/bin/run-heavy. It diffs the working tree's tracked files against the merge base of HEAD and
|
||||
--base, by default the default branch: origin's as last fetched, else main or master. It picks the tool by each changed file's
|
||||
extension, and leaves out test files: test_*.py, *_test.py, conftest.py, *.test.* and *.spec.*, and any file under a tests/, test/
|
||||
extension, and leaves out test files: test_*.py, *_test.py, *_test.go, conftest.py, *.test.* and *.spec.*, and any file under a tests/, test/
|
||||
or __tests__/ folder.
|
||||
|
||||
- .py: mutmut 3, in a scratch copy of the nearest folder above the file that holds a setup.cfg or a pyproject.toml, else of the
|
||||
@@ -16,6 +16,9 @@ or __tests__/ folder.
|
||||
in its own process, so a mutant that only a child process reaches, such as a tool a test starts, shows as `no tests`. A
|
||||
pyproject.toml with a [tool.mutmut] section fails the run. A crash inside mutmut, such as a KeyError in its worker pool,
|
||||
becomes one line for its folder, naming the error and telling you to mutate that folder's changed lines by hand.
|
||||
- .go: Gremlins, as `gremlins unleash --diff`, in the checkout's top, where its go.mod must be. Gremlins reads the diff from git, so
|
||||
the run adds settings to the GIT_CONFIG_* variables, after any the environment holds: no context lines, since Gremlins mutates
|
||||
every line of a hunk, so one hunk per change; a/ and b/ as the file prefixes; and no colour.
|
||||
- .rs: cargo-mutants, as `cargo mutants --in-diff`, in the checkout's top.
|
||||
- .cjs, .cts, .js, .jsx, .mjs, .mts, .ts, .tsx: Stryker, as `npx --no --package=@stryker-mutator/core stryker run --mutate` on the
|
||||
changed lines, in the nearest folder above the file that holds a package.json. A file with no package.json above it isn't
|
||||
@@ -25,8 +28,14 @@ or __tests__/ folder.
|
||||
--prefix is shell text that runs each tool: sh runs `<prefix> "$@"` in the tool's folder, which $PWD names, with the tool's command
|
||||
as "$@". For example, 'uvx --from "mutmut>=3,<4"'. Without --prefix, the command runs as it is. A container must mount $PWD and
|
||||
run there, as 'docker compose -f "$(git rev-parse --show-toplevel)/compose.yaml" run --rm -v "$PWD:$PWD" -w "$PWD" app' does. "$1"
|
||||
is the command's first word, mutmut, cargo, npx or vendor/bin/infection, so a prefix can pick what to run per tool:
|
||||
is the command's first word, mutmut, cargo, gremlins, npx or vendor/bin/infection, so a prefix can pick what to run per tool:
|
||||
'case "$1" in mutmut) set -- uvx --from "mutmut>=3,<4" "$@";; esac;'.
|
||||
Gremlins' git runs in the container too, so a Go prefix mounts git's data and passes on the GIT_CONFIG_* variables, and allows git
|
||||
to read a checkout that the container's root user doesn't own. In a git worktree, git's data is in the main checkout, outside $PWD:
|
||||
|
||||
common="$(git rev-parse --path-format=absolute --git-common-dir)"
|
||||
docker run --rm -v "$PWD:$PWD" -v "$common:$common" -w "$PWD" $(printenv | sed -n 's/^\(GIT_CONFIG_[A-Z0-9_]*\)=.*/-e \1/p') \
|
||||
<image holding gremlins> sh -c 'git config --global --add safe.directory "*" && exec "$@"' sh
|
||||
|
||||
--jobs sets how many tests each tool runs at once, 4 by default: one heavy slot's CPUs. Under run-heavy, a container that sets
|
||||
neither memory limit gets the slot's share, in MiB in RUN_HEAVY_MEMORY; set --jobs so that that many test runs fit in it.
|
||||
@@ -34,6 +43,8 @@ neither memory limit gets the slot's share, in MiB in RUN_HEAVY_MEMORY; set --jo
|
||||
It prints:
|
||||
- for mutmut, per folder, a line with the mutant count, then a line per mutant on a changed line that is not killed, as
|
||||
`<path>:<line>: <status>: <the line as mutated>`, or `<path>: <status>: <name>, on a line mutate-changes cannot find`;
|
||||
- for Gremlins, a line with the mutant count and how many it did not kill, then a line per mutant that lived or is not covered, as
|
||||
`<path>:<line>:<column>: <status>: <mutator>`;
|
||||
- for cargo-mutants, a line with the mutant count, then a line per mutant it missed or that timed out, as
|
||||
`<path>:<line>:<column>: <outcome>: <mutation>`;
|
||||
- for Stryker and Infection, a line naming the folder the tool ran in, then the tool's own report;
|
||||
@@ -42,9 +53,9 @@ It prints:
|
||||
- last, the wall time.
|
||||
|
||||
Exit 1, saying why, on an unknown or repeated argument, an option without its value, or a --jobs that is no whole number from 1.
|
||||
Exit 1 too when git fails, a file cannot be read or copied, a tool cannot start, or a tool fails before it reports. Stryker and
|
||||
Infection also exit nonzero on a mutation score below a threshold the project sets, which fails the run as well and prints their
|
||||
whole report.
|
||||
Exit 1 too when git fails, a file cannot be read or copied, a tool cannot start, or a tool fails before it reports. Stryker, Infection
|
||||
and Gremlins also exit nonzero on a mutation score below a threshold the project sets, which fails the run as well and prints their
|
||||
whole output.
|
||||
"""
|
||||
import functools
|
||||
import os
|
||||
@@ -57,7 +68,7 @@ from machine_load.slots import CPUS_PER_SLOT
|
||||
from git_cli.git_repo import default_ref, Git, top
|
||||
from failure import Failure
|
||||
from git_cli import git_repo
|
||||
from mutate_changes.in_diff_tools import run_cargo_mutants, run_infection, run_stryker
|
||||
from mutate_changes.in_diff_tools import run_cargo_mutants, run_gremlins, run_infection, run_stryker, with_plain_git_diff
|
||||
from mutate_changes.mutmut_tool import run_mutmut
|
||||
from mutate_changes.tool_process import run_tool
|
||||
|
||||
@@ -125,6 +136,8 @@ def mutate(given: dict[str, str], env: typing.Mapping[str, str]) -> list[str] |
|
||||
for tool, files in sorted(changed.items()):
|
||||
if tool == "cargo-mutants":
|
||||
lines = run_cargo_mutants(checkout, merge_base, sorted(files), run, git, jobs)
|
||||
elif tool == "gremlins":
|
||||
lines = run_gremlins(checkout, merge_base, functools.partial(run, env=with_plain_git_diff(env)), jobs)
|
||||
elif tool == "infection":
|
||||
lines = run_infection(checkout, merge_base, run, jobs)
|
||||
elif tool == "stryker":
|
||||
|
||||
@@ -22,6 +22,7 @@ from test_env import isolated_env
|
||||
from mutate_changes.mutmut_setup import left_out, mutmut_config
|
||||
from mutate_changes import main
|
||||
from mutate_changes.tool_process import run_tool, scratch_folder
|
||||
from mutate_changes.in_diff_tools import with_plain_git_diff
|
||||
|
||||
SOURCE = "def double(n):\n\treturn n * 2\n\n\nclass Pt:\n\tdef moved(self, by=1):\n\t\treturn self.x + by\n"
|
||||
# The shape of the file mutmut 3 writes under mutants/: the original of each function and its mutants, with the original's lines.
|
||||
@@ -66,12 +67,14 @@ class ChangedLines(unittest.TestCase):
|
||||
"src/Order.php": "infection",
|
||||
"web/app.tsx": "stryker",
|
||||
"web/app.mjs": "stryker",
|
||||
"cmd/serve/main.go": "gremlins",
|
||||
"bin/x/test_main.py": None,
|
||||
"bin/x/main_test.py": None,
|
||||
"pkg/conftest.py": None,
|
||||
"tests/helpers.py": None,
|
||||
"web/app.test.ts": None,
|
||||
"web/app.spec.js": None,
|
||||
"cmd/serve/main_test.go": None,
|
||||
"web/__tests__/a.js": None,
|
||||
"README.md": None,
|
||||
}
|
||||
@@ -140,8 +143,24 @@ class ChangedLines(unittest.TestCase):
|
||||
self.assertEqual(crash_in_mutmut(trace((f"{lib}/x.py", 1), error="")), None)
|
||||
|
||||
def test_not_mutated_names_the_files_no_tool_covers_then_the_test_files(self):
|
||||
self.assertEqual(not_mutated(["README.md", "a.py", "b.go", "test_a.py", "web/a.test.ts"]), ["Not mutated, as no tool covers their language: README.md, b.go.", "Not mutated, as test files: test_a.py, web/a.test.ts."])
|
||||
self.assertEqual((not_mutated(["a.py", "test_a.py"]), not_mutated(["a.py", "b.go"]), not_mutated([])), (["Not mutated, as test files: test_a.py."], ["Not mutated, as no tool covers their language: b.go."], []))
|
||||
self.assertEqual(not_mutated(["README.md", "a.py", "b.rb", "test_a.py", "web/a.test.ts"]), ["Not mutated, as no tool covers their language: README.md, b.rb.", "Not mutated, as test files: test_a.py, web/a.test.ts."])
|
||||
self.assertEqual((not_mutated(["a.py", "test_a.py"]), not_mutated(["a.py", "b.rb"]), not_mutated([])), (["Not mutated, as test files: test_a.py."], ["Not mutated, as no tool covers their language: b.rb."], []))
|
||||
|
||||
|
||||
class InDiffTools(unittest.TestCase):
|
||||
def test_the_git_settings_gremlins_needs_are_added_after_the_ones_the_environment_holds(self):
|
||||
held = {"HOME": "/h", "GIT_CONFIG_COUNT": "1", "GIT_CONFIG_KEY_0": "safe.directory", "GIT_CONFIG_VALUE_0": "*"}
|
||||
env = with_plain_git_diff(held)
|
||||
self.assertEqual(env["GIT_CONFIG_COUNT"], "8")
|
||||
self.assertEqual({name: value for name, value in env.items() if name in held}, held | {"GIT_CONFIG_COUNT": "8"})
|
||||
settings = [(env[f"GIT_CONFIG_KEY_{index}"], env[f"GIT_CONFIG_VALUE_{index}"]) for index in range(1, 8)]
|
||||
self.assertEqual(settings, [("diff.context", "0"), ("diff.noprefix", "false"), ("diff.mnemonicPrefix", "false"), ("diff.srcPrefix", "a/"), ("diff.dstPrefix", "b/"), ("color.diff", "never"), ("diff.interHunkContext", "0")])
|
||||
|
||||
def test_the_git_settings_start_at_the_first_for_an_environment_without_a_count_or_with_one_that_is_not_a_number(self):
|
||||
for held in ({}, {"GIT_CONFIG_COUNT": "many"}, {"GIT_CONFIG_COUNT": ""}):
|
||||
with self.subTest(held):
|
||||
env = with_plain_git_diff(held)
|
||||
self.assertEqual((env["GIT_CONFIG_COUNT"], env["GIT_CONFIG_KEY_0"], env["GIT_CONFIG_KEY_6"], "GIT_CONFIG_KEY_7" in env), ("7", "diff.context", "diff.interHunkContext", False))
|
||||
|
||||
|
||||
class MutmutSetup(unittest.TestCase):
|
||||
|
||||
@@ -62,10 +62,12 @@ else:
|
||||
print("Traceback (most recent call last):\n File \"/uv/site-packages/mutmut/__main__.py\", line 1050, in _run\n File \"/uv/site-packages/mutmut/workers/isolation.py\", line 889, in wait_for_result\n worker = pop(pid)\nKeyError: 974598")
|
||||
sys.exit(1)
|
||||
"""
|
||||
# cargo-mutants, Infection and Stryker, which log the folder they run in and how they were called, and exit with the code the test
|
||||
# cargo-mutants, Gremlins, Infection and Stryker, which log the folder they run in and how they were called, and exit with the code the test
|
||||
# gives. cargo-mutants also logs the diff it is given and writes its outcome files, all but unviable.txt, as a run without an
|
||||
# unviable mutant does; with CAUGHT_UNREADABLE set, nobody may read caught.txt, and with OUTPUT_LOCKED, nobody may delete from mutants.out.
|
||||
FAKE_TOOLS = r"""import os, pathlib, sys
|
||||
# Gremlins also logs the git settings it finds in its environment and writes a JSON report in Gremlins' shape. With REPORT set, it
|
||||
# writes that text in its place, or with REPORT=none, no report, as Gremlins does where it finds no mutant.
|
||||
FAKE_TOOLS = r"""import json, os, pathlib, sys
|
||||
log, code, args = pathlib.Path(sys.argv[1]), int(sys.argv[2]), sys.argv[3:]
|
||||
with log.open("a") as written:
|
||||
written.write(f"{os.getcwd()}\n{' '.join(args)}\n")
|
||||
@@ -80,6 +82,14 @@ with log.open("a") as written:
|
||||
(outcomes / "caught.txt").chmod(0)
|
||||
if os.environ.get("OUTPUT_LOCKED"):
|
||||
outcomes.chmod(0o555)
|
||||
if args[0] == "gremlins":
|
||||
git_config = (name for name in sorted(os.environ) if name.startswith(("GIT_CONFIG_COUNT", "GIT_CONFIG_KEY_", "GIT_CONFIG_VALUE_")))
|
||||
written.write("".join(f"{name}={os.environ[name]}\n" for name in git_config))
|
||||
killed, skipped, uncovered, lived = (dict(type=kind, status=status, line=line, column=column) for kind, status, line, column in (("ARITHMETIC_BASE", "KILLED", 4, 11), ("CONDITIONALS_BOUNDARY", "SKIPPED", 8, 11), ("CONDITIONALS_BOUNDARY", "NOT COVERED", 12, 11), ("ARITHMETIC_BASE", "LIVED", 4, 15)))
|
||||
files = [dict(file_name="calc.go", mutations=[killed, skipped, uncovered, lived]), dict(file_name="sub/other.go", mutations=[dict(type="INVERT_NEGATIVES", status="LIVED", line=2, column=9)])]
|
||||
report = os.environ.get("REPORT") or json.dumps(dict(go_module="x", files=files))
|
||||
if report != "none":
|
||||
pathlib.Path(args[args.index("--output") + 1]).write_text(report)
|
||||
print(f"{args[0]} reports")
|
||||
sys.exit(code)
|
||||
"""
|
||||
@@ -404,6 +414,53 @@ class MutateChangesRuns(unittest.TestCase):
|
||||
self.assertRegex(ran.stderr, rf"^could not hand cargo-mutants the diff or read its outcomes in {re.escape(str(self.checkout.resolve()))}/\.mutate-changes-\w+: ")
|
||||
self.assertEqual(sorted(path.name for path in self.checkout.iterdir()), [".git", "README.md", "package.json", "pkg", "src", "web"])
|
||||
|
||||
def change_go(self):
|
||||
"""Commit a Go file and its test, then change both."""
|
||||
(self.checkout / "calc.go").write_text("package calc\n\nfunc Add(a, b int) int {\n\treturn a + b\n}\n")
|
||||
(self.checkout / "calc_test.go").write_text("package calc\n")
|
||||
self.commit()
|
||||
(self.checkout / "calc.go").write_text("package calc\n\nfunc Add(a, b int) int {\n\treturn a + b + 0\n}\n")
|
||||
(self.checkout / "calc_test.go").write_text("package calc\n\nfunc helper() {}\n")
|
||||
|
||||
def test_go_goes_to_gremlins_at_the_top_and_prints_each_mutant_that_lived_or_is_not_covered(self):
|
||||
self.change_go()
|
||||
fake = f"python3 {self.temp / 'fake_tools.py'} {self.log} 0"
|
||||
ran = self.run_tool("--base", "HEAD", "--jobs=3", "--prefix", fake, cwd=self.checkout / "pkg")
|
||||
report = ["gremlins: 4 mutants on changed lines, 3 not killed.", "calc.go:4:15: LIVED: ARITHMETIC_BASE", "calc.go:12:11: NOT COVERED: CONDITIONALS_BOUNDARY", "sub/other.go:2:9: LIVED: INVERT_NEGATIVES"]
|
||||
self.assertEqual((ran.returncode, ran.stdout.splitlines()[:-2], ran.stderr), (0, report, ""))
|
||||
self.assertEqual(ran.stdout.splitlines()[-2], "Not mutated, as test files: calc_test.go.")
|
||||
folder, command, config = self.log.read_text().split("\n", 2)
|
||||
self.assertEqual(folder, str(self.checkout.resolve()))
|
||||
base = self.git("rev-parse", "HEAD").strip()
|
||||
self.assertRegex(command, rf"^gremlins unleash --diff {base} --output (\.mutate-changes-\w+)/report\.json --workers 3$")
|
||||
keys = ["diff.context", "diff.noprefix", "diff.mnemonicPrefix", "diff.srcPrefix", "diff.dstPrefix", "color.diff", "diff.interHunkContext"]
|
||||
values = ["0", "false", "false", "a/", "b/", "never", "0"]
|
||||
self.assertEqual(config, "GIT_CONFIG_COUNT=7\n" + "".join(f"GIT_CONFIG_KEY_{n}={key}\n" for n, key in enumerate(keys)) + "".join(f"GIT_CONFIG_VALUE_{n}={value}\n" for n, value in enumerate(values)))
|
||||
self.assertEqual(sorted(path.name for path in self.checkout.iterdir()), [".git", "README.md", "calc.go", "calc_test.go", "pkg"])
|
||||
|
||||
def test_a_gremlins_run_that_finds_no_mutant_reports_none(self):
|
||||
self.change_go()
|
||||
for report in ("none", '{"files": null}', '{"files": [{"file_name": "calc.go", "mutations": null}]}'):
|
||||
with self.subTest(report):
|
||||
ran = self.run_tool("--base", "HEAD", "--prefix", f"python3 {self.temp / 'fake_tools.py'} {self.log} 0", env={"REPORT": report})
|
||||
self.assertEqual((ran.returncode, ran.stdout.splitlines()[0], ran.stderr), (0, "gremlins: 0 mutants on changed lines, 0 not killed.", ""))
|
||||
|
||||
def test_a_gremlins_run_that_fails_or_leaves_a_report_it_cannot_read_fails_the_run(self):
|
||||
self.change_go()
|
||||
fake = f"python3 {self.temp / 'fake_tools.py'} {self.log}"
|
||||
# Gremlins also exits nonzero on a mutation score below the project's threshold, where its output lists what survived.
|
||||
ran = self.run_tool("--base", "HEAD", "--prefix", f"{fake} 10")
|
||||
self.assertEqual((ran.returncode, ran.stderr), (1, "Gremlins exited 10:\ngremlins reports\n"))
|
||||
for report in ("not json", "[]", '{"files": 3}'):
|
||||
with self.subTest(report):
|
||||
ran = self.run_tool("--base", "HEAD", "--prefix", f"{fake} 0", env={"REPORT": report})
|
||||
self.assertEqual(ran.returncode, 1)
|
||||
self.assertRegex(ran.stderr, rf"^could not read Gremlins' report in {re.escape(str(self.checkout.resolve()))}/\.mutate-changes-\w+: ")
|
||||
ran = self.run_tool("--base", "HEAD", env={"PATH": str(self.path_with_git())})
|
||||
self.assertEqual(ran.returncode, 1)
|
||||
self.assertRegex(ran.stderr, "^gremlins unleash .* could not start: ")
|
||||
self.assertEqual(sorted(path.name for path in self.checkout.iterdir()), [".git", "README.md", "calc.go", "calc_test.go", "pkg"])
|
||||
|
||||
def test_a_branch_that_adds_no_line_a_tool_mutates_says_so_and_names_what_it_changes(self):
|
||||
# A file the branch deletes is no longer there to mutate, so the report doesn't name it.
|
||||
(self.checkout / "pkg" / "old.md").write_text("Gone.\n")
|
||||
@@ -425,6 +482,11 @@ class MutateChangesRuns(unittest.TestCase):
|
||||
ran = self.run_tool(flag)
|
||||
self.assertEqual((ran.returncode, ran.stdout.splitlines()[0]), (0, "Mutation-test the lines a branch adds or changes, and print each mutant its tests don't kill."))
|
||||
|
||||
def test_help_shows_the_gremlins_prefix_that_mounts_gits_data_in_a_worktree(self):
|
||||
text = self.run_tool("--help").stdout
|
||||
self.assertIn("-v \"$common:$common\"", text)
|
||||
self.assertIn("--git-common-dir", text)
|
||||
|
||||
def test_unknown_arguments_fail_saying_why(self):
|
||||
usage = "usage: mutate-changes [--base <ref>] [--prefix <shell text>] [--jobs <n>]"
|
||||
for args, reason in (
|
||||
|
||||
@@ -18,6 +18,7 @@ Each project principle states one rule that the decisions below apply; their ent
|
||||
- bin/: the tools send no OTLP traces, logs or metrics, and loop-claude's archive holds the trace of each cycle
|
||||
- bin/: the tools support only the latest stable Python, and the tests, written in plain unittest, run on it in four pytest-xdist workers
|
||||
- chunk-pr: the chunks of a project merge one at a time on a machine, each finishing its plan items in its turn, and a pull request that waits for an approval leaves the queue while it waits
|
||||
- Claude Code in a home of its own: a `CLAUDE_CONFIG_DIR` home loads `CLAUDE.md`, rules, agents, skills, hooks and plugins, and `CLAUDE_CODE_OAUTH_TOKEN` signs in there, but never under `--bare`
|
||||
- deploy: this repo keeps no `AGENTS.md`, since `~/.claude` is its checkout; its principles and decision titles head `docs/decisions.md`, its working rules follow them there, and its `.claude/settings.json` excludes every `.claude/CLAUDE.md`, so a session in the clone loads only the clone's `CLAUDE.md`
|
||||
- do-epic: drives one Jira epic whose items span repos, or have none, ranked by the epic's goals, then code health, then each repo's goals, and closes the epic only once the human confirms every goal
|
||||
- epic-branch: an epic's chunks branch off and merge into a working branch its project's config names, without a person, and a person merges that branch into the default branch in one pull request
|
||||
@@ -123,6 +124,16 @@ Valid while a chunk's worker runs `chunk-pr merge` itself, and one machine runs
|
||||
|
||||
The cap is 5, `bin/plan/todo_table.py`'s `BUNDLE_MAX_R`, from the same chunks: by the highest R among a chunk's items, the median was 0.9M to 1.5M tokens at R 1 to 5, and 2.7M at R 6, where the 90th percentile was 8.8M. A risky item that runs long re-runs review over every item bundled with it. R 5 is also the least R an I 4 item gets, so an item that could take everything down shares a chunk only where its L + E is 3 or less, and an irreversible I 5 item, at R 7 or more, never does. Valid while a review's cost is mostly fixed per chunk; revisit once 30 chunks have shipped bundles under it.
|
||||
|
||||
## Claude Code in a home of its own: a `CLAUDE_CONFIG_DIR` home loads `CLAUDE.md`, rules, agents, skills, hooks and plugins, and `CLAUDE_CODE_OAUTH_TOKEN` signs in there, but never under `--bare`
|
||||
|
||||
2026-11-20, lilleman's standalone-install plan. Serves goal 2, so the setup installs without taking over the human's `~/.claude`, and goal 4: a home of its own keeps the loop's transcripts and token apart from the human's. The plan therefore runs every claude the loop starts in a home of its own, and doesn't load the setup with `--plugin-dir`, `--settings` and `--append-system-prompt-file` over the human's home. The probes ran on Claude Code 2.1.330, on throwaway homes, never the human's. A mock Messages API at `ANTHROPIC_BASE_URL` recorded each request and answered the first with a tool call that reads a file, so the system prompt and the tool results show what loaded, with no real token. Valid for Claude Code 2.1.330; repeat the probes when its version changes.
|
||||
|
||||
- **Loads from `$CLAUDE_CONFIG_DIR`:** `CLAUDE.md` (in the system prompt), `rules/*.md` (one with `paths:` entered the context after the model read a matching file, not before), `agents/`, `skills/`, the hooks of `settings.json` (a `SessionStart` hook ran) and plugins. `claude plugin marketplace add` and `claude plugin install` write to `<home>/plugins`, and the installed plugin's skill is listed. `.claude.json` lands in the home too, so the human's `~/.claude.json` stays theirs.
|
||||
- **Signs in by `CLAUDE_CODE_OAUTH_TOKEN`:** the request carried `Authorization: Bearer <token>` with the `oauth-2025-04-20` beta, and `claude auth status` said `loggedIn: true, authMethod: oauth_token`. The home didn't hold a credentials file. Without the variable the home is `loggedIn: false`. Not tested: that Anthropic accepts a real token, since `claude setup-token` needs the human's browser sign-in. The authentication docs say the token lasts one year and can only make model requests; neither is tested, and the first real run shows whether it signs in.
|
||||
- **`--bare` can't use that token.** With `CLAUDE_CODE_OAUTH_TOKEN`, it says `Not logged in · Please run /login` and doesn't send a request. With `ANTHROPIC_API_KEY` it signs in, but doesn't load anything from the home (`CLAUDE.md`, rules, agents or skills) and offers only Bash, Edit and Read. So a container that signs in by a long-lived token runs without `--bare`; its fresh home is empty, so nothing of the human's loads there either. The project's `CLAUDE.md` loads from the working folder, and its `AGENTS.md` only where the project has no `CLAUDE.md`.
|
||||
- **Transcripts** land in `<home>/projects/<working folder's path, with each / as ->/<session id>.jsonl`. `--no-session-persistence` doesn't write one, only an empty `memory` folder, so a bench that uses it reads the run from `--output-format stream-json`.
|
||||
- **The model catalog** sits in `<home>/cache/model-catalog/` on the human's machine, as `bin/loop_state/paths.py` expects. The throwaway homes didn't write one, since the mock was the only server they reached. Not tested: whether a long-lived-token session writes it. Where it doesn't, `bin/chunk_driver/project/model_choice.py` prints `no model catalog cached` and takes a typed model name.
|
||||
|
||||
## comprehension floor: every dimension and the overall at the project's readability floor or above is the target, and no score blocks a merge or release
|
||||
|
||||
2026-09-27, lilleman, restating the 2026-09-24 target and extending it to the overall. Serves goal 1. 2026-10-13, lilleman: the floor doesn't block anything, and `review: the comprehension check records …` says why. Valid while the scoring run keeps its four seats. 2026-11-12, lilleman, through todo item 358: a project's config sets this floor as `readability-floor`, its readability floor, since legacy code scores low. It is 7.0 where left out, and the signal stays that the score rises. Readers seated as the person who writes the config picked the key, 2 of 3.
|
||||
|
||||
@@ -6,18 +6,17 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 511`
|
||||
`Next ID = 512`
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 479 | 0.15.0 | | owner | **Record in a decision which of `CLAUDE.md`, path-scoped rules, agents, skills, hooks and plugins Claude Code loads from a `CLAUDE_CONFIG_DIR` home, whether a long-lived token logs in there, and where its transcripts and model catalogue land.** | 1 | 2 | 1 | 2 | 3 | 1.6 by 4 | 1.3 |
|
||||
| 500 | 0.15.0 | | owner | **Mutate the Go lines a branch changes with Gremlins in `mutate-changes`, as it mutates Python, Rust, TypeScript and PHP, and print each mutant the tests do not kill.** | 2 | 2 | 2 | 4 | 4 | 1.1 by 4 | 1.2 |
|
||||
| 501 | 0.15.0 | 479, 500 | owner | **Ship `task-bench`, which plants a bug the tests catch at the places the comprehension seats found hardest, has Haiku fix it in a vanilla Claude Code at a base and a head, and prints for each side how often the fix passes, with its median tokens, turns, wall time and reads before it opens a file the bug is in.** | 4 | 2 | 1 | 5 | 9 | 1.1 by 5, code health by 4 | 0.6 |
|
||||
| 501 | 0.15.0 | | owner | **Ship `task-bench`, which plants a bug the tests catch at the places the comprehension seats found hardest, has Haiku fix it in a vanilla Claude Code at a base and a head, and prints for each side how often the fix passes, with its median tokens, turns, wall time and reads before it opens a file the bug is in.** | 4 | 2 | 1 | 5 | 9 | 1.1 by 5, code health by 4 | 0.6 |
|
||||
| 502 | 0.15.0 | 501 | owner | **Distil each `task-bench` run into a digest of short traces and decoys, the files most failing runs read but no fix touches, and have a stronger model turn it into plan items before the transcripts are deleted.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 4, code health by 4 | 4.8 |
|
||||
| 469 | 0.15.0 | | | **Move a do-epic parent issue whose sub-tasks a person filed to Release Ready once every sub-task has landed, and to Done once every sub-task is done, by transitions its config names.** | 3 | 2 | 2 | 5 | 4 | 1.2 by 4, 1.5 by 4, 1.6 by 3 | 3.1 |
|
||||
| 503 | 0.15.0 | 502 | owner | **Calibrate `task-bench` on fejkdata: bench a head against itself for the noise floor, then one finished chunk at its base and head, and record in a decision how many tasks and runs the bench needs.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3 | 1.2 |
|
||||
| 511 | 0.15.0 | | | **Hand the product-owner and prose reviewers the real output of each command a changed doc line says prints something, so a claim about output is checked by running it, not by reading code.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 3 | 1.1 |
|
||||
| 509 | 0.16.0 | 503 | owner | **Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.** | 3 | 2 | 3 | 6 | 5 | code health by 5, 1.1 by 3 | 5.6 |
|
||||
| 504 | 0.16.0 | 503 | owner | **Bench new features in `task-bench`, each a spec for Haiku with acceptance tests it never sees and a reference patch that proves it can be built, kept in a `bench-tasks` folder the project config names, starting with six for fejkdata.** | 3 | 2 | 1 | 4 | 6 | 1.1 by 4, code health by 3 | 2.6 |
|
||||
| 505 | 0.16.0 | 504 | owner | **Bench adding source data in `task-bench`: hand Haiku a fake register that a seeded script generates for a made-up country, and check the importer, the data and the licence entry it adds, starting with five tasks for fejkdata.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 3, code health by 2 | 2.6 |
|
||||
@@ -28,9 +27,9 @@
|
||||
| 478 | 0.17.0 | | question | **Name the project, its repo and its two commands, today proposed as `do-release` and `do-epic`.** | 1 | 1 | 2 | 2 | 1 | | 0.0 |
|
||||
| 480 | 0.18.0 | | owner | **Resolve every path the tools use in one module: the release root (the folder the running code ships in), the config folder, the state folder and the Claude home. Then only that module names `~/.claude`, and the loop still runs from `~/.claude`.** | 3 | 2 | 3 | 6 | 6 | 1.6 by 3, code health by 3 | 2.4 |
|
||||
| 481 | 0.18.0 | 480 | owner | **Put the release's tools on `PATH` for every session the loop starts and every hook it runs, name them bare in the prose in place of `~/.claude/bin/<tool>`, and refuse the old spelling in new prose.** | 3 | 2 | 3 | 6 | 6 | 1.4 by 2, code health by 2 | 1.6 |
|
||||
| 482 | 0.19.0 | 478, 479, 481 | owner | **Install the setup with `uv tool install` from a release tag, with `do-release` and `do-epic` on `PATH`, and replace `deploy-release` with `do-release self-update`.** | 3 | 2 | 3 | 6 | 7 | 1.6 by 4, 1.3 by 3 | 1.3 |
|
||||
| 482 | 0.19.0 | 478, 481 | owner | **Install the setup with `uv tool install` from a release tag, with `do-release` and `do-epic` on `PATH`, and replace `deploy-release` with `do-release self-update`.** | 3 | 2 | 3 | 6 | 7 | 1.6 by 4, 1.3 by 3 | 1.3 |
|
||||
| 485 | 0.19.0 | 478, 482 | owner | **Rename `loop-claude` to `do-release`, and keep `loop-claude` as an alias that warns, for one release.** | 2 | 2 | 3 | 5 | 4 | 1.3 by 2 | 2.1 |
|
||||
| 483 | 0.19.0 | 479, 482 | owner | **Run every claude the loop starts in a Claude home the install generates from the release, with hooks that point into it and `do-release login` to sign in there, so the human's `~/.claude` stays theirs.** | 4 | 2 | 3 | 7 | 7 | 1.6 by 6 | 0.5 |
|
||||
| 483 | 0.19.0 | 482 | owner | **Run every claude the loop starts in a Claude home the install generates from the release, with hooks that point into it and `do-release login` to sign in there, so the human's `~/.claude` stays theirs.** | 4 | 2 | 3 | 7 | 7 | 1.6 by 6 | 0.5 |
|
||||
| 486 | 0.19.0 | 483 | owner | **Open an interactive claude in the loop's home with `do-release claude`, for a manual `/larv-review` or a talk with the setup loaded.** | 2 | 1 | 1 | 2 | 3 | 1.6 by 2 | 0.7 |
|
||||
| 484 | 0.19.0 | 482 | owner | **Keep project configs, this machine's rules, transcripts and review reports in `~/.config/do-release/` and `~/.local/state/do-release/`, move them there with `do-release migrate`, and have rule-hook guard the new places.** | 3 | 2 | 4 | 7 | 6 | 1.6 by 3 | 0.4 |
|
||||
| 487 | 0.20.0 | 483 | owner | **Run larv-review's passes from a program that picks them from the diff and the item, counts rounds against the cap and decides each rerun, with each reviewer returning its findings as JSON against a schema.** | 4 | 2 | 3 | 7 | 8 | 1.1 by 5, 1.4 by 5 | 0.5 |
|
||||
@@ -108,35 +107,13 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 479. Record in a decision which of `CLAUDE.md`, path-scoped rules, agents, skills, hooks and plugins Claude Code loads from a `CLAUDE_CONFIG_DIR` home, whether a long-lived token logs in there, and where its transcripts and model catalogue land.
|
||||
|
||||
Part of the standalone-install plan lilleman approved on 2026-11-20. `do-release` and `do-epic` stand for the names item 478 picks. The plan runs every claude the loop starts in a Claude home of its own, so the human's `~/.claude` stays theirs. Find out by testing on a throwaway home, never the human's. The result decides whether the loop gets a home of its own, or loads the setup with `--plugin-dir`, `--settings` and `--append-system-prompt-file` instead.
|
||||
|
||||
lilleman moved it into 0.15.0 on 2026-11-20, ahead of item 501: `task-bench` runs Haiku in a container, which is a fresh Claude home signed in by a long-lived token, so the decision also says how that container signs in.
|
||||
|
||||
Touches:
|
||||
|
||||
- `docs/decisions.md`: `## Claude Code in a home of its own: what loads from CLAUDE_CONFIG_DIR, and how a long-lived token signs in`
|
||||
|
||||
### 500. Mutate the Go lines a branch changes with Gremlins in `mutate-changes`, as it mutates Python, Rust, TypeScript and PHP, and print each mutant the tests do not kill.
|
||||
|
||||
fejkdata runs Gremlins v0.6.0 from a compose service of its own today, so a Go project's review mutates outside the shared tool. Run `gremlins unleash --diff <merge base> --output <file>` through `--prefix`, as the other tools run, read its JSON report, and print each mutant that `LIVED` or is `NOT COVERED` as `<path>:<line>:<column>: <status>: <mutator>`. Leave out `*_test.go`. Gremlins v0.6.0 mutates one run of lines per hunk, so its git runs with `diff.context=0`, as fejkdata's compose file sets. A worktree keeps git's data outside the folder a container mounts: `--help` shows the prefix that mounts it, as fejkdata's README does with `GIT_COMMON_DIR`.
|
||||
|
||||
Touches:
|
||||
|
||||
- `bin/mutate_changes/main.py`: `(module)` (small), `mutate` (small)
|
||||
- `bin/mutate_changes/changed_lines.py`: `TOOLS` (small), `TEST_NAME` (small)
|
||||
- `bin/mutate_changes/in_diff_tools.py`: `run_gremlins`
|
||||
- `bin/mutate_changes/test_mutate_changes.py`
|
||||
- `README.md`: `## Tools` (small)
|
||||
|
||||
### 501. Ship `task-bench`, which plants a bug the tests catch at the places the comprehension seats found hardest, has Haiku fix it in a vanilla Claude Code at a base and a head, and prints for each side how often the fix passes, with its median tokens, turns, wall time and reads before it opens a file the bug is in.
|
||||
|
||||
The panel's scores barely move between chunks, and `reading-cost` measures chunks whose tasks differ. A planted task stays fixed, so a change in its cost comes from the code. The bench runs only programs; no model judges anything in it.
|
||||
|
||||
- **Plant.** Run Gremlins, as item 500 runs it, without `--diff`, over the files that hold the places `larv-review places` prints. Draw 5 of its `KILLED` mutants by a seeded pick, and re-apply each at its file, line and column. The task's text is the failing assertion's message, without the test's name, file or line, so it says what goes wrong and not where. Go is the first language; item 508 adds the others.
|
||||
- **Copy.** The tree at the commit, without `.git`, committed once with the bug in it, so `git diff` and `git log` show Haiku nothing.
|
||||
- **Run.** Inside a container that holds only the copy: the project's development image with Claude Code added, such as `golang` for fejkdata, signed in by a long-lived token in an environment variable, with a memory limit and the network only for the API. In it, `claude -p --bare --strict-mcp-config --disable-slash-commands --no-session-persistence --model claude-haiku-5-5 --effort low --output-format stream-json` runs with every tool, Bash included, so Haiku greps, builds, runs the CLI and runs single tests as an agent normally does. A narrower tool list would raise the figures for reasons outside the code, and an allowlist can't fence Bash anyway: a rule that allows `cat` allows it on any path, such as the real checkout, whose diff would give away the bug. `--bare` drops the human's instructions, memories, hooks, skills and plugins, which would skew the figures and change with claude-config. The repo's README, `AGENTS.md` and `docs/` stay, since they are part of what the bench measures; a project's `CLAUDE.md`, which `--bare` skips, is passed in, so every project is measured alike.
|
||||
- **Run.** Inside a container that holds only the copy: the project's development image with Claude Code added, such as `golang` for fejkdata, signed in by a long-lived token in an environment variable, with a memory limit and the network only for the API. In it, `claude -p --strict-mcp-config --disable-slash-commands --no-session-persistence --model claude-haiku-5-5 --effort low --output-format stream-json` runs with every tool, Bash included, so Haiku greps, builds, runs the CLI and runs single tests as an agent normally does. A narrower tool list would raise the figures for reasons outside the code, and an allowlist can't fence Bash anyway: a rule that allows `cat` allows it on any path, such as the real checkout, whose diff would give away the bug. Run without `--bare`: it can't sign in by a long-lived token (the decision `Claude Code in a home of its own`). The container's fresh Claude home is empty, so none of the human's instructions, memories, hooks, skills or plugins load; they would skew the figures and change with claude-config. The repo's README and `docs/` stay in the copy, since they are part of what the bench measures. A project's `CLAUDE.md` loads from the working folder, or its `AGENTS.md` where it has no `CLAUDE.md`; the bench takes that as the project's own setup.
|
||||
- **Pair.** 5 runs per task at the base and at the head, in one bench, so a model or CLI update moves both sides. Each result names the Claude Code version and the model id.
|
||||
- **Check.** The project's test command over Haiku's changes to files other than tests, with the tests restored from the commit, so deleting a test doesn't pass.
|
||||
- **Measure.** From the stream: passed or not, tokens split into input, cache write, cache read and output, turns, wall time, and the reads before the first open of the file the bug is in.
|
||||
@@ -211,6 +188,16 @@ Touches:
|
||||
- `docs/decisions.md`: `## task-bench: how many tasks and runs a bench takes`
|
||||
- `bin/task_bench/main.py`: `main` (small)
|
||||
|
||||
### 511. Hand the product-owner and prose reviewers the real output of each command a changed doc line says prints something, so a claim about output is checked by running it, not by reading code.
|
||||
|
||||
An outside reviewer found a README line, in a persona's description, that said a CLI error names the fix. Our product-owner pass checked the exit code claim against the source and corrected it, but passed the "names the fix" half, since the pass has Read, Grep and Glob and cannot run the command: the one-character input that triggers the error prints an error with no fix in it. A deterministic check would be better than a reviewer's reading, but no tool can tell which doc lines claim an output. The orchestrator can: where a doc line a chunk changes says what a command, flag or error prints, larv-review's text has it run that command in the chunk's worktree, through the project's container setup, and give each reviewer the real output beside the claim. Applies to every project whose docs state outputs; a reviewer still gets no shell.
|
||||
|
||||
Touches:
|
||||
|
||||
- `skills/larv-review/SKILL.md`
|
||||
- `agents/product-owner-reviewer.md`
|
||||
- `agents/prose-reviewer.md`
|
||||
|
||||
### 509. Run `task-bench` beside the scoring run in larv-review comprehension check, and record its figures in the comprehension history beside the panel scores.
|
||||
|
||||
It stays a report beside the panel until about 10 chunks show whether the two agree; whether it can replace the scoring run or gate a merge is the human's call then.
|
||||
|
||||
Reference in New Issue
Block a user