Run the tests once, on the latest stable Python, across the CPUs

This commit is contained in:
2026-10-28 12:17:05 +01:00
parent d7806d0b16
commit 53e663c8f8
18 changed files with 33 additions and 71 deletions
+1 -1
View File
@@ -3,8 +3,8 @@
Decisions live in `docs/decisions.md`:
- bin/: an outcome is an entry of a `.outcomes.md` file in loop-claude's archive, a record is a document tools or people keep entries in, and a place is what a comprehension seat names
- bin/: the tests rerun themselves on a `uv`-fetched Python at README Install's floor
- bin/: the tools and their tests run on the host's Python, with no container
- bin/: the tools support only the latest stable Python, and the tests run once on it, spread across the CPUs by pytest-xdist
- chunks: a chunk takes every unblocked item of its release that fires the same review passes, whatever its purpose, and an item above R 5 ships alone
- comprehension floor: every dimension and the overall at 7.0 or above is the target, and blocks no merge or release
- larv-review: within a chunk lifting a project towards the comprehension floor, work on the project's lowest-scoring dimension comes first, and the floor doesn't block anything
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- The tools and their tests now support only the latest stable Python, today 3.14, which README Install now asks for as `python3`, in place of 3.11 or newer. README's Test line spreads the tests across the CPUs with pytest-xdist through `uvx`, so a full run takes about 23 s on 16 CPUs, where it took about 3.3 minutes. It used to rerun the whole suite on 3.11.
- `routing-figures.md` now reports dollars by project, subagent type, rung and tool, in place of tokens. Cache reads are included; the tokens left them out, though they were about a quarter of the cost. The dollars are what each worker transcript's last `cost-state` entry records for each model, split across the messages of the worker and its subagents in proportion to their cost at list prices. The judge's runs are not counted, since the archive doesn't hold them.
- larv-review's principles pass now launches one auditor per round, which checks the README's goals, code health and every principles file that applies, so it reads the project once per round. It used to launch one auditor for the goals with code health and one for each principles file, each reading the whole project. Its report is `principles-round-<n>.md`, like every other pass's.
- A session running larv-review now enters the chunk's worktree with `EnterWorktree` before the first review pass, so `/security-review` and the full scan read the branch's diff. Run from the checkout root, they used to find it empty.
+2 -2
View File
@@ -47,7 +47,7 @@ git remote add origin ssh://git@gitea.larvit.se:21022/lilleman/claude-config.git
Then run Update's commands, skipping `git add settings.json`: the new repository tracks no file yet.
Have Python 3.11 or newer as `python3` on `PATH`, and for Rust projects `rust-analyzer` on `PATH`, which the `rust-analyzer-lsp` plugin needs. Install every plugin `settings.json` enables:
Have the latest stable Python, today 3.14, as `python3` on `PATH`, and for Rust projects `rust-analyzer` on `PATH`, which the `rust-analyzer-lsp` plugin needs. Install every plugin `settings.json` enables:
```sh
claude plugin marketplace add anthropics/claude-plugins-official
@@ -110,7 +110,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
- `subagent-rung [--plan] <model> <L>` prints the Agent tool's model and effort for a subagent's task, picked by its L from the project's model, and with `--plan` never below the project's rung; `subagent-rung --help` states the rule.
Test on the host, with `uv` on `PATH` to fetch mypy and the Python version Install names (`docs/decisions.md` → `bin/`): `python3 -m unittest discover -s bin -p 'test_*.py'`. Test the bench's git wrapper with `python3 -m unittest discover -s workflow-bench -p 'test_*.py'`.
Test on the host, with `uv` on `PATH` to fetch mypy and pytest-xdist, which spreads the tests across the CPUs (`docs/decisions.md` → `bin/`): `uvx --python "$(command -v python3)" --with pytest-xdist pytest -n auto bin`. `python3 -m unittest discover -s bin -p 'test_*.py'` runs the same tests one at a time. Test the bench's git wrapper with `python3 -m unittest discover -s workflow-bench -p 'test_*.py'`.
To mutation-test a changed module, copy `bin/` to a scratch directory (`cp -r bin "$scratch"`): mutmut writes files beside its config. In the copy, name the module and its tests in a `setup.cfg`, as below for `score_outcomes/results.py`. Leave out each test class that starts a tool's launcher through `subprocess`: a mutated module fails to load in a process mutmut did not start.
+1 -1
View File
@@ -247,7 +247,7 @@ class Checkout:
os.write(controller, typed.encode())
output, deadline = b"", time.monotonic() + 60
while time.monotonic() < deadline:
if select.select([controller], [], [], 1)[0]:
if select.select([controller], [], [], 0.05)[0]:
try:
chunk = os.read(controller, 4096)
except OSError:
+3 -10
View File
@@ -10,7 +10,6 @@ import stat
import subprocess
import sys
import tempfile
import time
import unittest
import unittest.mock
import uuid
@@ -113,12 +112,6 @@ class FakeEffects:
return [what for called, what in self.calls if called == seam]
def uuid7():
# Python 3.11, the floor, has no uuid.uuid7.
random = int.from_bytes(os.urandom(10))
return uuid.UUID(int=int(time.time() * 1000) << 80 | 7 << 76 | (random >> 62 & 0xFFF) << 64 | 2 << 62 | random & (1 << 62) - 1)
def ran(stdout="", code=0, stderr=""):
return subprocess.CompletedProcess([], code, stdout, stderr)
@@ -143,7 +136,7 @@ class Worker:
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "a") as handle:
if flag == "--session-id":
handle.write(json.dumps({"type": "custom-title"}) + "\n" + json.dumps({"type": "queue-operation", "timestamp": str(uuid7())}) + "\n")
handle.write(json.dumps({"type": "custom-title"}) + "\n" + json.dumps({"type": "queue-operation", "timestamp": str(uuid.uuid7())}) + "\n")
if self.prompted:
handle.write(json.dumps({**human(args[args.index("-p") + 1]), "promptSource": "sdk"}) + "\n")
handle.write(json.dumps(assistant({"type": "text", "text": self.text})) + "\n")
@@ -640,7 +633,7 @@ class Interrupts(unittest.TestCase):
path = pathlib.Path(home, "t.jsonl")
path.write_text(json.dumps(human("a")) + "\n")
fake = FakeEffects(on_terminal_claude=[0], git=[""])
stop = interrupt.talk(home, uuid7(), Rung("m", "high"), str(path), paths.paths_from(pathlib.Path(home)), {}, fake.effects())
stop = interrupt.talk(home, uuid.uuid7(), Rung("m", "high"), str(path), paths.paths_from(pathlib.Path(home)), {}, fake.effects())
self.assertEqual(fake.made("claude_on_terminal")[0][:4], ["--model", "m", "--effort", "high"])
self.assertEqual(stop, Stop(2, "the human is needed to answer the question above, but the interrupt ended with nothing typed"))
@@ -650,7 +643,7 @@ class Interrupts(unittest.TestCase):
path.write_text(json.dumps(human("a")) + "\n")
fake = FakeEffects(on_terminal_claude=[0], git=[""])
handed = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort"}
interrupt.talk(home, uuid7(), Rung("claude-haiku-5-5", "max"), str(path), paths.paths_from(pathlib.Path(home)), handed, fake.effects())
interrupt.talk(home, uuid.uuid7(), Rung("claude-haiku-5-5", "max"), str(path), paths.paths_from(pathlib.Path(home)), handed, fake.effects())
self.assertEqual(fake.terminal_environments, [handed])
+4
View File
@@ -0,0 +1,4 @@
import pathlib
# The checkout root, which the repo checks read.
ROOT = pathlib.Path(__file__).resolve().parents[2]
+1 -1
View File
@@ -7,7 +7,7 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from loop_claude.project.records import HUMAN_RULE_DIRS
from repo_checks.test_python_floor import ROOT
from repo_checks import ROOT
RULE_PATH = rf"(?:{'|'.join(map(re.escape, HUMAN_RULE_DIRS))})/[\w-]+\.md"
CITATION = re.compile(rf'`?(?:~/\.claude/)?(CLAUDE\.md|{RULE_PATH})`? → ("[^"]+"(?:,? (?:and )?"[^"]+")*)')
+1 -1
View File
@@ -5,7 +5,7 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from repo_checks.test_python_floor import ROOT
from repo_checks import ROOT
PRIVATE = [
".claude/settings.local.json",
+1 -1
View File
@@ -6,7 +6,7 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from repo_checks.test_python_floor import ROOT
from repo_checks import ROOT
def bound(node: ast.Import | ast.ImportFrom) -> list[str]:
+1 -1
View File
@@ -5,7 +5,7 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from repo_checks.test_python_floor import ROOT
from repo_checks import ROOT
class Install(unittest.TestCase):
-27
View File
@@ -1,27 +0,0 @@
import pathlib
import re
import shutil
import subprocess
import sys
import unittest
ROOT = pathlib.Path(__file__).resolve().parents[2]
FLOOR = re.compile(r"Python 3\.(\d+) or newer")
def readme_floor() -> tuple[int, int] | None:
named = FLOOR.search((ROOT / "README.md").read_text(encoding="utf-8"))
return (3, int(named.group(1))) if named else None
class PythonFloor(unittest.TestCase):
def test_the_suite_passes_on_the_python_the_readme_names_as_floor(self):
floor = readme_floor()
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
if sys.version_info[:2] == floor:
self.skipTest("this run is on the floor")
if shutil.which("uv") is None:
self.fail("uv on PATH fetches the floor interpreter; see README.md's Test line")
suite = ["python", "-m", "unittest", "discover", "-s", "bin", "-p", "test_*.py"]
ran = subprocess.run(["uv", "run", "--no-project", "--python", "%d.%d" % floor, *suite], capture_output=True, cwd=ROOT, text=True)
self.assertEqual(ran.returncode, 0, ran.stderr)
+1 -1
View File
@@ -6,7 +6,7 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from review_passes import PASS_AGENTS, PASSES
from repo_checks.test_python_floor import ROOT
from repo_checks import ROOT
from rule_hook.reviewer_rung import SEAT, SECURITY_OPENER
OPENING = re.compile(r"^## Opening\n+```\n([^\n]+?\.)(?: |\n)", re.MULTILINE)
+1 -1
View File
@@ -7,7 +7,7 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from settings_filter import cleaned
from git_repo import git
from repo_checks.test_python_floor import ROOT
from repo_checks import ROOT
INSTALL = "git config filter.settings.clean bin/settings_filter.py"
+3 -5
View File
@@ -6,14 +6,12 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from repo_checks.test_python_floor import readme_floor, ROOT
from repo_checks import ROOT
class Types(unittest.TestCase):
def test_mypy_passes_on_the_tools_at_the_floor(self):
floor = readme_floor()
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
def test_mypy_passes_on_the_tools_for_the_python_running_the_tests(self):
if shutil.which("uv") is None:
self.fail("uv on PATH fetches mypy; see README.md's Test line")
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "chunk_preflight", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-p", "reading_cost", "-m", "review_passes", "-p", "review_replay", "-m", "round_titles", "-p", "rule_hook", "-p", "rung_replay", "-p", "score_outcomes", "-m", "settings_filter", "-m", "shell_command", "-m", "statusline", "-p", "subagent_rung", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
ran = subprocess.run(["uv", "tool", "run", "--python", sys.executable, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "chunk_preflight", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-p", "reading_cost", "-m", "review_passes", "-p", "review_replay", "-m", "round_titles", "-p", "rule_hook", "-p", "rung_replay", "-p", "score_outcomes", "-m", "settings_filter", "-m", "shell_command", "-m", "statusline", "-p", "subagent_rung", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
+1 -1
View File
@@ -10,7 +10,7 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from failure import Failure
import rank_todo
from repo_checks.test_python_floor import ROOT
from repo_checks import ROOT
import todo_table
# Voids code health, so these tests weigh only the project's own goals; class CodeHealth weighs it.
+4 -4
View File
@@ -10,14 +10,14 @@
Valid while the archive's file names, the Vocabulary and the panel keep these words.
## bin/: the tests rerun themselves on a `uv`-fetched Python at README Install's floor
2026-09-26, larv-review. Serves goal 2. The suite reruns itself on a `uv`-fetched interpreter of the version Install names: `ast.parse(feature_version=)` on the host's Python accepts the nested f-string 186181a fixed, a compile check passes a call newer than the floor, like 3.12's `itertools.batched`, and a `python3.N` on `PATH` is installed by hand on every machine and again at each bump. Valid while `uv` can fetch the floor interpreter and no machine taking the checkout ships an older Python.
## bin/: the tools and their tests run on the host's Python, with no container
2026-09-24, larv-review. Serves goals 1 and 2. The tools drive the host's claude CLI and read the host's session files, which a container cannot see; the tests run on the same host Python the tools meet. Valid while claude runs on the host.
## bin/: the tools support only the latest stable Python, and the tests run once on it, spread across the CPUs by pytest-xdist
2026-10-28, lilleman, through todo item 245. Serves goal 1.5: the suite gates every chunk. It took 198 s, 102 s of which reran it on the oldest Python README Install named, 3.11. README Install now asks for the latest stable Python as `python3`, and mypy checks the tools for the Python running the tests. README's Test line runs pytest-xdist through `uvx`, which spreads the tests across the CPUs: 23 s on a 16-CPU machine. It passes the path of `python3`, since `uvx --python python3` picks a Python uv installed itself, which may be older. Plain `python3 -m unittest` runs the same tests one at a time. Valid while every machine taking the checkout gets the latest stable Python soon after its release.
## chunks: a chunk takes every unblocked item of its release that fires the same review passes, whatever its purpose, and an item above R 5 ships alone
2026-10-22, lilleman, through todo item 217. Serves goal 1.4: each review has a fixed cost, so more items per chunk cost less per item. Across the 165 finished chunks loop-claude archived up to that day, a chunk took a median of 1.30M tokens with one item and 1.33M with two, and three in four chunks shipped one item. The old rule also asked one PR title to state what the items achieve together; lilleman dropped it, since unrelated items that fire the same passes cost one review.
+4
View File
@@ -883,3 +883,7 @@ Read most (opens and searches):
## Reading cost, chunk 22d616a7a912, finished 2026-10-28T10:14:46Z
116420 new input tokens before the first commit.
## Reading cost, chunk b25d0b8f6319, finished 2026-10-28T11:11:03Z
221959 new input tokens before the first commit.
+3 -14
View File
@@ -12,10 +12,9 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 245 | 0.4.0 | | owner | **Run the test suite once, on the latest stable Python alone, with its test files spread across the machine's CPUs, so a full run takes under a minute instead of about 3.3.** | 3 | 1 | 2 | 4 | 4 | 1.5 by 5 | 3.2 |
| 239 | 0.4.0 | 245 | owner | **Block a Read, `cat`, `sed` or `head` of a file range already in the session's context, with a note saying so, and let it through once a compaction dropped the range or the file changed.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 2.0 |
| 244 | 0.4.0 | 245 | owner | **Open every line loop-claude prints to its terminal with the machine's local time, as in `2026-10-28 11:46:01`.** | 2 | 2 | 1 | 3 | 3 | 1.5 by 2 | 1.9 |
| 243 | 0.4.0 | 245 | owner | **Show every loop-claude on the machine that waits for the human in one terminal app, and let the human answer each one there, `/talk` included.** | 4 | 2 | 3 | 7 | 6 | 1.5 by 4 | 0.9 |
| 239 | 0.4.0 | | owner | **Block a Read, `cat`, `sed` or `head` of a file range already in the session's context, with a note saying so, and let it through once a compaction dropped the range or the file changed.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 2.0 |
| 244 | 0.4.0 | | owner | **Open every line loop-claude prints to its terminal with the machine's local time, as in `2026-10-28 11:46:01`.** | 2 | 2 | 1 | 3 | 3 | 1.5 by 2 | 1.9 |
| 243 | 0.4.0 | | owner | **Show every loop-claude on the machine that waits for the human in one terminal app, and let the human answer each one there, `/talk` included.** | 4 | 2 | 3 | 7 | 6 | 1.5 by 4 | 0.9 |
| 226 | 0.5.0 | | question | **Answer whether rule-hook's `run-heavy` rule should also catch a mutation run or docker build behind `uv run`, `poetry run`, `python3 -m`, `sudo`, `docker image build` and `docker compose up --build`, or narrow the `machine load:` decision's wording.** | 2 | 2 | 3 | 5 | 3 | 1.2 by 2, 1.5 by 2 | 4.0 |
| 235 | 0.5.0 | | decision | **Compare the first 20 chunks whose worker ran under the 200k compaction window with the 20 before them, on review rounds past each pass's first, unfinished verdicts, 🔴, 🟡 and misread findings, dollars and hours per chunk, and file an item where any rose.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 2, 1.4 by 3 | 6.8 |
| 167 | 0.5.0 | | | **Refuse a `/security-review` or `/claude-security` Skill call below the project's rung, as rule-hook refuses such an Agent call.** | 3 | 2 | 3 | 6 | 2 | 1.1 by 4 | 5.6 |
@@ -79,16 +78,6 @@
## Details
### 245. Run the test suite once, on the latest stable Python alone, with its test files spread across the machine's CPUs, so a full run takes under a minute instead of about 3.3.
lilleman's call, 2026-10-28: support only the latest stable Python, today 3.14, and drop every older one. It blocks the other 0.4.0 items so their chunks get the faster runs. Measured on 2026-10-28, on a 16-CPU machine: 765 tests in 198 s at 68% of one CPU. 102 s of that is `repo_checks/test_python_floor.py` rerunning the whole suite on 3.11. About 50 s is `loop_claude/test_black_box.py`, whose tests start loop-claude, a fake claude and git repos, 1 to 8 s each. About 30 s is the other tests that start the tools. mypy takes 3 s with its cache off.
- Drop the floor rerun, `test_python_floor.py` and `readme_floor`, and make README Install ask for the latest stable Python as `python3`. Run mypy for that version.
- Replace `docs/decisions.md` → `bin/: the tests rerun themselves on a uv-fetched Python at README Install's floor` with the new rule, and update its title in `AGENTS.md`. Check that `bin/: the tools and their tests run on the host's Python` still holds.
- Spread the test files across the CPUs, with a small runner or pytest-xdist through `uvx`, and give README's Test line the one command that does it. Keep plain `python3 -m unittest` working.
- In `run_on_terminal` in `test_black_box.py`, check for the prompt every 50 ms instead of every second.
- Then, if the run is still over a minute, have the black-box tests copy one prepared git repo per class instead of building one per test.
### 239. Block a Read, `cat`, `sed` or `head` of a file range already in the session's context, with a note saying so, and let it through once a compaction dropped the range or the file changed.
lilleman's call, 2026-10-27, filed from PR #376's measurement of loop-claude's archive since 2026-10-21: a read of a file the agent read earlier in the thread, more than three messages before and without editing it in between, is 13% of the worker's reading, about $6 of the $45 its reads cost, and 1 to 2% of a reviewer's. How many of those repeat a range exactly is unmeasured, so the figures are a ceiling. A hook on `Read` and on the Bash reads `cat`, `sed` and `head` blocks such a read with a short note that the content is already in context and unchanged. It lets the read through after a compaction, when the content has left the context, or when the file changed since the read. A PreToolUse hook gets a `transcript_path`, which holds the earlier reads and each `compact_boundary`, so it may need no state of its own; check whether a hook inside a subagent gets the subagent's own transcript.