Merge pull request 'Hold each heavy command for one of a heavy slot per 4 CPUs, and each worker until a quarter of memory is available' (#366) from item-222 into main
This commit was merged in pull request #366.
This commit is contained in:
@@ -39,6 +39,8 @@
|
||||
!bin/loop_claude/
|
||||
!bin/loop_claude/**/
|
||||
!bin/loop_claude/**/*.py
|
||||
!bin/machine_load/
|
||||
!bin/machine_load/*.py
|
||||
!bin/markdown_text.py
|
||||
!bin/metric-bench
|
||||
!bin/metric_bench/
|
||||
@@ -65,6 +67,9 @@
|
||||
!bin/rule_hook/*.py
|
||||
!bin/rung_replay/
|
||||
!bin/rung_replay/*.py
|
||||
!bin/run-heavy
|
||||
!bin/run_heavy/
|
||||
!bin/run_heavy/*.py
|
||||
!bin/score-outcomes
|
||||
!bin/score_outcomes/
|
||||
!bin/score_outcomes/*.py
|
||||
|
||||
@@ -15,6 +15,7 @@ Decisions live in `docs/decisions.md`:
|
||||
- loop-claude: every claude it starts bypasses permissions inside its sandbox
|
||||
- loop-claude: one session per project, its id derived from the checkout's path relative to `~/code`, or to `/` outside it
|
||||
- loop-claude: the archive keeps the raw transcript, and a failed archive stops the loop
|
||||
- machine load: a heavy command waits for one of the machine's heavy slots, one for every 4 CPUs, which every session shares, and loop-claude starts a worker only while at least a quarter of the machine's memory is available
|
||||
- principles: about seven things per level, four or five where they act on each other, more where a reader finds them by name; crossing a limit asks for a grouping or a recorded reason
|
||||
- reading cost and `score-outcomes`: each waits for 10 finished chunks before it shows a median
|
||||
- reading cost: a rise is the last 5 chunks' median above the 5 before them, and files or refreshes one plan item
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
## Unreleased
|
||||
|
||||
- Reviewers now mark a 🔴 or 🟡 `[misread]` where it arose because the implementer misread code that had already shipped. A round's commit title now counts the marked findings, as in `Answer stability round 1: 1 🔴, 2 🟡 (1 misread); …`, and rule-hook's `round-title-counts` blocks a round title without that count. `score-outcomes` shows the count on each finished item, and lists it after the medians for each chunk whose round titles carry it, newest first. A chunk whose round titles predate this change reads as unknown.
|
||||
- Sessions running at once no longer starve the machine. A new tool, `run-heavy <command>`, runs a heavy command, such as a full gate, mutation tests, an e2e run or a `docker build`, once it holds one of the machine's heavy slots: one per 4 CPUs, shared by every session. `CLAUDE.md` has every session run its heavy commands under it, and rule-hook blocks a `docker build` or a mutation run outside it. loop-claude now waits to start a worker while less than a quarter of the machine's memory is available.
|
||||
- larv-review's skill now says a commit answering the external auto-reviewer takes no `Answer <pass> round` title, and that `larv-review round-count` doesn't list the auto-reviewer. A worker who followed the skill used to get a "not a pass" warning.
|
||||
- rule-hook now blocks a larv-review round commit titled past round 3, such as `Answer stability round 4: …`, unless a `round-cap` override quotes the human's yes.
|
||||
- loop-claude's first prompt for a chunk that no plan item routed now also names the rung on which the worker launches its reviews for an item it takes in, as the prompt after a step-up or an unfinished verdict already did.
|
||||
|
||||
@@ -65,6 +65,7 @@ The rules only reviewers apply, for quality, comments, READMEs and prose, live i
|
||||
- Keep the project's main checkout (e.g. `~/code/log`, never a worktree) current: when it is behind `origin` — a merge landed, or a fetch shows it — run `git pull --ff-only` there instead of reporting the lag, never with `--autostash`: a dirty checkout is a finding to report, not to carry along.
|
||||
- Run a project's language runtime, package manager, and task runner through its container/tooling setup, never directly on the host unless a decision puts them there. Example: `docker compose run --rm app <tool> …`.
|
||||
- Tear down test containers after a run.
|
||||
- A heavy command loads the machine for minutes: a full gate, mutation tests, an e2e run, a `docker build`. Run each one under `~/.claude/bin/run-heavy`, as in `~/.claude/bin/run-heavy ./ci.sh`. run-heavy waits for one of the heavy slots that every session on the machine shares. Run it in the background, since the wait can outlast the Bash tool's timeout.
|
||||
- Delegate a task to a subagent only when you need just its conclusion: a search or read across many files, or work independent enough to run in parallel. Keep edits, and whatever you must check line by line, in your own thread: you would read a delegated diff again to check it. Score the task's L as for a `todo.md` item; a task that changes no file, such as a search, is L 1. `<model>` below is the project's model id, such as `claude-opus-5-5`: the one loop-claude's prompt names as the project's, or else your own. Launch the subagent with the model and effort of the first case that fits:
|
||||
1. A reviewer, a comprehension seat or the security pass: those loop-claude's prompt gives it. Given none, pass no model or effort, so it runs on your model.
|
||||
2. An escalated task, below: the rung loop-claude's prompt names for escalation, or case 3's rung where the subagent files or scores `todo.md` items and that rung is higher.
|
||||
|
||||
@@ -104,6 +104,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure.
|
||||
- `review-replay <pass> <model>` replays a review pass's archived rounds from the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again; `review-replay --help` states the replay. Each replayed reviewer skips every permission prompt, stability-reviewer has Bash, and an archived prompt may name paths on the host.
|
||||
- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit, Write and Agent call, and `rule-hook --help` states the rules and the override.
|
||||
- `run-heavy <command>` runs a heavy command once it holds one of the machine's heavy slots, one per 4 CPUs, which every session shares; `run-heavy --help` states the slots.
|
||||
- `score-outcomes [L E I S]` lists the finished items whose guessed scores sit nearest the given ones, and the shipped items the project's `docs/score-anchors.md` names for each level of L, E and I, each with what its chunk measured; then the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings; then each chunk's 🔴 and 🟡 findings with how many a reviewer marked `[misread]`, code health's counter-signal; `score-outcomes --help` states the measures.
|
||||
- `settings_filter.py` is git's clean filter for `settings.json`: it drops `"model"` and sorts the keys. `.gitattributes` sets the filter `settings` on `settings.json`, and Install's `git config` line makes this script that filter.
|
||||
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
|
||||
@@ -124,7 +125,7 @@ pytest_add_cli_args=-k
|
||||
|
||||
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
|
||||
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py`, `settings_filter.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `larv_review` imports `rank_todo.py`'s ranking, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and `loop_claude` runs `chunk-preflight` as a process of its own. `judge_replay` and `review_replay` share `rung_replay/`: the rung search, the argument reader, the candidate rungs and the exits. `rung_replay/` reads `loop_claude`'s retry count and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude, `rule_hook`, both replay tools and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py`, `settings_filter.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `larv_review` imports `rank_todo.py`'s ranking, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and `loop_claude` runs `chunk-preflight` as a process of its own. `judge_replay` and `review_replay` share `rung_replay/`: the rung search, the argument reader, the candidate rungs and the exits. `rung_replay/` reads `loop_claude`'s retry count and models file. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude, `rule_hook`, both replay tools and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `machine_load/` takes the machine's heavy slots, its shared lock files, for `run_heavy`, and reads the memory share that loop-claude waits for. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
|
||||
## Tracking a new file
|
||||
|
||||
|
||||
@@ -12,8 +12,8 @@ You are an elite stability-focused code reviewer with deep expertise in defensiv
|
||||
1. **Scope the change.** You review every PR; a fresh invocation has no prior context, so determine the diff yourself. Diff the PR branch against its base (`gh pr diff`, or `git merge-base <base> HEAD` then `git diff`), and review the changed lines **plus enough surrounding code and callers** to judge correctness — read the whole changed file when practical; a diff alone hides whether a caller breaks. Read the PR description and, when the project uses an issue tracker (project-specific — check its docs) and the change references a ticket, that ticket: the diff is judged against its stated intent, and a scope mismatch in either direction is a finding — claimed-but-unimplemented, and implemented-but-unasked (#2).
|
||||
2. **Read every file under `~/.claude/principles/`**, the shared principles, and `~/.claude/skills/larv-review/standards/quality.md`. These are mandatory org guidelines; all code MUST comply with each that applies. A file's `Applies to` line says what it covers; one without the line covers every change. Re-read each time — they change. Flag violations prominently.
|
||||
3. **Read README.md, AGENTS.md and `docs/decisions.md`** in the current project for project-specific conventions, build instructions, tooling, and the accepted trade-offs a reviewer should honor rather than re-flag. Note the documented test, lint, and static-analysis commands — you will run them.
|
||||
4. **Verify, don't just read.** Run the test suite and static analysis/linters through the project's container tooling, unless a decision puts them on the host; tear down test containers afterwards. A full-suite run catches broken callers that diff-reading misses. When a suspected bug is cheap to confirm, reproduce it — a scratch script, a targeted test run — before reporting it. Make scratch work with Bash, since Write is for your report alone. Put it in a directory you create for this run, and delete it when done; delete nothing else, since a directory the prompt names may hold other agents' files. Leave the working tree exactly as you found it. Anything you could not run goes in Coverage & Verification, never silently skipped.
|
||||
5. **Mutation-test the changed lines.** Run the project's documented mutation command, or else a widely deployed mutation tool for the language: cargo-mutants for Rust, Infection for PHP, mutmut for Python, Stryker for JavaScript and TypeScript. Limit it to the lines the change adds or modifies, through the tool's own option where it has one (cargo-mutants `--in-diff <diff file>`, Infection `--git-diff-lines`, Stryker `--mutate <file>:<lines>`); otherwise mutate only the changed files and keep only the mutants on changed lines. Run it through the project's tooling, as in step 4, on a scratch copy: the tools write files. Report its wall time and mutant count in Coverage & Verification, or why nothing was mutated. Ignore a surviving mutant that leaves behaviour unchanged, or changes only message or log text no caller or user depends on. Report one 🟡 per test to add or strengthen, naming the surviving mutants it would catch.
|
||||
4. **Verify, don't just read.** Run the test suite and static analysis/linters through the project's container tooling, unless a decision puts them on the host; tear down test containers afterwards. Run the full suite under `~/.claude/bin/run-heavy`. Run it in the background or with the Bash tool's longest timeout: that timeout includes the wait for a slot. A full-suite run catches broken callers that diff-reading misses. When a suspected bug is cheap to confirm, reproduce it — a scratch script, a targeted test run — before reporting it. Make scratch work with Bash, since Write is for your report alone. Put it in a directory you create for this run, and delete it when done; delete nothing else, since a directory the prompt names may hold other agents' files. Leave the working tree exactly as you found it. Anything you could not run goes in Coverage & Verification, never silently skipped.
|
||||
5. **Mutation-test the changed lines.** Run the project's documented mutation command, or else a widely deployed mutation tool for the language: cargo-mutants for Rust, Infection for PHP, mutmut for Python, Stryker for JavaScript and TypeScript. Limit it to the lines the change adds or modifies, through the tool's own option where it has one (cargo-mutants `--in-diff <diff file>`, Infection `--git-diff-lines`, Stryker `--mutate <file>:<lines>`); otherwise mutate only the changed files and keep only the mutants on changed lines. Run it through the project's tooling and run-heavy, as in step 4, on a scratch copy: the tools write files. Report its wall time and mutant count in Coverage & Verification, or why nothing was mutated. Ignore a surviving mutant that leaves behaviour unchanged, or changes only message or log text no caller or user depends on. Report one 🟡 per test to add or strengthen, naming the surviving mutants it would catch.
|
||||
6. **Two passes minimum, then hunt absences.**
|
||||
- *Breadth:* every hunk in every changed file, no skimming. Track which files you have covered; batch large diffs, never truncate the review.
|
||||
- *Depth:* trace the riskiest data paths end-to-end (input → logic → persistence/response), the interactions between changed pieces, and between the change and config (timeout hierarchies, retries × idempotency, queue worker counts, env parity).
|
||||
|
||||
@@ -114,7 +114,7 @@ class Main(unittest.TestCase):
|
||||
result = reply(Rung(args[args.index("--model") + 1], args[args.index("--effort") + 1]), "said Verdict: interrupt" in prompt.rsplit("latest last:", 1)[-1])
|
||||
return subprocess.CompletedProcess(args, 0, json.dumps(result), "")
|
||||
|
||||
effects = Effects(claude=claude, claude_on_terminal=unused, discard_typed=unused, execv=unused, git=unused, input=unused, local_time=unused, now=unused, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused)
|
||||
effects = Effects(claude=claude, claude_on_terminal=unused, discard_typed=unused, execv=unused, git=unused, input=unused, local_time=unused, memory=unused, now=unused, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused)
|
||||
out, err = io.StringIO(), io.StringIO()
|
||||
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
code = main.main(argv, {}, self.home, effects)
|
||||
|
||||
@@ -1,15 +1,21 @@
|
||||
import sys
|
||||
|
||||
from claude_code.entries import assistant_parts, ends_stream, hit_limit, limit_said, parse_entry, rejected_until
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from claude_code.reply import Limited
|
||||
from claude_code.transcript import part_line
|
||||
from machine_load.memory import ROOM_SHARE
|
||||
from model_routing.ladder import Rung
|
||||
from claude_code.session import Session
|
||||
|
||||
PRINTED_TOOL_INPUT = 160
|
||||
# Seconds between reads of the machine's memory while too little of it is available to start a worker.
|
||||
MEMORY_POLL = 60
|
||||
|
||||
|
||||
def run_worker(root: str, args: list[str], env: dict[str, str], effects: Effects) -> Failure | Limited | None:
|
||||
wait_for_memory(effects)
|
||||
try:
|
||||
proc = effects.stream_claude(args, root, env)
|
||||
except OSError as failure:
|
||||
@@ -28,6 +34,16 @@ def run_worker(root: str, args: list[str], env: dict[str, str], effects: Effects
|
||||
return worker_outcome(proc.wait(), result, resets)
|
||||
|
||||
|
||||
def wait_for_memory(effects: Effects) -> None:
|
||||
"""Returns once the machine has room for a worker, or its memory cannot be read, so loops running at once do not push it into swap. While it waits, it says so once on stderr and reads the memory again each MEMORY_POLL seconds."""
|
||||
said = False
|
||||
while (memory := effects.memory()) is not None and not memory.has_room():
|
||||
if not said:
|
||||
print(f"waiting to start the worker until at least {ROOM_SHARE:.0%} of the machine's memory is available; {memory.available // 1024} MiB of {memory.total // 1024} MiB is available now", file=sys.stderr, flush=True)
|
||||
said = True
|
||||
effects.sleep(MEMORY_POLL)
|
||||
|
||||
|
||||
def worker_outcome(code: int, result: dict | None, resets: list[int]) -> Failure | Limited | None:
|
||||
if result is not None and hit_limit(result):
|
||||
return Limited(max(resets, default=None), limit_said(result))
|
||||
|
||||
@@ -8,6 +8,7 @@ import typing
|
||||
|
||||
from failure import Failure
|
||||
import git_repo
|
||||
from machine_load.memory import Memory, read_memory
|
||||
|
||||
# Seconds before a Python script loop-claude runs is stopped, so a hung one, such as chunk-preflight, cannot hold the loop.
|
||||
PYTHON_TIMEOUT = 600
|
||||
@@ -37,6 +38,8 @@ class Effects:
|
||||
git: Git
|
||||
input: typing.Callable[[str], str]
|
||||
local_time: typing.Callable[[float], str]
|
||||
# The machine's memory; None where it cannot be read.
|
||||
memory: typing.Callable[[], Memory | None]
|
||||
now: typing.Callable[[], float]
|
||||
on_terminal: typing.Callable[[], bool]
|
||||
# Runs a Python script in a directory, as run_python does.
|
||||
@@ -66,6 +69,7 @@ def system_effects() -> Effects:
|
||||
git=git_repo.git,
|
||||
input=input,
|
||||
local_time=lambda seconds: time.strftime("%Y-%m-%d %H:%M %Z", time.localtime(seconds)),
|
||||
memory=read_memory,
|
||||
now=time.time,
|
||||
on_terminal=lambda: sys.stdin.isatty() and sys.stdout.isatty(),
|
||||
python=lambda args, cwd: run_python(args, cwd, PYTHON_TIMEOUT),
|
||||
|
||||
@@ -44,6 +44,9 @@ copied into a folder per chunk under $LOOP_CLAUDE_TRANSCRIPTS/<key> (default roo
|
||||
then each verdict, quoted answer, rejected judge reply, step-up and stop, each opened by its UTC
|
||||
time. The layout, and when a copy is replaced, are in bin/transcript_archive/__init__.py.
|
||||
|
||||
Before each worker starts, loop-claude waits until at least a quarter of the machine's memory is available
|
||||
(bin/machine_load/memory.py).
|
||||
|
||||
Every claude loop-claude starts runs the project's model, saved per <key> in
|
||||
~/.claude/loop-claude-models.json and read each cycle, unless routing, below, puts the worker,
|
||||
the session /talk opens and the judge on another rung, a model at an effort. A project without an
|
||||
|
||||
@@ -116,6 +116,11 @@ if os.path.basename(sys.argv[0]) == "loop-claude":
|
||||
record({"argv": sys.argv[1:]})
|
||||
pause = time.sleep
|
||||
time.sleep = lambda seconds: record({"sleep": seconds}) if seconds >= 1 else pause(seconds)
|
||||
# The host's own memory doesn't matter: every read finds all of it available.
|
||||
sys.path.insert(0, os.path.dirname(os.path.realpath(sys.argv[0])))
|
||||
import pathlib
|
||||
from machine_load import memory
|
||||
memory.MEMINFO = pathlib.Path(os.environ["FAKE_MEMINFO"])
|
||||
"""
|
||||
|
||||
PREMISES = "# Project\n\n## Goals\n\n1. Work.\n\n## Audience\n\nMe.\n"
|
||||
@@ -181,8 +186,10 @@ class Checkout:
|
||||
site = self.temp / "site"
|
||||
site.mkdir()
|
||||
(site / "sitecustomize.py").write_text(SITECUSTOMIZE)
|
||||
(self.temp / "meminfo").write_text("MemTotal: 16 kB\nMemAvailable: 16 kB\n")
|
||||
self.env.update({
|
||||
"FAKE_CLAUDE_LOG": str(self.temp / "calls.jsonl"),
|
||||
"FAKE_MEMINFO": str(self.temp / "meminfo"),
|
||||
"FAKE_CLAUDE_SCENARIO": str(self.temp / "scenario.json"),
|
||||
"GIT_CONFIG_GLOBAL": os.devnull,
|
||||
"GIT_CONFIG_NOSYSTEM": "1",
|
||||
|
||||
@@ -30,6 +30,7 @@ from model_routing import ladder
|
||||
from model_routing.ladder import LADDER, Rung
|
||||
from claude_code.reply import Limited
|
||||
from loop_claude.loop import main, reload, routing_check
|
||||
from machine_load import memory
|
||||
from transcript_archive import outcomes, write
|
||||
from transcript_archive.location import project_archive
|
||||
from model_routing.route import Route, route_from, WorkerLine
|
||||
@@ -59,9 +60,11 @@ def human(content):
|
||||
class FakeEffects:
|
||||
"""Effects that record each call and answer from a script, one answer per call."""
|
||||
|
||||
def __init__(self, claude=(), on_terminal_claude=(), git=(), inputs=(), python=(), streams=(), on_terminal=True):
|
||||
def __init__(self, claude=(), on_terminal_claude=(), git=(), inputs=(), python=(), streams=(), on_terminal=True, free_memory=()):
|
||||
self.calls = []
|
||||
self.scripts = {"claude": list(claude), "claude_on_terminal": list(on_terminal_claude), "git": list(git), "input": list(inputs), "python": list(python), "stream_claude": list(streams)}
|
||||
# Without a script, every read finds all memory available.
|
||||
self.free_memory = list(free_memory)
|
||||
self.terminal = on_terminal
|
||||
self.worker_environments = []
|
||||
self.terminal_environments = []
|
||||
@@ -75,6 +78,7 @@ class FakeEffects:
|
||||
git=lambda *args, cwd: self.answer("git", list(args)),
|
||||
input=lambda prompt: self.answer("input", prompt),
|
||||
local_time=lambda seconds: f"<{seconds}>",
|
||||
memory=self.memory,
|
||||
now=lambda: NOW,
|
||||
on_terminal=lambda: self.terminal,
|
||||
python=lambda args, cwd: self.answer("python", (args, cwd)),
|
||||
@@ -82,6 +86,10 @@ class FakeEffects:
|
||||
stream_claude=self.stream,
|
||||
)
|
||||
|
||||
def memory(self):
|
||||
self.calls.append(("memory", None))
|
||||
return self.free_memory.pop(0) if self.free_memory else memory.Memory(available=16, total=16)
|
||||
|
||||
def on_terminal_claude(self, args, cwd, env):
|
||||
self.terminal_environments.append(env)
|
||||
return self.answer("claude_on_terminal", args)
|
||||
@@ -583,6 +591,21 @@ class Limits(unittest.TestCase):
|
||||
self.assertEqual(entries.rejected_until({"type": "rate_limit_event", "rate_limit_info": info}), want, info)
|
||||
|
||||
|
||||
class MemoryWait(unittest.TestCase):
|
||||
def test_a_worker_starts_only_once_a_quarter_of_memory_is_available(self):
|
||||
fake = FakeEffects(free_memory=[memory.Memory(available=3, total=16), memory.Memory(available=3, total=16), memory.Memory(available=4, total=16)], streams=[Streamed(0)])
|
||||
with contextlib.redirect_stderr(io.StringIO()) as said:
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, fake.effects()))
|
||||
self.assertEqual([seam for seam, _ in fake.calls], ["memory", "sleep", "memory", "sleep", "memory", "stream_claude"])
|
||||
self.assertEqual(fake.made("sleep"), [worker.MEMORY_POLL] * 2)
|
||||
self.assertEqual(said.getvalue().count("waiting"), 1, said.getvalue())
|
||||
|
||||
def test_a_worker_starts_at_once_where_memory_cannot_be_read(self):
|
||||
fake = FakeEffects(free_memory=[None], streams=[Streamed(0)])
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, fake.effects()))
|
||||
self.assertEqual([seam for seam, _ in fake.calls], ["memory", "stream_claude"])
|
||||
|
||||
|
||||
class Interrupts(unittest.TestCase):
|
||||
def put(self, fake, answer_here=True):
|
||||
with contextlib.redirect_stderr(io.StringIO()) as said:
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
"""How much of the machine's memory is available, as /proc/meminfo's MemAvailable and MemTotal state it, and whether at least ROOM_SHARE of it is available, which leaves room for a worker."""
|
||||
import dataclasses
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
MEMINFO = pathlib.Path("/proc/meminfo")
|
||||
# A worker starts only while at least this share of the machine's memory is available; the human chose it, as no standard sets one.
|
||||
ROOM_SHARE = 0.25
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Memory:
|
||||
"""The machine's memory, in kB as /proc/meminfo counts it: what a new process can have without swapping, and all of it."""
|
||||
available: int
|
||||
total: int
|
||||
|
||||
def has_room(self) -> bool:
|
||||
return self.available >= ROOM_SHARE * self.total
|
||||
|
||||
|
||||
def parsed(meminfo: str) -> Memory | None:
|
||||
"""The memory a /proc/meminfo text states; None where it lacks MemAvailable or MemTotal."""
|
||||
fields = dict(re.findall(r"^(MemAvailable|MemTotal):\s+(\d+) kB$", meminfo, re.MULTILINE))
|
||||
if len(fields) != 2:
|
||||
return None
|
||||
return Memory(available=int(fields["MemAvailable"]), total=int(fields["MemTotal"]))
|
||||
|
||||
|
||||
def read_memory(meminfo: pathlib.Path | None = None) -> Memory | None:
|
||||
"""The machine's memory, read from meminfo or else MEMINFO; None where it cannot be read, as off Linux."""
|
||||
try:
|
||||
return parsed((meminfo or MEMINFO).read_text())
|
||||
except OSError:
|
||||
return None
|
||||
@@ -0,0 +1,51 @@
|
||||
"""The machine's heavy slots: one per 4 CPUs, and at least one, each a file locked with flock that every session on the machine shares."""
|
||||
import fcntl
|
||||
import os
|
||||
import pathlib
|
||||
import time
|
||||
import typing
|
||||
|
||||
CPUS_PER_SLOT = 4
|
||||
# Seconds between tries while every slot is held; a slot frees as its command exits.
|
||||
POLL_SECONDS = 1
|
||||
|
||||
|
||||
def usable_cpus() -> int | None:
|
||||
"""The CPUs this process may run on, which an affinity mask can make fewer than the machine's; None where the count is unknown."""
|
||||
return len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else os.cpu_count()
|
||||
|
||||
|
||||
def slot_count(cpus: int | None) -> int:
|
||||
return max(1, (cpus or 0) // CPUS_PER_SLOT)
|
||||
|
||||
|
||||
def slot_folder(env: typing.Mapping[str, str]) -> pathlib.Path:
|
||||
state = env.get("XDG_STATE_HOME") or str(pathlib.Path.home() / ".local" / "state")
|
||||
return pathlib.Path(state) / "run-heavy"
|
||||
|
||||
|
||||
def taken_slot(folder: pathlib.Path, slots: int) -> tuple[int, int] | None:
|
||||
"""The number and open file of a slot this call locked; None while every slot is held. Raises OSError where a slot file cannot be opened or locked."""
|
||||
for slot in range(slots):
|
||||
fd = os.open(folder / f"slot-{slot}", os.O_RDWR | os.O_CREAT, 0o644)
|
||||
try:
|
||||
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
os.close(fd)
|
||||
continue
|
||||
except OSError:
|
||||
os.close(fd)
|
||||
raise
|
||||
return slot, fd
|
||||
return None
|
||||
|
||||
|
||||
def waited_slot(folder: pathlib.Path, slots: int, say: typing.Callable[[], None], sleep: typing.Callable[[float], None] = time.sleep) -> tuple[int, int]:
|
||||
"""The number and open file of a slot this call locked, once one frees. While every slot is held, it calls say once and tries again each POLL_SECONDS."""
|
||||
said = False
|
||||
while (taken := taken_slot(folder, slots)) is None:
|
||||
if not said:
|
||||
say()
|
||||
said = True
|
||||
sleep(POLL_SECONDS)
|
||||
return taken
|
||||
@@ -0,0 +1,85 @@
|
||||
import errno
|
||||
import fcntl
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import unittest.mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from machine_load import memory
|
||||
from machine_load.slots import slot_count
|
||||
from machine_load import slots
|
||||
|
||||
|
||||
class SlotCount(unittest.TestCase):
|
||||
def test_one_slot_per_four_cpus_and_never_none(self):
|
||||
for cpus, slots in ((16, 4), (8, 2), (7, 1), (4, 1), (1, 1), (None, 1)):
|
||||
self.assertEqual(slot_count(cpus), slots, cpus)
|
||||
|
||||
|
||||
class MemoryShare(unittest.TestCase):
|
||||
def test_has_room_from_a_quarter_of_memory_available(self):
|
||||
self.assertFalse(memory.Memory(available=3, total=16).has_room())
|
||||
self.assertTrue(memory.Memory(available=4, total=16).has_room())
|
||||
|
||||
def test_reads_available_and_total_memory_from_meminfo(self):
|
||||
meminfo = "MemTotal: 16387072 kB\nMemFree: 981236 kB\nMemAvailable: 4096000 kB\n"
|
||||
self.assertEqual(memory.parsed(meminfo), memory.Memory(available=4096000, total=16387072))
|
||||
for unread in ("MemTotal: 16 kB\n", "MemTotal: x kB\nMemAvailable: 1 kB\n", ""):
|
||||
self.assertIsNone(memory.parsed(unread), unread)
|
||||
|
||||
def test_reads_a_meminfo_file_and_none_where_it_cannot(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
meminfo = pathlib.Path(temp, "meminfo")
|
||||
meminfo.write_text("MemTotal: 8 kB\nMemAvailable: 2 kB\n")
|
||||
self.assertEqual(memory.read_memory(meminfo), memory.Memory(available=2, total=8))
|
||||
self.assertIsNone(memory.read_memory(pathlib.Path(temp, "gone")))
|
||||
|
||||
|
||||
class SlotFolder(unittest.TestCase):
|
||||
def test_every_session_shares_the_slots_under_the_users_state_folder(self):
|
||||
self.assertEqual(slots.slot_folder({}), pathlib.Path.home() / ".local" / "state" / "run-heavy")
|
||||
self.assertEqual(slots.slot_folder({"XDG_STATE_HOME": "/s"}), pathlib.Path("/s/run-heavy"))
|
||||
|
||||
|
||||
class TakenSlot(unittest.TestCase):
|
||||
def test_a_lock_that_fails_for_another_reason_than_a_holder_closes_its_file_and_raises(self):
|
||||
closed = []
|
||||
with tempfile.TemporaryDirectory() as folder, unittest.mock.patch("fcntl.flock", side_effect=OSError(errno.ENOLCK, "No locks available")), unittest.mock.patch("os.close", side_effect=closed.append):
|
||||
with self.assertRaises(OSError):
|
||||
slots.taken_slot(pathlib.Path(folder), 2)
|
||||
self.assertEqual(len(closed), 1)
|
||||
self.assertIsInstance(closed[0], int)
|
||||
os.close(closed[0])
|
||||
|
||||
|
||||
class WaitedSlot(unittest.TestCase):
|
||||
def test_waits_while_every_slot_is_held_saying_so_once_then_takes_the_freed_one(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
held = [open(pathlib.Path(folder, f"slot-{slot}"), "a") for slot in range(2)]
|
||||
for file in held:
|
||||
fcntl.flock(file, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
said, slept = [], []
|
||||
|
||||
def sleep(seconds):
|
||||
slept.append(seconds)
|
||||
if len(slept) == 2:
|
||||
held[1].close()
|
||||
|
||||
slot, fd = slots.waited_slot(pathlib.Path(folder), 2, lambda: said.append(True), sleep)
|
||||
os.close(fd)
|
||||
held[0].close()
|
||||
self.assertEqual((slot, said, slept), (1, [True], [slots.POLL_SECONDS] * 2))
|
||||
|
||||
def test_takes_a_free_slot_at_once(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
slot, fd = slots.waited_slot(pathlib.Path(folder), 2, lambda: self.fail("said it waits"), lambda seconds: self.fail("slept"))
|
||||
os.close(fd)
|
||||
self.assertEqual(slot, 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -215,7 +215,7 @@ class Main(unittest.TestCase):
|
||||
result = result if "api_error_status" in result else {"type": "result", "structured_output": {"found": [n + 1 for n in result["found"]]}}
|
||||
return subprocess.CompletedProcess(args, 0, json.dumps(result), "")
|
||||
|
||||
effects = Effects(claude=claude, claude_on_terminal=unused, discard_typed=unused, execv=unused, git=system_effects().git, input=unused, local_time=unused, now=unused, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused)
|
||||
effects = Effects(claude=claude, claude_on_terminal=unused, discard_typed=unused, execv=unused, git=system_effects().git, input=unused, local_time=unused, memory=unused, now=unused, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused)
|
||||
out, err = io.StringIO(), io.StringIO()
|
||||
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
code = main.main(argv, {}, self.home, effects)
|
||||
|
||||
+58
-1
@@ -8,7 +8,7 @@ from git_command import commits
|
||||
from review_folder import REPORTS
|
||||
from shell_command import Simple
|
||||
|
||||
RuleName = typing.Literal["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "reviewer-report", "reviewer-rung", "round-cap", "round-title-counts", "todo-rank-check"]
|
||||
RuleName = typing.Literal["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "reviewer-report", "reviewer-rung", "round-cap", "round-title-counts", "run-heavy", "todo-rank-check"]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -60,6 +60,10 @@ RULES: dict[RuleName, Rule] = {
|
||||
"a larv-review round's commit title gives the counts of 🔴 and 🟡 findings the round reported, and how many of them a reviewer marked `[misread]`, at most the two together, such as `Answer stability round 1: 1 🔴, 2 🟡 (1 misread); flag a deleted test`, since score-outcomes counts them",
|
||||
"a git commit whose message opens `Answer <pass> round <n>:`",
|
||||
),
|
||||
"run-heavy": Rule(
|
||||
"a heavy command, which loads the machine for minutes, runs under ~/.claude/bin/run-heavy, so sessions running at once cannot starve the machine",
|
||||
"a docker build, docker buildx build or docker compose build, and a run of cargo mutants, Infection, mutmut or Stryker, the mutation tools stability-reviewer names, whether direct, through a launcher such as uvx, or inside docker run, docker exec or docker compose run or exec. A gate script is not checked, since no program can tell it from a light one",
|
||||
),
|
||||
"todo-rank-check": Rule(
|
||||
"while todo.md has changes, staged or not, a git commit waits until ~/.claude/bin/rank_todo.py --check passes on it",
|
||||
"a git commit in a checkout whose todo.md has changes",
|
||||
@@ -96,6 +100,56 @@ class Checked:
|
||||
overridden: list[tuple[Breach, str]]
|
||||
|
||||
|
||||
# Launchers that run the tool named later in their words, such as uvx --with pytest mutmut run.
|
||||
LAUNCHERS = frozenset({"bunx", "env", "nice", "npx", "pipx", "pnpm", "time", "uvx", "yarn"})
|
||||
# Each mutation tool, and the words that make its call a mutation run when one of them follows it within two words; with no words, every call is one.
|
||||
MUTATORS = {"cargo": {"mutants"}, "infection": set(), "mutmut": {"run"}, "stryker": {"run"}}
|
||||
# Flags with which a tool only prints its help, version or list, so a call that carries one is light. Not -h, which docker run reads as the container's hostname.
|
||||
LIGHT_FLAGS = frozenset({"--help", "--list", "--version"})
|
||||
# The options of docker and docker compose that take a value, so the value is not read as a subcommand.
|
||||
DOCKER_VALUED = frozenset({"--ansi", "--config", "--context", "--env-file", "--file", "--host", "--log-level", "--parallel", "--profile", "--progress", "--project-directory", "--project-name", "-c", "-f", "-H", "-l", "-p"})
|
||||
|
||||
|
||||
def tool_name(word: str) -> str:
|
||||
"""The tool a word names, such as mutmut for `mutmut>=3,<4` or `.venv/bin/mutmut`; empty for a word that names none, such as a file name."""
|
||||
named = re.fullmatch(r"([A-Za-z_-]+)(?:[<>=!~@].*)?", posixpath.basename(word))
|
||||
return named.group(1) if named else ""
|
||||
|
||||
|
||||
def heavy(words: list[str]) -> bool:
|
||||
"""Whether the simple command runs a docker build, or a mutation run directly, through a launcher or in a container, outside run-heavy."""
|
||||
names = [tool_name(word) for word in words]
|
||||
if not names or "run-heavy" in names or LIGHT_FLAGS & set(words):
|
||||
return False
|
||||
# Past a launcher, the tool may follow the launcher's own options and assignments, as in nice -n 19 docker build or uvx --with pytest mutmut run; without one, the command itself is the tool.
|
||||
launched = names[0] in LAUNCHERS
|
||||
docker = next((i for i, name in enumerate(names) if name == "docker"), None) if launched else (0 if names[0] == "docker" else None)
|
||||
if docker is not None:
|
||||
subcommand = docker_subcommand(words[docker + 1:])
|
||||
return subcommand in (["build"], ["buildx", "build"], ["compose", "build"]) or (subcommand[-1:] in (["run"], ["exec"]) and mutation_run(names[docker + 1:]))
|
||||
return (launched or names[0] in MUTATORS) and mutation_run(names)
|
||||
|
||||
|
||||
def mutation_run(names: list[str]) -> bool:
|
||||
return any(name in MUTATORS and (not MUTATORS[name] or MUTATORS[name] & set(names[i + 1:i + 3])) for i, name in enumerate(names))
|
||||
|
||||
|
||||
def docker_subcommand(words: list[str]) -> list[str]:
|
||||
"""The subcommand docker's words name, past its options: such as ["build"], ["compose", "run"] or ["buildx", "build"]."""
|
||||
positional: list[str] = []
|
||||
skip = False
|
||||
for word in words:
|
||||
if skip:
|
||||
skip = False
|
||||
elif word.startswith("-"):
|
||||
skip = word in DOCKER_VALUED
|
||||
else:
|
||||
positional.append(word)
|
||||
if positional[0] not in ("buildx", "compose") or len(positional) == 2:
|
||||
break
|
||||
return positional
|
||||
|
||||
|
||||
def overrides(written: str) -> tuple[dict[str, str], list[str]]:
|
||||
"""The rules the text overrides with their reasons, and the markers that name no rule or give no reason; a marker for a rule that takes no override is neither."""
|
||||
given = [(rule, reason.strip().removesuffix("-->").strip()) for rule, reason in OVERRIDE.findall(written)]
|
||||
@@ -124,6 +178,9 @@ def command_breaches(command: str, simples: list[Simple], home: str) -> list[Bre
|
||||
publishing = [simple for simple in simples if simple.words[0] in ("gh", "git", "tea")]
|
||||
if publishing and ATTRIBUTION.search(command):
|
||||
breaches.append(Breach("no-ai-attribution", "the command carries an attribution", publishing[0].directory))
|
||||
for simple in simples:
|
||||
if heavy(simple.words):
|
||||
breaches.append(Breach("run-heavy", f"run `{' '.join(simple.words)[:80]}` under `~/.claude/bin/run-heavy`, in the background", simple.directory))
|
||||
for commit in found:
|
||||
if commit.from_file:
|
||||
breaches.append(Breach("commit-one-line", "pass the message as one -m '<line>', not from a file", commit.directory))
|
||||
|
||||
@@ -106,6 +106,16 @@ class CommandBreaches(unittest.TestCase):
|
||||
self.assertEqual(checked("git commit -m 'Answer stability round 1: 1 🔴, 2 🟡 (1 misread); fix the parser'"), [])
|
||||
self.assertEqual(checked("git commit -m 'Answer the question in the README'"), [])
|
||||
|
||||
def test_a_docker_build_or_a_mutation_run_outside_run_heavy_breaks_run_heavy(self):
|
||||
for command in ["docker build -t x .", "cd /w && docker buildx build .", "docker compose build", "docker compose -f x.yml --verbose build app", "uvx --with pytest 'mutmut>=3,<4' run", "cargo mutants --in-diff d", "npx stryker run", "vendor/bin/infection --git-diff-lines", "docker compose run --rm app mutmut run", "docker run --rm -v /w:/w rust:1.90.0 cargo mutants", "docker compose exec app npx stryker run", "docker run -h box rust:1.90.0 cargo mutants", "nice -n 19 docker build .", "env DOCKER_BUILDKIT=1 docker build .", "time -p docker compose build"]:
|
||||
with self.subTest(command):
|
||||
self.assertEqual(checked(command), ["run-heavy"])
|
||||
|
||||
def test_a_heavy_command_under_run_heavy_or_a_light_docker_or_mutmut_call_passes(self):
|
||||
for command in ["~/.claude/bin/run-heavy docker build -t x .", "/home/me/.claude/bin/run-heavy uvx mutmut run", "nice -n 19 ~/.claude/bin/run-heavy docker build .", "env X=1 run-heavy docker compose run app mutmut run", "docker images", "docker compose run --rm app go vet", "docker compose run app make build", "uvx 'mutmut>=3,<4' results", "echo docker build", "npx prettier --check infection.json5", "echo mutmut run", "env", "docker build --help", "cargo mutants --list", "infection --version"]:
|
||||
with self.subTest(command):
|
||||
self.assertEqual(checked(command), [])
|
||||
|
||||
def test_a_round_title_past_round_3_breaks_round_cap(self):
|
||||
self.assertEqual(checked("git commit -m 'Answer stability round 4: 0 🔴, 1 🟡 (0 misread); fix the parser'"), ["round-cap"])
|
||||
self.assertEqual(checked("git commit -m 'Answer stability round 3: 0 🔴, 1 🟡 (0 misread); fix the parser'"), [])
|
||||
@@ -457,7 +467,7 @@ class Hook(unittest.TestCase):
|
||||
|
||||
def test_help_lists_every_rule(self):
|
||||
helped = subprocess.run([str(HOOK), "--help"], capture_output=True, check=True, env=isolated_env(), text=True).stdout
|
||||
for rule in ["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "reviewer-report", "reviewer-rung", "round-cap", "round-title-counts", "todo-rank-check"]:
|
||||
for rule in ["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "reviewer-report", "reviewer-rung", "round-cap", "round-title-counts", "run-heavy", "todo-rank-check"]:
|
||||
self.assertIn(f" {rule}\n The rule: ", helped)
|
||||
self.assertIn("reviewer-report\n The rule: a reviewer writes nothing but its report", helped)
|
||||
self.assertRegex(helped, r" reviewer-report\n( .*\n){2} Takes no override\.\n")
|
||||
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env python3
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
|
||||
from machine_load.slots import usable_cpus
|
||||
from run_heavy import main
|
||||
|
||||
sys.exit(main.main(sys.argv[1:], os.environ, usable_cpus()))
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Run a heavy command once it holds one of the machine's heavy slots, so sessions running at once cannot starve the machine.
|
||||
|
||||
usage: run-heavy <command> [<arg>...]
|
||||
|
||||
A heavy command is one that loads the machine for minutes: a full gate, mutation tests, an e2e run,
|
||||
a docker build. The machine has one heavy slot for every 4 CPUs that run-heavy may use, and at least one, shared by every
|
||||
session that runs run-heavy. Each slot is a file locked with flock, under
|
||||
$XDG_STATE_HOME/run-heavy (default ~/.local/state/run-heavy). While every slot is held, run-heavy
|
||||
says so once on stderr and waits. It then replaces itself with the command. The slot's lock stays held
|
||||
until the command, and every process that inherited the lock, exits. Its exit code is the command's.
|
||||
|
||||
A command it runs carries the slot's number in RUN_HEAVY_SLOT, and run-heavy called with that set
|
||||
runs its command at once, so a gate that wraps its own steps cannot wait on itself.
|
||||
|
||||
As GNU env, nice and timeout do, it exits 125, saying why, when no command is given or the slot
|
||||
folder or a slot file cannot be made or locked; 126 when the command is found but cannot run; and 127 when it is not found.
|
||||
"""
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import typing
|
||||
|
||||
from failure import Failure
|
||||
from machine_load.slots import slot_count, slot_folder, waited_slot
|
||||
|
||||
OWN_FAILURE = 125
|
||||
CANNOT_RUN = 126
|
||||
NOT_FOUND = 127
|
||||
HELD = "RUN_HEAVY_SLOT"
|
||||
USAGE = "usage: run-heavy <command> [<arg>...]"
|
||||
|
||||
|
||||
def main(argv: list[str], env: typing.Mapping[str, str], cpus: int | None) -> int:
|
||||
if argv[:1] in (["-h"], ["--help"]):
|
||||
print(__doc__.strip())
|
||||
return 0
|
||||
if not argv:
|
||||
print(f"run-heavy: no command given; {USAGE}", file=sys.stderr)
|
||||
return OWN_FAILURE
|
||||
if HELD not in env:
|
||||
slot = held_slot(slot_folder(env), slot_count(cpus))
|
||||
if isinstance(slot, Failure):
|
||||
print(f"run-heavy: {slot.reason}", file=sys.stderr)
|
||||
return OWN_FAILURE
|
||||
env = {**env, HELD: str(slot)}
|
||||
return run(argv, env)
|
||||
|
||||
|
||||
def held_slot(folder: pathlib.Path, slots: int) -> int | Failure:
|
||||
"""The number of the slot this call holds, its lock left for the command to inherit, or why no slot could be taken."""
|
||||
try:
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
slot, fd = waited_slot(folder, slots, lambda: print(f"run-heavy: waiting for a heavy slot, of {slots} on this machine, in {folder}; `fuser {folder}/slot-*` names the processes that hold them", file=sys.stderr, flush=True))
|
||||
except OSError as failure:
|
||||
return Failure(f"cannot take a heavy slot in {folder}: {failure.strerror}")
|
||||
os.set_inheritable(fd, True)
|
||||
return slot
|
||||
|
||||
|
||||
def run(argv: list[str], env: typing.Mapping[str, str]) -> int:
|
||||
"""Replaces this process with the command; returns only where the command cannot start."""
|
||||
try:
|
||||
os.execvpe(argv[0], argv, env)
|
||||
except FileNotFoundError:
|
||||
print(f"run-heavy: {argv[0]}: command not found", file=sys.stderr)
|
||||
return NOT_FOUND
|
||||
except OSError as failure:
|
||||
print(f"run-heavy: cannot run {argv[0]}: {failure.strerror}", file=sys.stderr)
|
||||
return CANNOT_RUN
|
||||
@@ -0,0 +1,95 @@
|
||||
import contextlib
|
||||
import fcntl
|
||||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from run_heavy import main
|
||||
from machine_load.slots import slot_count, usable_cpus
|
||||
|
||||
TOOL = pathlib.Path(__file__).resolve().parents[1] / "run-heavy"
|
||||
|
||||
|
||||
class RunHeavy(unittest.TestCase):
|
||||
def setUp(self):
|
||||
state = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(state.cleanup)
|
||||
self.env = {**os.environ, "XDG_STATE_HOME": state.name}
|
||||
self.env.pop(main.HELD, None)
|
||||
self.slots = pathlib.Path(state.name) / "run-heavy"
|
||||
|
||||
def run_tool(self, *args: str, env=None) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, text=True, env=env or self.env, timeout=30)
|
||||
|
||||
def start_tool(self, *args: str, env=None) -> subprocess.Popen[str]:
|
||||
tool = subprocess.Popen([sys.executable, str(TOOL), *args], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, env=env or self.env)
|
||||
self.addCleanup(tool.kill)
|
||||
return tool
|
||||
|
||||
@contextlib.contextmanager
|
||||
def every_slot_held(self):
|
||||
self.slots.mkdir(parents=True, exist_ok=True)
|
||||
with contextlib.ExitStack() as held:
|
||||
for slot in range(slot_count(usable_cpus())):
|
||||
file = held.enter_context(open(self.slots / f"slot-{slot}", "a"))
|
||||
fcntl.flock(file, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
yield
|
||||
|
||||
def free_slots(self) -> int:
|
||||
free = 0
|
||||
for slot in range(slot_count(usable_cpus())):
|
||||
with open(self.slots / f"slot-{slot}", "a") as file:
|
||||
try:
|
||||
fcntl.flock(file, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
free += 1
|
||||
except BlockingIOError:
|
||||
pass
|
||||
return free
|
||||
|
||||
def test_runs_the_command_and_exits_with_its_code(self):
|
||||
result = self.run_tool("sh", "-c", "echo out; echo err >&2; exit 3")
|
||||
self.assertEqual((result.returncode, result.stdout, result.stderr), (3, "out\n", "err\n"))
|
||||
|
||||
def test_holds_a_slot_while_the_command_runs_and_frees_it_after(self):
|
||||
tool = self.start_tool("sh", "-c", "echo started; read line")
|
||||
self.assertEqual(tool.stdout.readline(), "started\n")
|
||||
self.assertEqual(self.free_slots(), slot_count(usable_cpus()) - 1)
|
||||
tool.communicate("\n", timeout=30)
|
||||
self.assertEqual(self.free_slots(), slot_count(usable_cpus()))
|
||||
|
||||
def test_a_command_inside_a_held_slot_runs_at_once(self):
|
||||
with self.every_slot_held():
|
||||
result = self.run_tool("echo", "nested", env={**self.env, main.HELD: "0"})
|
||||
self.assertEqual((result.returncode, result.stdout), (0, "nested\n"))
|
||||
|
||||
def test_exits_125_saying_why_where_the_slot_folder_cannot_be_made(self):
|
||||
with tempfile.NamedTemporaryFile() as state:
|
||||
result = self.run_tool("echo", "ran", env={**self.env, "XDG_STATE_HOME": state.name})
|
||||
self.assertEqual((result.returncode, result.stdout), (125, ""))
|
||||
self.assertIn(state.name, result.stderr)
|
||||
|
||||
def test_a_command_not_found_exits_127_and_one_that_cannot_run_126_saying_why(self):
|
||||
missing = self.run_tool("no-such-command-anywhere")
|
||||
self.assertEqual((missing.returncode, missing.stdout), (127, ""))
|
||||
self.assertIn("no-such-command-anywhere", missing.stderr)
|
||||
with tempfile.NamedTemporaryFile(suffix=".sh") as unrunnable:
|
||||
denied = self.run_tool(unrunnable.name)
|
||||
self.assertEqual((denied.returncode, denied.stdout), (126, ""))
|
||||
self.assertIn(unrunnable.name, denied.stderr)
|
||||
|
||||
def test_no_command_exits_125_with_the_usage_and_help_exits_0(self):
|
||||
bare = self.run_tool()
|
||||
self.assertEqual((bare.returncode, bare.stdout), (125, ""))
|
||||
self.assertIn("usage: run-heavy <command>", bare.stderr)
|
||||
helped = self.run_tool("--help")
|
||||
self.assertEqual(helped.returncode, 0)
|
||||
self.assertIn("usage: run-heavy <command>", helped.stdout)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -2648,3 +2648,21 @@ Inherited architect, decided by:
|
||||
- Locality: `bin/round_titles.py:45` `title_counts`: one function parses the counts after a round title's colon, so a change to the title format happens in one place
|
||||
- Locality: `bin/score_outcomes/results.py:143` `counted`: `render` and `misreads` share one spelling of a chunk's findings, so the two output paths cannot drift apart
|
||||
- Shape: `bin/round_titles.py:34` `lacks_counts`: the name now also covers a misread count above 🔴 plus 🟡, which a reader chasing a refusal learns only from the body; a cost that doesn't outweigh the gains
|
||||
|
||||
## 2026-10-23T09:51:56Z, PR #366 at e05c54a, against 4a0ba59
|
||||
|
||||
Ruling: same
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Mid A | same | same | worse | same | same |
|
||||
| Inherited architect | same | same | worse | same | same |
|
||||
|
||||
Mid A, decided by:
|
||||
|
||||
- Shape: `bin/rule_hook/rules.py:103` `heavy`: about 50 lines of command-word parsing sit in the module that holds the rule table, while rules of similar weight have their own modules; line 126 picks docker's index by a nested conditional
|
||||
|
||||
Inherited architect, decided by:
|
||||
|
||||
- Shape: `bin/rule_hook/rules.py:103` `heavy`: argv parsing for docker and mutation tools joins the rule registry, breaking the pattern of a sibling module per non-trivial rule, and rules.py now mixes registry, file checks and a shell parser
|
||||
- Shape: `bin/rule_hook/rules.py:126` `heavy`: a conditional nested in another decides where docker sits in the words, which a 3am reader unpicks by hand
|
||||
|
||||
@@ -72,6 +72,10 @@ The cap is 5, `bin/todo_table.py`'s `BUNDLE_MAX_R`, from the same chunks: by the
|
||||
|
||||
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. Valid while transcripts stay small enough to copy whole.
|
||||
|
||||
## machine load: a heavy command waits for one of the machine's heavy slots, one for every 4 CPUs, which every session shares, and loop-claude starts a worker only while at least a quarter of the machine's memory is available
|
||||
|
||||
2026-10-23, lilleman. Serves goal 1.5. On 2026-10-22 ai-sandbox, with 8 CPUs and 16 GiB, thrashed for hours: four to six loops ran mutation tests, full gates, `docker build`s and QEMU-booting e2e tests at once, and nothing told one loop how busy the machine was. lilleman chose both limits; no standard sets either, and GNU parallel's `--memfree`, which does the memory wait's job, has no default. A slot is a file lock, so a session that dies frees its slot. The memory wait reads `/proc/meminfo` only, and off Linux a worker starts at once. `bin/machine_load/` holds the slots and the memory share, `bin/run-heavy` takes a slot, `CLAUDE.md` → "Working with the codebase" has every session use it, and loop-claude's worker waits for the share. rule-hook's rule `run-heavy` blocks a `docker build`, and a run of a mutation tool stability-reviewer names, outside run-heavy, whether direct or inside a container. No program can tell a gate script from a light one, so only `CLAUDE.md` asks a session to run a gate under run-heavy. Valid while one machine runs several loops at once.
|
||||
|
||||
## principles: about seven things per level, four or five where they act on each other, more where a reader finds them by name; crossing a limit asks for a grouping or a recorded reason
|
||||
|
||||
2026-10-22, lilleman asked for a rule of seven that holds at every level and is never a hard rule. Serves code health: a reader new to the project should take in each level at once. The numbers follow the sources below, since no single study sets them.
|
||||
|
||||
@@ -6,12 +6,13 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 223`
|
||||
`Next ID = 224`
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 223 | 0.4.0 | | | **Move the `run-heavy` rule's command parsing out of `bin/rule_hook/rules.py` into a module of its own, as `reviewer_rung.py` holds its rule, and pick docker's position in `heavy` without a nested conditional.** | 2 | 1 | 2 | 3 | 2 | code health by 2 | 12.0 |
|
||||
| 59 | 0.4.0 | | | **Bound a run of unfinished verdicts on one chunk, so a worker and judge that disagree cannot cycle unattended without end.** | 2 | 1 | 3 | 4 | 3 | 1.4 by 6 | 6.0 |
|
||||
| 167 | 0.4.0 | | | **Refuse a `/security-review` or `/claude-security` Skill call below the project's rung, as rule-hook refuses such an Agent call.** | 3 | 2 | 3 | 6 | 2 | 1.1 by 4 | 5.6 |
|
||||
| 6 | 0.4.0 | | decision | **Move what `loop-claude` writes out of `~/.claude.json`, and ask the human about each write that must stay.** | 3 | 3 | 3 | 7 | 3 | 1.1 by 5 | 5.3 |
|
||||
@@ -69,6 +70,10 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 223. Move the `run-heavy` rule's command parsing out of `bin/rule_hook/rules.py` into a module of its own, as `reviewer_rung.py` holds its rule, and pick docker's position in `heavy` without a nested conditional.
|
||||
|
||||
PR #366's paired ruling, at e05c54a against 4a0ba59, had Mid A and the inherited architect both rule Shape worse at `bin/rule_hook/rules.py:103` `heavy`. About 50 lines of command-word parsing (`LAUNCHERS`, `MUTATORS`, `DOCKER_VALUED`, `LIGHT_FLAGS`, `tool_name`, `heavy`, `mutation_run`, `docker_subcommand`) joined the module that holds the rule table, while the other non-trivial rules each have a sibling module. Both also named line 126, where a conditional nested in another picks docker's index. Make that place easier to read.
|
||||
|
||||
### 59. Bound a run of unfinished verdicts on one chunk, so a worker and judge that disagree cannot cycle unattended without end.
|
||||
|
||||
Raised by the product-owner review of PR #211. The resumed worker gets the judge's quote of the work remaining, but a worker that still thinks the chunk done can answer each cycle the same way, and each cycle costs a worker and a judge run (goal 1.4).
|
||||
|
||||
Reference in New Issue
Block a user