Answer principles round 1: 0 🔴, 5 🟡; enforce run-heavy in rule-hook, take slots through one tested function, exit 125 on lock errors, fake meminfo in black-box tests
This commit is contained in:
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Sessions running at once no longer starve the machine. A new tool, `run-heavy <command>`, runs a heavy command, such as a full gate, mutation tests, an e2e run or a `docker build`, once it holds one of the machine's heavy slots: one per 4 CPUs, shared by every session. `CLAUDE.md` has every session run its heavy commands under it, and the stability reviewer its mutation tests. loop-claude now waits to start a worker while less than a quarter of the machine's memory is available.
|
||||
- Sessions running at once no longer starve the machine. A new tool, `run-heavy <command>`, runs a heavy command, such as a full gate, mutation tests, an e2e run or a `docker build`, once it holds one of the machine's heavy slots: one per 4 CPUs, shared by every session. `CLAUDE.md` has every session run its heavy commands under it, and rule-hook blocks a `docker build` or a mutation run outside it. loop-claude now waits to start a worker while less than a quarter of the machine's memory is available.
|
||||
- larv-review's skill now says a commit answering the external auto-reviewer takes no `Answer <pass> round` title, and that `larv-review round-count` doesn't list the auto-reviewer. A worker who followed the skill used to get a "not a pass" warning.
|
||||
- rule-hook now blocks a larv-review round commit titled past round 3, such as `Answer stability round 4: …`, unless a `round-cap` override quotes the human's yes.
|
||||
- loop-claude's first prompt for a chunk that no plan item routed now also names the rung on which the worker launches its reviews for an item it takes in, as the prompt after a step-up or an unfinished verdict already did.
|
||||
|
||||
@@ -12,8 +12,8 @@ You are an elite stability-focused code reviewer with deep expertise in defensiv
|
||||
1. **Scope the change.** You review every PR; a fresh invocation has no prior context, so determine the diff yourself. Diff the PR branch against its base (`gh pr diff`, or `git merge-base <base> HEAD` then `git diff`), and review the changed lines **plus enough surrounding code and callers** to judge correctness — read the whole changed file when practical; a diff alone hides whether a caller breaks. Read the PR description and, when the project uses an issue tracker (project-specific — check its docs) and the change references a ticket, that ticket: the diff is judged against its stated intent, and a scope mismatch in either direction is a finding — claimed-but-unimplemented, and implemented-but-unasked (#2).
|
||||
2. **Read every file under `~/.claude/principles/`**, the shared principles, and `~/.claude/skills/larv-review/standards/quality.md`. These are mandatory org guidelines; all code MUST comply with each that applies. A file's `Applies to` line says what it covers; one without the line covers every change. Re-read each time — they change. Flag violations prominently.
|
||||
3. **Read README.md, AGENTS.md and `docs/decisions.md`** in the current project for project-specific conventions, build instructions, tooling, and the accepted trade-offs a reviewer should honor rather than re-flag. Note the documented test, lint, and static-analysis commands — you will run them.
|
||||
4. **Verify, don't just read.** Run the test suite and static analysis/linters through the project's container tooling, unless a decision puts them on the host; tear down test containers afterwards. A full-suite run catches broken callers that diff-reading misses. When a suspected bug is cheap to confirm, reproduce it — a scratch script, a targeted test run — before reporting it. Make scratch work with Bash, since Write is for your report alone. Put it in a directory you create for this run, and delete it when done; delete nothing else, since a directory the prompt names may hold other agents' files. Leave the working tree exactly as you found it. Anything you could not run goes in Coverage & Verification, never silently skipped.
|
||||
5. **Mutation-test the changed lines.** Run the project's documented mutation command, or else a widely deployed mutation tool for the language: cargo-mutants for Rust, Infection for PHP, mutmut for Python, Stryker for JavaScript and TypeScript. Limit it to the lines the change adds or modifies, through the tool's own option where it has one (cargo-mutants `--in-diff <diff file>`, Infection `--git-diff-lines`, Stryker `--mutate <file>:<lines>`); otherwise mutate only the changed files and keep only the mutants on changed lines. Run it through the project's tooling, as in step 4, under `~/.claude/bin/run-heavy`, on a scratch copy: the tools write files. Report its wall time and mutant count in Coverage & Verification, or why nothing was mutated. Ignore a surviving mutant that leaves behaviour unchanged, or changes only message or log text no caller or user depends on. Report one 🟡 per test to add or strengthen, naming the surviving mutants it would catch.
|
||||
4. **Verify, don't just read.** Run the test suite and static analysis/linters through the project's container tooling, unless a decision puts them on the host, and a full suite under `~/.claude/bin/run-heavy`; tear down test containers afterwards. A full-suite run catches broken callers that diff-reading misses. When a suspected bug is cheap to confirm, reproduce it — a scratch script, a targeted test run — before reporting it. Make scratch work with Bash, since Write is for your report alone. Put it in a directory you create for this run, and delete it when done; delete nothing else, since a directory the prompt names may hold other agents' files. Leave the working tree exactly as you found it. Anything you could not run goes in Coverage & Verification, never silently skipped.
|
||||
5. **Mutation-test the changed lines.** Run the project's documented mutation command, or else a widely deployed mutation tool for the language: cargo-mutants for Rust, Infection for PHP, mutmut for Python, Stryker for JavaScript and TypeScript. Limit it to the lines the change adds or modifies, through the tool's own option where it has one (cargo-mutants `--in-diff <diff file>`, Infection `--git-diff-lines`, Stryker `--mutate <file>:<lines>`); otherwise mutate only the changed files and keep only the mutants on changed lines. Run it through the project's tooling and run-heavy, as in step 4, on a scratch copy: the tools write files. Report its wall time and mutant count in Coverage & Verification, or why nothing was mutated. Ignore a surviving mutant that leaves behaviour unchanged, or changes only message or log text no caller or user depends on. Report one 🟡 per test to add or strengthen, naming the surviving mutants it would catch.
|
||||
6. **Two passes minimum, then hunt absences.**
|
||||
- *Breadth:* every hunk in every changed file, no skimming. Track which files you have covered; batch large diffs, never truncate the review.
|
||||
- *Depth:* trace the riskiest data paths end-to-end (input → logic → persistence/response), the interactions between changed pieces, and between the change and config (timeout hierarchies, retries × idempotency, queue worker counts, env parity).
|
||||
|
||||
@@ -5,6 +5,7 @@ from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from claude_code.reply import Limited
|
||||
from claude_code.transcript import part_line
|
||||
from machine_load.memory import ROOM_SHARE
|
||||
from model_routing.ladder import Rung
|
||||
from claude_code.session import Session
|
||||
|
||||
@@ -34,11 +35,11 @@ def run_worker(root: str, args: list[str], env: dict[str, str], effects: Effects
|
||||
|
||||
|
||||
def wait_for_memory(effects: Effects) -> None:
|
||||
"""Returns once the machine has room for a worker, or its memory cannot be read, so loops running at once do not push it into swap."""
|
||||
"""Returns once the machine has room for a worker, or its memory cannot be read, so loops running at once do not push it into swap. While it waits, it says so once on stderr and reads the memory again each MEMORY_POLL seconds."""
|
||||
said = False
|
||||
while (memory := effects.memory()) is not None and not memory.has_room():
|
||||
if not said:
|
||||
print(f"waiting to start the worker until a quarter of the machine's memory is available; {memory.available // 1024} MiB of {memory.total // 1024} MiB is", file=sys.stderr, flush=True)
|
||||
print(f"waiting to start the worker until {ROOM_SHARE:.0%} of the machine's memory is available; {memory.available // 1024} MiB of {memory.total // 1024} MiB is", file=sys.stderr, flush=True)
|
||||
said = True
|
||||
effects.sleep(MEMORY_POLL)
|
||||
|
||||
|
||||
@@ -116,6 +116,11 @@ if os.path.basename(sys.argv[0]) == "loop-claude":
|
||||
record({"argv": sys.argv[1:]})
|
||||
pause = time.sleep
|
||||
time.sleep = lambda seconds: record({"sleep": seconds}) if seconds >= 1 else pause(seconds)
|
||||
# The host's own memory decides nothing: every read finds all of it available.
|
||||
sys.path.insert(0, os.path.dirname(os.path.realpath(sys.argv[0])))
|
||||
import pathlib
|
||||
from machine_load import memory
|
||||
memory.MEMINFO = pathlib.Path(os.environ["FAKE_MEMINFO"])
|
||||
"""
|
||||
|
||||
PREMISES = "# Project\n\n## Goals\n\n1. Work.\n\n## Audience\n\nMe.\n"
|
||||
@@ -181,8 +186,10 @@ class Checkout:
|
||||
site = self.temp / "site"
|
||||
site.mkdir()
|
||||
(site / "sitecustomize.py").write_text(SITECUSTOMIZE)
|
||||
(self.temp / "meminfo").write_text("MemTotal: 16 kB\nMemAvailable: 16 kB\n")
|
||||
self.env.update({
|
||||
"FAKE_CLAUDE_LOG": str(self.temp / "calls.jsonl"),
|
||||
"FAKE_MEMINFO": str(self.temp / "meminfo"),
|
||||
"FAKE_CLAUDE_SCENARIO": str(self.temp / "scenario.json"),
|
||||
"GIT_CONFIG_GLOBAL": os.devnull,
|
||||
"GIT_CONFIG_NOSYSTEM": "1",
|
||||
|
||||
@@ -23,7 +23,6 @@ from chunk_pick import DefaultPlan, LastCut
|
||||
from loop_claude.effects import Effects, PYTHON_TIMEOUT, run_python
|
||||
from claude_code import entries, reply, session
|
||||
from loop_claude import escalation_mark, paths, routing
|
||||
from machine_load import memory
|
||||
from failure import Failure
|
||||
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished
|
||||
from loop_claude.cycle import interrupt, judge, prompts, retry, run, states, worker
|
||||
@@ -31,6 +30,7 @@ from model_routing import ladder
|
||||
from model_routing.ladder import LADDER, Rung
|
||||
from claude_code.reply import Limited
|
||||
from loop_claude.loop import main, reload, routing_check
|
||||
from machine_load import memory
|
||||
from transcript_archive import outcomes, write
|
||||
from transcript_archive.location import project_archive
|
||||
from model_routing.route import Record, Route, route_from
|
||||
|
||||
@@ -1,10 +1,4 @@
|
||||
"""How much of the machine's memory is available, so loop-claude starts a worker only while a quarter is.
|
||||
|
||||
loop-claude reads MemAvailable and MemTotal from /proc/meminfo before each worker starts. While less
|
||||
than a quarter is available, it says so once on stderr and reads again each minute, so loops running
|
||||
at once do not push the machine into swap. Where the file cannot be read, as off Linux, the worker
|
||||
starts at once.
|
||||
"""
|
||||
"""How much of the machine's memory is available, as /proc/meminfo's MemAvailable and MemTotal state it, and whether that leaves room for a worker: a quarter of it."""
|
||||
import dataclasses
|
||||
import pathlib
|
||||
import re
|
||||
@@ -32,9 +26,9 @@ def parsed(meminfo: str) -> Memory | None:
|
||||
return Memory(available=int(fields["MemAvailable"]), total=int(fields["MemTotal"]))
|
||||
|
||||
|
||||
def read_memory(meminfo: pathlib.Path = MEMINFO) -> Memory | None:
|
||||
"""The machine's memory; None where meminfo cannot be read, as off Linux."""
|
||||
def read_memory(meminfo: pathlib.Path | None = None) -> Memory | None:
|
||||
"""The machine's memory, read from meminfo or else MEMINFO; None where it cannot be read, as off Linux."""
|
||||
try:
|
||||
return parsed(meminfo.read_text())
|
||||
return parsed((meminfo or MEMINFO).read_text())
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
@@ -2,9 +2,17 @@
|
||||
import fcntl
|
||||
import os
|
||||
import pathlib
|
||||
import time
|
||||
import typing
|
||||
|
||||
CPUS_PER_SLOT = 4
|
||||
# Seconds between tries while every slot is held; a slot frees as its command exits.
|
||||
POLL_SECONDS = 1
|
||||
|
||||
|
||||
def usable_cpus() -> int | None:
|
||||
"""The CPUs this process may run on, which an affinity mask can make fewer than the machine's; None where the count is unknown."""
|
||||
return len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else os.cpu_count()
|
||||
|
||||
|
||||
def slot_count(cpus: int | None) -> int:
|
||||
@@ -17,7 +25,7 @@ def slot_folder(env: typing.Mapping[str, str]) -> pathlib.Path:
|
||||
|
||||
|
||||
def taken_slot(folder: pathlib.Path, slots: int) -> tuple[int, int] | None:
|
||||
"""The number and open file of a slot this call locked; None while every slot is held."""
|
||||
"""The number and open file of a slot this call locked; None while every slot is held. Raises OSError where a slot file cannot be opened or locked."""
|
||||
for slot in range(slots):
|
||||
fd = os.open(folder / f"slot-{slot}", os.O_RDWR | os.O_CREAT, 0o644)
|
||||
try:
|
||||
@@ -25,5 +33,19 @@ def taken_slot(folder: pathlib.Path, slots: int) -> tuple[int, int] | None:
|
||||
except BlockingIOError:
|
||||
os.close(fd)
|
||||
continue
|
||||
except OSError:
|
||||
os.close(fd)
|
||||
raise
|
||||
return slot, fd
|
||||
return None
|
||||
|
||||
|
||||
def waited_slot(folder: pathlib.Path, slots: int, say: typing.Callable[[], None], sleep: typing.Callable[[float], None] = time.sleep) -> tuple[int, int]:
|
||||
"""The number and open file of a slot this call locked, once one frees. While every slot is held, it calls say once and tries again each POLL_SECONDS."""
|
||||
said = False
|
||||
while (taken := taken_slot(folder, slots)) is None:
|
||||
if not said:
|
||||
say()
|
||||
said = True
|
||||
sleep(POLL_SECONDS)
|
||||
return taken
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import fcntl
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
@@ -7,6 +9,7 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from machine_load import memory
|
||||
from machine_load.slots import slot_count
|
||||
from machine_load import slots
|
||||
|
||||
|
||||
class SlotCount(unittest.TestCase):
|
||||
@@ -15,7 +18,7 @@ class SlotCount(unittest.TestCase):
|
||||
self.assertEqual(slot_count(cpus), slots, cpus)
|
||||
|
||||
|
||||
class Memory(unittest.TestCase):
|
||||
class MemoryShare(unittest.TestCase):
|
||||
def test_has_room_from_a_quarter_of_memory_available(self):
|
||||
self.assertFalse(memory.Memory(available=3, total=16).has_room())
|
||||
self.assertTrue(memory.Memory(available=4, total=16).has_room())
|
||||
@@ -34,6 +37,30 @@ class Memory(unittest.TestCase):
|
||||
self.assertIsNone(memory.read_memory(pathlib.Path(temp, "gone")))
|
||||
|
||||
|
||||
class WaitedSlot(unittest.TestCase):
|
||||
def test_waits_while_every_slot_is_held_saying_so_once_then_takes_the_freed_one(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
held = [open(pathlib.Path(folder, f"slot-{slot}"), "a") for slot in range(2)]
|
||||
for file in held:
|
||||
fcntl.flock(file, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
said, slept = [], []
|
||||
|
||||
def sleep(seconds):
|
||||
slept.append(seconds)
|
||||
if len(slept) == 2:
|
||||
held[1].close()
|
||||
|
||||
slot, fd = slots.waited_slot(pathlib.Path(folder), 2, lambda: said.append(True), sleep)
|
||||
os.close(fd)
|
||||
held[0].close()
|
||||
self.assertEqual((slot, said, slept), (1, [True], [slots.POLL_SECONDS] * 2))
|
||||
|
||||
def test_takes_a_free_slot_at_once(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
slot, fd = slots.waited_slot(pathlib.Path(folder), 2, lambda: self.fail("said it waits"), lambda seconds: self.fail("slept"))
|
||||
os.close(fd)
|
||||
self.assertEqual(slot, 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
+23
-1
@@ -8,7 +8,7 @@ from git_command import commits
|
||||
from review_folder import REPORTS
|
||||
from shell_command import Simple
|
||||
|
||||
RuleName = typing.Literal["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "reviewer-report", "reviewer-rung", "round-cap", "round-title-counts", "todo-rank-check"]
|
||||
RuleName = typing.Literal["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "reviewer-report", "reviewer-rung", "round-cap", "round-title-counts", "run-heavy", "todo-rank-check"]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -60,6 +60,10 @@ RULES: dict[RuleName, Rule] = {
|
||||
"a larv-review round's commit title gives the counts of 🔴 and 🟡 findings the round reported, such as `Answer stability round 1: 1 🔴, 2 🟡; flag a deleted test`, since score-outcomes counts them",
|
||||
"a git commit whose message opens `Answer <pass> round <n>:`",
|
||||
),
|
||||
"run-heavy": Rule(
|
||||
"a heavy command, which loads the machine for minutes, runs under ~/.claude/bin/run-heavy, so sessions running at once cannot starve the machine",
|
||||
"a docker build, docker buildx build or docker compose build, and a run of mutmut, cargo mutants, Stryker or Infection, the mutation tools stability-reviewer names; a gate script it cannot tell from a light one",
|
||||
),
|
||||
"todo-rank-check": Rule(
|
||||
"while todo.md has changes, staged or not, a git commit waits until ~/.claude/bin/rank_todo.py --check passes on it",
|
||||
"a git commit in a checkout whose todo.md has changes",
|
||||
@@ -95,6 +99,21 @@ class Outcome:
|
||||
overridden: list[tuple[Breach, str]]
|
||||
|
||||
|
||||
# Launchers that run the tool named later in their words, such as uvx --with pytest mutmut run.
|
||||
LAUNCHERS = frozenset({"bunx", "env", "nice", "npx", "pipx", "pnpm", "time", "uvx", "yarn"})
|
||||
# Each heavy tool, and the words of which one must follow it within two words for its call to be heavy; an empty set makes every call heavy.
|
||||
HEAVY = {"cargo": {"mutants"}, "docker": {"build"}, "infection": set(), "mutmut": {"run"}, "stryker": {"run"}}
|
||||
|
||||
|
||||
def heavy(words: list[str]) -> bool:
|
||||
"""Whether the simple command runs a heavy tool, itself or through a launcher, outside run-heavy."""
|
||||
names = [re.sub(r"[^A-Za-z_-].*", "", posixpath.basename(word)) for word in words]
|
||||
if names[0] == "run-heavy":
|
||||
return False
|
||||
at = range(len(names)) if names[0] in LAUNCHERS else range(1)
|
||||
return any(names[i] in HEAVY and (not HEAVY[names[i]] or HEAVY[names[i]] & set(names[i + 1:i + 3])) for i in at)
|
||||
|
||||
|
||||
def overrides(written: str) -> tuple[dict[str, str], list[str]]:
|
||||
"""The rules the text overrides with their reasons, and the markers that name no rule or give no reason; a marker for a rule that takes no override is neither."""
|
||||
given = [(rule, reason.strip().removesuffix("-->").strip()) for rule, reason in OVERRIDE.findall(written)]
|
||||
@@ -123,6 +142,9 @@ def command_breaches(command: str, simples: list[Simple], home: str) -> list[Bre
|
||||
publishing = [simple for simple in simples if simple.words[0] in ("gh", "git", "tea")]
|
||||
if publishing and ATTRIBUTION.search(command):
|
||||
breaches.append(Breach("no-ai-attribution", "the command carries an attribution", publishing[0].directory))
|
||||
for simple in simples:
|
||||
if heavy(simple.words):
|
||||
breaches.append(Breach("run-heavy", f"run `{' '.join(simple.words[:3])}` as `~/.claude/bin/run-heavy {' '.join(simple.words[:2])} …`, in the background", simple.directory))
|
||||
for commit in found:
|
||||
if commit.from_file:
|
||||
breaches.append(Breach("commit-one-line", "pass the message as one -m '<line>', not from a file", commit.directory))
|
||||
|
||||
@@ -105,6 +105,16 @@ class CommandBreaches(unittest.TestCase):
|
||||
self.assertEqual(checked("git commit -m 'Answer stability round 1: 1 🔴, 2 🟡; fix the parser'"), [])
|
||||
self.assertEqual(checked("git commit -m 'Answer the question in the README'"), [])
|
||||
|
||||
def test_a_docker_build_or_a_mutation_run_outside_run_heavy_breaks_run_heavy(self):
|
||||
for command in ["docker build -t x .", "cd /w && docker buildx build .", "docker compose build", "uvx --with pytest 'mutmut>=3,<4' run", "cargo mutants --in-diff d", "npx stryker run", "vendor/bin/infection --git-diff-lines"]:
|
||||
with self.subTest(command):
|
||||
self.assertEqual(checked(command), ["run-heavy"])
|
||||
|
||||
def test_a_heavy_command_under_run_heavy_or_a_light_docker_or_mutmut_call_passes(self):
|
||||
for command in ["~/.claude/bin/run-heavy docker build -t x .", "/home/me/.claude/bin/run-heavy uvx mutmut run", "docker images", "docker compose run --rm app go vet", "uvx 'mutmut>=3,<4' results", "echo docker build"]:
|
||||
with self.subTest(command):
|
||||
self.assertEqual(checked(command), [])
|
||||
|
||||
def test_a_round_title_past_round_3_breaks_round_cap(self):
|
||||
self.assertEqual(checked("git commit -m 'Answer stability round 4: 0 🔴, 1 🟡; fix the parser'"), ["round-cap"])
|
||||
self.assertEqual(checked("git commit -m 'Answer stability round 3: 0 🔴, 1 🟡; fix the parser'"), [])
|
||||
@@ -456,7 +466,7 @@ class Hook(unittest.TestCase):
|
||||
|
||||
def test_help_lists_every_rule(self):
|
||||
helped = subprocess.run([str(HOOK), "--help"], capture_output=True, check=True, env=isolated_env(), text=True).stdout
|
||||
for rule in ["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "reviewer-report", "reviewer-rung", "round-cap", "round-title-counts", "todo-rank-check"]:
|
||||
for rule in ["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "reviewer-report", "reviewer-rung", "round-cap", "round-title-counts", "run-heavy", "todo-rank-check"]:
|
||||
self.assertIn(f" {rule}\n The rule: ", helped)
|
||||
self.assertIn("reviewer-report\n The rule: a reviewer writes nothing but its report", helped)
|
||||
self.assertRegex(helped, r" reviewer-report\n( .*\n){2} Takes no override\.\n")
|
||||
|
||||
+2
-1
@@ -5,6 +5,7 @@ import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
|
||||
from machine_load.slots import usable_cpus
|
||||
from run_heavy import main
|
||||
|
||||
sys.exit(main.main(sys.argv[1:], os.environ))
|
||||
sys.exit(main.main(sys.argv[1:], os.environ, usable_cpus()))
|
||||
|
||||
+25
-22
@@ -3,7 +3,7 @@
|
||||
usage: run-heavy <command> [<arg>...]
|
||||
|
||||
A heavy command is one that loads the machine for minutes: a full gate, mutation tests, an e2e run,
|
||||
a docker build. The machine has one heavy slot per 4 CPUs, and at least one, shared by every
|
||||
a docker build. The machine has one heavy slot per 4 CPUs it may run on, and at least one, shared by every
|
||||
session that runs run-heavy. Each slot is a file locked with flock, under
|
||||
$XDG_STATE_HOME/run-heavy (default ~/.local/state/run-heavy). While every slot is held, run-heavy
|
||||
says so once on stderr and waits. It then runs the command in its place, which holds the slot until
|
||||
@@ -13,14 +13,14 @@ A command it runs carries the slot's number in RUN_HEAVY_SLOT, and run-heavy cal
|
||||
runs its command at once, so a gate that wraps its own steps cannot wait on itself.
|
||||
|
||||
As GNU env, nice and timeout do, it exits 125, saying why, when no command is given or the slot
|
||||
folder cannot be made; 126 when the command is found but cannot run; and 127 when it is not found.
|
||||
folder or a slot file cannot be made or locked; 126 when the command is found but cannot run; and 127 when it is not found.
|
||||
"""
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import time
|
||||
import typing
|
||||
|
||||
from machine_load.slots import slot_count, slot_folder, taken_slot
|
||||
from machine_load.slots import slot_count, slot_folder, waited_slot
|
||||
|
||||
# Exit codes for a command run-heavy never ran, apart from any the command exits with, as GNU env, nice and timeout use them.
|
||||
OWN_FAILURE = 125
|
||||
@@ -28,12 +28,10 @@ CANNOT_RUN = 126
|
||||
NOT_FOUND = 127
|
||||
# Names the slot a command runs in, so a heavy command it runs does not wait for a second slot.
|
||||
HELD = "RUN_HEAVY_SLOT"
|
||||
# Seconds between tries while every slot is held; a slot frees as its command exits.
|
||||
POLL_SECONDS = 1
|
||||
USAGE = "usage: run-heavy <command> [<arg>...]"
|
||||
|
||||
|
||||
def main(argv: list[str], env: typing.Mapping[str, str]) -> int:
|
||||
def main(argv: list[str], env: typing.Mapping[str, str], cpus: int | None) -> int:
|
||||
if argv[:1] in (["-h"], ["--help"]):
|
||||
print(__doc__.strip())
|
||||
return 0
|
||||
@@ -41,23 +39,28 @@ def main(argv: list[str], env: typing.Mapping[str, str]) -> int:
|
||||
print(f"run-heavy: no command given; {USAGE}", file=sys.stderr)
|
||||
return OWN_FAILURE
|
||||
if HELD not in env:
|
||||
folder = slot_folder(env)
|
||||
try:
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as failure:
|
||||
print(f"run-heavy: cannot make the slot folder {folder}: {failure.strerror}", file=sys.stderr)
|
||||
slot = held_slot(slot_folder(env), slot_count(cpus))
|
||||
if isinstance(slot, str):
|
||||
print(f"run-heavy: {slot}", file=sys.stderr)
|
||||
return OWN_FAILURE
|
||||
slots = slot_count(os.cpu_count())
|
||||
said = False
|
||||
while (taken := taken_slot(folder, slots)) is None:
|
||||
if not said:
|
||||
print(f"run-heavy: waiting for one of the machine's {slots} heavy slots", file=sys.stderr, flush=True)
|
||||
said = True
|
||||
time.sleep(POLL_SECONDS)
|
||||
slot, fd = taken
|
||||
# The command inherits the locked file, so the slot stays held until it exits.
|
||||
os.set_inheritable(fd, True)
|
||||
env = {**env, HELD: str(slot)}
|
||||
return run(argv, env)
|
||||
|
||||
|
||||
def held_slot(folder: pathlib.Path, slots: int) -> int | str:
|
||||
"""The number of the slot this call holds, its lock left for the command to inherit, or why no slot could be taken."""
|
||||
try:
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
slot, fd = waited_slot(folder, slots, lambda: print(f"run-heavy: waiting for one of the machine's {slots} heavy slots", file=sys.stderr, flush=True))
|
||||
except OSError as failure:
|
||||
return f"cannot take a heavy slot in {folder}: {failure.strerror}"
|
||||
# The command inherits the locked file, so the slot stays held until it exits.
|
||||
os.set_inheritable(fd, True)
|
||||
return slot
|
||||
|
||||
|
||||
def run(argv: list[str], env: typing.Mapping[str, str]) -> int:
|
||||
"""Runs the command in run-heavy's place; returns only where it cannot start."""
|
||||
try:
|
||||
os.execvpe(argv[0], argv, env)
|
||||
except FileNotFoundError:
|
||||
|
||||
@@ -5,13 +5,12 @@ import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from machine_load.slots import slot_count
|
||||
from run_heavy import main
|
||||
from machine_load.slots import slot_count, usable_cpus
|
||||
|
||||
TOOL = pathlib.Path(__file__).resolve().parents[1] / "run-heavy"
|
||||
|
||||
@@ -36,14 +35,14 @@ class RunHeavy(unittest.TestCase):
|
||||
def every_slot_held(self):
|
||||
self.slots.mkdir(parents=True, exist_ok=True)
|
||||
with contextlib.ExitStack() as held:
|
||||
for slot in range(slot_count(os.cpu_count())):
|
||||
for slot in range(slot_count(usable_cpus())):
|
||||
file = held.enter_context(open(self.slots / f"slot-{slot}", "a"))
|
||||
fcntl.flock(file, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
yield
|
||||
|
||||
def free_slots(self) -> int:
|
||||
free = 0
|
||||
for slot in range(slot_count(os.cpu_count())):
|
||||
for slot in range(slot_count(usable_cpus())):
|
||||
with open(self.slots / f"slot-{slot}", "a") as file:
|
||||
try:
|
||||
fcntl.flock(file, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
@@ -59,18 +58,9 @@ class RunHeavy(unittest.TestCase):
|
||||
def test_holds_a_slot_while_the_command_runs_and_frees_it_after(self):
|
||||
tool = self.start_tool("sh", "-c", "echo started; read line")
|
||||
self.assertEqual(tool.stdout.readline(), "started\n")
|
||||
self.assertEqual(self.free_slots(), slot_count(os.cpu_count()) - 1)
|
||||
self.assertEqual(self.free_slots(), slot_count(usable_cpus()) - 1)
|
||||
tool.communicate("\n", timeout=30)
|
||||
self.assertEqual(self.free_slots(), slot_count(os.cpu_count()))
|
||||
|
||||
def test_waits_while_every_slot_is_held_and_says_so_once(self):
|
||||
with self.every_slot_held():
|
||||
tool = self.start_tool("echo", "ran")
|
||||
time.sleep(3 * main.POLL_SECONDS)
|
||||
self.assertIsNone(tool.poll())
|
||||
out, err = tool.communicate(timeout=30)
|
||||
self.assertEqual((tool.returncode, out), (0, "ran\n"))
|
||||
self.assertEqual(err.count("waiting"), 1, err)
|
||||
self.assertEqual(self.free_slots(), slot_count(usable_cpus()))
|
||||
|
||||
def test_a_command_inside_a_held_slot_runs_at_once(self):
|
||||
with self.every_slot_held():
|
||||
|
||||
+1
-1
@@ -64,7 +64,7 @@ The cap is 5, `bin/todo_table.py`'s `BUNDLE_MAX_R`, from the same chunks: by the
|
||||
|
||||
## machine load: a heavy command waits for one of a slot per 4 CPUs that every session on the machine shares, and loop-claude starts a worker only while a quarter of the machine's memory is available
|
||||
|
||||
2026-10-23, lilleman. Serves goal 1.5. On 2026-10-22 ai-sandbox, with 8 CPUs and 16 GiB, thrashed for hours: four to six loops ran mutation tests, full gates, `docker build`s and QEMU-booting e2e tests at once, and nothing told one loop how busy the machine was. The human chose both limits; no standard sets either, and GNU parallel's `--memfree`, which does the memory wait's job, has no default. A slot is a file lock, so a session that dies frees its slot. A heavy command run inside a slot already held runs at once, so a gate that wraps its own steps cannot wait on itself. The memory wait reads `/proc/meminfo` only, and off Linux a worker starts at once. `bin/machine_load/` holds the slots and the memory share, `bin/run-heavy` takes a slot, `CLAUDE.md` → "Working with the codebase" has every session use it, and loop-claude's worker waits for the share. Valid while one machine runs several loops at once.
|
||||
2026-10-23, lilleman. Serves goal 1.5. On 2026-10-22 ai-sandbox, with 8 CPUs and 16 GiB, thrashed for hours: four to six loops ran mutation tests, full gates, `docker build`s and QEMU-booting e2e tests at once, and nothing told one loop how busy the machine was. The human chose both limits; no standard sets either, and GNU parallel's `--memfree`, which does the memory wait's job, has no default. A slot is a file lock, so a session that dies frees its slot. A heavy command run inside a slot already held runs at once, so a gate that wraps its own steps cannot wait on itself. The memory wait reads `/proc/meminfo` only, and off Linux a worker starts at once. `bin/machine_load/` holds the slots and the memory share, `bin/run-heavy` takes a slot, `CLAUDE.md` → "Working with the codebase" has every session use it, and loop-claude's worker waits for the share. rule-hook's rule `run-heavy` blocks the heavy commands a program can tell apart, a `docker build` and a run of the mutation tools stability-reviewer names, outside run-heavy; a gate script it cannot tell from a light one, so that part of the rule rests on `CLAUDE.md`. Valid while one machine runs several loops at once.
|
||||
|
||||
## principles: about seven things per level, four or five where they act on each other, more where a reader finds them by name; crossing a limit asks for a grouping or a recorded reason
|
||||
|
||||
|
||||
Reference in New Issue
Block a user