Cap the memory of each container a heavy command makes under run-heavy at its slot's share, and have stability-reviewer size a mutation run's concurrency to fit it

This commit is contained in:
2026-10-24 01:06:56 +02:00
parent eb7b50698c
commit 4c1ab3f185
16 changed files with 149 additions and 27 deletions
+2
View File
@@ -71,6 +71,8 @@
!bin/run-heavy
!bin/run_heavy/
!bin/run_heavy/*.py
!bin/run_heavy/path_entry/
!bin/run_heavy/path_entry/docker
!bin/score-outcomes
!bin/score_outcomes/
!bin/score_outcomes/*.py
+1 -1
View File
@@ -16,7 +16,7 @@ Decisions live in `docs/decisions.md`:
- loop-claude: one session per project, its id derived from the checkout's path relative to `~/code`, or to `/` outside it
- loop-claude: the archive keeps the raw transcript, and a failed archive stops the loop
- machine load: a heavy command waits for one of the machine's heavy slots, one for every 4 CPUs, which every session shares, and loop-claude starts a worker only while at least a quarter of the machine's memory is available
- machine load: every container a session starts has a memory limit and a swap limit as large as its work needs, which rule-hook's `docker-run-memory` enforces on `docker run` and `docker create`, and no daemon-wide cap limits all containers together
- machine load: every container a session starts has a memory limit and a swap limit as large as its work needs, which rule-hook's `docker-run-memory` enforces on `docker run` and `docker create`; under run-heavy, a container that lacks them gets the slot's share of three quarters of the memory; and no daemon-wide cap limits all containers together
- principles: about seven things per level, four or five where they act on each other, more where a reader finds them by name; crossing a limit asks for a grouping or a recorded reason
- reading cost and `score-outcomes`: each waits for 10 finished chunks before it shows a median
- reading cost: a rise is the last 5 chunks' median above the 5 before them, and files or refreshes one plan item
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- `run-heavy` now caps the memory of each container its command makes. Where a `docker run` or `docker create` sets no limit, it gets a memory limit and a swap limit, both the slot's share: three quarters of the machine's memory, split across the heavy slots. A container that sets its own memory limit keeps it. This covers containers a project's own scripts start, which rule-hook can't see. rule-hook's `docker-run-memory` now lets a `docker run` under run-heavy pass.
- rule-hook has a new rule, `docker-run-memory`. It blocks a `docker run` or `docker create` that lacks a memory limit or a swap limit, such as `--memory 4g --memory-swap 4g`, sized to what the work needs. `--memory-swap -1`, unlimited swap, counts as none. `CLAUDE.md` asks for the same limits on a container a script or a compose file starts, which rule-hook can't see. rule-hook's `run-heavy` rule also now reads docker's `--tlscacert`, `--tlscert`, `--tlskey` and buildx's `--builder` as options that take a value, and reads `docker container run` as a run: `docker --tlscacert ca.pem build .` was not blocked outside run-heavy before.
- Reviewers now mark a 🔴 or 🟡 `[misread]` where it arose because the implementer misread code that had already shipped. A round's commit title now counts the marked findings, as in `Answer stability round 1: 1 🔴, 2 🟡 (1 misread); …`, and rule-hook's `round-title-counts` blocks a round title without that count. `score-outcomes` shows the count on each finished item, and lists it after the medians for each chunk whose round titles carry it, newest first. A chunk whose round titles predate this change reads as unknown.
- Sessions running at once no longer starve the machine. A new tool, `run-heavy <command>`, runs a heavy command, such as a full gate, mutation tests, an e2e run or a `docker build`, once it holds one of the machine's heavy slots: one per 4 CPUs, shared by every session. `CLAUDE.md` has every session run its heavy commands under it, and rule-hook blocks a `docker build` or a mutation run outside it. loop-claude now waits to start a worker while less than a quarter of the machine's memory is available.
+1 -1
View File
@@ -66,7 +66,7 @@ The rules only reviewers apply, for quality, comments, READMEs and prose, live i
- Run a project's language runtime, package manager, and task runner through its container/tooling setup, never directly on the host unless a decision puts them there. Example: `docker compose run --rm app <tool> …`.
- Tear down test containers after a run.
- Give every container you start a memory limit and a swap limit, as large as its work needs: `--memory 4g --memory-swap 4g` on `docker run`, or `mem_limit` and `memswap_limit` in a compose file, scripts included. rule-hook checks only the `docker run` and `docker create` in your Bash commands.
- A heavy command loads the machine for minutes: a full gate, mutation tests, an e2e run, a `docker build`. Run each one under `~/.claude/bin/run-heavy`, as in `~/.claude/bin/run-heavy ./ci.sh`. run-heavy waits for one of the heavy slots that every session on the machine shares. Run it in the background, since the wait can outlast the Bash tool's timeout.
- A heavy command loads the machine for minutes: a full gate, mutation tests, an e2e run, a `docker build`. Run each one under `~/.claude/bin/run-heavy`, as in `~/.claude/bin/run-heavy ./ci.sh`. run-heavy waits for one of the heavy slots that every session on the machine shares, and gives each container the command makes without limits the slot's share of memory. Run it in the background, since the wait can outlast the Bash tool's timeout.
- Delegate a task to a subagent only when you need just its conclusion: a search or read across many files, or work independent enough to run in parallel. Keep edits, and whatever you must check line by line, in your own thread: you would read a delegated diff again to check it. Score the task's L as for a `todo.md` item; a task that changes no file, such as a search, is L 1. `<model>` below is the project's model id, such as `claude-opus-5-5`: the one loop-claude's prompt names as the project's, or else your own. Launch the subagent with the model and effort of the first case that fits:
1. A reviewer, a comprehension seat or the security pass: those loop-claude's prompt gives it. Given none, pass no model or effort, so it runs on your model.
2. An escalated task, below: the rung loop-claude's prompt names for escalation, or case 3's rung where the subagent files or scores `todo.md` items and that rung is higher.
+2 -2
View File
@@ -104,7 +104,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure.
- `review-replay <pass> <model>` replays a review pass's archived rounds from the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again; `review-replay --help` states the replay. Each replayed reviewer skips every permission prompt, stability-reviewer has Bash, and an archived prompt may name paths on the host.
- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit, Write and Agent call, and `rule-hook --help` states the rules and the override.
- `run-heavy <command>` runs a heavy command once it holds one of the machine's heavy slots, one per 4 CPUs, which every session shares; `run-heavy --help` states the slots.
- `run-heavy <command>` runs a heavy command once it holds one of the machine's heavy slots, one per 4 CPUs, which every session shares; `run-heavy --help` states the slots, and the memory share it gives each container the command makes that lacks a limit.
- `score-outcomes [L E I S]` lists the finished items whose guessed scores sit nearest the given ones, and the shipped items the project's `docs/score-anchors.md` names for each level of L, E and I, each with what its chunk measured; then the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings; then each chunk's 🔴 and 🟡 findings with how many a reviewer marked `[misread]`, code health's counter-signal; `score-outcomes --help` states the measures.
- `settings_filter.py` is git's clean filter for `settings.json`: it drops `"model"` and sorts the keys. `.gitattributes` sets the filter `settings` on `settings.json`, and Install's `git config` line makes this script that filter.
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
@@ -125,7 +125,7 @@ pytest_add_cli_args=-k
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py`, `settings_filter.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `larv_review` imports `rank_todo.py`'s ranking, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and `loop_claude` runs `chunk-preflight` as a process of its own. `judge_replay` and `review_replay` share `rung_replay/`: the rung search, the argument reader, the candidate rungs and the exits. `rung_replay/` reads `loop_claude`'s retry count and models file. `docker_command.py` reads a docker command's subcommand and memory limits for `rule_hook`, and sits at the root so `run_heavy` can read them too. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude, `rule_hook`, both replay tools and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `machine_load/` takes the machine's heavy slots, its shared lock files, for `run_heavy`, and reads the memory share that loop-claude waits for. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py`, `settings_filter.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `larv_review` imports `rank_todo.py`'s ranking, `score_outcomes` imports `reading_cost`'s record path, `judge_replay` imports `loop_claude`'s judge, and `loop_claude` runs `chunk-preflight` as a process of its own. `judge_replay` and `review_replay` share `rung_replay/`: the rung search, the argument reader, the candidate rungs and the exits. `rung_replay/` reads `loop_claude`'s retry count and models file. The other non-test modules at `bin/`'s root, `docker_command.py` among them, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude, `rule_hook`, both replay tools and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `machine_load/` takes the machine's heavy slots, its shared lock files, for `run_heavy`, and reads the memory share that loop-claude waits for. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
## Tracking a new file
+1 -1
View File
@@ -13,7 +13,7 @@ You are an elite stability-focused code reviewer with deep expertise in defensiv
2. **Read every file under `~/.claude/principles/`**, the shared principles, and `~/.claude/skills/larv-review/standards/quality.md`. These are mandatory org guidelines; all code MUST comply with each that applies. A file's `Applies to` line says what it covers; one without the line covers every change. Re-read each time — they change. Flag violations prominently.
3. **Read README.md, AGENTS.md and `docs/decisions.md`** in the current project for project-specific conventions, build instructions, tooling, and the accepted trade-offs a reviewer should honor rather than re-flag. Note the documented test, lint, and static-analysis commands — you will run them.
4. **Verify, don't just read.** Run the test suite and static analysis/linters through the project's container tooling, unless a decision puts them on the host; tear down test containers afterwards. Run the full suite under `~/.claude/bin/run-heavy`. Run it in the background or with the Bash tool's longest timeout: that timeout includes the wait for a slot. A full-suite run catches broken callers that diff-reading misses. When a suspected bug is cheap to confirm, reproduce it — a scratch script, a targeted test run — before reporting it. Make scratch work with Bash, since Write is for your report alone. Put it in a directory you create for this run, and delete it when done; delete nothing else, since a directory the prompt names may hold other agents' files. Leave the working tree exactly as you found it. Anything you could not run goes in Coverage & Verification, never silently skipped.
5. **Mutation-test the changed lines.** Run the project's documented mutation command, or else a widely deployed mutation tool for the language: cargo-mutants for Rust, Infection for PHP, mutmut for Python, Stryker for JavaScript and TypeScript. Limit it to the lines the change adds or modifies, through the tool's own option where it has one (cargo-mutants `--in-diff <diff file>`, Infection `--git-diff-lines`, Stryker `--mutate <file>:<lines>`); otherwise mutate only the changed files and keep only the mutants on changed lines. Run it through the project's tooling and run-heavy, as in step 4, on a scratch copy: the tools write files. Report its wall time and mutant count in Coverage & Verification, or why nothing was mutated. Ignore a surviving mutant that leaves behaviour unchanged, or changes only message or log text no caller or user depends on. Report one 🟡 per test to add or strengthen, naming the surviving mutants it would catch.
5. **Mutation-test the changed lines.** Run the project's documented mutation command, or else a widely deployed mutation tool for the language: cargo-mutants for Rust, Infection for PHP, mutmut for Python, Stryker for JavaScript and TypeScript. Limit it to the lines the change adds or modifies, through the tool's own option where it has one (cargo-mutants `--in-diff <diff file>`, Infection `--git-diff-lines`, Stryker `--mutate <file>:<lines>`); otherwise mutate only the changed files and keep only the mutants on changed lines. Run it through the project's tooling and run-heavy, as in step 4, on a scratch copy: the tools write files. Under run-heavy, a container that sets no memory limit gets the slot's share, `RUN_HEAVY_MEMORY` in the command's environment: set the tool's concurrency so that many test runs at once fit in it. Report its wall time and mutant count in Coverage & Verification, or why nothing was mutated. Ignore a surviving mutant that leaves behaviour unchanged, or changes only message or log text no caller or user depends on. Report one 🟡 per test to add or strengthen, naming the surviving mutants it would catch.
6. **Two passes minimum, then hunt absences.**
- *Breadth:* every hunk in every changed file, no skimming. Track which files you have covered; batch large diffs, never truncate the review.
- *Depth:* trace the riskiest data paths end-to-end (input → logic → persistence/response), the interactions between changed pieces, and between the change and config (timeout hierarchies, retries × idempotency, queue worker counts, env parity).
+23 -6
View File
@@ -35,20 +35,37 @@ def unlimited(arguments: list[str]) -> bool:
The container's own arguments, past its image, are read too: telling them apart needs every value-taking option of docker run. So a -m 4g there reads as a limit, while python -m pytest does not, since pytest is no size.
"""
subcommand = docker_subcommand(arguments)
limited = given(arguments, ("-m", "--memory"), SIZE) and given(arguments, ("--memory-swap",), SWAP)
limited = option_value(arguments, ("-m", "--memory"), SIZE) and option_value(arguments, ("--memory-swap",), SWAP)
return subcommand in STARTS_CONTAINER and not limited and not helped(arguments, subcommand)
def given(arguments: list[str], options: tuple[str, ...], pattern: str) -> bool:
"""Whether the arguments give one of the options a value pattern matches, as its next word or attached, as in -m4g or --memory=4g."""
attached = re.compile("|".join(re.escape(option) + ("" if len(option) == 2 else "=") + f"(?:{pattern})" for option in options))
return any(attached.fullmatch(word) or (word in options and re.fullmatch(pattern, following)) for word, following in zip(arguments, [*arguments[1:], ""]))
def capped(arguments: list[str], share: str) -> list[str]:
"""docker's arguments with the limits a container they make lacks, put right after the subcommand: share for each, except a swap limit beside the container's own memory limit, which takes that limit, as docker refuses less swap than memory."""
if not unlimited(arguments):
return arguments
memory = option_value(arguments, ("-m", "--memory"), SIZE)
after = arguments.index(docker_subcommand(arguments)[-1]) + 1
limits = ["--memory-swap", memory] if memory else ["--memory", share, "--memory-swap", share]
return [*arguments[:after], *limits, *arguments[after:]]
def option_value(arguments: list[str], options: tuple[str, ...], pattern: str) -> str | None:
"""The value the arguments give one of the options, where pattern matches it, as its next word or attached, as in -m4g or --memory=4g; None where they give none."""
for word, following in zip(arguments, [*arguments[1:], ""]):
for option in options:
attached = word.removeprefix(option if len(option) == 2 else option + "=")
if attached != word and re.fullmatch(pattern, attached):
return attached
if word == option and re.fullmatch(pattern, following):
return following
return None
def unlimited_container(words: list[str]) -> bool:
"""Whether the simple command, directly or through a launcher, starts a container that lacks a memory limit or a swap limit."""
docker = launched(words, {"docker"})
return docker is not None and unlimited(docker.arguments)
# Under run-heavy, its docker adds the limits a container lacks.
return docker is not None and "run-heavy" not in docker.launchers and unlimited(docker.arguments)
def helped(arguments: list[str], subcommand: list[str]) -> bool:
+2 -2
View File
@@ -117,12 +117,12 @@ class CommandBreaches(unittest.TestCase):
self.assertEqual(checked(command), [])
def test_a_docker_run_or_create_without_a_memory_limit_breaks_docker_run_memory(self):
for command in ["docker run --rm node:24.22.1-alpine3.24 node x.js", "cd /w && docker run -v /w:/w alpine:3.22.2 true", "docker create alpine:3.22.2", "docker container run alpine:3.22.2 true", "docker container create alpine:3.22.2", "nice -n 19 docker run alpine:3.22.2 true", "docker -H unix:///d.sock run alpine:3.22.2 true", "docker run --memory-swap 4g alpine:3.22.2 true", "docker run -m 4g alpine:3.22.2 true", "docker run --memory=4g alpine:3.22.2 sh -c 'x --memory-swap 4g'", "docker run -m 4g alpine:3.22.2 tool --memory-swap", "docker run -m 4g --memory-swap -1 alpine:3.22.2 true", "docker run -m 4g --memory-swap=0 alpine:3.22.2 true", "~/.claude/bin/run-heavy docker run alpine:3.22.2 true", "docker run --rm python:3.13.1 python -m pytest", "docker run alpine:3.22.2 git commit -m x", "timeout 60 docker run alpine:3.22.2 true", "docker run node:24.22.1-alpine3.24 node --help", "docker run alpine:3.22.2 --help", "docker run -m 4g alpine:3.22.2 sleep 60"]:
for command in ["docker run --rm node:24.22.1-alpine3.24 node x.js", "cd /w && docker run -v /w:/w alpine:3.22.2 true", "docker create alpine:3.22.2", "docker container run alpine:3.22.2 true", "docker container create alpine:3.22.2", "nice -n 19 docker run alpine:3.22.2 true", "docker -H unix:///d.sock run alpine:3.22.2 true", "docker run --memory-swap 4g alpine:3.22.2 true", "docker run -m 4g alpine:3.22.2 true", "docker run --memory=4g alpine:3.22.2 sh -c 'x --memory-swap 4g'", "docker run -m 4g alpine:3.22.2 tool --memory-swap", "docker run -m 4g --memory-swap -1 alpine:3.22.2 true", "docker run -m 4g --memory-swap=0 alpine:3.22.2 true", "docker run --rm python:3.13.1 python -m pytest", "docker run alpine:3.22.2 git commit -m x", "timeout 60 docker run alpine:3.22.2 true", "docker run node:24.22.1-alpine3.24 node --help", "docker run alpine:3.22.2 --help", "docker run -m 4g alpine:3.22.2 sleep 60"]:
with self.subTest(command):
self.assertEqual(checked(command), ["docker-run-memory"])
def test_a_docker_run_with_a_memory_limit_or_another_docker_call_passes_docker_run_memory(self):
for command in ["docker run -m 4g --memory-swap 4g alpine:3.22.2 true", "docker run -m4g --memory-swap=4g alpine:3.22.2 true", "docker run --memory 512M --memory-swap 1g alpine:3.22.2 true", "docker run -m 1073741824 --memory-swap 2g alpine:3.22.2 true", "timeout 60 docker run -m 4g --memory-swap 4g alpine:3.22.2 true", "docker run --rm --memory=1.5GiB --memory-swap=1.5GiB alpine:3.22.2 true", "docker container create -m 1g --memory-swap 1g alpine:3.22.2", "docker run --help", "docker container run --help", "docker --help", "docker ps", "docker images", "docker container ls", "docker compose run --rm app go vet", "docker exec box true", "echo docker run alpine:3.22.2"]:
for command in ["docker run -m 4g --memory-swap 4g alpine:3.22.2 true", "docker run -m4g --memory-swap=4g alpine:3.22.2 true", "docker run --memory 512M --memory-swap 1g alpine:3.22.2 true", "docker run -m 1073741824 --memory-swap 2g alpine:3.22.2 true", "timeout 60 docker run -m 4g --memory-swap 4g alpine:3.22.2 true", "docker run --rm --memory=1.5GiB --memory-swap=1.5GiB alpine:3.22.2 true", "docker container create -m 1g --memory-swap 1g alpine:3.22.2", "docker run --help", "docker container run --help", "docker --help", "docker ps", "docker images", "docker container ls", "docker compose run --rm app go vet", "docker exec box true", "echo docker run alpine:3.22.2", "~/.claude/bin/run-heavy docker run alpine:3.22.2 true"]:
with self.subTest(command):
self.assertEqual(checked(command), [])
+2 -1
View File
@@ -5,7 +5,8 @@ import sys
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from machine_load.memory import read_memory
from machine_load.slots import usable_cpus
from run_heavy import main
sys.exit(main.main(sys.argv[1:], os.environ, usable_cpus()))
sys.exit(main.main(sys.argv[1:], os.environ, usable_cpus(), read_memory()))
+23
View File
@@ -0,0 +1,23 @@
"""The docker a heavy command finds first on its PATH: it gives a container the command makes the limits it lacks, the slot's memory share in RUN_HEAVY_MEMORY, then runs the real docker."""
import os
import pathlib
import shutil
import sys
import typing
from docker_command import capped
FOLDER = pathlib.Path(__file__).resolve().parent / "path_entry"
SHARE = "RUN_HEAVY_MEMORY"
NOT_FOUND = 127
def main(argv: list[str], env: typing.Mapping[str, str]) -> int:
"""Replaces this process with the real docker; returns only where none is found."""
path = os.pathsep.join(entry for entry in env.get("PATH", "").split(os.pathsep) if pathlib.Path(entry or ".").resolve() != FOLDER)
docker = shutil.which("docker", path=path)
if docker is None:
print("run-heavy: docker: command not found", file=sys.stderr)
return NOT_FOUND
share = env.get(SHARE)
os.execve(docker, [docker, *(capped(argv, share) if share else argv)], {**env, "PATH": path})
+17 -2
View File
@@ -9,6 +9,12 @@ $XDG_STATE_HOME/run-heavy (default ~/.local/state/run-heavy). While every slot i
says so once on stderr and waits. It then replaces itself with the command. The slot's lock stays held
until the command, and every process that inherited the lock, exits. Its exit code is the command's.
Each container the command makes with docker run or docker create gets a memory limit and a swap limit,
both the slot's share, where it sets neither: three quarters of the machine's memory, split across the
slots. A docker first on the command's PATH adds them, and passes every other call on unchanged. A
container that sets its own memory limit keeps it, with a swap limit as large. The share is in
RUN_HEAVY_MEMORY. A docker compose service, or a docker named by its full path, gets no limit.
A command it runs carries the slot's number in RUN_HEAVY_SLOT, and run-heavy called with that set
runs its command at once, so a gate that wraps its own steps cannot wait on itself.
@@ -21,6 +27,8 @@ import sys
import typing
from failure import Failure
from run_heavy.capped_docker import FOLDER, SHARE
from machine_load.memory import Memory
from machine_load.slots import slot_count, slot_folder, waited_slot
OWN_FAILURE = 125
@@ -30,7 +38,7 @@ HELD = "RUN_HEAVY_SLOT"
USAGE = "usage: run-heavy <command> [<arg>...]"
def main(argv: list[str], env: typing.Mapping[str, str], cpus: int | None) -> int:
def main(argv: list[str], env: typing.Mapping[str, str], cpus: int | None, memory: Memory | None) -> int:
if argv[:1] in (["-h"], ["--help"]):
print(__doc__.strip())
return 0
@@ -42,10 +50,17 @@ def main(argv: list[str], env: typing.Mapping[str, str], cpus: int | None) -> in
if isinstance(slot, Failure):
print(f"run-heavy: {slot.reason}", file=sys.stderr)
return OWN_FAILURE
env = {**env, HELD: str(slot)}
env = {**env, HELD: str(slot), **capping(memory, slot_count(cpus), env)}
return run(argv, env)
def capping(memory: Memory | None, slots: int, env: typing.Mapping[str, str]) -> dict[str, str]:
"""What the command's environment needs for its docker to cap each container it makes at the slot's share: three quarters of the memory, which the quarter loop-claude keeps free for a worker leaves, split across the slots. Nothing where the memory is unknown, as off Linux."""
if memory is None:
return {}
return {SHARE: f"{memory.total * 3 // 4 // slots // 1024}m", "PATH": os.pathsep.join([str(FOLDER), env.get("PATH", "")])}
def held_slot(folder: pathlib.Path, slots: int) -> int | Failure:
"""The number of the slot this call holds, its lock left for the command to inherit, or why no slot could be taken."""
try:
+10
View File
@@ -0,0 +1,10 @@
#!/usr/bin/env python3
import os
import pathlib
import sys
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2]))
from run_heavy import capped_docker
sys.exit(capped_docker.main(sys.argv[1:], os.environ))
+32
View File
@@ -10,6 +10,7 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from run_heavy import main
from machine_load.memory import read_memory
from machine_load.slots import slot_count, usable_cpus
TOOL = pathlib.Path(__file__).resolve().parents[1] / "run-heavy"
@@ -82,6 +83,37 @@ class RunHeavy(unittest.TestCase):
self.assertEqual((denied.returncode, denied.stdout), (126, ""))
self.assertIn(unrunnable.name, denied.stderr)
def fake_docker(self) -> dict[str, str]:
"""An environment whose PATH finds, ahead of any real one, a docker that prints the arguments it was given."""
folder = tempfile.TemporaryDirectory()
self.addCleanup(folder.cleanup)
docker = pathlib.Path(folder.name) / "docker"
docker.write_text("#!/bin/sh\necho \"$@\"\n")
docker.chmod(0o755)
return {**self.env, "PATH": f"{folder.name}:{self.env['PATH']}"}
def test_a_docker_run_without_limits_gets_three_quarters_of_the_memory_split_across_the_slots(self):
memory = read_memory()
assert memory is not None
share = f"{memory.total * 3 // 4 // slot_count(usable_cpus()) // 1024}m"
result = self.run_tool("docker", "run", "--rm", "img", "true", env=self.fake_docker())
self.assertEqual((result.returncode, result.stdout), (0, f"run --memory {share} --memory-swap {share} --rm img true\n"))
def test_a_docker_call_with_its_own_limits_or_that_starts_no_container_is_passed_on_unchanged(self):
env = self.fake_docker()
for args in [["run", "-m", "40g", "--memory-swap", "40g", "img"], ["ps", "-a"], ["build", "."]]:
with self.subTest(args):
result = self.run_tool("docker", *args, env=env)
self.assertEqual((result.returncode, result.stdout), (0, " ".join(args) + "\n"))
def test_a_capped_docker_that_finds_no_docker_exits_127_saying_so(self):
folder = tempfile.TemporaryDirectory()
self.addCleanup(folder.cleanup)
(pathlib.Path(folder.name) / "python3").symlink_to(sys.executable)
result = self.run_tool("docker", "ps", env={**self.env, "PATH": folder.name})
self.assertEqual((result.returncode, result.stdout), (127, ""))
self.assertIn("docker", result.stderr)
def test_no_command_exits_125_with_the_usage_and_help_exits_0(self):
bare = self.run_tool()
self.assertEqual((bare.returncode, bare.stdout), (125, ""))
+28
View File
@@ -0,0 +1,28 @@
import unittest
from docker_command import capped
class Capped(unittest.TestCase):
def test_a_container_without_limits_gets_the_share_for_both(self):
for given, expected in [
(["run", "--rm", "img", "true"], ["run", "--memory", "6144m", "--memory-swap", "6144m", "--rm", "img", "true"]),
(["-H", "unix:///d.sock", "run", "img"], ["-H", "unix:///d.sock", "run", "--memory", "6144m", "--memory-swap", "6144m", "img"]),
(["container", "create", "img"], ["container", "create", "--memory", "6144m", "--memory-swap", "6144m", "img"]),
(["run", "img", "python", "-m", "pytest"], ["run", "--memory", "6144m", "--memory-swap", "6144m", "img", "python", "-m", "pytest"]),
]:
with self.subTest(given):
self.assertEqual(capped(given, "6144m"), expected)
def test_a_container_s_own_memory_limit_is_kept_and_caps_its_swap(self):
self.assertEqual(capped(["run", "-m", "10g", "img"], "6144m"), ["run", "--memory-swap", "10g", "-m", "10g", "img"])
self.assertEqual(capped(["run", "--memory=12g", "img"], "6144m"), ["run", "--memory-swap", "12g", "--memory=12g", "img"])
def test_a_container_with_both_limits_or_another_docker_call_is_left_alone(self):
for given in [["run", "-m", "10g", "--memory-swap", "12g", "img"], ["ps"], ["build", "."], ["run", "--help"], ["compose", "run", "app"], ["exec", "box", "true"]]:
with self.subTest(given):
self.assertEqual(capped(given, "6144m"), given)
if __name__ == "__main__":
unittest.main()
+4 -2
View File
@@ -76,9 +76,11 @@ The cap is 5, `bin/todo_table.py`'s `BUNDLE_MAX_R`, from the same chunks: by the
2026-10-23, lilleman. Serves goal 1.5. On 2026-10-22 ai-sandbox, with 8 CPUs and 16 GiB, thrashed for hours: four to six loops ran mutation tests, full gates, `docker build`s and QEMU-booting e2e tests at once, and nothing told one loop how busy the machine was. lilleman chose both limits; no standard sets either, and GNU parallel's `--memfree`, which does the memory wait's job, has no default. A slot is a file lock, so a session that dies frees its slot. The memory wait reads `/proc/meminfo` only, and off Linux a worker starts at once. `bin/machine_load/` holds the slots and the memory share, `bin/run-heavy` takes a slot, `CLAUDE.md` → "Working with the codebase" has every session use it, and loop-claude's worker waits for the share. rule-hook's rule `run-heavy` blocks a `docker build`, and a run of a mutation tool stability-reviewer names, outside run-heavy, whether direct or inside a container. No program can tell a gate script from a light one, so only `CLAUDE.md` asks a session to run a gate under run-heavy. Valid while one machine runs several loops at once.
## machine load: every container a session starts has a memory limit and a swap limit as large as its work needs, which rule-hook's `docker-run-memory` enforces on `docker run` and `docker create`, and no daemon-wide cap limits all containers together
## machine load: every container a session starts has a memory limit and a swap limit as large as its work needs, which rule-hook's `docker-run-memory` enforces on `docker run` and `docker create`; under run-heavy, a container that lacks them gets the slot's share of three quarters of the memory; and no daemon-wide cap limits all containers together
2026-10-24, lilleman. Serves goal 1.5. On 2026-10-23 ai-sandbox, then with 8 CPUs and 7.6 GiB of memory, hung until a hard reset, although its one heavy command held a slot. Beside that mutation run, a reviewer's stress probes ran in `docker run` with no memory limit, each to 2 GiB. A limit makes a runaway container fail alone instead of taking the machine with it. A swap limit is needed too: without one, docker lets a container swap as much again as its memory limit, and swapping is how the machine hung. The session sizes both limits, since lilleman wants a project to run heavier work where it needs to. rule-hook reads only the `docker run` and `docker create` in a session's Bash commands. A container a script starts is out of its sight, and so is a compose service, whose limits live in its compose file. `CLAUDE.md` → "Working with the codebase" asks a session for limits there, and todo item 227 caps the containers a heavy command starts under run-heavy.
2026-10-24, lilleman. Serves goal 1.5. On 2026-10-23 ai-sandbox, then with 8 CPUs and 7.6 GiB of memory, hung until a hard reset, although its one heavy command held a slot. Beside that mutation run, a reviewer's stress probes ran in `docker run` with no memory limit, each to 2 GiB. A limit makes a runaway container fail alone instead of taking the machine with it. A swap limit is needed too: without one, docker lets a container swap as much again as its memory limit, and swapping is how the machine hung. The session sizes both limits, since lilleman wants a project to run heavier work where it needs to. rule-hook reads only the `docker run` and `docker create` in a session's Bash commands. A container a script starts is out of its sight, and so is a compose service, whose limits live in its compose file. `CLAUDE.md` → "Working with the codebase" asks a session for limits there.
The heavy command in that hang started its container from the project's own script, out of rule-hook's sight. So run-heavy puts first on its command's `PATH` a `docker`, `bin/run_heavy/path_entry/docker`, which gives each `docker run` or `docker create` that lacks a limit the slot's share: three quarters of MemTotal, which the quarter a worker waits for leaves, split evenly across the slots (todo item 227). A container that sets its own memory limit keeps it, with a swap limit as large, since docker refuses less swap than memory. run-heavy cannot place the command in a cgroup instead: none is delegated to the user, and dockerd starts the containers outside the command's process tree. A compose service, and a docker named by its full path, keep no cap.
A daemon-wide cap would see every container: dockerd's `cgroup-parent`, given a memory limit. lilleman turned it down, since it is host config and would cap a project's heavy work too. Valid while sessions start their work in containers on a machine they share.
-9
View File
@@ -16,7 +16,6 @@
| 59 | 0.4.0 | | | **Bound a run of unfinished verdicts on one chunk, so a worker and judge that disagree cannot cycle unattended without end.** | 2 | 1 | 3 | 4 | 3 | 1.4 by 6 | 6.0 |
| 167 | 0.4.0 | | | **Refuse a `/security-review` or `/claude-security` Skill call below the project's rung, as rule-hook refuses such an Agent call.** | 3 | 2 | 3 | 6 | 2 | 1.1 by 4 | 5.6 |
| 6 | 0.4.0 | | decision | **Move what `loop-claude` writes out of `~/.claude.json`, and ask the human about each write that must stay.** | 3 | 3 | 3 | 7 | 3 | 1.1 by 5 | 5.3 |
| 227 | 0.4.0 | | | **Cap the memory of each container a heavy command starts under run-heavy at its slot's share, and have stability-reviewer size a mutation run's concurrency to fit that cap.** | 3 | 2 | 3 | 6 | 5 | 1.5 by 5 | 1.9 |
| 1 | 0.5.0 | | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 1 | 2 | 1 | 2 | 1 | | 0.0 |
| 219 | 0.5.0 | | | **Name the project's rung `project` in `bin/judge_replay/main.py`, `bin/subagent_rung/main.py` and `bin/review_replay/main.py`, in place of `anchor`, as `bin/model_routing/ladder.py` and `bin/loop_claude` name it.** | 1 | 1 | 3 | 3 | 2 | code health by 2 | 12.0 |
| 224 | 0.5.0 | | | **Split `docs/decisions.md` → `todo.md: one table of items, …`'s 420-word scoring paragraph into one paragraph per choice, and drop the sentences that tell how the goal weights changed.** | 1 | 1 | 1 | 1 | 2 | code health by 2 | 12.0 |
@@ -87,14 +86,6 @@ rule-hook checks Agent calls (item 163) but no Skill calls. A worker that invoke
Claude Code owns that file and writes it. `trust` reads it, changes it and writes it back, so a write that lands in between is lost.
### 227. Cap the memory of each container a heavy command starts under run-heavy at its slot's share, and have stability-reviewer size a mutation run's concurrency to fit that cap.
On 2026-10-23 ai-sandbox, then with 8 CPUs and 7.6 GiB, hung in swap until a hard reset. One heavy command held its slot, as the decision `machine load: a heavy command waits for one of the machine's heavy slots, …` asks: adf-codec's stability reviewer ran Stryker with concurrency 4, each copy running the whole `node --test` suite, in one container that reached 4 GiB. The container was started by the project's own script, so rule-hook's `docker-run-memory` can't see one like it. The slots cap how many heavy commands run at once, not how much memory one takes.
run-heavy cannot place the command in a cgroup: none is delegated to the user, and dockerd starts the containers outside the command's process tree. So run-heavy puts first on the command's `PATH` a `docker` that adds `--memory` and `--memory-swap`, both at the slot's share, to each `docker run`, `docker create` and `docker container run|create` that leaves either without a limit, then runs the real docker. It tells such a call apart with `bin/docker_command.py`'s `unlimited`, which rule-hook's `docker-run-memory` reads too. A container that sets its own limits keeps them, so a project runs heavier work where it needs to (lilleman, 2026-10-24). A compose service, and a docker named by its full path, keep no cap.
The slot's share is a new choice, to record in the decision `machine load: every container a session starts has …`: that decision's elder, `machine load: a heavy command waits …`, starts a worker only while a quarter of the memory is available, so the share is the other three quarters of MemTotal, split evenly across the slots. Update both decisions, run-heavy's help and stability-reviewer's mutation step, which sizes Stryker's or mutmut's concurrency so the run fits the cap. Once run-heavy caps them, let `docker-run-memory` pass a `docker run` under run-heavy, which `shell_command.launched` reports among a call's launchers. Test first in `bin/run_heavy/test_run_heavy.py`.
### 1. Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.
It lives under `~/.claude`, which `.gitignore` keeps from the remote, so a lost disk loses every unattended run's record. Back it up only to a destination private to me and encrypted at rest: the worker runs with permissions bypassed, so the archived tool results can hold credentials it read.