Judge a worker that stalls before its result with a headless claude, and report the verdict without a kill #633
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- loop-claude asks a headless claude whether a worker that went quiet before its result is stuck, and prints the verdict and appends it to stuck-workers.md, without killing anything.
|
||||
- loop-claude stops the background shells that hold a worker after its final result, once they have been quiet for a few minutes, and records each stop
|
||||
- Add `do-epic-e2e`, a command that runs `do-epic` end to end against a sandbox epic on a real Jira site and deletes the epic afterwards.
|
||||
- `do-epic <EPIC-KEY>` plans the epic before it starts chunks: it files a sub-task per repo under an open issue that changes several repos, labels each other unlabelled issue with where it is worked, asking on the terminal where the repo rule leaves that unclear, and puts the scored issues it filed in order of score.
|
||||
|
||||
@@ -18,12 +18,12 @@ RECORD = "lingering-workers.md"
|
||||
|
||||
|
||||
def stopped_shells(proc: Streaming, printed_result: bool, record: pathlib.Path, effects: Effects) -> None:
|
||||
"""Stops the shells of a claude that is still alive CHECK_EVERY seconds after it printed its final result (printed_result), saying each stop on the terminal and in record. A claude whose latest line is not a result is working and keeps its shells, and so does one with a shell that wrote output within QUIET seconds or runs a busy tool, as `Shell.runs_busy_tool` says. A subagent's shells are children of the claude too, so they count like its own: a subagent that prints keeps them, and one inside a single quiet call loses it."""
|
||||
"""Stops the shells of a claude that is still alive CHECK_EVERY seconds after it printed its final result (printed_result), saying each stop on the terminal and in record. A claude whose latest line is not a result is working and keeps its shells, and so does one with a shell that wrote output within QUIET seconds or runs a busy tool or holds a heavy slot, as `Shell.runs_busy_tool` and `Shell.holds_slot` say. A subagent's shells are children of the claude too, so they count like its own: a subagent that prints keeps them, and one inside a single quiet call loses it."""
|
||||
if not printed_result:
|
||||
return
|
||||
now = effects.now()
|
||||
shells = proc.shells()
|
||||
if any(now - shell.written <= QUIET or shell.runs_busy_tool for shell in shells):
|
||||
if any(now - shell.written <= QUIET or shell.runs_busy_tool or shell.holds_slot for shell in shells):
|
||||
return
|
||||
for shell in shells:
|
||||
proc.stop(shell)
|
||||
|
||||
@@ -122,7 +122,7 @@ def work(cycle: Cycle, answered: QuotedAnswer | TypedAnswer | None, stepped: boo
|
||||
if isinstance(answered, QuotedAnswer):
|
||||
say(f"\n=== {cycle.prompts.quote_sources} answered: {answered.question} ===\n{answered.answer}", cycle.effects, stdout=True)
|
||||
# worker_start runs inside the lambda: a failed attempt may have opened the session.
|
||||
if stop := retried("worker", lambda: run_worker(cycle.root, worker_start(cycle, answered, stepped), worker_environment(cycle), cycle.project_archive, cycle.effects), cycle.effects):
|
||||
if stop := retried("worker", lambda: run_worker(cycle.root, worker_start(cycle, answered, stepped), worker_environment(cycle), cycle.project_archive, cycle.project.rung if cycle.project else cycle.route.rung, cycle.effects), cycle.effects):
|
||||
return stop
|
||||
path = transcript_file(cycle.session)
|
||||
return WorkerRan(path) if path is not None else Stop(1, f"claude left no transcript for session {cycle.session.id}")
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
"""Reports whether a worker that has gone quiet before its final result is stuck, as a headless claude judges it.
|
||||
|
||||
A worker can stall on a command that hangs, as a full test run did on a shell that ignored SIGINT (item 402). The check runs on the tick `lingering_workers` uses: CHECK_EVERY seconds after the worker's last line. It looks for two signs in each of the worker's shells. One is a shell that used no CPU since the last check and wrote no output for QUIET seconds. The other is a shell older than OLD. A shell that runs a busy tool, as `Shell.runs_busy_tool` says, shows neither sign, since waiting for a slot or a merge is work. A shell that holds a heavy slot can show them, since a hung test run holds one.
|
||||
|
||||
Where a shell shows a sign, a headless claude judges the worker from each shell's command, processes and output tail. These hold what the worker's tmux pane shows, so each chunk's loop checks its own worker, and no process watches a whole tmux session. A shell is judged again only where it used CPU since its last judgment and then went idle, so a stall that lasts all night costs one call.
|
||||
|
||||
The check only reports, on the terminal and in RECORD in the project's transcript archive. Killing the process it names waits until the recorded verdicts match what the human would have done.
|
||||
"""
|
||||
import dataclasses
|
||||
import json
|
||||
import pathlib
|
||||
import typing
|
||||
|
||||
from agent_cycle.lingering_workers import CHECK_EVERY, QUIET
|
||||
from claude_code.reply import claude_reply, Limited
|
||||
from loop_effects.effects import Effects, say, Streaming
|
||||
from failure import Failure
|
||||
from model_routing.ladder import Rung
|
||||
from loop_effects.child_shells import Shell
|
||||
from claude_code.entries import structured_output
|
||||
from text_files.utc_stamp import utc
|
||||
|
||||
# Seconds after which a shell holds a worker longer than a chunk usually runs.
|
||||
OLD = 4 * 3600
|
||||
RECORD = "stuck-workers.md"
|
||||
VERDICTS = ("working", "waiting on a person", "stuck")
|
||||
SCHEMA = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"verdict": {"enum": list(VERDICTS)},
|
||||
"process": {"type": "integer", "description": "The pid of the process that holds the worker, from the evidence."},
|
||||
"why": {"type": "string", "description": "One sentence on why that process holds it."},
|
||||
},
|
||||
"required": ["verdict", "process", "why"],
|
||||
"additionalProperties": False,
|
||||
}
|
||||
PROMPT = f"""A headless Claude Code worker hasn't printed anything for {CHECK_EVERY} seconds. Judge from its shells below whether it is `working`, `waiting on a person` or `stuck`. Name the pid of the process it waits on, and say why.
|
||||
- Working: it waits on a process that will finish, such as a test run that still writes output or uses CPU, or run-heavy waiting for a slot.
|
||||
- Waiting on a person: it waits on input or an approval only a person can give, such as a prompt read from a terminal.
|
||||
- Stuck: it waits on a process that will never finish on its own, such as a command waiting on a child that never exits.
|
||||
"""
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class Watch:
|
||||
"""What checks a worker for stuck shells: the rung its investigations run on, the project's transcript archive, each shell's CPU seconds at the last check, and the CPU seconds of each shell that showed a sign at the latest judgment."""
|
||||
rung: Rung
|
||||
archive: pathlib.Path
|
||||
cpu: dict[int, float] = dataclasses.field(default_factory=dict)
|
||||
judged: dict[int, float] = dataclasses.field(default_factory=dict)
|
||||
|
||||
def check(self, proc: Streaming, root: str, effects: Effects) -> None:
|
||||
"""Has a headless claude judge the worker whose claude is proc, where a shell shows a sign and is due a judgment. The verdict, or why there is none, goes to the terminal and to RECORD."""
|
||||
now = effects.now()
|
||||
shells = proc.shells()
|
||||
before, self.cpu = self.cpu, {shell.pid: shell.cpu for shell in shells}
|
||||
signs = {shell.pid: sign for shell in shells if (sign := sign_of(shell, before.get(shell.pid), now))}
|
||||
# A shell is judged again only where it used CPU since its judgment and then went idle, so one stall costs one call, and a stall after a `working` verdict still gets one.
|
||||
if not any(shell.pid not in self.judged or self.judged[shell.pid] != shell.cpu and idle(shell, before.get(shell.pid), now) for shell in shells if shell.pid in signs):
|
||||
return
|
||||
judged = investigated(shells, self.rung, root, effects)
|
||||
if isinstance(judged, Failure):
|
||||
line = f"stuck check: could not judge whether the worker is stuck: {judged.reason}"
|
||||
else:
|
||||
self.judged = {shell.pid: shell.cpu for shell in shells if shell.pid in signs}
|
||||
line = f"stuck check: a headless claude judged the worker {judged['verdict']}, naming process {judged['process']} ({command_of(judged['process'], shells)}): {judged['why']}"
|
||||
line = " ".join(line.split()) + "".join(f"\n - {' '.join(sign.split())}" for sign in signs.values())
|
||||
say(line, effects, stdout=not isinstance(judged, Failure))
|
||||
record = self.archive / RECORD
|
||||
try:
|
||||
record.parent.mkdir(parents=True, exist_ok=True)
|
||||
with record.open("a") as handle:
|
||||
handle.write(f"- {utc(now)} {line}\n")
|
||||
except OSError as failure:
|
||||
say(f"could not record the stuck check in {record}: {failure.strerror}", effects)
|
||||
|
||||
|
||||
def sign_of(shell: Shell, cpu_before: float | None, now: float) -> str:
|
||||
"""The sign that shell may hold its worker stuck, given the CPU seconds it had used at the last check; empty where it shows none."""
|
||||
if shell.runs_busy_tool:
|
||||
return ""
|
||||
quiet = idle(shell, cpu_before, now)
|
||||
if not (quiet or now - shell.started > OLD):
|
||||
return ""
|
||||
return f"shell {shell.pid} has run {int(now - shell.started)} s and written no output for {int(now - shell.written)} s{f', and used no CPU for {CHECK_EVERY} s' if quiet else ''}: {shell.command}"
|
||||
|
||||
|
||||
def idle(shell: Shell, cpu_before: float | None, now: float) -> bool:
|
||||
"""Whether shell used no CPU since the last check, when it had used cpu_before, and wrote no output for QUIET seconds."""
|
||||
return cpu_before == shell.cpu and now - shell.written > QUIET
|
||||
|
||||
|
||||
def command_of(pid: int, shells: list[Shell]) -> str:
|
||||
"""The command line of the process pid among shells and the processes below them, as the evidence shows it; `not in the evidence` where neither holds pid, and `no command line` where its line is empty, as a defunct process's is."""
|
||||
for shell in shells:
|
||||
if shell.pid == pid:
|
||||
return shell.command
|
||||
for line in shell.processes:
|
||||
if line.startswith(f"{pid} "):
|
||||
return line.split(" ", 4)[4] or "no command line"
|
||||
return "not in the evidence"
|
||||
|
||||
|
||||
def investigated(shells: list[Shell], rung: Rung, root: str, effects: Effects) -> dict[str, typing.Any] | Failure:
|
||||
"""A headless claude's verdict on the worker holding shells, as SCHEMA has it, or why there is none. It runs without tools, so it judges only the evidence it is handed."""
|
||||
evidence = "\n\n".join(f"Shell {shell.pid}: {shell.command}\nIts processes, as pid, state, CPU and command line:\n{chr(10).join(shell.processes) or 'none'}\nThe end of its output:\n{shell.output or 'none'}" for shell in shells)
|
||||
reply = claude_reply(effects.claude, ["--json-schema", json.dumps(SCHEMA), *rung.flags(), "--tools", ""], root, PROMPT + "\n" + evidence, "the stuck check")
|
||||
if isinstance(reply, Limited):
|
||||
return Failure(f"claude hit a limit: {reply.said}")
|
||||
if isinstance(reply, Failure):
|
||||
return reply
|
||||
output = structured_output(reply)
|
||||
if not (isinstance(output, dict) and output.get("verdict") in VERDICTS and isinstance(output.get("process"), int) and isinstance(output.get("why"), str)):
|
||||
return Failure(f"answered outside its schema: {output!r}")
|
||||
return output
|
||||
@@ -0,0 +1,132 @@
|
||||
import contextlib
|
||||
import dataclasses
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from loop_claude.test_loop_claude import FakeEffects, NOW, ran, Streamed
|
||||
from model_routing.ladder import Rung
|
||||
from loop_effects.child_shells import Shell
|
||||
from agent_cycle import stuck_workers, worker
|
||||
from text_files.utc_stamp import utc
|
||||
|
||||
RUNG = Rung("claude-opus-5-5", "medium")
|
||||
ASSISTANT = {"type": "assistant", "message": {"content": []}}
|
||||
|
||||
|
||||
def verdict(said="stuck", process=8, why="pytest waits on a shell that ignores SIGINT"):
|
||||
return ran(json.dumps({"type": "result", "structured_output": {"verdict": said, "process": process, "why": why}}))
|
||||
|
||||
|
||||
def shell(pid=7, cpu=2.0, started=NOW - 1000, written=NOW - 1000, runs_busy_tool=False, processes=("8 S 2.0 s python3 -m pytest", "9 S 0.0 s sh -c trap '' INT; sleep 999"), output="collected 40 items", holds_slot=False):
|
||||
return Shell(pid=pid, started=started, command="bin/run-tests", written=written, runs_busy_tool=runs_busy_tool, cpu=cpu, processes=processes, output=output, holds_slot=holds_slot)
|
||||
|
||||
|
||||
def sign(run, quiet, idle=True):
|
||||
return f"\n - shell 7 has run {run} s and written no output for {quiet} s{', and used no CPU for 300 s' if idle else ''}: bin/run-tests"
|
||||
|
||||
|
||||
class StuckWorkers(unittest.TestCase):
|
||||
def worker_with(self, *events, held=(), replies=(), archive=None, each_check=None):
|
||||
"""Runs a worker whose claude prints events and holds the shells held, while each investigation replies in turn; returns the claude, the fake, what was said on stdout and stderr and the record's text. The archive folder is archive, or else one that doesn't exist yet, as for a project's first stop. Where each_check is given, each check finds the next of its shells alone."""
|
||||
clock = [NOW]
|
||||
streamed = Streamed(0, *events, clock=clock, held=held)
|
||||
if each_check:
|
||||
streamed.shells = lambda: [each_check.pop(0)]
|
||||
fake = FakeEffects(streams=[streamed], claude=replies)
|
||||
effects = dataclasses.replace(fake.effects(), now=lambda: clock[0])
|
||||
with tempfile.TemporaryDirectory() as folder, contextlib.redirect_stdout(io.StringIO()) as said, contextlib.redirect_stderr(io.StringIO()) as failed:
|
||||
archive = archive or pathlib.Path(folder, "transcripts", "project")
|
||||
worker.run_worker("/r", [], {}, archive, RUNG, effects)
|
||||
record = archive / stuck_workers.RECORD
|
||||
return streamed, fake, said.getvalue(), failed.getvalue(), record.read_text() if record.exists() else ""
|
||||
|
||||
def test_a_silent_worker_whose_shell_used_no_cpu_since_the_last_check_is_investigated_once_and_the_verdict_said_and_recorded_without_a_kill(self):
|
||||
streamed, fake, said, _, record = self.worker_with(ASSISTANT, None, None, None, held=[shell()], replies=[verdict()])
|
||||
[(args, prompt)] = fake.made("claude")
|
||||
self.assertEqual(args[args.index("--model") + 1 :][:3], ["claude-opus-5-5", "--effort", "medium"])
|
||||
self.assertEqual(args[args.index("--tools") + 1], "")
|
||||
self.assertEqual(json.loads(args[args.index("--json-schema") + 1])["properties"]["verdict"]["enum"], ["working", "waiting on a person", "stuck"])
|
||||
self.assertTrue(prompt.endswith("\n\nShell 7: bin/run-tests\nIts processes, as pid, state, CPU and command line:\n8 S 2.0 s python3 -m pytest\n9 S 0.0 s sh -c trap '' INT; sleep 999\nThe end of its output:\ncollected 40 items"))
|
||||
line = "stuck check: a headless claude judged the worker stuck, naming process 8 (python3 -m pytest): pytest waits on a shell that ignores SIGINT" + sign(1600, 1600)
|
||||
self.assertEqual(said, f"<{NOW + 600}> {line}\n")
|
||||
self.assertEqual(record, f"- {utc(NOW + 600)} {line}\n")
|
||||
self.assertEqual(streamed.stopped, [])
|
||||
|
||||
def test_a_judged_shell_is_judged_again_only_after_it_used_cpu_and_went_idle_once_more(self):
|
||||
cpu = [2.0, 2.0, 2.0, 3.0, 3.0]
|
||||
_, fake, _, _, record = self.worker_with(ASSISTANT, *[None] * 5, each_check=[shell(cpu=each) for each in cpu], replies=[verdict("working", 8, "the tests run"), verdict()])
|
||||
self.assertEqual(len(fake.made("claude")), 2)
|
||||
self.assertEqual([line.split(" ")[1] for line in record.splitlines() if line.startswith("- ")], [utc(NOW + 600), utc(NOW + 1500)])
|
||||
|
||||
def test_an_old_shell_that_keeps_using_cpu_is_judged_once(self):
|
||||
_, fake, _, _, _ = self.worker_with(ASSISTANT, None, None, None, each_check=[shell(cpu=each, started=NOW - stuck_workers.OLD) for each in (2.0, 3.0, 4.0)], replies=[verdict()])
|
||||
self.assertEqual(len(fake.made("claude")), 1)
|
||||
|
||||
def test_a_shell_whose_investigation_failed_is_judged_at_the_next_check(self):
|
||||
_, fake, said, _, _ = self.worker_with(ASSISTANT, None, None, None, held=[shell()], replies=[ran(code=1, stderr="no model"), verdict()])
|
||||
self.assertEqual(len(fake.made("claude")), 2)
|
||||
self.assertIn(f"<{NOW + 900}> stuck check: a headless claude judged the worker stuck", said)
|
||||
|
||||
def test_a_named_process_without_a_command_line_is_said_so(self):
|
||||
_, _, said, _, _ = self.worker_with(ASSISTANT, None, None, held=[shell(processes=("8 Z 0.0 s ",))], replies=[verdict()])
|
||||
self.assertIn("naming process 8 (no command line):", said)
|
||||
|
||||
def test_hands_the_investigation_none_for_a_shell_without_processes_or_output(self):
|
||||
_, fake, _, _, _ = self.worker_with(ASSISTANT, None, None, held=[shell(processes=(), output="")], replies=[verdict()])
|
||||
[(_, prompt)] = fake.made("claude")
|
||||
self.assertTrue(prompt.endswith("\n\nShell 7: bin/run-tests\nIts processes, as pid, state, CPU and command line:\nnone\nThe end of its output:\nnone"))
|
||||
|
||||
def test_a_shell_older_than_a_chunk_usually_runs_is_investigated_at_the_first_check(self):
|
||||
old = [shell(pid, started=NOW - stuck_workers.OLD, written=NOW + 290) for pid in (7, 10)]
|
||||
_, fake, said, _, _ = self.worker_with(ASSISTANT, None, held=old, replies=[verdict("working", 7, "the tests still print")])
|
||||
[(_, prompt)] = fake.made("claude")
|
||||
self.assertIn("collected 40 items\n\nShell 10: bin/run-tests\n", prompt)
|
||||
self.assertIn(f"stuck check: a headless claude judged the worker working, naming process 7 (bin/run-tests): the tests still print{sign(stuck_workers.OLD + 300, 10, idle=False)}{sign(stuck_workers.OLD + 300, 10, idle=False).replace('shell 7', 'shell 10')}\n", said)
|
||||
|
||||
def test_a_shell_holding_a_heavy_slot_is_investigated_and_the_verdict_said_on_one_line_with_each_sign_under_it(self):
|
||||
_, _, said, _, _ = self.worker_with(ASSISTANT, None, None, held=[shell(holds_slot=True)], replies=[verdict(process=99, why="the tests\nhang")])
|
||||
self.assertEqual(said, f"<{NOW + 600}> stuck check: a headless claude judged the worker stuck, naming process 99 (not in the evidence): the tests hang{sign(1600, 1600)}\n")
|
||||
|
||||
def test_a_worker_whose_shells_show_no_sign_is_not_investigated(self):
|
||||
for events, held in (
|
||||
((ASSISTANT, None, None), [shell(runs_busy_tool=True)]),
|
||||
((ASSISTANT, None, None), [shell(written=NOW + 600 - stuck_workers.QUIET)]),
|
||||
((ASSISTANT, None), [shell()]),
|
||||
((ASSISTANT, None, None), []),
|
||||
(({"type": "result"}, None, None), [shell()]),
|
||||
((ASSISTANT, None), [shell(started=NOW + 300 - stuck_workers.OLD)]),
|
||||
):
|
||||
with self.subTest(events=events, held=held):
|
||||
_, fake, said, _, record = self.worker_with(*events, held=held)
|
||||
self.assertEqual((fake.made("claude"), record), ([], ""))
|
||||
|
||||
def test_a_shell_that_used_cpu_since_the_last_check_is_not_investigated(self):
|
||||
clock = [NOW]
|
||||
used = [shell(cpu=2.0), shell(cpu=2.5)]
|
||||
streamed = Streamed(0, ASSISTANT, None, None, clock=clock)
|
||||
streamed.shells = lambda: [used.pop(0)]
|
||||
fake = FakeEffects(streams=[streamed])
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
worker.run_worker("/r", [], {}, pathlib.Path("/archive"), RUNG, dataclasses.replace(fake.effects(), now=lambda: clock[0]))
|
||||
self.assertEqual(fake.made("claude"), [])
|
||||
|
||||
def test_an_investigation_that_fails_or_answers_outside_the_schema_is_said_on_stderr_and_recorded(self):
|
||||
limited = ran(json.dumps({"type": "result", "api_error_status": 429, "result": "You've hit your limit"}))
|
||||
for reply, reason in ((ran(code=1, stderr="no model"), "no model"), (verdict("hung"), "answered outside its schema: {'verdict': 'hung', 'process': 8, 'why': 'pytest waits on a shell that ignores SIGINT'}"), (ran("crashed"), "the stuck check printed no json reply; its last line was: crashed"), (limited, "claude hit a limit: You've hit your limit")):
|
||||
with self.subTest(reason=reason):
|
||||
_, _, said, failed, record = self.worker_with(ASSISTANT, None, None, held=[shell()], replies=[reply])
|
||||
line = f"stuck check: could not judge whether the worker is stuck: {reason}{sign(1600, 1600)}"
|
||||
self.assertEqual((said, failed, record), ("", f"<{NOW + 600}> {line}\n", f"- {utc(NOW + 600)} {line}\n"))
|
||||
|
||||
def test_a_verdict_that_cannot_be_recorded_is_still_said_and_the_failure_said_on_stderr(self):
|
||||
with tempfile.NamedTemporaryFile() as blocker:
|
||||
_, _, said, failed, _ = self.worker_with(ASSISTANT, None, None, held=[shell()], replies=[verdict()], archive=pathlib.Path(blocker.name))
|
||||
self.assertIn("judged the worker stuck", said)
|
||||
self.assertIn(f"could not record the stuck check in {blocker.name}/{stuck_workers.RECORD}", failed)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -9,6 +9,7 @@ from claude_code.transcript import part_line
|
||||
from machine_load.memory import ROOM_SHARE
|
||||
from model_routing.ladder import Rung
|
||||
from claude_code.session import Session
|
||||
from agent_cycle.stuck_workers import Watch
|
||||
|
||||
PRINTED_TOOL_INPUT = 160
|
||||
# Seconds between reads of the machine's memory while too little of it is available to start a worker.
|
||||
@@ -18,13 +19,14 @@ AUTO_COMPACT_WINDOW = "CLAUDE_CODE_AUTO_COMPACT_WINDOW"
|
||||
COMPACT_WINDOW = 200_000
|
||||
|
||||
|
||||
def run_worker(root: str, args: list[str], env: dict[str, str], project_archive: pathlib.Path, effects: Effects) -> Failure | Limited | None:
|
||||
def run_worker(root: str, args: list[str], env: dict[str, str], project_archive: pathlib.Path, investigator: Rung, effects: Effects) -> Failure | Limited | None:
|
||||
"""Runs the worker's claude once the machine has room for it, as run_claude does with a Watch whose stuck checks run on investigator, the project's rung."""
|
||||
wait_for_memory(effects)
|
||||
return run_claude(root, args, env, effects, project_archive / RECORD)
|
||||
return run_claude(root, args, env, effects, Watch(investigator, project_archive))
|
||||
|
||||
|
||||
def run_claude(root: str, args: list[str], env: dict[str, str], effects: Effects, record: pathlib.Path | None = None) -> Failure | Limited | None:
|
||||
"""Runs a headless claude at once, printing what it says, and returns how it failed, or None. Where record is given, a claude that lingers after its final result loses its quiet shells, as `lingering_workers.stopped_shells` says, and each stop is appended to record."""
|
||||
def run_claude(root: str, args: list[str], env: dict[str, str], effects: Effects, watch: Watch | None = None) -> Failure | Limited | None:
|
||||
"""Runs a headless claude at once, printing what it says, and returns how it failed, or None. Where watch is given, a claude that lingers after its final result loses its quiet shells, as `lingering_workers.stopped_shells` says, each stop appended to RECORD in watch's archive; and a silent claude that has printed no result is checked for stuck shells, as `stuck_workers` says."""
|
||||
try:
|
||||
proc = effects.stream_claude(args, root, env)
|
||||
except OSError as failure:
|
||||
@@ -35,8 +37,10 @@ def run_claude(root: str, args: list[str], env: dict[str, str], effects: Effects
|
||||
resets: list[int] = []
|
||||
for raw in proc.lines(CHECK_EVERY):
|
||||
if raw is None:
|
||||
if record is not None:
|
||||
stopped_shells(proc, printed_result, record, effects)
|
||||
if watch is not None:
|
||||
stopped_shells(proc, printed_result, watch.archive / RECORD, effects)
|
||||
if not printed_result:
|
||||
watch.check(proc, root, effects)
|
||||
continue
|
||||
event = json_object(raw) or {}
|
||||
printed_result = ends_stream(event)
|
||||
|
||||
@@ -88,6 +88,17 @@ minutes, runs chunk-pr or run-heavy, or holds a heavy slot. It prints each stop
|
||||
the UTC time, to lingering-workers.md at the project's folder in that archive
|
||||
(bin/agent_cycle/lingering_workers.py).
|
||||
|
||||
A worker can also stall before its result, on a command that hangs. Every 5 minutes without output
|
||||
from its claude, loop-claude looks for a shell that may be stuck: one that used no CPU since the
|
||||
last check and wrote no output for 3 minutes, or one that has run over 4 hours. A shell that runs
|
||||
chunk-pr or run-heavy doesn't count, but one that holds a heavy slot can. Where such a shell is
|
||||
found, a headless claude on the project's rung reads every shell's command, processes and output.
|
||||
The headless claude judges the worker working, waiting on a person or stuck, and names the process
|
||||
the worker waits on. It judges a shell again only after the shell used CPU and went idle once more.
|
||||
loop-claude prints the verdict and the shells that showed a sign, and appends them to
|
||||
stuck-workers.md beside lingering-workers.md. It doesn't kill anything yet
|
||||
(bin/agent_cycle/stuck_workers.py).
|
||||
|
||||
Before each worker starts, loop-claude waits until at least a quarter of the machine's memory is
|
||||
available (bin/machine_load/memory.py).
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ from claude_code import entries, reply, session
|
||||
from failure import Failure
|
||||
from agent_cycle.verdicts import Finished, Interrupt, Judged, QuotedAnswer, TakenIn, Unfinished
|
||||
from git_cli.git_repo import git
|
||||
from agent_cycle import interrupt, judge, lingering_workers, retry, run, spend, states, worker, worker_prompts
|
||||
from agent_cycle import interrupt, judge, lingering_workers, retry, run, spend, states, stuck_workers, worker, worker_prompts
|
||||
from claude_code.reply import Limited
|
||||
import loop_claude.routed_item
|
||||
from loop_claude.loop import main, reload, routing_check, this_loop, usage
|
||||
@@ -384,6 +384,23 @@ Worker: {OFF_LADDER}
|
||||
self.assertEqual(run.run_cycle(cycle).verdict, Finished("working"))
|
||||
self.assertEqual(fake.worker_environments, [{"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5", "CLAUDE_CODE_AUTO_COMPACT_WINDOW": "200000", "BASH_MAX_OUTPUT_LENGTH": "6000"}])
|
||||
|
||||
def test_a_stuck_check_of_the_worker_runs_on_the_projects_rung_and_off_the_ladder_on_the_workers_own(self):
|
||||
for project, model in ((OPUS_PROJECT, "claude-opus-5-5"), (None, "claude-haiku-5-5")):
|
||||
with self.subTest(project=project):
|
||||
clock = [NOW]
|
||||
quiet = Shell(pid=7, started=NOW, command="bin/run-tests", written=NOW, runs_busy_tool=False)
|
||||
events = ({"type": "assistant", "message": {"content": []}}, None, None, {"type": "result"})
|
||||
stream = lambda args: (Worker(self.paths.claude_code_transcripts.parent)(args), Streamed(0, *events, clock=clock, held=[quiet]))[1]
|
||||
verdict = ran(json.dumps({"type": "result", "structured_output": {"verdict": "stuck", "process": 7, "why": "it waits"}}))
|
||||
fake = FakeEffects(streams=[stream], claude=[verdict, judged("finished")])
|
||||
cycle = dataclasses.replace(self.cycle(fake), project=project, route=Route(Rung("claude-haiku-5-5", "max")))
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
run.run_cycle(dataclasses.replace(cycle, effects=dataclasses.replace(cycle.effects, now=lambda: clock[0])))
|
||||
[investigator, _] = [args for args, _ in fake.made("claude")]
|
||||
self.assertEqual(investigator[investigator.index("--model") + 1], model)
|
||||
self.assertEqual(fake.worker_folders, [str(self.home)])
|
||||
self.assertIn("judged the worker stuck", (self.archive / stuck_workers.RECORD).read_text())
|
||||
|
||||
def test_the_claude_an_interrupt_resumes_is_handed_the_same_rungs_as_the_worker(self):
|
||||
def answer(args):
|
||||
with open(session.transcript_file(self.session), "a") as handle:
|
||||
@@ -795,7 +812,7 @@ class Limits(unittest.TestCase):
|
||||
allowed = {"type": "rate_limit_event", "rate_limit_info": {"status": "allowed", "resetsAt": NOW + 9000}}
|
||||
limited = {"type": "result", "api_error_status": 429, "result": "You've hit your session limit"}
|
||||
fake = FakeEffects(streams=[Streamed(1, rejected[1], allowed, rejected[0], limited)])
|
||||
self.assertEqual(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), fake.effects()), Limited(NOW + 600, "You've hit your session limit"))
|
||||
self.assertEqual(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), Rung("m"), fake.effects()), Limited(NOW + 600, "You've hit your session limit"))
|
||||
|
||||
def test_a_limit_outranks_the_exit_code_and_its_message_fits_one_line(self):
|
||||
def limited(**said):
|
||||
@@ -829,12 +846,12 @@ class LingeringWorkers(unittest.TestCase):
|
||||
with tempfile.TemporaryDirectory() as folder, contextlib.redirect_stdout(io.StringIO()) as said:
|
||||
# The archive folder of a project's first stop doesn't exist yet.
|
||||
archive = pathlib.Path(folder, "transcripts", "project")
|
||||
worker.run_worker("/r", [], {}, archive, effects)
|
||||
worker.run_worker("/r", [], {}, archive, Rung("m"), effects)
|
||||
record = archive / lingering_workers.RECORD
|
||||
return streamed, said.getvalue(), record.read_text() if record.exists() else ""
|
||||
|
||||
def shell(self, written=NOW - 1000, runs_busy_tool=False):
|
||||
return Shell(pid=7, started=NOW - 1000, command="until false; do sleep 15; done", written=written, runs_busy_tool=runs_busy_tool)
|
||||
def shell(self, written=NOW - 1000, runs_busy_tool=False, holds_slot=False):
|
||||
return Shell(pid=7, started=NOW - 1000, command="until false; do sleep 15; done", written=written, runs_busy_tool=runs_busy_tool, holds_slot=holds_slot)
|
||||
|
||||
def test_a_worker_that_lingers_after_its_result_loses_its_quiet_shell_and_the_stop_is_said_and_recorded_in_utc(self):
|
||||
shell = self.shell()
|
||||
@@ -850,8 +867,8 @@ class LingeringWorkers(unittest.TestCase):
|
||||
streamed, said, record = self.worker_with(*events, held=[self.shell()])
|
||||
self.assertEqual((streamed.stopped, said, record), ([], "", ""))
|
||||
|
||||
def test_a_worker_keeps_every_shell_while_one_of_them_writes_output_or_runs_a_busy_tool(self):
|
||||
for busy in (self.shell(written=NOW + 200), self.shell(runs_busy_tool=True)):
|
||||
def test_a_worker_keeps_every_shell_while_one_of_them_writes_output_runs_a_busy_tool_or_holds_a_heavy_slot(self):
|
||||
for busy in (self.shell(written=NOW + 200), self.shell(runs_busy_tool=True), self.shell(holds_slot=True)):
|
||||
with self.subTest(busy=busy):
|
||||
streamed, _, _ = self.worker_with(self.RESULT, None, held=[self.shell(), busy])
|
||||
self.assertEqual(streamed.stopped, [])
|
||||
@@ -866,7 +883,7 @@ class LingeringWorkers(unittest.TestCase):
|
||||
def test_a_worker_reads_on_past_a_check_that_found_it_still_working(self):
|
||||
clock = [NOW]
|
||||
effects = dataclasses.replace(FakeEffects(streams=[Streamed(0, None, self.RESULT, clock=clock)]).effects(), now=lambda: clock[0])
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), effects))
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), Rung("m"), effects))
|
||||
|
||||
def test_a_stop_that_cannot_be_recorded_is_said_on_stderr_and_every_shell_is_still_stopped(self):
|
||||
clock = [NOW]
|
||||
@@ -874,7 +891,7 @@ class LingeringWorkers(unittest.TestCase):
|
||||
streamed = Streamed(0, self.RESULT, None, clock=clock, held=held)
|
||||
effects = dataclasses.replace(FakeEffects(streams=[streamed]).effects(), now=lambda: clock[0])
|
||||
with tempfile.NamedTemporaryFile() as blocker, contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()) as failed:
|
||||
worker.run_worker("/r", [], {}, pathlib.Path(blocker.name), effects)
|
||||
worker.run_worker("/r", [], {}, pathlib.Path(blocker.name), Rung("m"), effects)
|
||||
self.assertEqual(streamed.stopped, held)
|
||||
self.assertIn("could not record the stop", failed.getvalue())
|
||||
|
||||
@@ -888,14 +905,14 @@ class MemoryWait(unittest.TestCase):
|
||||
def test_a_worker_starts_only_once_a_quarter_of_memory_is_available(self):
|
||||
fake = FakeEffects(free_memory=[memory.Memory(available=3, total=16), memory.Memory(available=3, total=16), memory.Memory(available=4, total=16)], streams=[Streamed(0)])
|
||||
with contextlib.redirect_stderr(io.StringIO()) as said:
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), fake.effects()))
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), Rung("m"), fake.effects()))
|
||||
self.assertEqual([seam for seam, _ in fake.calls], ["memory", "sleep", "memory", "sleep", "memory", "stream_claude"])
|
||||
self.assertEqual(fake.made("sleep"), [worker.MEMORY_POLL] * 2)
|
||||
self.assertEqual(said.getvalue().count("waiting"), 1, said.getvalue())
|
||||
|
||||
def test_a_worker_starts_at_once_where_memory_cannot_be_read(self):
|
||||
fake = FakeEffects(free_memory=[None], streams=[Streamed(0)])
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), fake.effects()))
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), Rung("m"), fake.effects()))
|
||||
self.assertEqual([seam for seam, _ in fake.calls], ["memory", "stream_claude"])
|
||||
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ from machine_load.slots import slot_folder
|
||||
# The command names of a shell claude starts for a Bash call.
|
||||
SHELL_NAMES = frozenset({"bash", "dash", "sh", "zsh"})
|
||||
COMMAND_CHARS = 300
|
||||
OUTPUT_BYTES = 2000
|
||||
# Scripts whose process below a shell shows work that may go quiet for long: run-heavy while it waits for a slot, and chunk-pr, which waits for a merge's approval and checks.
|
||||
BUSY_TOOLS = frozenset({"chunk-pr", "run-heavy"})
|
||||
|
||||
@@ -26,21 +27,42 @@ class Shell:
|
||||
command: str
|
||||
# Epoch seconds of the last write to the file the shell's output goes to; the shell's start where the output goes to no file, which the check cannot read.
|
||||
written: float
|
||||
# Whether a process below the shell runs one of BUSY_TOOLS or holds a heavy slot, a file of slot_folder that run-heavy leaves open in the command it runs.
|
||||
# Whether a process below the shell runs one of BUSY_TOOLS.
|
||||
runs_busy_tool: bool
|
||||
# CPU seconds the shell and every process below it have used.
|
||||
cpu: float = 0.0
|
||||
# Each process below the shell, as `<pid> <state> <CPU seconds> s <command line>`, with the state letter /proc gives, such as S for sleeping.
|
||||
processes: tuple[str, ...] = ()
|
||||
# The last OUTPUT_BYTES of the file the shell's output goes to; empty where it goes to a pipe or a terminal.
|
||||
output: str = ""
|
||||
# Whether a process below the shell holds a heavy slot: a file of slot_folder, which run-heavy leaves open in the command it runs.
|
||||
holds_slot: bool = False
|
||||
|
||||
|
||||
def parent_and_start(pid: int) -> tuple[int, int, str] | None:
|
||||
"""The parent pid, the start time in clock ticks since boot and the command name of a process; None where it is gone."""
|
||||
def stat_of(pid: int) -> tuple[str, list[str]] | None:
|
||||
"""A process's command name and the fields of /proc/<pid>/stat after it, from the state on; None where it is gone."""
|
||||
try:
|
||||
with open(f"/proc/{pid}/stat") as handle:
|
||||
text = handle.read()
|
||||
except OSError:
|
||||
return None
|
||||
# The name sits in parentheses and may hold spaces or parentheses itself, so split at the last one.
|
||||
name = text[text.index("(") + 1 : text.rindex(")")]
|
||||
fields = text[text.rindex(")") + 2 :].split()
|
||||
return int(fields[1]), int(fields[19]), name
|
||||
return text[text.index("(") + 1 : text.rindex(")")], text[text.rindex(")") + 2 :].split()
|
||||
|
||||
|
||||
def parent_and_start(pid: int) -> tuple[int, int, str] | None:
|
||||
"""The parent pid, the start time in clock ticks since boot and the command name of a process; None where it is gone."""
|
||||
found = stat_of(pid)
|
||||
return None if found is None else (int(found[1][1]), int(found[1][19]), found[0])
|
||||
|
||||
|
||||
def cpu_and_state(pid: int) -> tuple[float, str]:
|
||||
"""The CPU seconds a process has used, in user and kernel mode, and its state letter; 0.0 and `gone` where it is gone."""
|
||||
found = stat_of(pid)
|
||||
if found is None:
|
||||
return 0.0, "gone"
|
||||
fields = found[1]
|
||||
return (int(fields[11]) + int(fields[12])) / os.sysconf("SC_CLK_TCK"), fields[0]
|
||||
|
||||
|
||||
def children_of(pid: int) -> list[int]:
|
||||
@@ -77,6 +99,31 @@ def output_written(pid: int, started: float) -> float:
|
||||
return info.st_mtime if stat.S_ISREG(info.st_mode) else started
|
||||
|
||||
|
||||
def output_tail(pid: int) -> str:
|
||||
"""The last OUTPUT_BYTES of the file that pid's standard output goes to; empty where it goes to a pipe or a terminal."""
|
||||
path = f"/proc/{pid}/fd/1"
|
||||
try:
|
||||
# Before the open, which can block on a pipe.
|
||||
if not stat.S_ISREG(os.stat(path).st_mode):
|
||||
return ""
|
||||
with open(path, "rb") as handle:
|
||||
handle.seek(max(0, os.fstat(handle.fileno()).st_size - OUTPUT_BYTES))
|
||||
return handle.read().decode(errors="replace")
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def tree_of(pid: int) -> tuple[float, tuple[str, ...]]:
|
||||
"""The CPU seconds pid and every process below it have used, and each process below it as `Shell.processes` shows one."""
|
||||
total = cpu_and_state(pid)[0]
|
||||
shown = []
|
||||
for each in descendants(pid):
|
||||
cpu, state = cpu_and_state(each)
|
||||
total += cpu
|
||||
shown.append(f"{each} {state} {cpu:.1f} s {' '.join(argv_of(each))[:COMMAND_CHARS]}")
|
||||
return total, tuple(shown)
|
||||
|
||||
|
||||
def command_of(shell_argv: list[str]) -> str:
|
||||
"""What a shell runs: the text of its command line after Claude Code's `eval '`, up to the last ` < /dev/null`, which a call that began in the foreground lacks; the whole line where it holds no `eval`."""
|
||||
line = " ".join(shell_argv)
|
||||
@@ -84,7 +131,7 @@ def command_of(shell_argv: list[str]) -> str:
|
||||
return (command.rsplit(" < /dev/null", 1)[0].removeprefix("'").removesuffix("'") if found else line)[:COMMAND_CHARS]
|
||||
|
||||
|
||||
def holds_slot(pid: int, slots: pathlib.Path) -> bool:
|
||||
def has_slot(pid: int, slots: pathlib.Path) -> bool:
|
||||
"""Whether pid has a file of the heavy slots' folder open. run-heavy replaces itself with its command once it holds a slot, and the command keeps the slot's file open."""
|
||||
try:
|
||||
return any(os.readlink(f"/proc/{pid}/fd/{fd}").startswith(f"{slots}/") for fd in os.listdir(f"/proc/{pid}/fd"))
|
||||
@@ -92,9 +139,14 @@ def holds_slot(pid: int, slots: pathlib.Path) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def runs_busy_tool(pid: int, slots: pathlib.Path) -> bool:
|
||||
"""Whether a process below pid holds a heavy slot or has one of BUSY_TOOLS as its program or its script, as `python3 /x/chunk-pr`. A command line that merely names a tool, such as a `pgrep -f chunk-pr`, doesn't count."""
|
||||
return any(holds_slot(each, slots) or any(os.path.basename(name) in BUSY_TOOLS for name in argv_of(each)[:2]) for each in descendants(pid))
|
||||
def runs_busy_tool(pid: int) -> bool:
|
||||
"""Whether a process below pid has one of BUSY_TOOLS as its program or its script, as `python3 /x/chunk-pr`. A command line that merely names a tool, such as a `pgrep -f chunk-pr`, doesn't count."""
|
||||
return any(os.path.basename(name) in BUSY_TOOLS for each in descendants(pid) for name in argv_of(each)[:2])
|
||||
|
||||
|
||||
def holds_slot(pid: int, slots: pathlib.Path) -> bool:
|
||||
"""Whether a process below pid holds a heavy slot."""
|
||||
return any(has_slot(each, slots) for each in descendants(pid))
|
||||
|
||||
|
||||
def shells_of(claude_pid: int) -> list[Shell]:
|
||||
@@ -110,7 +162,8 @@ def shells_of(claude_pid: int) -> list[Shell]:
|
||||
if found is None or found[2] not in SHELL_NAMES:
|
||||
continue
|
||||
started = booted + found[1] / ticks
|
||||
shells.append(Shell(pid, started, command_of(argv_of(pid)), output_written(pid, started), runs_busy_tool(pid, slots)))
|
||||
cpu, processes = tree_of(pid)
|
||||
shells.append(Shell(pid, started, command_of(argv_of(pid)), output_written(pid, started), runs_busy_tool(pid), cpu, processes, output_tail(pid), holds_slot(pid, slots)))
|
||||
return shells
|
||||
|
||||
|
||||
|
||||
@@ -27,6 +27,13 @@ def clocked():
|
||||
return dataclasses.replace(system_effects(), local_time=lambda seconds: f"<{seconds}>", now=lambda: NOW)
|
||||
|
||||
|
||||
def ticks(pid):
|
||||
"""The user and the kernel clock ticks a process has used, read from /proc apart from the code under test."""
|
||||
with open(f"/proc/{pid}/stat") as handle:
|
||||
fields = handle.read().rpartition(") ")[2].split()
|
||||
return int(fields[11]), int(fields[12])
|
||||
|
||||
|
||||
def wait_until(condition):
|
||||
"""Polls up to 5 s for condition, so a test doesn't depend on how fast a process starts."""
|
||||
for _ in range(100):
|
||||
@@ -150,12 +157,18 @@ class PythonEffect(unittest.TestCase):
|
||||
|
||||
class ChildShells(unittest.TestCase):
|
||||
def started(self, command, **options):
|
||||
"""A shell running command, once a process runs below it."""
|
||||
proc = subprocess.Popen(["bash", "-c", command], **options)
|
||||
self.addCleanup(lambda: (proc.kill(), proc.wait()))
|
||||
"""A shell running command, once a process runs below it. The shell leads a process group of its own, which the cleanup kills, so that no process it started outlives the test."""
|
||||
proc = subprocess.Popen(["bash", "-c", command], start_new_session=True, **options)
|
||||
self.addCleanup(self.killed, proc)
|
||||
wait_until(lambda: child_shells.descendants(proc.pid))
|
||||
return proc
|
||||
|
||||
def killed(self, proc):
|
||||
# A test that stopped the shell has emptied its group already.
|
||||
with contextlib.suppress(ProcessLookupError):
|
||||
os.killpg(proc.pid, signal.SIGKILL)
|
||||
proc.wait()
|
||||
|
||||
def shell_of(self, proc):
|
||||
[found] = [each for each in child_shells.shells_of(os.getpid()) if each.pid == proc.pid]
|
||||
return found
|
||||
@@ -168,18 +181,20 @@ class ChildShells(unittest.TestCase):
|
||||
# Like the shell of a loop that only names the tool, as the one that held a worker on 2026-11-18 did.
|
||||
naming = self.started("source /dev/null && eval 'until false; do sleep 60; done # chunk-pr merge' < /dev/null && pwd")
|
||||
busy = self.started(f"{tool} merge; true")
|
||||
# The tool's name as the third word, after a program and a script read from stdin.
|
||||
third = self.started("python3 - chunk-pr <<< 'import time; time.sleep(60)'; true")
|
||||
self.assertEqual(self.shell_of(naming).command, "until false; do sleep 60; done # chunk-pr merge")
|
||||
self.assertEqual((self.shell_of(naming).runs_busy_tool, self.shell_of(busy).runs_busy_tool), (False, True))
|
||||
self.assertEqual([self.shell_of(each).runs_busy_tool for each in (naming, third, busy)], [False, False, True])
|
||||
self.assertLess(abs(self.shell_of(naming).started - time.time()), 5)
|
||||
|
||||
def test_a_shell_below_which_a_process_holds_a_heavy_slot_file_runs_a_busy_tool(self):
|
||||
def test_a_shell_below_which_a_process_holds_a_heavy_slot_file_holds_a_slot_and_runs_no_busy_tool(self):
|
||||
# run-heavy replaces itself with its command, which keeps the slot's file open.
|
||||
with tempfile.TemporaryDirectory() as state, unittest.mock.patch.dict(os.environ, {"XDG_STATE_HOME": state}):
|
||||
slot = pathlib.Path(state, "run-heavy", "slot-0")
|
||||
slot.parent.mkdir()
|
||||
holding = self.started(f"exec 9>{slot}; sleep 60; true")
|
||||
plain = self.started("sleep 60; true")
|
||||
self.assertEqual((self.shell_of(holding).runs_busy_tool, self.shell_of(plain).runs_busy_tool), (True, False))
|
||||
self.assertEqual([(self.shell_of(each).holds_slot, self.shell_of(each).runs_busy_tool) for each in (holding, plain)], [(True, False), (False, False)])
|
||||
|
||||
def test_a_shell_written_output_is_the_modified_time_of_its_output_file_and_else_its_start(self):
|
||||
with tempfile.NamedTemporaryFile() as output:
|
||||
@@ -189,6 +204,33 @@ class ChildShells(unittest.TestCase):
|
||||
self.assertEqual(self.shell_of(to_file).written, 1000)
|
||||
self.assertEqual(self.shell_of(to_pipe).written, self.shell_of(to_pipe).started)
|
||||
|
||||
def test_reads_the_end_of_a_shells_output_file_and_no_output_from_a_pipe(self):
|
||||
for written, want in ((b"x" * child_shells.OUTPUT_BYTES + b"last line\n", "x" * (child_shells.OUTPUT_BYTES - 10) + "last line\n"), (b"short \xff\n", "short \ufffd\n")):
|
||||
with self.subTest(want=want[-12:]), tempfile.NamedTemporaryFile() as output:
|
||||
output.write(written)
|
||||
output.flush()
|
||||
self.assertEqual(self.shell_of(self.started("sleep 60; true", stdout=output)).output, want)
|
||||
self.assertEqual(self.shell_of(self.started("sleep 60; true", stdout=subprocess.PIPE)).output, "")
|
||||
|
||||
def test_reads_no_cpu_state_or_output_of_a_process_that_is_gone(self):
|
||||
gone = subprocess.Popen(["true"])
|
||||
gone.wait()
|
||||
self.assertEqual((child_shells.cpu_and_state(gone.pid), child_shells.output_tail(gone.pid)), ((0.0, "gone"), ""))
|
||||
|
||||
def test_reads_each_process_below_a_shell_and_the_user_and_kernel_cpu_the_whole_tree_used(self):
|
||||
# The shell counts in user mode before it starts dd, which spends its time in the kernel. Stopping dd freezes every figure, so the total must equal the ticks of all three processes, each mode counted once.
|
||||
busy = self.started("i=0; while [ $i -lt 100000 ]; do i=$((i+1)); done; dd if=/dev/zero of=/dev/null bs=1M & sleep 60; true")
|
||||
wait_until(lambda: len(child_shells.descendants(busy.pid)) == 2)
|
||||
dd, sleep = child_shells.descendants(busy.pid)
|
||||
wait_until(lambda: ticks(dd)[1] > 0)
|
||||
os.kill(dd, signal.SIGSTOP)
|
||||
wait_until(lambda: child_shells.cpu_and_state(dd)[1] == "T")
|
||||
self.assertGreater(ticks(busy.pid)[0], 0)
|
||||
self.assertAlmostEqual(self.shell_of(busy).cpu, sum(sum(ticks(pid)) for pid in (busy.pid, dd, sleep)) / os.sysconf("SC_CLK_TCK"))
|
||||
processes = self.shell_of(busy).processes
|
||||
self.assertEqual([line.split(" ", 4)[4] for line in processes], ["dd if=/dev/zero of=/dev/null bs=1M", "sleep 60"])
|
||||
self.assertEqual([line.split(" ")[:2] for line in processes], [[str(dd), "T"], [str(sleep), "S"]])
|
||||
|
||||
def test_stops_a_shell_and_the_process_it_started_and_no_other(self):
|
||||
shell = self.started("sleep 60; true")
|
||||
other = subprocess.Popen(["sleep", "60"])
|
||||
@@ -219,7 +261,7 @@ class ChildShells(unittest.TestCase):
|
||||
wait_until(lambda: len(child_shells.descendants(shell.pid)) == 2)
|
||||
|
||||
def test_a_process_that_is_gone_holds_no_slot_and_a_vanished_child_does_not_hide_the_shells_after_it(self):
|
||||
self.assertFalse(child_shells.holds_slot(2**30, pathlib.Path("/")))
|
||||
self.assertFalse(child_shells.has_slot(2**30, pathlib.Path("/")))
|
||||
shell = self.started("sleep 60; true")
|
||||
real = child_shells.children_of
|
||||
with unittest.mock.patch.object(child_shells, "children_of", side_effect=lambda pid: [2**30, shell.pid] if pid == os.getpid() else real(pid)):
|
||||
|
||||
@@ -4011,3 +4011,111 @@ Inherited architect, decided by:
|
||||
- Locality: `bin/chunk_pr/main.py:130` `step`: the config is passed in once, where `misfit` shelled out to `project-config` twice
|
||||
- Shape: `bin/project_config/settings.py:320` `SCALARS`: one row per one-value setting, where `load` handled each at four sites
|
||||
- Navigation, Self-sufficiency: `bin/chunk_preflight/gathered.py:60` `Gathered.plan`: a stale pointer to `choices.plan_name`, with two more in `next_chunk.py` and the README, small next to the gains
|
||||
|
||||
## 2026-11-19T09:23:24Z, PR #633 at 81e8d8a1
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Junior A | 7 | 6 | 6 | 5.5 | 6 |
|
||||
| Mid A | 7 | 6 | 6.5 | 5.5 | 6.5 |
|
||||
| Maintainability senior | 7.5 | 6.5 | 6.5 | 6.5 | 7 |
|
||||
| Inherited architect | 7 | 7 | 6 | 6 | 6.5 |
|
||||
| Mean | 7.13 | 6.38 | 6.25 | 5.88 | 6.50 |
|
||||
|
||||
Junior A, hardest first:
|
||||
|
||||
- Locality: `bin/do_epic/main.py:283` `parked_through`: one polling loop carries five mutable locals against a `now` of four types, and a nested conditional must be traced by hand
|
||||
- Locality: `bin/do_epic/scheduler.py:136` `start_chunks`: four overlapping sets feed an elif chain whose arm order decides the outcome, under a lock with a second one inside
|
||||
- Self-sufficiency: `skills/larv-review/SKILL.md:97` "Re-entry": dense rules held at once with the round cap, pointing out to other files
|
||||
- Locality, Self-sufficiency: `bin/agent_cycle/routing.py:115` `unfinished_route`: five rungs, a counter and a mark live at once; the why of the caps is only in the decision log
|
||||
- Self-sufficiency: `bin/agent_cycle/judge.py:235` `quotes_checked`: six quote checks each with its own fallback; the prompt constants are behaviour guarded by one comment
|
||||
- Locality: `bin/loop_claude/loop/main.py:108` `next_cycle`: the order of steps carries meaning the code doesn't state
|
||||
- Locality: `bin/chunk_pr/main.py:429` `levelled_and_finished`: order-dependent steps that comments state and nothing enforces
|
||||
- Self-sufficiency: `bin/rule_hook/rules.py:27` `RULES`: a rule's only specification is one long sentence with nested unless clauses
|
||||
|
||||
Mid A, hardest first:
|
||||
|
||||
- Locality: `bin/agent_cycle/routing.py:115` `unfinished_route`: interacting rungs, a counter and an escalation mark; the module table resolves it
|
||||
- Self-sufficiency: `bin/agent_cycle/judge.py:235` `quotes_checked`: the prompt text is behaviour written as English
|
||||
- Shape: `bin/loop_claude/cycle/prompts.py:132` `escalation`: a prompt assembled from pieces that may each be empty
|
||||
- Locality: `bin/agent_cycle/run.py:150` `rule`: the order of stamping, archiving and stepping matters across functions
|
||||
- Self-sufficiency: `bin/rule_hook/rules.py:27` `RULES`: rule text doubles as specification and help
|
||||
- Shape: `bin/rule_hook/endless_wait.py:93` `self_match`: long lines and a regex whose cases need a second read
|
||||
- Locality: `bin/loop_claude/loop/chunk_windows.py:107` `may_start`: depends on state kept out of sight
|
||||
- Self-sufficiency: `bin/larv_review/plan.py:80` `plan_reasons`: reasons whose meaning lives in the skill's text
|
||||
- Locality: `bin/model_routing/route.py:42` `Route`: fields used as archive keys elsewhere; the unit this seat least wanted to modify
|
||||
|
||||
Maintainability senior, hardest first:
|
||||
|
||||
- Locality: `bin/do_epic/main.py:283` `parked_through`: five mutable locals in one polling loop, and an empty string that means "never parked"
|
||||
- Locality: `bin/chunk_pr/main.py:429` `levelled_and_finished`: order-dependent steps, and a context manager that yields a Failure instead of raising
|
||||
- Locality, Shape: `bin/do_epic/scheduler.py:136` `start_chunks`: overlapping sets and an elif chain under nested locks
|
||||
- Self-sufficiency: `bin/agent_cycle/routing.py:115` `unfinished_route`: the why of the caps sits in the decision log
|
||||
- Self-sufficiency: `bin/agent_cycle/judge.py:235` `quotes_checked`: fallbacks change the verdict's type without a signal at the call site
|
||||
- Self-sufficiency: `bin/rule_hook/rules.py:27` `RULES` env-file-read: a 130-word sentence is the rule's only specification
|
||||
- Shape: `bin/plan/ranking.py:239` `score`: two table shapes double the branches
|
||||
- Locality: `bin/agent_cycle/run.py:240` `close`: closing steps whose order matters across the module
|
||||
|
||||
Inherited architect, would restructure first:
|
||||
|
||||
- Shape: `bin/` root: about 85 entries at once with no split between tools and libraries
|
||||
- Locality: `bin/do_epic/main.py:11` imports: do_epic reaches into loop_claude's internals
|
||||
- Navigation: `bin/loop_claude/cycle/` the folder: the name misleads after its code moved to agent_cycle
|
||||
- Navigation: `bin/agent_cycle/verdicts.py:1` the module: names left from before the move
|
||||
- Shape: `bin/agent_cycle/run.py:38` `Cycle`: the config, judge and project fields widen a 16-field value
|
||||
- Shape: `bin/larv_review/reviewer_header.py:121` `Launch`: built positionally
|
||||
- Self-sufficiency: `bin/larv_review/plan.py:25` the module: no docstring says what it holds
|
||||
|
||||
## 2026-11-19T09:27:43Z, PR #633 rerun at 81e8d8a1
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Junior A | 6.5 | 5 | 6 | 4.5 | 5.5 |
|
||||
| Mid A | 7 | 6 | 6.5 | 6 | 6.5 |
|
||||
| Maintainability senior | 7 | 5.5 | 6 | 5.5 | 6 |
|
||||
| Inherited architect | 6.5 | 6 | 5 | 5 | 5.5 |
|
||||
| Mean | 6.75 | 5.63 | 5.88 | 5.25 | 5.88 |
|
||||
|
||||
Junior A, hardest first:
|
||||
|
||||
- Locality, Self-sufficiency: `bin/agent_cycle/routing.py:115` `unfinished_route`: five rungs, a count and the escalation mark held at once; the why sits in the decision log and `escalation_mark.py` `checked` must be open beside it
|
||||
- Self-sufficiency: `bin/agent_cycle/judge.py:235` `quotes_checked`: six kinds of quote each fail differently, and half the behaviour is prompt text whose load-bearing sentences can't be told apart
|
||||
- Self-sufficiency: `skills/larv-review/SKILL.md:97` "Re-entry": rules with exceptions to exceptions, cross-referencing other sections and files
|
||||
- Locality, Shape: `bin/agent_cycle/run.py:150` `rule`: a 16-field `Cycle`, long one-line calls, and an archive-before-step-up order only reading order tells
|
||||
- Self-sufficiency: `bin/rule_hook/rules.py:27` `RULES`: each rule is one enormous sentence, and its check lives in another module
|
||||
- Locality: `bin/loop_claude/loop/chunk_windows.py:107` `may_start`: sets that depend on tmux and lock state out of sight, a lock a second program also takes
|
||||
- Locality: `bin/loop_claude/loop/reload.py:69` `decoded_route`: renaming a field stops running loops, and nothing warns at the other end
|
||||
- Locality: `bin/do_epic/main.py:118` `run`: argv parsed through interacting booleans, and imports of loop_claude internals
|
||||
|
||||
Mid A, hardest first:
|
||||
|
||||
- Locality: `bin/do_epic/main.py:283` `parked_through`: a polling loop holds four variables across three result types, a three-deep chained conditional, and `on is None` doubling as the first pass; the unit this seat least wanted to modify
|
||||
- Locality, Self-sufficiency: `bin/agent_cycle/routing.py:115` `unfinished_route`: five rungs plus a count and a mark held together; the why is cited to the decision log
|
||||
- Locality: `bin/do_epic/scheduler.py:136` `start_chunks`: four window sets feed a five-arm chain whose meaning changes with `own`, inside two nested flocks shared with loop-claude
|
||||
- Self-sufficiency: `bin/agent_cycle/judge.py:235` `quotes_checked`: six quote kinds fail differently; much behaviour is an English prompt, and `TASK_HEADING` is silently an archive format
|
||||
- Locality: `bin/project_config/settings.py:182` `load`: a 20-entry list of cross-setting rules, and an assert that depends on it being complete
|
||||
- Locality: `bin/chunk_pr/main.py:429` `levelled_and_finished`: a load-bearing order and a short-circuit hidden in `failed or run(...)`
|
||||
- Self-sufficiency: `bin/rule_hook/rules.py:27` `RULES`: each contract is a long English sentence, and nothing ties it to the checking code in eight sibling modules
|
||||
- Shape: `bin/agent_cycle/stuck_workers.py:52` `Watch.check`: the condition at line 59 mixes `or` and `and` without parentheses over dicts mutated across ticks
|
||||
|
||||
Maintainability senior, hardest first:
|
||||
|
||||
- Locality: `bin/do_epic/main.py:283` `parked_through`: four mutable locals across iterations, a side effect correct only because `on` is reassigned later, and a crash-ordering invariant across three stores; the unit this seat least wanted to modify
|
||||
- Locality: `bin/do_epic/scheduler.py:136` `start_chunks`: four overlapping sets from two stores feed a five-arm ladder, and a parked chunk silently restarts down the no-repo path
|
||||
- Locality: `bin/loop_claude/loop/chunk_windows.py:107` `may_start`: depends on window names and options, dead flags, archived lines, a flock and predictions, with the schema in `agent_cycle/tmux.py`
|
||||
- Locality: `bin/agent_cycle/routing.py:115` `unfinished_route`: five rungs plus a count and the mark; the table resolves it, the why is in the decision log
|
||||
- Locality: `bin/chunk_pr/main.py:429` `levelled_and_finished`: a load-bearing order held together by one-liners where a Failure's truthiness carries the control flow
|
||||
- Shape: `bin/agent_cycle/run.py:150` `rule`: archive-then-step-up order, several exits, and very long single-line expressions
|
||||
- Locality: `bin/project_config/settings.py:182` `load`: one list encodes the whole validation matrix, whose interactions don't read
|
||||
- Self-sufficiency: `bin/agent_cycle/judge.py:235` `quotes_checked`: six quote kinds with different fallbacks; `INTERRUPT_RULES` is behaviour no code can check
|
||||
|
||||
Inherited architect, would restructure first:
|
||||
|
||||
- Locality, Self-sufficiency: `skills/larv-review/SKILL.md:89` "Round cap": the cap is spread across three packages and this prose, and nothing in code counts a reviewer's launches
|
||||
- Locality: `bin/do_epic/main.py:11` imports: do_epic reaches into eight loop_claude modules, over a wire format a renamed field breaks
|
||||
- Navigation: `bin/loop_claude/cycle/` the folder: it holds prompts and chunk items while the cycle is agent_cycle
|
||||
- Shape: `bin/agent_cycle/verdicts.py:1` the module: its docstring names a `cycle.` package that is now agent_cycle
|
||||
- Shape: `bin/` root: about 80 entries with no grouping, indexed by one dense README paragraph
|
||||
- Navigation: `bin/loop_state/paths.py:6` `Paths`: neither state nor loop-specific
|
||||
- Self-sufficiency: `bin/larv_review/plan.py:25` `scoring_run_due`: the module holds the scoring-run trigger and has no docstring
|
||||
- Locality: `bin/review_rules/review_passes.py:8` `PASS_AGENTS`: code constants that must match sentences in SKILL.md
|
||||
|
||||
@@ -3071,3 +3071,45 @@ Read most (opens and searches):
|
||||
- `docs/decisions.md` 2
|
||||
- `bin/do_epic/backend/protocol.py` 1
|
||||
- `bin/do_epic/findings.py` 1
|
||||
|
||||
## Reading cost, chunk 59bf63f1876b, finished 2026-11-19T04:43:46Z
|
||||
|
||||
97876 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `README.md` 6
|
||||
- `skills/larv-review/SKILL.md` 4
|
||||
- `.gitignore` 3
|
||||
- `bin/do_epic/main.py` 3
|
||||
- `bin/project_config/settings.py` 3
|
||||
- `bin/atlassian/client.py` 2
|
||||
- `bin/do_epic/backend/contract.py` 2
|
||||
- `bin/do_epic/epic_record.py` 2
|
||||
- `bin/do_epic/findings.py` 2
|
||||
- `bin/do_epic/finish.py` 2
|
||||
|
||||
## Reading cost, chunk 476ff8575248, finished 2026-11-19T04:48:45Z
|
||||
|
||||
22763 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `bin/agent_cycle/interrupt.py` 1
|
||||
|
||||
## Reading cost, chunk 8cb7c9836e2d, finished 2026-11-19T07:47:33Z
|
||||
|
||||
106608 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `bin/loop_claude/test_loop_claude.py` 8
|
||||
- `bin/loop_claude/loop/usage.py` 6
|
||||
- `bin/loop_effects/effects.py` 5
|
||||
- `README.md` 4
|
||||
- `bin/loop_claude/test_black_box.py` 4
|
||||
- `bin/loop_claude/loop/main.py` 3
|
||||
- `bin/loop_effects/test_effects.py` 2
|
||||
- `docs/decisions.md` 2
|
||||
- `bin/repo_checks/test_import_order.py` 1
|
||||
- `bin/run-heavy` 1
|
||||
|
||||
@@ -6,13 +6,13 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 446`
|
||||
`Next ID = 448`
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 422 | 0.12.0 | | | **Check every 5 minutes whether a tmux window of the loop looks stuck by other signs than a lingering worker, investigate it with a headless claude, and kill only the process the investigation names.** | 3 | 2 | 3 | 6 | 5 | 1.5 by 4 | 0.8 |
|
||||
| 446 | 0.13.0 | | question | **Kill the process a stuck check names where its verdict is `stuck`, once the human says the recorded verdicts match what they would have done.** | 2 | 2 | 3 | 5 | 3 | 1.5 by 3 | 1.0 |
|
||||
| 445 | 0.13.0 | | | **Refuse a plan that holds an item a `Finish item <ID>` commit already removed, so a merge resolution cannot bring a finished item back.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3, 1.1 by 2 | 1.8 |
|
||||
| 365 | 0.13.0 | | | **Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
|
||||
| 368 | 0.13.0 | | | **Call a release review due, call a release ready to cut, and tell whether a chunk owes a scoring run, for a plan outside the checkout, from the project repo's history or from Jira.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
|
||||
@@ -25,6 +25,7 @@
|
||||
| 298 | 0.13.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 0.9 |
|
||||
| 395 | 0.13.0 | | decision | **Keep each block a save didn't change, with its localId, and the spacing paragraphs beside it as Jira or Confluence holds them.** | 3 | 2 | 2 | 5 | 5 | 1.6 by 4 | 0.8 |
|
||||
| 425 | 0.13.0 | | | **Show on loop-claude's terminal why it doesn't start a chunk beside the next item, as chunk-preflight prints it.** | 1 | 1 | 2 | 2 | 2 | 1.5 by 2 | 0.7 |
|
||||
| 447 | 0.13.0 | | | **Stop a headless claude call that runs past a time limit, and report it as a failure, so a hung judge or stuck check can't hold the loop.** | 2 | 2 | 2 | 4 | 3 | 1.5 by 2 | 0.7 |
|
||||
| 367 | 0.13.0 | | | **Check that every write to a Jira plan leaves it ranked, as rule-hook's `todo-rank-check` checks `todo.md` today.** | 2 | 2 | 3 | 5 | 3 | 1.6 by 2 | 0.7 |
|
||||
| 423 | 0.13.0 | | | **Skip the judge's model call on a worker's final message that the offline classifier decides is finished.** | 3 | 3 | 3 | 7 | 5 | 1.4 by 2 | 0.4 |
|
||||
| 77 | 0.14.0 | | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 4 | 2 | 1 | 5 | 6 | 1.1 by 7, 1.4 by 7 | 2.1 |
|
||||
@@ -33,7 +34,7 @@
|
||||
| 273 | 0.14.0 | | decision | **Re-tune the E + I steps of `REVIEW_STEPS` for 6 to 8 once 10 finished chunks have scored an E + I of 6 or more, from how well E + I orders their review rounds and 🔴 findings.** | 1 | 2 | 2 | 3 | 2 | 1.1 by 2, 1.4 by 3 | 2.0 |
|
||||
| 268 | 0.14.0 | | | **End a run whose release is ready to cut with a notice on loop-claude's terminal, in place of an interrupt that waits for an answer.** | 2 | 2 | 3 | 5 | 3 | 1 by 5 | 1.9 |
|
||||
| 171 | 0.14.0 | | decision | **Group the routing figures' review rounds by the reviewers' rung as well as the worker's, so a call blames a rung only for rounds its own reviews raised.** | 3 | 2 | 2 | 5 | 4 | 1.2 by 3, 1.4 by 3 | 1.8 |
|
||||
| 300 | 0.14.0 | | owner | **Reach the comprehension floor: restructure the places the comprehension seats found hardest to follow, so the project reads at 7.0 or above on every dimension, Locality first; today the scoring run at 55235a0 scored Navigation 7.38, Locality 6.13, Shape 6.50, Self-sufficiency 6.50 and Overall 6.50.** | 4 | 2 | 3 | 7 | 8 | code health by 8 | 1.7 |
|
||||
| 300 | 0.14.0 | | owner | **Reach the comprehension floor: restructure the places the comprehension seats found hardest to follow, so the project reads at 7.0 or above on every dimension, Self-sufficiency first; today the scoring runs at 81e8d8a1 scored Navigation 7.13 and 6.75, Locality 6.38 and 5.63, Shape 6.25 and 5.88, Self-sufficiency 5.88 and 5.25, and Overall 6.50 and 5.88.** | 4 | 2 | 3 | 7 | 8 | code health by 8 | 1.7 |
|
||||
| 264 | 0.14.0 | | | **Report the comprehension seats and the escalated subagents as subagent types of their own in `routing-figures.md`, with their dollars.** | 2 | 2 | 1 | 3 | 3 | 1.4 by 5 | 1.7 |
|
||||
| 323 | 0.14.0 | | | **Count a release review as landed once its pull request has merged, whatever its commits are titled.** | 3 | 2 | 3 | 6 | 3 | 1.4 by 3, 1.5 by 2 | 1.7 |
|
||||
| 342 | 0.14.0 | | | **Research the tooling agents read, search and run commands with once more, and file an item for each tool that cuts tokens a replay shows keep quality.** | 1 | 2 | 1 | 2 | 3 | 1.4 by 3, 1.5 by 2 | 1.7 |
|
||||
@@ -68,19 +69,15 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 422. Check every 5 minutes whether a tmux window of the loop looks stuck by other signs than a lingering worker, investigate it with a headless claude, and kill only the process the investigation names.
|
||||
### 446. Kill the process a stuck check names where its verdict is `stuck`, once the human says the recorded verdicts match what they would have done.
|
||||
|
||||
Item 420 ends a worker that printed its final result and lingers. A window can also stall earlier: a test run that hangs on a shell ignoring SIGINT (item 402's first full run), a pane with no output while its process tree does no work, or a background shell older than the chunk's usual length.
|
||||
|
||||
Every 5 minutes loop-claude lists its own session's windows and, for one with such a sign, launches a headless claude on the project's rung that reads the pane, the process tree and the chunk's log files and returns `working`, `waiting on a person` or `stuck`, with the process that holds the window and why. Only `stuck` kills, and only the named process, never the worker or the loop. A window waiting on a gate (`run-heavy`, a test run that still writes output) or on the human's answer is not `stuck`. Each check and kill is printed on loop-claude's terminal with the evidence.
|
||||
|
||||
Run it first as a report that kills nothing, and turn the kill on once its verdicts over a few days match what the human would have done; record that in the item's decision. Blocked by 401, which moves loop-claude's cycle into a package; this item lands in the package.
|
||||
Item 422 judges a worker that stalls before its result, but only reports: each verdict goes to the terminal and to `stuck-workers.md` in the project's transcript archive. Question for the human: after a few days of verdicts there, do they match what you would have done? Once you say yes, kill only the process a `stuck` verdict names, and only where the pid is below the worker's claude, never the worker or the loop, and record the call in a decision.
|
||||
|
||||
Touches:
|
||||
|
||||
- `bin/loop_claude/loop/stuck_windows.py`
|
||||
- `bin/agent_cycle/tmux.py`: `Listed`, `listed`
|
||||
- `README.md`: `## Tools` (small)
|
||||
- `bin/agent_cycle/stuck_workers.py`: `Watch.check`
|
||||
- `bin/loop_effects/child_shells.py`: `stop` (small)
|
||||
- `bin/loop_claude/loop/usage.py`: `HELP` (small)
|
||||
|
||||
### 445. Refuse a plan that holds an item a `Finish item <ID>` commit already removed, so a merge resolution cannot bring a finished item back.
|
||||
|
||||
@@ -196,6 +193,17 @@ Today a Jira detail edited in one line, or a Confluence record with one entry ap
|
||||
|
||||
`chunks_beside` names each item it refuses and why, such as an unreadable `Touches:` list, and chunk-preflight prints those lines, but loop-claude's `chunk_windows.may_start` keeps only `picked` and drops them, so idle slots show no reason. Found by item 424's product-owner review; a list a person edits in Jira's UI, or one filed before item 424, still reaches loop-claude unchecked.
|
||||
|
||||
### 447. Stop a headless claude call that runs past a time limit, and report it as a failure, so a hung judge or stuck check can't hold the loop.
|
||||
|
||||
Every headless claude that `claude_reply` runs, the judge's and the stuck check's included, runs without a time limit, and the stuck check runs while the worker's own claude waits for its output to be read. A call that hangs holds the loop, and the worker with it, until a person notices. Found by item 422's stability review.
|
||||
|
||||
Give the `claude` effect a timeout, and have `claude_reply` turn its expiry into a Failure that names who timed out, as it names a non-zero exit. Pick the limit from the longest such call the transcript archive shows, with room to spare, and say where it comes from in a comment at the constant.
|
||||
|
||||
Touches:
|
||||
|
||||
- `bin/loop_effects/effects.py`: `system_effects` (small)
|
||||
- `bin/claude_code/reply.py`: `claude_reply`
|
||||
|
||||
### 367. Check that every write to a Jira plan leaves it ranked, as rule-hook's `todo-rank-check` checks `todo.md` today.
|
||||
|
||||
One of 0.13.0's history ports, per lilleman's call of 2026-11-12.
|
||||
@@ -267,29 +275,26 @@ Releases are the human's, and `CLAUDE.md` → "Working loop" already ends the lo
|
||||
|
||||
Raised by the goals audit of PR #296. `bin/routing_figures/calls.py` groups a chunk's review rounds by the rung its worker started on, and calls for a routing change when a lower rung took more of them. Since PR #296 the reviewers may run above the worker's rung, on the one their item's E + I picks, and no figure groups rounds by it, though the archived worker line names the E + I it follows from. A cheap worker on an item of high E + I gets stronger reviewers, who may raise more rounds, and a call could then blame the worker's rung for them: an interrupt of lilleman that settles nothing (goal 1.2).
|
||||
|
||||
### 300. Reach the comprehension floor: restructure the places the comprehension seats found hardest to follow, so the project reads at 7.0 or above on every dimension, Locality first; today the scoring run at 55235a0 scored Navigation 7.38, Locality 6.13, Shape 6.50, Self-sufficiency 6.50 and Overall 6.50.
|
||||
### 300. Reach the comprehension floor: restructure the places the comprehension seats found hardest to follow, so the project reads at 7.0 or above on every dimension, Self-sufficiency first; today the scoring runs at 81e8d8a1 scored Navigation 7.13 and 6.75, Locality 6.38 and 5.63, Shape 6.25 and 5.88, Self-sufficiency 5.88 and 5.25, and Overall 6.50 and 5.88.
|
||||
|
||||
Filed by the comprehension check of PR #473 at 13d1550 (`larv-review record-scoring`), which printed `Floor:` and `Next:`, and refreshed from the two runs of PR #560 at 55235a0. The lowest mean is Locality, 6.13 in both runs: the seats name rules held partly in agent-facing prose and partly in code, and state that one unit can't be read without three others open. Self-sufficiency and Shape follow at 6.50. Refresh this item's places and aim from the newest run before the chunk starts; do not file a second floor item.
|
||||
Filed by the comprehension check of PR #473 at 13d1550 (`larv-review record-scoring`), which printed `Floor:` and `Next:`, and refreshed from the two runs of PR #633 at 81e8d8a1. The lowest mean is Self-sufficiency: 5.88 in the first run and 5.25 in the second. Every seat says the same thing: the vocabulary lives only in `CLAUDE.md` → "Vocabulary", the why of thresholds and orderings sits behind `docs/decisions.md → …` pointers, and behaviour lives in prose that no code ties to its check. Locality follows, at 6.38 and 5.63: scheduling state lives in tmux window names, state files and flocks across three packages. Refresh this item's places and aim from the newest run before the chunk starts; do not file a second floor item.
|
||||
|
||||
Aim: one ambitious refactor of the areas below, sized so every mean reaches 7.0 on the chunk's own scoring run. Where a place is already easier to read when the chunk starts, drop it.
|
||||
|
||||
Places the seats named, hardest first (`larv-review places`), Locality first:
|
||||
Places the seats named, hardest first (`larv-review places`), Self-sufficiency first:
|
||||
|
||||
- Locality, Self-sufficiency: `bin/release_plan/release_standing.py:176` `next_release`, with `:162` `carried`: the cut from a tag, a dated heading or `BEFORE_FIRST_RELEASE`, release names replayed from the plan's history, and renames meet in the `is_earliest` test at `:198`; `carried` is reapplied in `chunk_item` and in `chunk_preflight/choices.py` `planned`, one relying on `zip` keeping order. All eight seats named it; four would least want to change it.
|
||||
- Locality: `bin/agent_cycle/routing.py:116` `unfinished_route`, with `:131` `escalated_route`: five rungs, the unfinished count and the escalation mark, which spans `escalation_mark.py`, `judge.py`, `verdicts.py`, `route.py` and `reload.py`. Seven seats named it; the module table is what makes it readable.
|
||||
- Locality: `bin/chunk_pr/main.py:298` `merged`, with `:384` `levelled_and_finished` and `:407` `finished`: a loop around the queue turn binds `done` only under some `if ready` branches, the approval wait is a fall-through, and `finished` destructures five subprocess results by position.
|
||||
- Locality: `skills/larv-review/SKILL.md:82` "Round cap" and `:90` "Re-entry", with `bin/review_rules/round_titles.py:45` and `:63` `past_cap`, `bin/rule_hook/round_cap.py:10` and `bin/larv_review/reviewer_header.py:73`: the round cap lives in five places, and SKILL.md's Done, Cap, Re-entry and Mechanical answers point at each other as prose.
|
||||
- Locality: `bin/loop_claude/loop/chunk_windows.py:107` `start_chunks`: under one flock it checks the session's owner, splits running from stopped windows, computes the room, reads the plan with a fallback, gathers held items and filters again, with the limit applied a second time in `may_start`.
|
||||
- Shape, Locality: `bin/release_plan/ranking.py:239` `score`, `:144` `weigh` and `:250` `weight_of`: every function forks on `SHAPE` versus `OLD_SHAPE`, and one Score depends on five inputs held in other files.
|
||||
- Locality, Shape: `bin/plan_store/stores.py:1` and `bin/release_plan/todo_table.py:1`: `release_plan` and `plan_store` import each other, the plan has three names, and the goals file sits under `release_plan`.
|
||||
- Shape: `bin/larv_review/plan.py:22` `scoring_run_due`, with `owed.py`, `unmeasured.py` and `scoring.py`: the scoring-run trigger is spread over four modules, three of whose names don't say it, in a flat 14-module package that mixes rounds, the comprehension check and its triggers.
|
||||
- Self-sufficiency: `bin/agent_cycle/judge.py:22` `INTERRUPT_RULES`, with `:197` `quotes_checked`: the judge's behaviour is held partly in prompt text and field metadata and partly in Python checks.
|
||||
- Shape: `bin/plan_store/jira_epic.py:356` `revision_of`: indexes the untyped tuple `scores` at `:530` returns by position.
|
||||
- Shape: `bin/agent_cycle/run.py:55` `Cycle.todo`, `bin/agent_cycle/hand_over.py:23` `hand_over` and `bin/release_plan/named_code.py:5`: names that say todo.md where the plan may be a Jira epic, a handover that is not the reload's, and a module with no code in it.
|
||||
- Navigation, Shape: `README.md:197` the `bin/` package list: 74 entries at one level, with the README's list as the only index.
|
||||
- Self-sufficiency: every seat names the decision-log titles that code cites for a reason (`routing.py:1`, `ranking.py:20`, `loop/routing_check.py:1`) as a cost; state each reason beside its constant and keep the citation as a footnote.
|
||||
|
||||
Scored L 4 because the refactor touches several components that hold known defects, E 2 because review and a manual check see a regression, I 3 because every session runs these tools. The chunk runs a scoring run, since it is a floor item.
|
||||
- Self-sufficiency, Locality: `bin/agent_cycle/routing.py:115` `unfinished_route`, with `:131` `escalated_route`: five rungs, a count and the escalation mark held at once. The module table resolves the what, but the why of the caps is only in the decision log. Every seat in both runs named it.
|
||||
- Self-sufficiency: `bin/agent_cycle/judge.py:235` `quotes_checked`, with `:23` `INTERRUPT_RULES` and `:76` `TASK_HEADING`: six quote kinds fail differently, the prompt is behaviour no code checks, and `TASK_HEADING` is silently an archive format. Seven seats named it.
|
||||
- Self-sufficiency: `bin/rule_hook/rules.py:27` `RULES`: each rule's only specification is one long English sentence, such as `env-file-read`, and its check lives in a sibling module that nothing ties it to. Five seats named it.
|
||||
- Self-sufficiency, Locality: `skills/larv-review/SKILL.md:89` "Round cap" and `:97` "Re-entry", with `bin/review_rules/review_passes.py:8` `PASS_AGENTS`: rules with exceptions to exceptions, a cap spread over three packages and this prose, and code constants that must match sentences here.
|
||||
- Locality: `bin/do_epic/main.py:283` `parked_through`: a polling loop with four mutable locals across three result types, and a crash-ordering invariant across a state file, a window name and Jira. Five seats named it; two would least want to change it.
|
||||
- Locality: `bin/do_epic/scheduler.py:136` `start_chunks`, with `bin/loop_claude/loop/chunk_windows.py:107` `may_start`: overlapping window sets from two stores feed an elif chain under nested flocks, with the tmux state's schema in `agent_cycle/tmux.py:92` `as_listed`.
|
||||
- Locality: `bin/chunk_pr/main.py:429` `levelled_and_finished`: a load-bearing order held together by one-liners where a Failure's truthiness carries the control flow.
|
||||
- Locality, Shape: `bin/agent_cycle/run.py:150` `rule`, with `:32` `Cycle` and `:240` `close`: a 16-field value, very long single-line expressions, and an archive-before-step-up order that only reading order tells.
|
||||
- Locality: `bin/do_epic/main.py:11` imports, with `bin/loop_claude/loop/reload.py:18` `LoopRun`: do_epic reaches into eight loop_claude modules over a wire format a renamed field breaks.
|
||||
- Shape, Navigation: `bin/agent_cycle/verdicts.py:1`, `bin/loop_claude/cycle/` and `bin/loop_state/paths.py:6`: stale `cycle.` names, and folders whose names mislead.
|
||||
- Locality: `bin/project_config/settings.py:182` `load`: one list encodes the whole validation matrix.
|
||||
- Shape: `bin/agent_cycle/stuck_workers.py:52` `Watch.check`: the condition at line 59 mixes `or` and `and` without parentheses over dicts mutated across ticks.
|
||||
|
||||
### 264. Report the comprehension seats and the escalated subagents as subagent types of their own in `routing-figures.md`, with their dollars.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user