Judge a worker that stalls before its result with a headless claude, and report the verdict without a kill
CI / tests (pull_request) Failing after 4m3s

This commit is contained in:
2026-11-19 08:55:14 +01:00
parent a559bbfcbf
commit 08038d93eb
10 changed files with 327 additions and 20 deletions
+1 -1
View File
@@ -122,7 +122,7 @@ def work(cycle: Cycle, answered: QuotedAnswer | TypedAnswer | None, stepped: boo
if isinstance(answered, QuotedAnswer):
say(f"\n=== {cycle.prompts.quote_sources} answered: {answered.question} ===\n{answered.answer}", cycle.effects, stdout=True)
# worker_start runs inside the lambda: a failed attempt may have opened the session.
if stop := retried("worker", lambda: run_worker(cycle.root, worker_start(cycle, answered, stepped), worker_environment(cycle), cycle.project_archive, cycle.effects), cycle.effects):
if stop := retried("worker", lambda: run_worker(cycle.root, worker_start(cycle, answered, stepped), worker_environment(cycle), cycle.project_archive, cycle.project.rung if cycle.project else cycle.route.rung, cycle.effects), cycle.effects):
return stop
path = transcript_file(cycle.session)
return WorkerRan(path) if path is not None else Stop(1, f"claude left no transcript for session {cycle.session.id}")
+91
View File
@@ -0,0 +1,91 @@
"""Asks a headless claude whether a worker that prints nothing before its final result is stuck, where one of its shells shows a sign of it, and reports the verdict.
A worker can stall on a command that hangs, as a full test run did on a shell that ignored SIGINT (item 402). The check runs on the tick `lingering_workers` uses, CHECK_EVERY seconds after the worker's last line, so a worker that prints is never checked. A shell shows a sign where it used no CPU since the last check and wrote nothing for QUIET seconds, or where it is older than OLD; one that runs a busy tool, as `Shell.runs_busy_tool` says, shows none, since a wait for a gate is work. Each chunk's loop checks its own worker: no process watches a whole tmux session, and the shells' commands, processes and output hold what the worker's pane shows.
The verdict is only reported, on the terminal and in RECORD in the project's transcript archive, and kills nothing until its verdicts have matched what the human would have done.
"""
import dataclasses
import json
import pathlib
import typing
from agent_cycle.lingering_workers import CHECK_EVERY, QUIET
from claude_code.entries import structured_output
from claude_code.reply import claude_reply, Limited
from failure import Failure
from loop_effects.child_shells import Shell
from loop_effects.effects import Effects, say, Streaming
from model_routing.ladder import Rung
from text_files.utc_stamp import utc
# Seconds after which a shell holds a worker longer than a chunk usually runs.
OLD = 4 * 3600
RECORD = "stuck-workers.md"
VERDICTS = ("working", "waiting on a person", "stuck")
SCHEMA = {
"type": "object",
"properties": {
"verdict": {"enum": list(VERDICTS)},
"process": {"type": "integer", "description": "The pid of the process that holds the worker, from the evidence."},
"why": {"type": "string", "description": "One sentence on why that process holds it."},
},
"required": ["verdict", "process", "why"],
"additionalProperties": False,
}
PROMPT = f"""A headless Claude Code worker has printed nothing for {CHECK_EVERY} seconds. Judge from its shells below whether it is `working`, `waiting on a person` or `stuck`, and name the pid of the process that holds it and why.
A worker that waits on a gate, such as run-heavy's wait for a slot or a test run that still writes output or uses CPU, is working. Stuck means the process will not finish on its own, as a command waiting on a child that never exits.
"""
@dataclasses.dataclass
class Watch:
"""What checks a worker for stuck shells: the rung its investigations run on, the project's transcript archive, and each shell's CPU seconds at the last check."""
rung: Rung
archive: pathlib.Path
cpu: dict[int, float] = dataclasses.field(default_factory=dict)
def check(self, proc: Streaming, root: str, effects: Effects) -> None:
"""Investigates the worker whose claude is proc where a shell shows a sign, and says the verdict on the terminal and in RECORD; a failed investigation is said on stderr."""
now = effects.now()
shells = proc.shells()
signs = [sign for shell in shells if (sign := sign_of(shell, self.cpu.get(shell.pid), now))]
self.cpu = {shell.pid: shell.cpu for shell in shells}
if not signs:
return
judged = investigated(shells, self.rung, root, effects)
if isinstance(judged, Failure):
say(f"could not judge whether the worker is stuck: {judged.reason}", effects)
return
line = f"stuck check: {'; '.join(signs)}; a headless claude judged the worker {judged['verdict']}, held by process {judged['process']}: {judged['why']}"
say(line, effects, stdout=True)
record = self.archive / RECORD
try:
record.parent.mkdir(parents=True, exist_ok=True)
with record.open("a") as handle:
handle.write(f"- {utc(now)} {line}\n")
except OSError as failure:
say(f"could not record the stuck check in {record}: {failure.strerror}", effects)
def sign_of(shell: Shell, cpu_before: float | None, now: float) -> str:
"""What says shell may hold its worker stuck, given the CPU seconds it had used at the last check; empty where nothing does."""
if shell.runs_busy_tool:
return ""
idle = cpu_before == shell.cpu and now - shell.written > QUIET
if not (idle or now - shell.started > OLD):
return ""
return f"shell {shell.pid} has run {int(now - shell.started)} s, written nothing for {int(now - shell.written)} s{f' and used no CPU for {CHECK_EVERY} s' if idle else ''}: {shell.command}"
def investigated(shells: list[Shell], rung: Rung, root: str, effects: Effects) -> dict[str, typing.Any] | Failure:
"""A headless claude's verdict on the worker holding shells, as SCHEMA has it, or why there is none. It runs without tools, so it judges only the evidence it is handed."""
evidence = "\n\n".join(f"Shell {shell.pid}: {shell.command}\nIts processes, as pid, state, CPU and command line:\n{chr(10).join(shell.processes) or 'none'}\nThe end of its output:\n{shell.output or 'none'}" for shell in shells)
reply = claude_reply(effects.claude, ["--json-schema", json.dumps(SCHEMA), *rung.flags(), "--tools", ""], root, PROMPT + "\n" + evidence, "the stuck check")
if isinstance(reply, Limited):
return Failure(f"claude hit a limit: {reply.said}")
if isinstance(reply, Failure):
return reply
output = structured_output(reply)
if not (isinstance(output, dict) and output.get("verdict") in VERDICTS and isinstance(output.get("process"), int) and isinstance(output.get("why"), str)):
return Failure(f"answered outside its schema: {output!r}")
return output
+89
View File
@@ -0,0 +1,89 @@
import contextlib
import dataclasses
import io
import json
import pathlib
import tempfile
import unittest
from agent_cycle import stuck_workers, worker
from loop_claude.test_loop_claude import FakeEffects, NOW, Streamed, ran
from loop_effects.child_shells import Shell
from model_routing.ladder import Rung
from text_files.utc_stamp import utc
RUNG = Rung("claude-opus-5-5", "medium")
ASSISTANT = {"type": "assistant", "message": {"content": []}}
def verdict(said="stuck", process=8, why="pytest waits on a shell that ignores SIGINT"):
return ran(json.dumps({"type": "result", "structured_output": {"verdict": said, "process": process, "why": why}}))
def shell(cpu=2.0, started=NOW - 1000, written=NOW - 1000, runs_busy_tool=False):
return Shell(pid=7, started=started, command="bin/run-tests", written=written, runs_busy_tool=runs_busy_tool, cpu=cpu, processes=("8 S 2.0 s python3 -m pytest",), output="collected 40 items")
class StuckWorkers(unittest.TestCase):
def worker_with(self, *events, held=(), replies=()):
"""Runs a worker whose claude prints events and holds the shells held, while each investigation replies in turn; returns the claude, the fake, what was said on stdout and stderr and the record's text."""
clock = [NOW]
streamed = Streamed(0, *events, clock=clock, held=held)
fake = FakeEffects(streams=[streamed], claude=replies)
effects = dataclasses.replace(fake.effects(), now=lambda: clock[0])
with tempfile.TemporaryDirectory() as folder, contextlib.redirect_stdout(io.StringIO()) as said, contextlib.redirect_stderr(io.StringIO()) as failed:
archive = pathlib.Path(folder, "transcripts", "project")
worker.run_worker("/r", [], {}, archive, RUNG, effects)
record = archive / stuck_workers.RECORD
return streamed, fake, said.getvalue(), failed.getvalue(), record.read_text() if record.exists() else ""
def test_a_silent_worker_whose_shell_used_no_cpu_since_the_last_check_is_investigated_and_the_verdict_said_and_recorded_without_a_kill(self):
streamed, fake, said, _, record = self.worker_with(ASSISTANT, None, None, held=[shell()], replies=[verdict()])
[(args, prompt)] = fake.made("claude")
self.assertEqual(args[args.index("--model") + 1 :][:3], ["claude-opus-5-5", "--effort", "medium"])
self.assertEqual(args[args.index("--tools") + 1], "")
for evidence in ("bin/run-tests", "8 S 2.0 s python3 -m pytest", "collected 40 items", "working", "waiting on a person", "stuck"):
self.assertIn(evidence, prompt)
line = "stuck check: shell 7 has run 1600 s, written nothing for 1600 s and used no CPU for 300 s: bin/run-tests; a headless claude judged the worker stuck, held by process 8: pytest waits on a shell that ignores SIGINT"
self.assertEqual(said, f"<{NOW + 600}> {line}\n")
self.assertEqual(record, f"- {utc(NOW + 600)} {line}\n")
self.assertEqual(streamed.stopped, [])
def test_a_shell_older_than_a_chunk_usually_runs_is_investigated_at_the_first_check(self):
_, fake, said, _, _ = self.worker_with(ASSISTANT, None, held=[shell(started=NOW - stuck_workers.OLD, written=NOW + 290)], replies=[verdict("working", 7, "the tests still print")])
self.assertEqual(len(fake.made("claude")), 1)
self.assertIn(f"shell 7 has run {stuck_workers.OLD + 300} s, written nothing for 10 s: bin/run-tests; a headless claude judged the worker working, held by process 7: the tests still print", said)
def test_a_worker_whose_shells_show_no_sign_is_not_investigated(self):
for events, held in (
((ASSISTANT, None, None), [shell(runs_busy_tool=True)]),
((ASSISTANT, None, None), [shell(written=NOW + 600 - 179)]),
((ASSISTANT, None), [shell()]),
((ASSISTANT, None, None), []),
(({"type": "result"}, None, None), [shell()]),
((ASSISTANT, None), [shell(started=NOW + 300 - stuck_workers.OLD)]),
):
with self.subTest(events=events, held=held):
_, fake, said, _, record = self.worker_with(*events, held=held)
self.assertEqual((fake.made("claude"), record), ([], ""))
def test_a_shell_that_used_cpu_since_the_last_check_is_not_investigated(self):
clock = [NOW]
used = [shell(cpu=2.0), shell(cpu=2.5)]
streamed = Streamed(0, ASSISTANT, None, None, clock=clock)
streamed.shells = lambda: [used.pop(0)]
fake = FakeEffects(streams=[streamed])
with contextlib.redirect_stdout(io.StringIO()):
worker.run_worker("/r", [], {}, pathlib.Path("/archive"), RUNG, dataclasses.replace(fake.effects(), now=lambda: clock[0]))
self.assertEqual(fake.made("claude"), [])
def test_an_investigation_that_fails_or_answers_outside_the_schema_is_said_on_stderr(self):
for reply, reason in ((ran(code=1, stderr="no model"), "no model"), (verdict("hung"), "answered outside its schema")):
with self.subTest(reason=reason):
_, _, said, failed, record = self.worker_with(ASSISTANT, None, None, held=[shell()], replies=[reply])
self.assertEqual((said, record), ("", ""))
self.assertIn(f"could not judge whether the worker is stuck: {reason}", failed)
if __name__ == "__main__":
unittest.main()
+10 -6
View File
@@ -9,6 +9,7 @@ from claude_code.transcript import part_line
from machine_load.memory import ROOM_SHARE
from model_routing.ladder import Rung
from claude_code.session import Session
from agent_cycle.stuck_workers import Watch
PRINTED_TOOL_INPUT = 160
# Seconds between reads of the machine's memory while too little of it is available to start a worker.
@@ -18,13 +19,14 @@ AUTO_COMPACT_WINDOW = "CLAUDE_CODE_AUTO_COMPACT_WINDOW"
COMPACT_WINDOW = 200_000
def run_worker(root: str, args: list[str], env: dict[str, str], project_archive: pathlib.Path, effects: Effects) -> Failure | Limited | None:
def run_worker(root: str, args: list[str], env: dict[str, str], project_archive: pathlib.Path, investigator: Rung, effects: Effects) -> Failure | Limited | None:
"""Runs the worker's claude once the machine has room for it, as run_claude does with a Watch whose stuck checks run on investigator, the project's rung."""
wait_for_memory(effects)
return run_claude(root, args, env, effects, project_archive / RECORD)
return run_claude(root, args, env, effects, Watch(investigator, project_archive))
def run_claude(root: str, args: list[str], env: dict[str, str], effects: Effects, record: pathlib.Path | None = None) -> Failure | Limited | None:
"""Runs a headless claude at once, printing what it says, and returns how it failed, or None. Where record is given, a claude that lingers after its final result loses its quiet shells, as `lingering_workers.stopped_shells` says, and each stop is appended to record."""
def run_claude(root: str, args: list[str], env: dict[str, str], effects: Effects, watch: Watch | None = None) -> Failure | Limited | None:
"""Runs a headless claude at once, printing what it says, and returns how it failed, or None. Where watch is given, a claude that lingers after its final result loses its quiet shells, as `lingering_workers.stopped_shells` says, each stop appended to RECORD in watch's archive; and a silent claude that has printed no result is checked for stuck shells, as `stuck_workers` says."""
try:
proc = effects.stream_claude(args, root, env)
except OSError as failure:
@@ -35,8 +37,10 @@ def run_claude(root: str, args: list[str], env: dict[str, str], effects: Effects
resets: list[int] = []
for raw in proc.lines(CHECK_EVERY):
if raw is None:
if record is not None:
stopped_shells(proc, printed_result, record, effects)
if watch is not None:
stopped_shells(proc, printed_result, watch.archive / RECORD, effects)
if not printed_result:
watch.check(proc, root, effects)
continue
event = json_object(raw) or {}
printed_result = ends_stream(event)
+8
View File
@@ -88,6 +88,14 @@ minutes, runs chunk-pr or run-heavy, or holds a heavy slot. It prints each stop
the UTC time, to lingering-workers.md at the project's folder in that archive
(bin/agent_cycle/lingering_workers.py).
A worker can also stall before its result, on a command that hangs. Every 5 minutes without output
from its claude, where a shell has used no CPU since the last check and written nothing for 3
minutes, or has run over 4 hours, and runs no chunk-pr or run-heavy, a headless claude on the
project's rung reads the shells' commands, processes and output. It judges the worker working,
waiting on a person or stuck, and names the process that holds it. loop-claude prints the verdict
with its evidence and appends it to stuck-workers.md beside lingering-workers.md. It kills nothing
yet (bin/agent_cycle/stuck_workers.py).
Before each worker starts, loop-claude waits until at least a quarter of the machine's memory is
available (bin/machine_load/memory.py).
+6 -6
View File
@@ -795,7 +795,7 @@ class Limits(unittest.TestCase):
allowed = {"type": "rate_limit_event", "rate_limit_info": {"status": "allowed", "resetsAt": NOW + 9000}}
limited = {"type": "result", "api_error_status": 429, "result": "You've hit your session limit"}
fake = FakeEffects(streams=[Streamed(1, rejected[1], allowed, rejected[0], limited)])
self.assertEqual(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), fake.effects()), Limited(NOW + 600, "You've hit your session limit"))
self.assertEqual(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), Rung("m"), fake.effects()), Limited(NOW + 600, "You've hit your session limit"))
def test_a_limit_outranks_the_exit_code_and_its_message_fits_one_line(self):
def limited(**said):
@@ -829,7 +829,7 @@ class LingeringWorkers(unittest.TestCase):
with tempfile.TemporaryDirectory() as folder, contextlib.redirect_stdout(io.StringIO()) as said:
# The archive folder of a project's first stop doesn't exist yet.
archive = pathlib.Path(folder, "transcripts", "project")
worker.run_worker("/r", [], {}, archive, effects)
worker.run_worker("/r", [], {}, archive, Rung("m"), effects)
record = archive / lingering_workers.RECORD
return streamed, said.getvalue(), record.read_text() if record.exists() else ""
@@ -866,7 +866,7 @@ class LingeringWorkers(unittest.TestCase):
def test_a_worker_reads_on_past_a_check_that_found_it_still_working(self):
clock = [NOW]
effects = dataclasses.replace(FakeEffects(streams=[Streamed(0, None, self.RESULT, clock=clock)]).effects(), now=lambda: clock[0])
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), effects))
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), Rung("m"), effects))
def test_a_stop_that_cannot_be_recorded_is_said_on_stderr_and_every_shell_is_still_stopped(self):
clock = [NOW]
@@ -874,7 +874,7 @@ class LingeringWorkers(unittest.TestCase):
streamed = Streamed(0, self.RESULT, None, clock=clock, held=held)
effects = dataclasses.replace(FakeEffects(streams=[streamed]).effects(), now=lambda: clock[0])
with tempfile.NamedTemporaryFile() as blocker, contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()) as failed:
worker.run_worker("/r", [], {}, pathlib.Path(blocker.name), effects)
worker.run_worker("/r", [], {}, pathlib.Path(blocker.name), Rung("m"), effects)
self.assertEqual(streamed.stopped, held)
self.assertIn("could not record the stop", failed.getvalue())
@@ -888,14 +888,14 @@ class MemoryWait(unittest.TestCase):
def test_a_worker_starts_only_once_a_quarter_of_memory_is_available(self):
fake = FakeEffects(free_memory=[memory.Memory(available=3, total=16), memory.Memory(available=3, total=16), memory.Memory(available=4, total=16)], streams=[Streamed(0)])
with contextlib.redirect_stderr(io.StringIO()) as said:
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), fake.effects()))
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), Rung("m"), fake.effects()))
self.assertEqual([seam for seam, _ in fake.calls], ["memory", "sleep", "memory", "sleep", "memory", "stream_claude"])
self.assertEqual(fake.made("sleep"), [worker.MEMORY_POLL] * 2)
self.assertEqual(said.getvalue().count("waiting"), 1, said.getvalue())
def test_a_worker_starts_at_once_where_memory_cannot_be_read(self):
fake = FakeEffects(free_memory=[None], streams=[Streamed(0)])
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), fake.effects()))
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), Rung("m"), fake.effects()))
self.assertEqual([seam for seam, _ in fake.calls], ["memory", "stream_claude"])
+52 -6
View File
@@ -13,6 +13,7 @@ from machine_load.slots import slot_folder
# The command names of a shell claude starts for a Bash call.
SHELL_NAMES = frozenset({"bash", "dash", "sh", "zsh"})
COMMAND_CHARS = 300
OUTPUT_BYTES = 2000
# Scripts whose process below a shell shows work that may go quiet for long: run-heavy while it waits for a slot, and chunk-pr, which waits for a merge's approval and checks.
BUSY_TOOLS = frozenset({"chunk-pr", "run-heavy"})
@@ -28,19 +29,38 @@ class Shell:
written: float
# Whether a process below the shell runs one of BUSY_TOOLS or holds a heavy slot, a file of slot_folder that run-heavy leaves open in the command it runs.
runs_busy_tool: bool
# CPU seconds the shell and every process below it have used.
cpu: float = 0.0
# Each process below the shell, as `<pid> <state> <CPU seconds> s <command line>`, with the state letter /proc gives, such as S for sleeping.
processes: tuple[str, ...] = ()
# The last OUTPUT_BYTES of the file the shell's output goes to; empty where it goes to no file.
output: str = ""
def parent_and_start(pid: int) -> tuple[int, int, str] | None:
"""The parent pid, the start time in clock ticks since boot and the command name of a process; None where it is gone."""
def stat_of(pid: int) -> tuple[str, list[str]] | None:
"""A process's command name and the fields of /proc/<pid>/stat after it, from the state on; None where it is gone."""
try:
with open(f"/proc/{pid}/stat") as handle:
text = handle.read()
except OSError:
return None
# The name sits in parentheses and may hold spaces or parentheses itself, so split at the last one.
name = text[text.index("(") + 1 : text.rindex(")")]
fields = text[text.rindex(")") + 2 :].split()
return int(fields[1]), int(fields[19]), name
return text[text.index("(") + 1 : text.rindex(")")], text[text.rindex(")") + 2 :].split()
def parent_and_start(pid: int) -> tuple[int, int, str] | None:
"""The parent pid, the start time in clock ticks since boot and the command name of a process; None where it is gone."""
found = stat_of(pid)
return None if found is None else (int(found[1][1]), int(found[1][19]), found[0])
def cpu_and_state(pid: int) -> tuple[float, str]:
"""The CPU seconds a process has used, in user and kernel mode, and its state letter; none and `gone` where it is gone."""
found = stat_of(pid)
if found is None:
return 0.0, "gone"
fields = found[1]
return (int(fields[11]) + int(fields[12])) / os.sysconf("SC_CLK_TCK"), fields[0]
def children_of(pid: int) -> list[int]:
@@ -77,6 +97,31 @@ def output_written(pid: int, started: float) -> float:
return info.st_mtime if stat.S_ISREG(info.st_mode) else started
def output_tail(pid: int) -> str:
"""The last OUTPUT_BYTES of the file that pid's standard output goes to; empty where it goes to a pipe or a terminal."""
path = f"/proc/{pid}/fd/1"
try:
# Before the open, which can block on a pipe.
if not stat.S_ISREG(os.stat(path).st_mode):
return ""
with open(path, "rb") as handle:
handle.seek(max(0, os.fstat(handle.fileno()).st_size - OUTPUT_BYTES))
return handle.read().decode(errors="replace")
except OSError:
return ""
def tree_of(pid: int) -> tuple[float, tuple[str, ...]]:
"""The CPU seconds pid and every process below it have used, and each process below it as `Shell.processes` shows one."""
total = cpu_and_state(pid)[0]
shown = []
for each in descendants(pid):
cpu, state = cpu_and_state(each)
total += cpu
shown.append(f"{each} {state} {cpu:.1f} s {' '.join(argv_of(each))[:COMMAND_CHARS]}")
return total, tuple(shown)
def command_of(shell_argv: list[str]) -> str:
"""What a shell runs: the text of its command line after Claude Code's `eval '`, up to the last ` < /dev/null`, which a call that began in the foreground lacks; the whole line where it holds no `eval`."""
line = " ".join(shell_argv)
@@ -110,7 +155,8 @@ def shells_of(claude_pid: int) -> list[Shell]:
if found is None or found[2] not in SHELL_NAMES:
continue
started = booted + found[1] / ticks
shells.append(Shell(pid, started, command_of(argv_of(pid)), output_written(pid, started), runs_busy_tool(pid, slots)))
cpu, processes = tree_of(pid)
shells.append(Shell(pid, started, command_of(argv_of(pid)), output_written(pid, started), runs_busy_tool(pid, slots), cpu, processes, output_tail(pid)))
return shells
+16
View File
@@ -189,6 +189,22 @@ class ChildShells(unittest.TestCase):
self.assertEqual(self.shell_of(to_file).written, 1000)
self.assertEqual(self.shell_of(to_pipe).written, self.shell_of(to_pipe).started)
def test_reads_the_end_of_a_shells_output_file_and_nothing_from_a_pipe(self):
with tempfile.NamedTemporaryFile() as output:
output.write(b"x" * child_shells.OUTPUT_BYTES + b"last line\n")
output.flush()
to_file = self.started("sleep 60; true", stdout=output)
to_pipe = self.started("sleep 60; true", stdout=subprocess.PIPE)
self.assertEqual(self.shell_of(to_file).output, "x" * (child_shells.OUTPUT_BYTES - 10) + "last line\n")
self.assertEqual(self.shell_of(to_pipe).output, "")
def test_reads_each_process_below_a_shell_and_the_cpu_the_whole_tree_used(self):
busy = self.started("timeout 5 sh -c 'while :; do :; done'; true")
wait_until(lambda: self.shell_of(busy).cpu > 0.1)
processes = self.shell_of(busy).processes
self.assertEqual([line.split(" ", 4)[4] for line in processes], ["timeout 5 sh -c while :; do :; done", "sh -c while :; do :; done"])
self.assertEqual(processes[1].split(" ")[1], "R")
def test_stops_a_shell_and_the_process_it_started_and_no_other(self):
shell = self.started("sleep 60; true")
other = subprocess.Popen(["sleep", "60"])
+42
View File
@@ -3071,3 +3071,45 @@ Read most (opens and searches):
- `docs/decisions.md` 2
- `bin/do_epic/backend/protocol.py` 1
- `bin/do_epic/findings.py` 1
## Reading cost, chunk 59bf63f1876b, finished 2026-11-19T04:43:46Z
97876 new input tokens before the first commit.
Read most (opens and searches):
- `README.md` 6
- `skills/larv-review/SKILL.md` 4
- `.gitignore` 3
- `bin/do_epic/main.py` 3
- `bin/project_config/settings.py` 3
- `bin/atlassian/client.py` 2
- `bin/do_epic/backend/contract.py` 2
- `bin/do_epic/epic_record.py` 2
- `bin/do_epic/findings.py` 2
- `bin/do_epic/finish.py` 2
## Reading cost, chunk 476ff8575248, finished 2026-11-19T04:48:45Z
22763 new input tokens before the first commit.
Read most (opens and searches):
- `bin/agent_cycle/interrupt.py` 1
## Reading cost, chunk 8cb7c9836e2d, finished 2026-11-19T07:47:33Z
106608 new input tokens before the first commit.
Read most (opens and searches):
- `bin/loop_claude/test_loop_claude.py` 8
- `bin/loop_claude/loop/usage.py` 6
- `bin/loop_effects/effects.py` 5
- `README.md` 4
- `bin/loop_claude/test_black_box.py` 4
- `bin/loop_claude/loop/main.py` 3
- `bin/loop_effects/test_effects.py` 2
- `docs/decisions.md` 2
- `bin/repo_checks/test_import_order.py` 1
- `bin/run-heavy` 1
+12 -1
View File
@@ -6,13 +6,14 @@
`Bar = 0.1`
`Next ID = 446`
`Next ID = 447`
## Items
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 422 | 0.12.0 | | | **Check every 5 minutes whether a tmux window of the loop looks stuck by other signs than a lingering worker, investigate it with a headless claude, and kill only the process the investigation names.** | 3 | 2 | 3 | 6 | 5 | 1.5 by 4 | 0.8 |
| 446 | 0.13.0 | | question | **Kill the process a stuck check names where its verdict is `stuck`, once the human says the recorded verdicts match what they would have done.** | 2 | 2 | 3 | 5 | 3 | 1.5 by 3 | 1.0 |
| 445 | 0.13.0 | | | **Refuse a plan that holds an item a `Finish item <ID>` commit already removed, so a merge resolution cannot bring a finished item back.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3, 1.1 by 2 | 1.8 |
| 365 | 0.13.0 | | | **Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
| 368 | 0.13.0 | | | **Call a release review due, call a release ready to cut, and tell whether a chunk owes a scoring run, for a plan outside the checkout, from the project repo's history or from Jira.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
@@ -82,6 +83,16 @@ Touches:
- `bin/agent_cycle/tmux.py`: `Listed`, `listed`
- `README.md`: `## Tools` (small)
### 446. Kill the process a stuck check names where its verdict is `stuck`, once the human says the recorded verdicts match what they would have done.
Item 422 judges a worker that stalls before its result, but only reports: each verdict goes to the terminal and to `stuck-workers.md` in the project's transcript archive. Question for the human: after a few days of verdicts there, do they match what you would have done? Once you say yes, kill only the process a `stuck` verdict names, and only where the pid is below the worker's claude, never the worker or the loop, and record the call in a decision.
Touches:
- `bin/agent_cycle/stuck_workers.py`: `Watch.check`
- `bin/loop_effects/child_shells.py`: `stop` (small)
- `bin/loop_claude/loop/usage.py`: `HELP` (small)
### 445. Refuse a plan that holds an item a `Finish item <ID>` commit already removed, so a merge resolution cannot bring a finished item back.
Merging main into item-442 in 610edb1a resolved a todo.md conflict by keeping item 310's row and detail, which e019a6b7 had just finished. loop-claude then started a chunk on item 310, whose work had already merged, and that chunk only removed the item again. No check noticed: `rank_todo.py --check` reads one file, and rule-hook's `todo-rank-check` sees only an agent's writes, not what a merge resolution keeps.