Judge a worker that stalls before its result with a headless claude, and report the verdict without a kill
CI / tests (pull_request) Failing after 4m3s
CI / tests (pull_request) Failing after 4m3s
This commit is contained in:
@@ -122,7 +122,7 @@ def work(cycle: Cycle, answered: QuotedAnswer | TypedAnswer | None, stepped: boo
|
||||
if isinstance(answered, QuotedAnswer):
|
||||
say(f"\n=== {cycle.prompts.quote_sources} answered: {answered.question} ===\n{answered.answer}", cycle.effects, stdout=True)
|
||||
# worker_start runs inside the lambda: a failed attempt may have opened the session.
|
||||
if stop := retried("worker", lambda: run_worker(cycle.root, worker_start(cycle, answered, stepped), worker_environment(cycle), cycle.project_archive, cycle.effects), cycle.effects):
|
||||
if stop := retried("worker", lambda: run_worker(cycle.root, worker_start(cycle, answered, stepped), worker_environment(cycle), cycle.project_archive, cycle.project.rung if cycle.project else cycle.route.rung, cycle.effects), cycle.effects):
|
||||
return stop
|
||||
path = transcript_file(cycle.session)
|
||||
return WorkerRan(path) if path is not None else Stop(1, f"claude left no transcript for session {cycle.session.id}")
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
"""Asks a headless claude whether a worker that prints nothing before its final result is stuck, where one of its shells shows a sign of it, and reports the verdict.
|
||||
|
||||
A worker can stall on a command that hangs, as a full test run did on a shell that ignored SIGINT (item 402). The check runs on the tick `lingering_workers` uses, CHECK_EVERY seconds after the worker's last line, so a worker that prints is never checked. A shell shows a sign where it used no CPU since the last check and wrote nothing for QUIET seconds, or where it is older than OLD; one that runs a busy tool, as `Shell.runs_busy_tool` says, shows none, since a wait for a gate is work. Each chunk's loop checks its own worker: no process watches a whole tmux session, and the shells' commands, processes and output hold what the worker's pane shows.
|
||||
|
||||
The verdict is only reported, on the terminal and in RECORD in the project's transcript archive, and kills nothing until its verdicts have matched what the human would have done.
|
||||
"""
|
||||
import dataclasses
|
||||
import json
|
||||
import pathlib
|
||||
import typing
|
||||
|
||||
from agent_cycle.lingering_workers import CHECK_EVERY, QUIET
|
||||
from claude_code.entries import structured_output
|
||||
from claude_code.reply import claude_reply, Limited
|
||||
from failure import Failure
|
||||
from loop_effects.child_shells import Shell
|
||||
from loop_effects.effects import Effects, say, Streaming
|
||||
from model_routing.ladder import Rung
|
||||
from text_files.utc_stamp import utc
|
||||
|
||||
# Seconds after which a shell holds a worker longer than a chunk usually runs.
|
||||
OLD = 4 * 3600
|
||||
RECORD = "stuck-workers.md"
|
||||
VERDICTS = ("working", "waiting on a person", "stuck")
|
||||
SCHEMA = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"verdict": {"enum": list(VERDICTS)},
|
||||
"process": {"type": "integer", "description": "The pid of the process that holds the worker, from the evidence."},
|
||||
"why": {"type": "string", "description": "One sentence on why that process holds it."},
|
||||
},
|
||||
"required": ["verdict", "process", "why"],
|
||||
"additionalProperties": False,
|
||||
}
|
||||
PROMPT = f"""A headless Claude Code worker has printed nothing for {CHECK_EVERY} seconds. Judge from its shells below whether it is `working`, `waiting on a person` or `stuck`, and name the pid of the process that holds it and why.
|
||||
A worker that waits on a gate, such as run-heavy's wait for a slot or a test run that still writes output or uses CPU, is working. Stuck means the process will not finish on its own, as a command waiting on a child that never exits.
|
||||
"""
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class Watch:
|
||||
"""What checks a worker for stuck shells: the rung its investigations run on, the project's transcript archive, and each shell's CPU seconds at the last check."""
|
||||
rung: Rung
|
||||
archive: pathlib.Path
|
||||
cpu: dict[int, float] = dataclasses.field(default_factory=dict)
|
||||
|
||||
def check(self, proc: Streaming, root: str, effects: Effects) -> None:
|
||||
"""Investigates the worker whose claude is proc where a shell shows a sign, and says the verdict on the terminal and in RECORD; a failed investigation is said on stderr."""
|
||||
now = effects.now()
|
||||
shells = proc.shells()
|
||||
signs = [sign for shell in shells if (sign := sign_of(shell, self.cpu.get(shell.pid), now))]
|
||||
self.cpu = {shell.pid: shell.cpu for shell in shells}
|
||||
if not signs:
|
||||
return
|
||||
judged = investigated(shells, self.rung, root, effects)
|
||||
if isinstance(judged, Failure):
|
||||
say(f"could not judge whether the worker is stuck: {judged.reason}", effects)
|
||||
return
|
||||
line = f"stuck check: {'; '.join(signs)}; a headless claude judged the worker {judged['verdict']}, held by process {judged['process']}: {judged['why']}"
|
||||
say(line, effects, stdout=True)
|
||||
record = self.archive / RECORD
|
||||
try:
|
||||
record.parent.mkdir(parents=True, exist_ok=True)
|
||||
with record.open("a") as handle:
|
||||
handle.write(f"- {utc(now)} {line}\n")
|
||||
except OSError as failure:
|
||||
say(f"could not record the stuck check in {record}: {failure.strerror}", effects)
|
||||
|
||||
|
||||
def sign_of(shell: Shell, cpu_before: float | None, now: float) -> str:
|
||||
"""What says shell may hold its worker stuck, given the CPU seconds it had used at the last check; empty where nothing does."""
|
||||
if shell.runs_busy_tool:
|
||||
return ""
|
||||
idle = cpu_before == shell.cpu and now - shell.written > QUIET
|
||||
if not (idle or now - shell.started > OLD):
|
||||
return ""
|
||||
return f"shell {shell.pid} has run {int(now - shell.started)} s, written nothing for {int(now - shell.written)} s{f' and used no CPU for {CHECK_EVERY} s' if idle else ''}: {shell.command}"
|
||||
|
||||
|
||||
def investigated(shells: list[Shell], rung: Rung, root: str, effects: Effects) -> dict[str, typing.Any] | Failure:
|
||||
"""A headless claude's verdict on the worker holding shells, as SCHEMA has it, or why there is none. It runs without tools, so it judges only the evidence it is handed."""
|
||||
evidence = "\n\n".join(f"Shell {shell.pid}: {shell.command}\nIts processes, as pid, state, CPU and command line:\n{chr(10).join(shell.processes) or 'none'}\nThe end of its output:\n{shell.output or 'none'}" for shell in shells)
|
||||
reply = claude_reply(effects.claude, ["--json-schema", json.dumps(SCHEMA), *rung.flags(), "--tools", ""], root, PROMPT + "\n" + evidence, "the stuck check")
|
||||
if isinstance(reply, Limited):
|
||||
return Failure(f"claude hit a limit: {reply.said}")
|
||||
if isinstance(reply, Failure):
|
||||
return reply
|
||||
output = structured_output(reply)
|
||||
if not (isinstance(output, dict) and output.get("verdict") in VERDICTS and isinstance(output.get("process"), int) and isinstance(output.get("why"), str)):
|
||||
return Failure(f"answered outside its schema: {output!r}")
|
||||
return output
|
||||
@@ -0,0 +1,89 @@
|
||||
import contextlib
|
||||
import dataclasses
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from agent_cycle import stuck_workers, worker
|
||||
from loop_claude.test_loop_claude import FakeEffects, NOW, Streamed, ran
|
||||
from loop_effects.child_shells import Shell
|
||||
from model_routing.ladder import Rung
|
||||
from text_files.utc_stamp import utc
|
||||
|
||||
RUNG = Rung("claude-opus-5-5", "medium")
|
||||
ASSISTANT = {"type": "assistant", "message": {"content": []}}
|
||||
|
||||
|
||||
def verdict(said="stuck", process=8, why="pytest waits on a shell that ignores SIGINT"):
|
||||
return ran(json.dumps({"type": "result", "structured_output": {"verdict": said, "process": process, "why": why}}))
|
||||
|
||||
|
||||
def shell(cpu=2.0, started=NOW - 1000, written=NOW - 1000, runs_busy_tool=False):
|
||||
return Shell(pid=7, started=started, command="bin/run-tests", written=written, runs_busy_tool=runs_busy_tool, cpu=cpu, processes=("8 S 2.0 s python3 -m pytest",), output="collected 40 items")
|
||||
|
||||
|
||||
class StuckWorkers(unittest.TestCase):
|
||||
def worker_with(self, *events, held=(), replies=()):
|
||||
"""Runs a worker whose claude prints events and holds the shells held, while each investigation replies in turn; returns the claude, the fake, what was said on stdout and stderr and the record's text."""
|
||||
clock = [NOW]
|
||||
streamed = Streamed(0, *events, clock=clock, held=held)
|
||||
fake = FakeEffects(streams=[streamed], claude=replies)
|
||||
effects = dataclasses.replace(fake.effects(), now=lambda: clock[0])
|
||||
with tempfile.TemporaryDirectory() as folder, contextlib.redirect_stdout(io.StringIO()) as said, contextlib.redirect_stderr(io.StringIO()) as failed:
|
||||
archive = pathlib.Path(folder, "transcripts", "project")
|
||||
worker.run_worker("/r", [], {}, archive, RUNG, effects)
|
||||
record = archive / stuck_workers.RECORD
|
||||
return streamed, fake, said.getvalue(), failed.getvalue(), record.read_text() if record.exists() else ""
|
||||
|
||||
def test_a_silent_worker_whose_shell_used_no_cpu_since_the_last_check_is_investigated_and_the_verdict_said_and_recorded_without_a_kill(self):
|
||||
streamed, fake, said, _, record = self.worker_with(ASSISTANT, None, None, held=[shell()], replies=[verdict()])
|
||||
[(args, prompt)] = fake.made("claude")
|
||||
self.assertEqual(args[args.index("--model") + 1 :][:3], ["claude-opus-5-5", "--effort", "medium"])
|
||||
self.assertEqual(args[args.index("--tools") + 1], "")
|
||||
for evidence in ("bin/run-tests", "8 S 2.0 s python3 -m pytest", "collected 40 items", "working", "waiting on a person", "stuck"):
|
||||
self.assertIn(evidence, prompt)
|
||||
line = "stuck check: shell 7 has run 1600 s, written nothing for 1600 s and used no CPU for 300 s: bin/run-tests; a headless claude judged the worker stuck, held by process 8: pytest waits on a shell that ignores SIGINT"
|
||||
self.assertEqual(said, f"<{NOW + 600}> {line}\n")
|
||||
self.assertEqual(record, f"- {utc(NOW + 600)} {line}\n")
|
||||
self.assertEqual(streamed.stopped, [])
|
||||
|
||||
def test_a_shell_older_than_a_chunk_usually_runs_is_investigated_at_the_first_check(self):
|
||||
_, fake, said, _, _ = self.worker_with(ASSISTANT, None, held=[shell(started=NOW - stuck_workers.OLD, written=NOW + 290)], replies=[verdict("working", 7, "the tests still print")])
|
||||
self.assertEqual(len(fake.made("claude")), 1)
|
||||
self.assertIn(f"shell 7 has run {stuck_workers.OLD + 300} s, written nothing for 10 s: bin/run-tests; a headless claude judged the worker working, held by process 7: the tests still print", said)
|
||||
|
||||
def test_a_worker_whose_shells_show_no_sign_is_not_investigated(self):
|
||||
for events, held in (
|
||||
((ASSISTANT, None, None), [shell(runs_busy_tool=True)]),
|
||||
((ASSISTANT, None, None), [shell(written=NOW + 600 - 179)]),
|
||||
((ASSISTANT, None), [shell()]),
|
||||
((ASSISTANT, None, None), []),
|
||||
(({"type": "result"}, None, None), [shell()]),
|
||||
((ASSISTANT, None), [shell(started=NOW + 300 - stuck_workers.OLD)]),
|
||||
):
|
||||
with self.subTest(events=events, held=held):
|
||||
_, fake, said, _, record = self.worker_with(*events, held=held)
|
||||
self.assertEqual((fake.made("claude"), record), ([], ""))
|
||||
|
||||
def test_a_shell_that_used_cpu_since_the_last_check_is_not_investigated(self):
|
||||
clock = [NOW]
|
||||
used = [shell(cpu=2.0), shell(cpu=2.5)]
|
||||
streamed = Streamed(0, ASSISTANT, None, None, clock=clock)
|
||||
streamed.shells = lambda: [used.pop(0)]
|
||||
fake = FakeEffects(streams=[streamed])
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
worker.run_worker("/r", [], {}, pathlib.Path("/archive"), RUNG, dataclasses.replace(fake.effects(), now=lambda: clock[0]))
|
||||
self.assertEqual(fake.made("claude"), [])
|
||||
|
||||
def test_an_investigation_that_fails_or_answers_outside_the_schema_is_said_on_stderr(self):
|
||||
for reply, reason in ((ran(code=1, stderr="no model"), "no model"), (verdict("hung"), "answered outside its schema")):
|
||||
with self.subTest(reason=reason):
|
||||
_, _, said, failed, record = self.worker_with(ASSISTANT, None, None, held=[shell()], replies=[reply])
|
||||
self.assertEqual((said, record), ("", ""))
|
||||
self.assertIn(f"could not judge whether the worker is stuck: {reason}", failed)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -9,6 +9,7 @@ from claude_code.transcript import part_line
|
||||
from machine_load.memory import ROOM_SHARE
|
||||
from model_routing.ladder import Rung
|
||||
from claude_code.session import Session
|
||||
from agent_cycle.stuck_workers import Watch
|
||||
|
||||
PRINTED_TOOL_INPUT = 160
|
||||
# Seconds between reads of the machine's memory while too little of it is available to start a worker.
|
||||
@@ -18,13 +19,14 @@ AUTO_COMPACT_WINDOW = "CLAUDE_CODE_AUTO_COMPACT_WINDOW"
|
||||
COMPACT_WINDOW = 200_000
|
||||
|
||||
|
||||
def run_worker(root: str, args: list[str], env: dict[str, str], project_archive: pathlib.Path, effects: Effects) -> Failure | Limited | None:
|
||||
def run_worker(root: str, args: list[str], env: dict[str, str], project_archive: pathlib.Path, investigator: Rung, effects: Effects) -> Failure | Limited | None:
|
||||
"""Runs the worker's claude once the machine has room for it, as run_claude does with a Watch whose stuck checks run on investigator, the project's rung."""
|
||||
wait_for_memory(effects)
|
||||
return run_claude(root, args, env, effects, project_archive / RECORD)
|
||||
return run_claude(root, args, env, effects, Watch(investigator, project_archive))
|
||||
|
||||
|
||||
def run_claude(root: str, args: list[str], env: dict[str, str], effects: Effects, record: pathlib.Path | None = None) -> Failure | Limited | None:
|
||||
"""Runs a headless claude at once, printing what it says, and returns how it failed, or None. Where record is given, a claude that lingers after its final result loses its quiet shells, as `lingering_workers.stopped_shells` says, and each stop is appended to record."""
|
||||
def run_claude(root: str, args: list[str], env: dict[str, str], effects: Effects, watch: Watch | None = None) -> Failure | Limited | None:
|
||||
"""Runs a headless claude at once, printing what it says, and returns how it failed, or None. Where watch is given, a claude that lingers after its final result loses its quiet shells, as `lingering_workers.stopped_shells` says, each stop appended to RECORD in watch's archive; and a silent claude that has printed no result is checked for stuck shells, as `stuck_workers` says."""
|
||||
try:
|
||||
proc = effects.stream_claude(args, root, env)
|
||||
except OSError as failure:
|
||||
@@ -35,8 +37,10 @@ def run_claude(root: str, args: list[str], env: dict[str, str], effects: Effects
|
||||
resets: list[int] = []
|
||||
for raw in proc.lines(CHECK_EVERY):
|
||||
if raw is None:
|
||||
if record is not None:
|
||||
stopped_shells(proc, printed_result, record, effects)
|
||||
if watch is not None:
|
||||
stopped_shells(proc, printed_result, watch.archive / RECORD, effects)
|
||||
if not printed_result:
|
||||
watch.check(proc, root, effects)
|
||||
continue
|
||||
event = json_object(raw) or {}
|
||||
printed_result = ends_stream(event)
|
||||
|
||||
@@ -88,6 +88,14 @@ minutes, runs chunk-pr or run-heavy, or holds a heavy slot. It prints each stop
|
||||
the UTC time, to lingering-workers.md at the project's folder in that archive
|
||||
(bin/agent_cycle/lingering_workers.py).
|
||||
|
||||
A worker can also stall before its result, on a command that hangs. Every 5 minutes without output
|
||||
from its claude, where a shell has used no CPU since the last check and written nothing for 3
|
||||
minutes, or has run over 4 hours, and runs no chunk-pr or run-heavy, a headless claude on the
|
||||
project's rung reads the shells' commands, processes and output. It judges the worker working,
|
||||
waiting on a person or stuck, and names the process that holds it. loop-claude prints the verdict
|
||||
with its evidence and appends it to stuck-workers.md beside lingering-workers.md. It kills nothing
|
||||
yet (bin/agent_cycle/stuck_workers.py).
|
||||
|
||||
Before each worker starts, loop-claude waits until at least a quarter of the machine's memory is
|
||||
available (bin/machine_load/memory.py).
|
||||
|
||||
|
||||
@@ -795,7 +795,7 @@ class Limits(unittest.TestCase):
|
||||
allowed = {"type": "rate_limit_event", "rate_limit_info": {"status": "allowed", "resetsAt": NOW + 9000}}
|
||||
limited = {"type": "result", "api_error_status": 429, "result": "You've hit your session limit"}
|
||||
fake = FakeEffects(streams=[Streamed(1, rejected[1], allowed, rejected[0], limited)])
|
||||
self.assertEqual(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), fake.effects()), Limited(NOW + 600, "You've hit your session limit"))
|
||||
self.assertEqual(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), Rung("m"), fake.effects()), Limited(NOW + 600, "You've hit your session limit"))
|
||||
|
||||
def test_a_limit_outranks_the_exit_code_and_its_message_fits_one_line(self):
|
||||
def limited(**said):
|
||||
@@ -829,7 +829,7 @@ class LingeringWorkers(unittest.TestCase):
|
||||
with tempfile.TemporaryDirectory() as folder, contextlib.redirect_stdout(io.StringIO()) as said:
|
||||
# The archive folder of a project's first stop doesn't exist yet.
|
||||
archive = pathlib.Path(folder, "transcripts", "project")
|
||||
worker.run_worker("/r", [], {}, archive, effects)
|
||||
worker.run_worker("/r", [], {}, archive, Rung("m"), effects)
|
||||
record = archive / lingering_workers.RECORD
|
||||
return streamed, said.getvalue(), record.read_text() if record.exists() else ""
|
||||
|
||||
@@ -866,7 +866,7 @@ class LingeringWorkers(unittest.TestCase):
|
||||
def test_a_worker_reads_on_past_a_check_that_found_it_still_working(self):
|
||||
clock = [NOW]
|
||||
effects = dataclasses.replace(FakeEffects(streams=[Streamed(0, None, self.RESULT, clock=clock)]).effects(), now=lambda: clock[0])
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), effects))
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), Rung("m"), effects))
|
||||
|
||||
def test_a_stop_that_cannot_be_recorded_is_said_on_stderr_and_every_shell_is_still_stopped(self):
|
||||
clock = [NOW]
|
||||
@@ -874,7 +874,7 @@ class LingeringWorkers(unittest.TestCase):
|
||||
streamed = Streamed(0, self.RESULT, None, clock=clock, held=held)
|
||||
effects = dataclasses.replace(FakeEffects(streams=[streamed]).effects(), now=lambda: clock[0])
|
||||
with tempfile.NamedTemporaryFile() as blocker, contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()) as failed:
|
||||
worker.run_worker("/r", [], {}, pathlib.Path(blocker.name), effects)
|
||||
worker.run_worker("/r", [], {}, pathlib.Path(blocker.name), Rung("m"), effects)
|
||||
self.assertEqual(streamed.stopped, held)
|
||||
self.assertIn("could not record the stop", failed.getvalue())
|
||||
|
||||
@@ -888,14 +888,14 @@ class MemoryWait(unittest.TestCase):
|
||||
def test_a_worker_starts_only_once_a_quarter_of_memory_is_available(self):
|
||||
fake = FakeEffects(free_memory=[memory.Memory(available=3, total=16), memory.Memory(available=3, total=16), memory.Memory(available=4, total=16)], streams=[Streamed(0)])
|
||||
with contextlib.redirect_stderr(io.StringIO()) as said:
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), fake.effects()))
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), Rung("m"), fake.effects()))
|
||||
self.assertEqual([seam for seam, _ in fake.calls], ["memory", "sleep", "memory", "sleep", "memory", "stream_claude"])
|
||||
self.assertEqual(fake.made("sleep"), [worker.MEMORY_POLL] * 2)
|
||||
self.assertEqual(said.getvalue().count("waiting"), 1, said.getvalue())
|
||||
|
||||
def test_a_worker_starts_at_once_where_memory_cannot_be_read(self):
|
||||
fake = FakeEffects(free_memory=[None], streams=[Streamed(0)])
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), fake.effects()))
|
||||
self.assertIsNone(worker.run_worker("/r", [], {}, pathlib.Path("/archive"), Rung("m"), fake.effects()))
|
||||
self.assertEqual([seam for seam, _ in fake.calls], ["memory", "stream_claude"])
|
||||
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ from machine_load.slots import slot_folder
|
||||
# The command names of a shell claude starts for a Bash call.
|
||||
SHELL_NAMES = frozenset({"bash", "dash", "sh", "zsh"})
|
||||
COMMAND_CHARS = 300
|
||||
OUTPUT_BYTES = 2000
|
||||
# Scripts whose process below a shell shows work that may go quiet for long: run-heavy while it waits for a slot, and chunk-pr, which waits for a merge's approval and checks.
|
||||
BUSY_TOOLS = frozenset({"chunk-pr", "run-heavy"})
|
||||
|
||||
@@ -28,19 +29,38 @@ class Shell:
|
||||
written: float
|
||||
# Whether a process below the shell runs one of BUSY_TOOLS or holds a heavy slot, a file of slot_folder that run-heavy leaves open in the command it runs.
|
||||
runs_busy_tool: bool
|
||||
# CPU seconds the shell and every process below it have used.
|
||||
cpu: float = 0.0
|
||||
# Each process below the shell, as `<pid> <state> <CPU seconds> s <command line>`, with the state letter /proc gives, such as S for sleeping.
|
||||
processes: tuple[str, ...] = ()
|
||||
# The last OUTPUT_BYTES of the file the shell's output goes to; empty where it goes to no file.
|
||||
output: str = ""
|
||||
|
||||
|
||||
def parent_and_start(pid: int) -> tuple[int, int, str] | None:
|
||||
"""The parent pid, the start time in clock ticks since boot and the command name of a process; None where it is gone."""
|
||||
def stat_of(pid: int) -> tuple[str, list[str]] | None:
|
||||
"""A process's command name and the fields of /proc/<pid>/stat after it, from the state on; None where it is gone."""
|
||||
try:
|
||||
with open(f"/proc/{pid}/stat") as handle:
|
||||
text = handle.read()
|
||||
except OSError:
|
||||
return None
|
||||
# The name sits in parentheses and may hold spaces or parentheses itself, so split at the last one.
|
||||
name = text[text.index("(") + 1 : text.rindex(")")]
|
||||
fields = text[text.rindex(")") + 2 :].split()
|
||||
return int(fields[1]), int(fields[19]), name
|
||||
return text[text.index("(") + 1 : text.rindex(")")], text[text.rindex(")") + 2 :].split()
|
||||
|
||||
|
||||
def parent_and_start(pid: int) -> tuple[int, int, str] | None:
|
||||
"""The parent pid, the start time in clock ticks since boot and the command name of a process; None where it is gone."""
|
||||
found = stat_of(pid)
|
||||
return None if found is None else (int(found[1][1]), int(found[1][19]), found[0])
|
||||
|
||||
|
||||
def cpu_and_state(pid: int) -> tuple[float, str]:
|
||||
"""The CPU seconds a process has used, in user and kernel mode, and its state letter; none and `gone` where it is gone."""
|
||||
found = stat_of(pid)
|
||||
if found is None:
|
||||
return 0.0, "gone"
|
||||
fields = found[1]
|
||||
return (int(fields[11]) + int(fields[12])) / os.sysconf("SC_CLK_TCK"), fields[0]
|
||||
|
||||
|
||||
def children_of(pid: int) -> list[int]:
|
||||
@@ -77,6 +97,31 @@ def output_written(pid: int, started: float) -> float:
|
||||
return info.st_mtime if stat.S_ISREG(info.st_mode) else started
|
||||
|
||||
|
||||
def output_tail(pid: int) -> str:
|
||||
"""The last OUTPUT_BYTES of the file that pid's standard output goes to; empty where it goes to a pipe or a terminal."""
|
||||
path = f"/proc/{pid}/fd/1"
|
||||
try:
|
||||
# Before the open, which can block on a pipe.
|
||||
if not stat.S_ISREG(os.stat(path).st_mode):
|
||||
return ""
|
||||
with open(path, "rb") as handle:
|
||||
handle.seek(max(0, os.fstat(handle.fileno()).st_size - OUTPUT_BYTES))
|
||||
return handle.read().decode(errors="replace")
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def tree_of(pid: int) -> tuple[float, tuple[str, ...]]:
|
||||
"""The CPU seconds pid and every process below it have used, and each process below it as `Shell.processes` shows one."""
|
||||
total = cpu_and_state(pid)[0]
|
||||
shown = []
|
||||
for each in descendants(pid):
|
||||
cpu, state = cpu_and_state(each)
|
||||
total += cpu
|
||||
shown.append(f"{each} {state} {cpu:.1f} s {' '.join(argv_of(each))[:COMMAND_CHARS]}")
|
||||
return total, tuple(shown)
|
||||
|
||||
|
||||
def command_of(shell_argv: list[str]) -> str:
|
||||
"""What a shell runs: the text of its command line after Claude Code's `eval '`, up to the last ` < /dev/null`, which a call that began in the foreground lacks; the whole line where it holds no `eval`."""
|
||||
line = " ".join(shell_argv)
|
||||
@@ -110,7 +155,8 @@ def shells_of(claude_pid: int) -> list[Shell]:
|
||||
if found is None or found[2] not in SHELL_NAMES:
|
||||
continue
|
||||
started = booted + found[1] / ticks
|
||||
shells.append(Shell(pid, started, command_of(argv_of(pid)), output_written(pid, started), runs_busy_tool(pid, slots)))
|
||||
cpu, processes = tree_of(pid)
|
||||
shells.append(Shell(pid, started, command_of(argv_of(pid)), output_written(pid, started), runs_busy_tool(pid, slots), cpu, processes, output_tail(pid)))
|
||||
return shells
|
||||
|
||||
|
||||
|
||||
@@ -189,6 +189,22 @@ class ChildShells(unittest.TestCase):
|
||||
self.assertEqual(self.shell_of(to_file).written, 1000)
|
||||
self.assertEqual(self.shell_of(to_pipe).written, self.shell_of(to_pipe).started)
|
||||
|
||||
def test_reads_the_end_of_a_shells_output_file_and_nothing_from_a_pipe(self):
|
||||
with tempfile.NamedTemporaryFile() as output:
|
||||
output.write(b"x" * child_shells.OUTPUT_BYTES + b"last line\n")
|
||||
output.flush()
|
||||
to_file = self.started("sleep 60; true", stdout=output)
|
||||
to_pipe = self.started("sleep 60; true", stdout=subprocess.PIPE)
|
||||
self.assertEqual(self.shell_of(to_file).output, "x" * (child_shells.OUTPUT_BYTES - 10) + "last line\n")
|
||||
self.assertEqual(self.shell_of(to_pipe).output, "")
|
||||
|
||||
def test_reads_each_process_below_a_shell_and_the_cpu_the_whole_tree_used(self):
|
||||
busy = self.started("timeout 5 sh -c 'while :; do :; done'; true")
|
||||
wait_until(lambda: self.shell_of(busy).cpu > 0.1)
|
||||
processes = self.shell_of(busy).processes
|
||||
self.assertEqual([line.split(" ", 4)[4] for line in processes], ["timeout 5 sh -c while :; do :; done", "sh -c while :; do :; done"])
|
||||
self.assertEqual(processes[1].split(" ")[1], "R")
|
||||
|
||||
def test_stops_a_shell_and_the_process_it_started_and_no_other(self):
|
||||
shell = self.started("sleep 60; true")
|
||||
other = subprocess.Popen(["sleep", "60"])
|
||||
|
||||
@@ -3071,3 +3071,45 @@ Read most (opens and searches):
|
||||
- `docs/decisions.md` 2
|
||||
- `bin/do_epic/backend/protocol.py` 1
|
||||
- `bin/do_epic/findings.py` 1
|
||||
|
||||
## Reading cost, chunk 59bf63f1876b, finished 2026-11-19T04:43:46Z
|
||||
|
||||
97876 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `README.md` 6
|
||||
- `skills/larv-review/SKILL.md` 4
|
||||
- `.gitignore` 3
|
||||
- `bin/do_epic/main.py` 3
|
||||
- `bin/project_config/settings.py` 3
|
||||
- `bin/atlassian/client.py` 2
|
||||
- `bin/do_epic/backend/contract.py` 2
|
||||
- `bin/do_epic/epic_record.py` 2
|
||||
- `bin/do_epic/findings.py` 2
|
||||
- `bin/do_epic/finish.py` 2
|
||||
|
||||
## Reading cost, chunk 476ff8575248, finished 2026-11-19T04:48:45Z
|
||||
|
||||
22763 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `bin/agent_cycle/interrupt.py` 1
|
||||
|
||||
## Reading cost, chunk 8cb7c9836e2d, finished 2026-11-19T07:47:33Z
|
||||
|
||||
106608 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `bin/loop_claude/test_loop_claude.py` 8
|
||||
- `bin/loop_claude/loop/usage.py` 6
|
||||
- `bin/loop_effects/effects.py` 5
|
||||
- `README.md` 4
|
||||
- `bin/loop_claude/test_black_box.py` 4
|
||||
- `bin/loop_claude/loop/main.py` 3
|
||||
- `bin/loop_effects/test_effects.py` 2
|
||||
- `docs/decisions.md` 2
|
||||
- `bin/repo_checks/test_import_order.py` 1
|
||||
- `bin/run-heavy` 1
|
||||
|
||||
@@ -6,13 +6,14 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 446`
|
||||
`Next ID = 447`
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 422 | 0.12.0 | | | **Check every 5 minutes whether a tmux window of the loop looks stuck by other signs than a lingering worker, investigate it with a headless claude, and kill only the process the investigation names.** | 3 | 2 | 3 | 6 | 5 | 1.5 by 4 | 0.8 |
|
||||
| 446 | 0.13.0 | | question | **Kill the process a stuck check names where its verdict is `stuck`, once the human says the recorded verdicts match what they would have done.** | 2 | 2 | 3 | 5 | 3 | 1.5 by 3 | 1.0 |
|
||||
| 445 | 0.13.0 | | | **Refuse a plan that holds an item a `Finish item <ID>` commit already removed, so a merge resolution cannot bring a finished item back.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3, 1.1 by 2 | 1.8 |
|
||||
| 365 | 0.13.0 | | | **Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
|
||||
| 368 | 0.13.0 | | | **Call a release review due, call a release ready to cut, and tell whether a chunk owes a scoring run, for a plan outside the checkout, from the project repo's history or from Jira.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
|
||||
@@ -82,6 +83,16 @@ Touches:
|
||||
- `bin/agent_cycle/tmux.py`: `Listed`, `listed`
|
||||
- `README.md`: `## Tools` (small)
|
||||
|
||||
### 446. Kill the process a stuck check names where its verdict is `stuck`, once the human says the recorded verdicts match what they would have done.
|
||||
|
||||
Item 422 judges a worker that stalls before its result, but only reports: each verdict goes to the terminal and to `stuck-workers.md` in the project's transcript archive. Question for the human: after a few days of verdicts there, do they match what you would have done? Once you say yes, kill only the process a `stuck` verdict names, and only where the pid is below the worker's claude, never the worker or the loop, and record the call in a decision.
|
||||
|
||||
Touches:
|
||||
|
||||
- `bin/agent_cycle/stuck_workers.py`: `Watch.check`
|
||||
- `bin/loop_effects/child_shells.py`: `stop` (small)
|
||||
- `bin/loop_claude/loop/usage.py`: `HELP` (small)
|
||||
|
||||
### 445. Refuse a plan that holds an item a `Finish item <ID>` commit already removed, so a merge resolution cannot bring a finished item back.
|
||||
|
||||
Merging main into item-442 in 610edb1a resolved a todo.md conflict by keeping item 310's row and detail, which e019a6b7 had just finished. loop-claude then started a chunk on item 310, whose work had already merged, and that chunk only removed the item again. No check noticed: `rank_todo.py --check` reads one file, and rule-hook's `todo-rank-check` sees only an agent's writes, not what a merge resolution keeps.
|
||||
|
||||
Reference in New Issue
Block a user