Merge pull request 'Resume a worker one rung up next cycle when it ends its run asking for a higher rung and the judge finds the escalation trigger it cites' (#278) from plan-143 into main

This commit was merged in pull request #278.
This commit is contained in:
2026-10-15 22:34:52 +02:00
18 changed files with 239 additions and 88 deletions
+1
View File
@@ -37,6 +37,7 @@ Decisions live in `docs/decisions.md`:
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model, and a chunk's worker may run below that rung, launching its reviewers on that rung; the worker routes each subagent it delegates to by the R + C of the subagent's task
- routing: a worker hands a task one rung up on each escalation trigger, up to two rungs above the rung its item picks unless the human agrees, and opens the subagent's prompt with the trigger so the routing figures count it
- routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, resumes one rung up next cycle, up to two rungs above the rung its item picks
- routing: on a model judge-replay covered, the judge runs on the rung its chunk's item picks, never below the project's rung
- routing: the stability reviewer stays on the project's rung while the worker runs below it, until `stability-replay` shows a lower rung finding every 🔴 and 🟡 the project's rung finds
+1
View File
@@ -5,6 +5,7 @@
- The new `bin/stability-replay <model>` replays archived stability reviews of the projects run on the model, each on the code it reviewed, first on the project's rung and then on each rung below, and prints the lowest rung that, like each rung above it, finds every 🔴 and 🟡 finding the project's rung finds. It shares its search with `judge-replay`, which now tries rungs from the project's rung down, where it tried them from the cheapest up. Its first run, on Opus 5.5 projects, found none: Sonnet 5.5 at high missed 13 of the 29 findings the project's rung found again. So the stability reviewer stays on the project's rung while the worker runs below it.
- A `todo.md` item you tell a session to keep is marked `owner` in its Exempt cell, and stays whatever its score.
- A `loop-claude` worker hands a task that proves too hard for its rung to a subagent one rung up, edits included, when a trigger fires: a re-score of the item on the project's rung or above picks a higher rung, the same failing test or finding survives two fix attempts, or a review pass reports a 🔴 in round 2 or later. Each further trigger climbs one more rung, to at most two rungs above the rung the chunk's `todo.md` item picks, unless you agree. `CLAUDE.md` states the rule, and `loop-claude`'s prompt names the rungs a trigger opens. The worker opens the subagent's prompt with `Escalation trigger <n>:`, and `routing-figures.md` counts the escalations by trigger. Expect a chunk that gets stuck to spend more on the stuck part, and fewer unfinished verdicts.
- A loop-claude worker may end its run asking for a higher rung, citing an escalation trigger. Once the judge quotes the transcript passage that shows the trigger fired, the next cycle resumes the worker one rung up, up to two rungs above the rung the chunk's `todo.md` item picks. The archive quotes that passage under `Escalation:` in the unfinished verdict.
- A reviewer that rules a chunk's work below standard, with systems-architect's BLOCK or a finding against larv-review's `standards/quality.md`, may ask to rerun one rung up, as may a comprehension seat whose vote in a paired ruling `record-ruling` counts as worse. The worker relaunches it once, on the rung `loop-claude`'s prompt names. A relaunched reviewer gets the report that asked and confirms or dismisses each of its findings. The relaunch's report stands in place of the one that asked, in the same round, so it adds no round. The ruling that stands is a fourth escalation trigger in `CLAUDE.md`. Expect a chunk whose work falls below standard to spend more on its review and its fix.
- A session or subagent below the project's rung, the cheapest rung of the project's model, no longer scores or reshapes `todo.md`: it drafts each new item and each change to a filed item, and before its next commit one subagent on the project's rung or above applies and scores the drafts and sets their release. It may still remove finished items. `rules/todo-items.md` states the rule. `CLAUDE.md` has a session launch a subagent that files or scores items on the rung the new `bin/subagent-rung --plan <model> <R+C>` prints, never below the project's rung, and `loop-claude`'s prompt to a worker below that rung names the command. Expect one extra subagent in a chunk that files items while its worker runs below the project's rung.
- `loop-claude` runs the judge of an Opus 5.5 project on the rung the chunk's `todo.md` item picks, never below Opus 5.5 at medium effort, the project's rung. Before, the judge ran at the effort `settings.json` sets, xhigh. A replay of 39 archived verdicts found every cheaper rung missing verdicts. The judge keeps its rung when the worker steps up or takes in another item, and runs on the project's rung when no item routed the chunk. A project on another model keeps its judge at its settings' effort. Each verdict in the archive now names the judge's rung on a `Judge:` line. The new `bin/judge-replay <model>` replays the judge on archived verdicts and prints the lowest rung that, like each rung above it, returns them. Expect the judge to cost less on items of R + C 12 or less, and more on items of R + C 15 or more.
+1 -1
View File
@@ -122,7 +122,7 @@ class Main(unittest.TestCase):
@staticmethod
def verdict(interrupt, evidence):
output = {"answer": "", "bundled": 0, "evidence": evidence, "finished": not interrupt, "interrupt": interrupt, "question": "Which?" if interrupt else ""}
output = {"answer": "", "bundled": 0, "escalation": "", "evidence": evidence, "finished": not interrupt, "interrupt": interrupt, "question": "Which?" if interrupt else ""}
return {"type": "result", "structured_output": output}
def test_replays_the_models_projects_on_the_reference_then_from_the_projects_rung_down_and_prints_the_lowest_that_returns_all(self):
+18 -6
View File
@@ -30,15 +30,16 @@ JUDGE_RULES = (
"- it chose between alternatives that neither README.md nor the records discriminate "
"between - neither rules the rejected one out - and the choice materially changes the deliverable\n"
"interrupt=false otherwise. Bookkeeping the agent settled silently - what to document, wording, "
"where a note goes - is not on its own a reason to interrupt. The human may have replied inside the transcript, on a [human] line: judge only what "
"where a note goes - is not on its own a reason to interrupt. A run that ends asking loop-claude for a higher "
"rung is no reason to interrupt either; one that asks the human's yes to a higher rung is. The human may have replied inside the transcript, on a [human] line: judge only what "
"is still open, never a question they already answered there. A [loop] line is what loop-claude told "
"the agent, such as an answer quoted from README.md or the records.\n"
"the agent, such as an answer quoted from README.md or the records.\n\n"
"Where the agent asked or flagged something itself, the question is that one, in the agent's own "
"wording; never construct another from facts it stated.\n"
"wording; never construct another from facts it stated.\n\n"
"Where README.md or the records answer the agent's own question and no [loop] line has "
"quoted that answer yet, set interrupt=false and quote the passage that answers it; the agent is "
"then given that passage in place of the human's reply. An interrupt outranks a quoted answer, and either "
"outranks finished.\n"
"outranks finished.\n\n"
"The chunk is the piece of work the agent is on, shipped as one pull request. "
"finished=true when the chunk is complete: merged, or nothing remains that the agent could do on "
f"its own. Even then, it is finished only once {ANSWERS_RECORDED}. An answer counts as recorded "
@@ -54,6 +55,8 @@ class Reply:
answer: str = dataclasses.field(metadata={"description": "With interrupt false, the passage of README.md or the records that answers the agent's question, quoted verbatim, without elisions. Empty otherwise."})
# Routes the next cycle's worker. Counting the item the chunk was routed on is harmless: loop-claude never steps the worker down.
bundled: int = dataclasses.field(metadata={"description": "The highest R + C the agent states for a todo.md item it took into this chunk; 0 when it states none."})
# Routes the next cycle's worker one rung up: docs/decisions.md → `routing: a worker whose run ends asking for a higher rung`.
escalation: str = dataclasses.field(metadata={"description": "Empty unless the chunk is unfinished and the agent's run ends asking loop-claude for a higher rung, citing an escalation trigger. Then: a verbatim quote, without elisions, of an earlier transcript passage showing the trigger fired after any earlier such request, such as a review round's commit title or the agent's report of the finding or failing test. Never the request itself."})
evidence: str = dataclasses.field(metadata={"description": "A verbatim quote of the one transcript passage the verdict rests on, without elisions: what triggers the interrupt; for a quoted answer, the agent's question; else what shows the chunk finished or the work remaining."})
finished: bool = dataclasses.field(metadata={"description": "Whether the chunk is finished."})
interrupt: bool = dataclasses.field(metadata={"description": "Whether the human must now be interrupted."})
@@ -94,6 +97,8 @@ class Unfinished:
evidence: str
# The bundled R + C the judge saw, 0 for none.
bundled: int = 0
# The judge's quote showing that the worker's cited escalation trigger fired; empty for none.
escalation: str = ""
Verdict = Interrupt | QuotedAnswer | Finished | Unfinished
@@ -118,7 +123,7 @@ def verdict_from(output: object) -> Verdict | Failure:
return Finished(output["evidence"])
if not 0 <= output["bundled"] <= HIGHEST_R_PLUS_C:
return Failure(f"the judge's bundled R + C {output['bundled']} is not from 0 to {HIGHEST_R_PLUS_C}")
return Unfinished(output["evidence"], output["bundled"])
return Unfinished(output["evidence"], output["bundled"], output["escalation"])
def judge(root: str, task: str, premises: str, records: str, rendered_transcript: str, rung: Rung, effects: Effects) -> Verdict | Stop:
@@ -142,6 +147,12 @@ def judge_args(rung: Rung) -> list[str]:
def run_judge(root: str, args: list[str], prompt: str, shown: str, quotable: str, effects: Effects) -> Verdict | Failure | Limited:
"""One judge run, its three quotes each checked against what they quote:
- evidence, against shown, its fragments in order: a misquote is a Failure, which judge retries;
- a quoted answer, against quotable, word for word: a misquote puts the question to the human;
- an escalation, against shown, word for word: a misquote drops it, so the worker's rung holds.
A misquoted answer or escalation is no Failure, which would otherwise exit an unattended loop. Neither may be elided, since stitched fragments can say what the records or the transcript do not.
"""
reply = claude_reply(effects.claude, args, root, prompt, "the judge")
if isinstance(reply, Failure | Limited):
return reply
@@ -150,9 +161,10 @@ def run_judge(root: str, args: list[str], prompt: str, shown: str, quotable: str
return Failure(f"{verdict.reason}; it replied: {json.dumps(reply, ensure_ascii=False)}")
if not grounded(verdict.evidence, shown):
return Failure(f"the judge quoted {verdict.evidence!r}, which is not in the transcript" + (f", for the question: {verdict.question}" if isinstance(verdict, Interrupt | QuotedAnswer) else ""))
# Not a Failure: a misquote would otherwise exit an unattended loop. Never elided: stitched fragments can say what the records do not.
if isinstance(verdict, QuotedAnswer) and not verbatim(verdict.answer, quotable):
return Interrupt(verdict.evidence, verdict.question)
if isinstance(verdict, Unfinished) and verdict.escalation and not verbatim(verdict.escalation, shown):
return dataclasses.replace(verdict, escalation="")
return verdict
+8 -5
View File
@@ -38,15 +38,17 @@ def off_the_projects_rung(route: Route, anchor: Rung | None) -> str:
def escalation(route: Route, anchor: Rung | None) -> str:
"""The prompt's escalation sentence: the rungs escalation_rungs opens, and the marker; empty off the ladder or on its top rung."""
"""The prompt's escalation sentences, the marker's among them. Empty off the ladder or on its top rung. Where escalation_rungs opens none, the worker escalates only with the human's yes.
Else they name the rungs escalation_rungs opens to a subagent, and offer to resume the worker one rung up for the rest of the chunk, up to the highest of those rungs."""
if anchor is None or route.rung not in LADDER[:-1]:
return ""
marked = f" Open the subagent's prompt with `{ESCALATION_MARKER} <n>:`, n the trigger's number."
rungs = [f"`{rung.agent_args()}`" for rung in escalation_rungs(anchor, route)]
if not rungs:
return f" An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `{climbed(route.rung, 1).agent_args()}`, one rung above yours, only once the human agrees.{marked}"
return f" An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `{climbed(route.rung, 1).agent_args()}`, one rung above yours, only once the human agrees; loop-claude will not resume you higher for a trigger either, so ask the human for that yes.{marked}"
further = f", and on each further trigger one rung higher, up to {rungs[-1]}" if len(rungs) > 1 else ""
return f" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to {rungs[0]}, one rung above yours{further}; above that, ask the human first.{marked}"
resumed_higher = f" Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last such request: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to {rungs[-1]}."
return f" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to {rungs[0]}, one rung above yours{further}; above that, ask the human first.{marked}{resumed_higher}"
def reruns(route: Route, anchor: Rung | None) -> str:
@@ -64,9 +66,10 @@ def reruns(route: Route, anchor: Rung | None) -> str:
def resumed(unfinished_evidence: str, route: Route, anchor: Rung | None) -> str:
"""Names the project's rung again, since a step-up between cycles may have taken the worker above it."""
"""Names the worker's rung, so it learns whether its request for a higher rung went through, and the project's again, since a step-up between cycles may have taken the worker above it."""
on = f" loop-claude runs you on {route.rung}." if anchor else ""
ruled = f" A judge ruled the chunk unfinished, quoting this passage of your transcript as the work remaining:\n\n{quoted(unfinished_evidence)}\n\n" if unfinished_evidence else " "
return f"Continue the task; the previous run ended before the chunk was finished.{ruled}The chunk is finished once it is merged, or nothing remains that you could do on your own. Even then, it is finished only once {ANSWERS_RECORDED}.{off_the_projects_rung(route, anchor)}{escalation(route, anchor)}{reruns(route, anchor)}"
return f"Continue the task; the previous run ended before the chunk was finished.{ruled}The chunk is finished once it is merged, or nothing remains that you could do on your own. Even then, it is finished only once {ANSWERS_RECORDED}.{on}{off_the_projects_rung(route, anchor)}{escalation(route, anchor)}{reruns(route, anchor)}"
def answered(question: str, answer: str) -> str:
+3 -3
View File
@@ -128,9 +128,9 @@ def question_for(ruling: Interrupt | QuotedAnswer) -> str:
def close(cycle: Cycle, transcript_path: str, verdict: Finished | Unfinished) -> Archived | Stop:
finished = isinstance(verdict, Finished)
if failure := archive(cycle.project_archive, transcript_path, cycle.task, outcomes.finished(str(cycle.route), verdict.evidence, str(judge_of(cycle))) if finished else outcomes.unfinished(str(cycle.route), verdict.evidence, str(judge_of(cycle))), cycle.effects.now()):
outcome = outcomes.finished(str(cycle.route), verdict.evidence, str(judge_of(cycle))) if isinstance(verdict, Finished) else outcomes.unfinished(str(cycle.route), verdict.evidence, str(judge_of(cycle)), verdict.escalation)
if failure := archive(cycle.project_archive, transcript_path, cycle.task, outcome, cycle.effects.now()):
return Stop(1, failure.reason)
if finished and (failure := delete_session(cycle.session)):
if isinstance(verdict, Finished) and (failure := delete_session(cycle.session)):
return Stop(1, failure.reason)
return Archived()
+9 -5
View File
@@ -28,9 +28,9 @@ interrupt, and as every cycle ends, Ctrl-C and a failure included, the session's
copied to $LOOP_CLAUDE_TRANSCRIPTS/<key>/<chunk>/<sha256>.jsonl (default
~/.claude/loop-claude-transcripts), named by its hash. <chunk> is the hash of the transcript's first
line with a timestamp, the same for every cycle of a chunk. Beside the copy, <sha256>.outcomes.md
holds the verdict, with the transcript passage the judge rested it on and the judge's rung; each
answer the judge quoted, with the question it answers; and why the loop stopped. Each of these opens with its UTC
time. Claude Code keeps a folder per session beside its transcript, holding the subagents'
holds the verdict, with the transcript passage the judge rested it on, any passage it quoted to
show an escalation trigger fired, and the judge's rung; each answer the judge quoted, with the
question it answers; and why the loop stopped. Each of these opens with its UTC time. Claude Code keeps a folder per session beside its transcript, holding the subagents'
transcripts among other files; that folder, and every other file named after the session except
the transcript, are copied into <chunk>/session/. A file there is replaced only by a version that
grew from it. A file Claude Code rewrote lands beside its copy as <name>.<hash of its content>.
@@ -54,7 +54,11 @@ the cheapest rung. The prompt after each unfinished verdict
says this again, since a step-up may have taken the worker above the cheapest rung. A reviewer or
seat that rules the work below standard may ask to rerun one rung up; each prompt names that rung. The
judge reports the highest R + C the worker states, and the next cycle steps up to the rung that
R + C picks, never down. Every second unfinished verdict on a chunk steps it up one rung. Each
R + C picks, never down. Every second unfinished verdict on a chunk steps it up one rung. So does
an unfinished verdict on a worker that ended its run asking for a higher rung, once the judge quotes
the transcript showing that the escalation trigger it named fired
(~/.claude/CLAUDE.md → "Working with the codebase" lists them). That step stops two rungs above the rung the chunk's item picks, and the
worker's prompts offer it only while the worker is below that limit. Each
finished or unfinished verdict, stop and step-up in the archive names the worker's rung, with the
item and R + C that routed the chunk; a rerun resumes an open chunk on the latest rung named there,
and counts its unfinished verdicts afresh. A rung no longer on the ladder, or below the one its
@@ -206,7 +210,7 @@ def routed(loop_run: LoopRun, anchor: Rung | None, archived: typing.Callable[[],
def after_verdict(loop_run: LoopRun, anchor: Rung | None, verdict: Finished | Unfinished) -> LoopRun:
if isinstance(verdict, Finished):
return dataclasses.replace(loop_run, route=None, unfinished_evidence="")
route = stepped(anchor, loop_run.route, verdict.bundled) if anchor and loop_run.route else loop_run.route
route = stepped(anchor, loop_run.route, verdict.bundled, bool(verdict.escalation)) if anchor and loop_run.route else loop_run.route
return dataclasses.replace(loop_run, route=route, unfinished_evidence=verdict.evidence)
+35 -15
View File
@@ -7,7 +7,7 @@ from todo_table import Item, items
from model_routing.route import Route
UNFINISHED_PER_STEP = 2
# How many rungs above item_rung the worker may escalate a task without the human's yes: docs/decisions.md → `routing: a worker hands a task one rung up`.
# How many rungs above item_rung an escalation trigger takes the worker or a task without the human's yes, as escalation_cap counts them.
ESCALATION_CAP = 2
# The models `judge-replay` found returning, on their cheapest rung, every verdict the reference judge reproduced: docs/decisions.md → `routing: on a model judge-replay covered`.
JUDGE_REPLAYED = ("claude-opus-5-5",)
@@ -31,8 +31,13 @@ def first_route(anchor: Rung, item: Item | None) -> Route:
def kept(route: Route, anchor: Rung) -> bool:
"""Whether a chunk keeps route: its rung is on the ladder at or above item_rung."""
return highest(route.rung, item_rung(anchor, route)) == route.rung
"""Whether a chunk keeps route: its rung is on the ladder and at or above item_rung."""
return kept_rung(anchor, route) == route.rung
def kept_rung(anchor: Rung, route: Route) -> Rung:
"""The route's rung, raised to item_rung where it is lower or off the ladder."""
return highest(route.rung, item_rung(anchor, route))
def item_rung(anchor: Rung, route: Route) -> Rung:
@@ -41,23 +46,38 @@ def item_rung(anchor: Rung, route: Route) -> Rung:
def escalation_rungs(anchor: Rung, route: Route) -> tuple[Rung, ...]:
"""The rungs an escalation trigger opens to the worker, least competent first: from one above its own up to ESCALATION_CAP above item_rung; none off the ladder.
A step-up between cycles raises the worker but not that limit, as lilleman set it from the rung the chunk started on.
"""
"""The rungs an escalation trigger opens to the worker, least competent first: from one above its own up to escalation_cap; none off the ladder."""
if route.rung not in LADDER:
return ()
return LADDER[position(route.rung) + 1:position(climbed(item_rung(anchor, route), ESCALATION_CAP)) + 1]
return LADDER[position(route.rung) + 1:position(escalation_cap(anchor, route)) + 1]
def stepped(anchor: Rung, route: Route, bundled: int) -> Route:
"""The route after one more unfinished verdict; bundled is the highest R + C the judge saw the worker take into the chunk, 0 for none."""
# An archived rung since removed from LADDER restarts from the project's rung.
rung = route.rung if route.rung in LADDER else anchor
rung = highest(rung, picked(anchor, bundled)) if bundled else rung
if route.unfinished + 1 >= UNFINISHED_PER_STEP:
def escalation_cap(anchor: Rung, route: Route) -> Rung:
"""The highest rung an escalation trigger takes the worker or a task to without the human's yes: docs/decisions.md → `routing: a worker hands a task one rung up`.
It counts from item_rung, the rung the chunk started on, so a step-up between cycles narrows what the worker may climb without a yes, and never widens it.
"""
return climbed(item_rung(anchor, route), ESCALATION_CAP)
def stepped(anchor: Rung, route: Route, bundled: int, escalated: bool) -> Route:
"""The route after one more unfinished verdict, never on a lower rung.
bundled and escalated are the verdict's fields, not the verdict, since loop_claude.cycle.judge imports this module through prompts.
"""
rung = taken_in(anchor, route, bundled)
unfinished = route.unfinished + 1
counted_out = unfinished >= UNFINISHED_PER_STEP
# The judge found the escalation trigger the worker cited in asking for a higher rung.
escalates = escalated and position(rung) < position(escalation_cap(anchor, route))
if counted_out or escalates:
return dataclasses.replace(route, rung=stepped_up(rung), unfinished=0)
return dataclasses.replace(route, rung=rung, unfinished=route.unfinished + 1)
return dataclasses.replace(route, rung=rung, unfinished=unfinished)
def taken_in(anchor: Rung, route: Route, bundled: int) -> Rung:
"""kept_rung, raised to the rung bundled picks from anchor; bundled is the highest R + C the judge saw the worker take into the chunk, 0 for none."""
return highest(kept_rung(anchor, route), picked(anchor, bundled)) if bundled else kept_rung(anchor, route)
def judge_rung(judge_model: str, route: Route) -> Rung:
+24 -2
View File
@@ -123,8 +123,8 @@ def worker(**step):
return {"role": "worker", **step}
def judge(interrupt=False, finished=False, evidence="working", question="", bundled=0, **step):
verdict = {"answer": "", "bundled": bundled, "evidence": evidence, "finished": finished, "interrupt": interrupt, "question": question}
def judge(interrupt=False, finished=False, evidence="working", question="", bundled=0, escalation="", **step):
verdict = {"answer": "", "bundled": bundled, "escalation": escalation, "evidence": evidence, "finished": finished, "interrupt": interrupt, "question": question}
return {"role": "judge", "verdict": verdict, **step}
@@ -395,6 +395,28 @@ class Cycles(unittest.TestCase):
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> three\n\nJudge: judge-model\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's R + C of 11\n\n<UTC> Stepped up\n\nWorker: claude-fable-5-1 at max effort, routed by item 4's R + C of 11\n", outcomes)
self.assertIn("<UTC> Verdict: finished\n\nEvidence:\n> four\n\nJudge: judge-model\nWorker: claude-fable-5-1 at max effort, routed by item 4's R + C of 11\n", outcomes)
def test_steps_the_worker_up_a_rung_when_the_judge_finds_in_the_transcript_the_escalation_trigger_it_cites(self):
checkout = Checkout(self)
checkout.env["LOOP_CLAUDE_WORKER_MODEL"] = "claude-opus-5-5"
(checkout.root / "todo.md").write_text(TODO)
run = checkout.run([
worker(say="Stability round 2: a 🔴. Trigger 3 fired; I ask for a higher rung."), judge(evidence="I ask for a higher rung.", escalation="Stability round 2: a 🔴."),
worker(say="Trigger 2 fired; I ask for a higher rung."), judge(evidence="I ask for a higher rung.", escalation="Trigger 2 fired; … a higher rung."),
worker(say="merged"), judge(finished=True, evidence="merged"),
worker(say="next chunk"),
])
self.assertEqual(run.code, 130)
workers = [call for call in run.calls if call["role"] == "worker"]
self.assertEqual([(argument(call, "--model"), argument(call, "--effort")) for call in workers], [
("claude-opus-5-5", "high"),
("claude-opus-5-5", "xhigh"),
("claude-opus-5-5", "xhigh"),
("claude-opus-5-5", "high"),
])
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at high effort, routed by item 4's R + C of 11\n\n<UTC> Stepped up\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's R + C of 11\n", outcomes)
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's R + C of 11\n", outcomes)
def test_runs_a_cheap_items_worker_below_the_projects_rung_with_its_reviewers_and_judge_on_it(self):
checkout = Checkout(self)
checkout.env["LOOP_CLAUDE_WORKER_MODEL"] = checkout.env["LOOP_CLAUDE_JUDGE_MODEL"] = "claude-opus-5-5"
+52 -28
View File
@@ -94,7 +94,7 @@ def ran(stdout="", code=0, stderr=""):
def judged(verdict, evidence="working", answer=""):
reply = {"answer": answer, "bundled": 0, "evidence": evidence, "finished": verdict == "finished", "interrupt": verdict == "interrupt", "question": "Which one?" if verdict in ("interrupt", "quoted") else ""}
reply = {"answer": answer, "bundled": 0, "escalation": "", "evidence": evidence, "finished": verdict == "finished", "interrupt": verdict == "interrupt", "question": "Which one?" if verdict in ("interrupt", "quoted") else ""}
return ran(json.dumps({"structured_output": reply}))
@@ -384,7 +384,7 @@ class RoutingCheck(unittest.TestCase):
folder = self.root / project / f"c{number}"
subagents = folder / "session" / "s" / "subagents"
subagents.mkdir(parents=True)
kept = "".join(f"2026-10-14T10:0{verdict}:00Z {outcomes.unfinished('w', 'e', 'j')}\n" for verdict in range(verdicts - 1))
kept = "".join(f"2026-10-14T10:0{verdict}:00Z {outcomes.unfinished('w', 'e', 'j', '')}\n" for verdict in range(verdicts - 1))
(folder / "copy.outcomes.md").write_text(f"Task: t\n\n{kept}2026-10-14T11:00:00Z {outcomes.finished('w', 'm', 'j')}\n")
launch = {"type": "assistant", "timestamp": "2026-10-14T09:00:00Z", "message": {"id": "m", "content": [{"type": "tool_use", "id": "a", "name": "Agent", "input": {"subagent_type": "Explore"}}]}}
report = {"type": "user", "message": {"content": [{"type": "tool_result", "tool_use_id": "a", "content": "long report"}]}}
@@ -515,25 +515,33 @@ class Routing(unittest.TestCase):
self.assertEqual(routing.first_route(self.OPUS, None), Route(self.OPUS))
def test_steps_up_a_rung_after_every_second_unfinished_verdict(self):
once = routing.stepped(self.OPUS, Route(self.OPUS, 3, 9), 0)
once = routing.stepped(self.OPUS, Route(self.OPUS, 3, 9), 0, False)
self.assertEqual(once, Route(self.OPUS, 3, 9, 1))
self.assertEqual(routing.stepped(self.OPUS, once, 0), Route(self.HIGH, 3, 9, 0))
self.assertEqual(routing.stepped(self.OPUS, Route(LADDER[-1], unfinished=1), 0).rung, LADDER[-1])
self.assertEqual(routing.stepped(self.OPUS, once, 0, False), Route(self.HIGH, 3, 9, 0))
self.assertEqual(routing.stepped(self.OPUS, Route(LADDER[-1], unfinished=1), 0, False).rung, LADDER[-1])
def test_steps_up_to_the_rung_a_bundled_items_r_plus_c_picks_and_never_down(self):
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS), 13).rung, self.XHIGH)
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 11).rung, self.XHIGH)
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS, unfinished=1), 13).rung, Rung("claude-opus-5-5", "max"))
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS), 13, False).rung, self.XHIGH)
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 11, False).rung, self.XHIGH)
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS, unfinished=1), 13, False).rung, Rung("claude-opus-5-5", "max"))
def test_steps_up_from_a_rung_below_the_projects_one_rung_at_a_time(self):
low = Route(Rung("claude-haiku-5-5", "xhigh"), 3, 2, 1)
self.assertEqual(routing.stepped(self.OPUS, low, 0).rung, Rung("claude-haiku-5-5", "max"))
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 2), 0).rung, low.rung)
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 2), 7).rung, Rung("claude-sonnet-5-5", "high"))
self.assertEqual(routing.stepped(self.OPUS, low, 0, False).rung, Rung("claude-haiku-5-5", "max"))
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 2), 0, False).rung, low.rung)
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 2), 7, False).rung, Rung("claude-sonnet-5-5", "high"))
def test_steps_up_a_rung_on_an_escalation_the_judge_found_up_to_two_above_the_items_rung(self):
self.assertEqual(routing.stepped(self.OPUS, Route(self.HIGH, 3, 11), 0, True), Route(self.XHIGH, 3, 11, 0))
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH, 3, 11), 0, True), Route(Rung("claude-opus-5-5", "max"), 3, 11, 0))
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "max"), 3, 11), 0, True), Route(Rung("claude-opus-5-5", "max"), 3, 11, 1))
self.assertEqual(routing.stepped(self.OPUS, Route(self.HIGH, 3, 11, 1), 0, True), Route(self.XHIGH, 3, 11, 0))
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 0, True), Route(self.XHIGH, unfinished=1))
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS, 3, 9), 13, True), Route(self.XHIGH, 3, 9, 1))
def test_a_rung_an_edit_took_off_the_ladder_counts_as_the_projects(self):
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 0).rung, self.OPUS)
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 2).rung, self.OPUS)
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 0, False).rung, self.OPUS)
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 2, False).rung, self.OPUS)
def test_runs_the_judge_on_the_rung_its_chunks_item_picks_never_below_the_projects(self):
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.HIGH, 3, 11)), self.HIGH)
@@ -594,7 +602,7 @@ class RouteSources(unittest.TestCase):
self.assertIsNone(main.archived_route(archive, opened))
route = Route(Rung("claude-opus-5-5", "high"), 3, 11)
with contextlib.redirect_stderr(io.StringIO()):
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(route), "working", "j"), NOW))
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(route), "working", "j", ""), NOW))
self.assertEqual(main.archived_route(archive, opened), route)
(chunk,) = archive.iterdir()
chunk.chmod(0)
@@ -623,7 +631,7 @@ class Steps(unittest.TestCase):
transcript.write_text(json.dumps({"type": "queue-operation", "timestamp": "1"}) + "\n")
before = Route(self.OPUS, 3, 9)
with contextlib.redirect_stderr(io.StringIO()):
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(before), "working", "j"), NOW))
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(before), "working", "j", ""), NOW))
after = main.after_verdict(main.LoopRun(cycle_number=1, root="/r", task="Do it.", unfinished_evidence="", route=before), self.OPUS, Unfinished("working", 13))
self.assertIsNone(main.archive_step(archive, opened, after, before, FakeEffects().effects()))
self.assertEqual(main.archived_route(archive, opened), Route(Rung("claude-opus-5-5", "xhigh"), 3, 9))
@@ -655,6 +663,8 @@ class WorkerArgs(unittest.TestCase):
stepped_up = prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 9), opus)
self.assertIn(" Your rung is above the project's, claude-opus-5-5 at medium effort: launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them. When an escalation trigger", stepped_up)
self.assertNotIn("the project's", prompts.resumed("", Route(opus, 3, 9), opus))
self.assertIn(f"{prompts.ANSWERS_RECORDED}. loop-claude runs you on claude-opus-5-5 at high effort. Your rung is above", stepped_up)
self.assertNotIn("runs you on", prompts.resumed("", Route(Rung("m")), None))
self.assertIn("whether or not it is item 3. Your rung is above the project's, claude-opus-5-5 at medium effort: launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them. When an escalation trigger", above)
def test_names_the_rungs_an_escalation_trigger_lets_the_worker_hand_a_task_to_at_most_two_above_its_items(self):
@@ -662,19 +672,26 @@ class WorkerArgs(unittest.TestCase):
fresh = prompts.fresh("Do it.", "## Goals", Route(opus, 3, 9), opus)
self.assertIn("whether or not it is item 3. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", fresh)
self.assertIn(" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort xhigh`, one rung above yours; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 9), opus))
self.assertIn(" An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `model opus, effort max`, one rung above yours, only once the human agrees. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 9), opus))
self.assertIn(" An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `model opus, effort max`, one rung above yours, only once the human agrees; loop-claude will not resume you higher for a trigger either, so ask the human for that yes. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 9), opus))
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("claude-fable-5-1", "max"), 3, 20), opus))
self.assertIn("hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort max`, one rung above yours;", prompts.resumed("", Route(Rung("claude-opus-5-5", "max"), 3, 13), opus))
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("m")), None))
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("claude-opus-5-5", "low"), 3, 9), opus))
self.assertIn("before creating a new one.\n\nWhen an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number. A reviewer", prompts.fresh("Do it.", "## Goals", Route(opus), opus))
self.assertIn("before creating a new one.\n\nWhen an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number. Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last such request: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`. A reviewer", prompts.fresh("Do it.", "## Goals", Route(opus), opus))
def test_offers_the_worker_a_higher_rung_for_the_rest_of_the_chunk_up_to_the_escalation_cap(self):
opus = Rung("claude-opus-5-5", "medium")
offer = " Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last such request: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`."
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 9), opus))
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 9), opus))
self.assertNotIn("Where the hard part", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 9), opus))
def test_names_the_rung_a_reviewer_or_seat_reruns_on_one_above_its_own(self):
opus = Rung("claude-opus-5-5", "medium")
both = " A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`."
self.assertTrue(prompts.resumed("", Route(opus, 3, 9), opus).endswith(both))
self.assertTrue(prompts.resumed("", Route(Rung("claude-sonnet-5-5", "high"), 3, 5), opus).endswith(both))
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{both}\n\nThe chunk's", prompts.fresh("Do it.", "## Goals", Route(opus), opus))
self.assertTrue(prompts.fresh("Do it.", "## Goals", Route(opus), opus).split("\n\nThe chunk's")[0].endswith(both))
self.assertTrue(prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 11), opus).endswith(" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort xhigh`, and a comprehension seat with `model opus, effort high`."))
self.assertTrue(prompts.resumed("", Route(Rung("claude-fable-5-1", "max"), 3, 20), opus).endswith(" A comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`."))
self.assertNotIn("rerun", prompts.resumed("", Route(Rung("m")), None))
@@ -768,19 +785,19 @@ class Grounded(unittest.TestCase):
class Verdict(unittest.TestCase):
def test_needs_every_field_the_schema_requires(self):
self.assertEqual(verdict_from({"bundled": 0, "answer": "", "interrupt": False, "evidence": "e", "finished": True, "question": ""}), Finished("e"))
self.assertEqual(verdict_from({"bundled": 0, "answer": "", "interrupt": False, "evidence": "e", "finished": False, "question": "ignored"}), Unfinished("e"))
self.assertEqual(verdict_from({"bundled": 0, "answer": "", "interrupt": True, "evidence": "e", "finished": True, "question": "Which one?"}), Interrupt("e", "Which one?"))
self.assertEqual(verdict_from({"bundled": 0, "answer": "A.", "interrupt": False, "evidence": "e", "finished": False, "question": "Which one?"}), QuotedAnswer("e", "Which one?", "A."))
self.assertEqual(verdict_from({"bundled": 0, "answer": "A.", "interrupt": True, "evidence": "e", "finished": False, "question": "Which one?"}), Interrupt("e", "Which one?"))
self.assertEqual(verdict_from({"bundled": 0, "answer": "A.", "interrupt": False, "evidence": "e", "finished": True, "question": ""}), Failure("the judge quoted an answer without a question"))
self.assertEqual(verdict_from({"bundled": 0, "answer": "", "interrupt": "no", "evidence": "", "finished": True, "question": ""}), Failure("the judge's reply has no interrupt of type boolean"))
self.assertEqual(verdict_from({"bundled": 0, "answer": "", "interrupt": False, "evidence": None, "finished": True, "question": ""}), Failure("the judge's reply has no evidence of type string"))
self.assertEqual(verdict_from({"bundled": 0, "answer": "", "interrupt": True, "evidence": "x", "finished": False, "question": " "}), Failure("the judge interrupted without a question"))
self.assertEqual(verdict_from({"bundled": 0, "escalation": "", "answer": "", "interrupt": False, "evidence": "e", "finished": True, "question": ""}), Finished("e"))
self.assertEqual(verdict_from({"bundled": 0, "escalation": "", "answer": "", "interrupt": False, "evidence": "e", "finished": False, "question": "ignored"}), Unfinished("e"))
self.assertEqual(verdict_from({"bundled": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "e", "finished": True, "question": "Which one?"}), Interrupt("e", "Which one?"))
self.assertEqual(verdict_from({"bundled": 0, "escalation": "", "answer": "A.", "interrupt": False, "evidence": "e", "finished": False, "question": "Which one?"}), QuotedAnswer("e", "Which one?", "A."))
self.assertEqual(verdict_from({"bundled": 0, "escalation": "", "answer": "A.", "interrupt": True, "evidence": "e", "finished": False, "question": "Which one?"}), Interrupt("e", "Which one?"))
self.assertEqual(verdict_from({"bundled": 0, "escalation": "", "answer": "A.", "interrupt": False, "evidence": "e", "finished": True, "question": ""}), Failure("the judge quoted an answer without a question"))
self.assertEqual(verdict_from({"bundled": 0, "escalation": "", "answer": "", "interrupt": "no", "evidence": "", "finished": True, "question": ""}), Failure("the judge's reply has no interrupt of type boolean"))
self.assertEqual(verdict_from({"bundled": 0, "escalation": "", "answer": "", "interrupt": False, "evidence": None, "finished": True, "question": ""}), Failure("the judge's reply has no evidence of type string"))
self.assertEqual(verdict_from({"bundled": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "x", "finished": False, "question": " "}), Failure("the judge interrupted without a question"))
self.assertEqual(verdict_from(None), Failure("the judge returned no reply"))
def test_takes_a_bundled_r_plus_c_from_0_to_20_on_an_unfinished_verdict(self):
reply = {"answer": "", "interrupt": False, "evidence": "e", "finished": False, "question": ""}
reply = {"answer": "", "escalation": "", "interrupt": False, "evidence": "e", "finished": False, "question": ""}
self.assertEqual(verdict_from({**reply, "bundled": 20}), Unfinished("e", 20))
self.assertEqual(verdict_from({**reply, "bundled": True}), Failure("the judge's reply has no bundled of type integer"))
for bundled in (21, -1):
@@ -788,6 +805,13 @@ class Verdict(unittest.TestCase):
self.assertEqual(verdict_from({**reply, "bundled": bundled}), Failure(f"the judge's bundled R + C {bundled} is not from 0 to 20"))
def test_takes_an_escalation_quote_on_an_unfinished_verdict(self):
reply = {"answer": "", "bundled": 0, "escalation": "round 2: 🔴", "interrupt": False, "evidence": "e", "finished": False, "question": ""}
self.assertEqual(verdict_from(reply), Unfinished("e", 0, "round 2: 🔴"))
self.assertEqual(verdict_from({**reply, "finished": True}), Finished("e"))
self.assertEqual(verdict_from({**reply, "escalation": None}), Failure("the judge's reply has no escalation of type string"))
class LoopRuns(unittest.TestCase):
def test_stops_naming_the_reload_when_it_hands_over_anything_but_the_state_with_a_task(self):
for handed in (["3", "2", "/r", "do it"], ["3"], ["[1]"], [json.dumps({"cycle_number": 3, "root": "/r", "unfinished_evidence": ""})], [json.dumps({"cycle_number": 3, "root": "/r", "task": "", "unfinished_evidence": ""})], [json.dumps({"cycle_number": "3", "root": "/r", "task": "do it", "unfinished_evidence": ""})], [json.dumps({"cycle_number": 3, "root": "", "task": "do it", "unfinished_evidence": ""})], [json.dumps({"cycle_number": 3, "root": "/r", "task": "do it", "turns": 1, "unfinished_evidence": ""})], [json.dumps({"cycle_number": 3, "failures": 0, "root": "/r", "task": "do it", "unfinished_evidence": ""})]):
+1 -1
View File
@@ -133,7 +133,7 @@ class ReadingCostTool(unittest.TestCase):
def test_appends_each_finished_chunk_once_in_the_order_they_finished_from_a_worktree(self):
self.archive_chunk(1790000200)
self.archive_chunk(1790000100)
self.archive_chunk(1790000000, outcomes.unfinished)
self.archive_chunk(1790000000, lambda *fields: outcomes.unfinished(*fields, ""))
chunks = finished_chunks(self.archive)
self.assertEqual([chunk.finished for chunk in chunks], ["2026-09-21T14:15:00Z", "2026-09-21T14:16:40Z"])
worktree = self.home / "code" / "proj-wt"
+5 -5
View File
@@ -49,7 +49,7 @@ class Measured(unittest.TestCase):
message("m2", "2026-10-14T10:05:00.000Z", {"type": "text", "text": "done"}),
message("m3", "2026-10-14T11:00:00.000Z", {"type": "text", "text": "on"}),
]
kept = stamped("2026-10-14T10:30:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j")) + stamped("2026-10-14T10:30:00Z", outcomes.stepped_up(str(HIGH))) + stamped("2026-10-14T12:00:00Z", outcomes.finished(str(HIGH), "merged", "j"))
kept = stamped("2026-10-14T10:30:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j", "")) + stamped("2026-10-14T10:30:00Z", outcomes.stepped_up(str(HIGH))) + stamped("2026-10-14T12:00:00Z", outcomes.finished(str(HIGH), "merged", "j"))
found = measured(Chunk("log", [kept], worker, []), "/h")
self.assertEqual(found.tokens["project"], {"log": 333})
self.assertEqual(found.tokens["subagent type"], {"worker": 333})
@@ -92,7 +92,7 @@ class Measured(unittest.TestCase):
def test_puts_a_subagent_on_the_model_it_ran_at_the_effort_of_the_cycle_that_launched_it(self):
kept = stamped("2026-10-14T10:00:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j")) + stamped("2026-10-14T11:00:00Z", outcomes.finished(str(HIGH), "m", "j"))
kept = stamped("2026-10-14T10:00:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j", "")) + stamped("2026-10-14T11:00:00Z", outcomes.finished(str(HIGH), "m", "j"))
worker = [
message("m1", "2026-10-14T10:00:00.900Z"),
message("m2", "2026-10-14T10:30:00Z", call("Agent", "a1", subagent_type="Explore", model="sonnet")),
@@ -183,14 +183,14 @@ class Archive(unittest.TestCase):
(folder / "session" / "s" / "subagents" / "agent-a.meta.json").write_text(json.dumps(meta))
def test_counts_the_verdicts_across_every_projects_chunks(self):
self.chunk("home/x", "c1", stamped("2026-10-14T10:00:00Z", outcomes.unfinished("w", "e", "j")) + stamped("2026-10-14T10:01:00Z", outcomes.stopped("w", "Ctrl-C")) + stamped("2026-10-14T10:02:00Z", outcomes.finished("w", "e", "j")), [])
self.chunk("home/x", "c1", stamped("2026-10-14T10:00:00Z", outcomes.unfinished("w", "e", "j", "")) + stamped("2026-10-14T10:01:00Z", outcomes.stopped("w", "Ctrl-C")) + stamped("2026-10-14T10:02:00Z", outcomes.finished("w", "e", "j")), [])
self.chunk("log", "c2", stamped("2026-10-14T10:00:00Z", outcomes.interrupted("q", "e", "j")), [])
self.assertEqual(archive.cycles(self.root), 2)
self.assertEqual(archive.cycles(self.root / "missing"), 0)
def test_measures_each_finished_chunk_keyed_by_its_project(self):
self.chunk("home/x", "c1", stamped("2026-10-14T10:00:00Z", outcomes.finished(LOW_ROUTE, "m", "j")), [message("m1", "2026-10-14T09:00:00Z", call("Agent", "a1", subagent_type="Explore"))], ({"agentType": "Explore", "toolUseId": "a1"}, [message("s1", "2026-10-14T09:00:01Z")]))
self.chunk("log", "c2", stamped("2026-10-14T10:00:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j")), [message("m1", "2026-10-14T09:00:00Z")])
self.chunk("log", "c2", stamped("2026-10-14T10:00:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j", "")), [message("m1", "2026-10-14T09:00:00Z")])
found = archive.measure_archive(self.root, "/h", "")
self.assertEqual([chunk.tokens["project"] for chunk in found.chunks], [{"home/x": 222}])
self.assertEqual(found.chunks[0].tokens["subagent type"], {"worker": 111, "Explore": 111})
@@ -202,7 +202,7 @@ class Archive(unittest.TestCase):
first = message("m1", "2026-09-21T00:00:00Z")
transcript.write_text(json.dumps(first) + "\n")
with contextlib.redirect_stderr(io.StringIO()):
self.assertIsNone(write.archive(kept / "log", str(transcript), "t", outcomes.unfinished(LOW_ROUTE, "e", "j"), 1790000000))
self.assertIsNone(write.archive(kept / "log", str(transcript), "t", outcomes.unfinished(LOW_ROUTE, "e", "j", ""), 1790000000))
transcript.write_text(json.dumps(first) + "\n" + json.dumps(message("m2", "2099-01-01T00:00:00Z")) + "\n")
self.assertIsNone(write.archive(kept / "log", str(transcript), "t", outcomes.finished(str(HIGH), "m", "j"), 4070908800))
(found,) = archive.measure_archive(kept, "/h", "").chunks
+4 -2
View File
@@ -27,8 +27,10 @@ def finished(worker: str, evidence: str, judge: str) -> str:
return f"{FINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{JUDGE}{judge}\n{WORKER}{worker}"
def unfinished(worker: str, evidence: str, judge: str) -> str:
return f"{UNFINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{JUDGE}{judge}\n{WORKER}{worker}"
def unfinished(worker: str, evidence: str, judge: str, escalation: str) -> str:
"""escalation is the judge's quote showing that the worker's cited escalation trigger fired; empty for none."""
escalated = f"Escalation:\n{quoted(escalation)}\n\n" if escalation else ""
return f"{UNFINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{JUDGE}{judge}\n{WORKER}{worker}"
def interrupted(question: str, evidence: str, judge: str) -> str:
@@ -24,7 +24,7 @@ class Archive(unittest.TestCase):
transcript = pathlib.Path(root, "t.jsonl")
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n')
with contextlib.redirect_stderr(io.StringIO()):
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.unfinished("m", "working", "j"), NOW))
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.unfinished("m", "working", "j", ""), NOW))
(kept_outcomes,) = kept.glob("*/*.outcomes.md")
kept_outcomes.unlink()
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.stopped("m", "Ctrl-C"), NOW))
@@ -49,13 +49,17 @@ class Outcomes(unittest.TestCase):
def test_reads_back_the_worker_of_each_outcome_that_closes_a_cycle(self):
kept = "Task: Do it.\n\n" + "\n".join(f"{utc(NOW + offset)} {outcome}\n" for offset, outcome in enumerate([
outcomes.unfinished("w1", "Worker: not this", "j"),
outcomes.unfinished("w1", "Worker: not this", "j", ""),
outcomes.interrupted("Which one?\nWorker: nor this", "e", "j"),
outcomes.stopped("w2", "the judge failed\nthree times"),
outcomes.finished("w3", "merged", "j"),
]))
self.assertEqual(outcomes.workers(kept), [(utc(NOW), "w1"), (utc(NOW + 2), "w2"), (utc(NOW + 3), "w3")])
def test_archives_the_escalation_an_unfinished_verdict_quotes_beside_its_evidence(self):
self.assertEqual(outcomes.unfinished("w", "e", "j", "round 2: 🔴"), "Verdict: unfinished\n\nEvidence:\n> e\n\nEscalation:\n> round 2: 🔴\n\nJudge: j\nWorker: w")
self.assertEqual(outcomes.unfinished("w", "e", "j", ""), "Verdict: unfinished\n\nEvidence:\n> e\n\nJudge: j\nWorker: w")
def test_reads_no_worker_from_an_outcome_written_before_the_worker_line(self):
kept = f"Task: Do it.\n\n{utc(NOW)} Stopped: Ctrl-C\n\n{utc(NOW)} Verdict: unfinished\n\nEvidence:\n> Worker: no\n"
self.assertEqual(outcomes.workers(kept), [])
@@ -68,7 +72,7 @@ class Outcomes(unittest.TestCase):
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n')
with contextlib.redirect_stderr(io.StringIO()):
for worker in ("z-first", "a-last"):
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.unfinished(worker, "working", "j"), NOW))
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.unfinished(worker, "working", "j", ""), NOW))
self.assertEqual(read.latest_worker(kept, str(transcript)), "a-last")
def test_names_the_worker_the_latest_outcome_of_the_transcripts_chunk_records(self):
@@ -78,7 +82,7 @@ class Outcomes(unittest.TestCase):
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n')
self.assertIsNone(read.latest_worker(kept, str(transcript)))
with contextlib.redirect_stderr(io.StringIO()):
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.unfinished("w1", "working", "j"), NOW))
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.unfinished("w1", "working", "j", ""), NOW))
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n{"more": 1}\n')
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.stopped("w2", "Ctrl-C"), NOW + 1))
self.assertEqual(read.latest_worker(kept, str(transcript)), "w2")
+38
View File
@@ -1807,3 +1807,41 @@ Junior A, decided by:
- Shape: `bin/rung_replay/search.py:37` `Replay`: the search is generic over three type parameters with two callbacks and a label, so a reader works out what `Item` and `Reply` are for each caller before the loop makes sense
- Shape: `bin/judge_replay/replay.py:65` `reproduced`: returns `[kind]` or `[]` only to fit the generic shape, and `report` rebuilds the unreproduced list by calling it again
- Shape: `bin/rung_replay/command.py:1` `command`: the name says little, and the module mixes argument parsing, a models-file lookup, rung arithmetic and stderr exits
## 2026-10-15T20:33:16Z, PR #278 at 0f71e33, against ddb86cb
Ruling: same
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|---|---|---|---|---|---|
| Mid A | same | better | same | better | better |
| Inherited architect | same | same | worse | better | same |
| Maintainability senior | same | same | worse | better | same |
| Junior A | same | same | same | better | better |
Mid A, decided by:
- Locality: `bin/loop_claude/routing.py:38` `kept_rung`: one rule now raises a route's rung to item_rung for both `kept` and `stepped`, where the base handled the off-ladder case in two places with two different targets
- Locality, Self-sufficiency: `bin/loop_claude/routing.py:55` `escalation_cap`: the escalation step and the prompt's offered rungs share one named cap, whose docstring gives the reason it counts from item_rung instead of naming who set it
- Self-sufficiency: `bin/loop_claude/cycle/judge.py:149` `run_judge`: a docstring lists the three quotes, what each is checked against and what a misquote does, where the base had one comment covering only the quoted answer
- Self-sufficiency: `bin/loop_claude/routing.py:63` `stepped`: the docstring says why it takes the verdict's fields and not the verdict, an import cycle through prompts
Inherited architect, decided by:
- Shape: `bin/loop_claude/routing.py:63` `stepped`: the starting rung now takes three helper hops, `taken_in`, `kept_rung`, `highest`, where the base had two inline lines, and the off-ladder handling is said only in `kept_rung`'s docstring
- Shape: `bin/loop_claude/routing.py:78` `taken_in`: used once, its name does not say what it returns, and `kept(route, anchor)` and `kept_rung(anchor, route)` take their arguments in different orders
- Self-sufficiency: `bin/loop_claude/routing.py:55` `escalation_cap`: the cap's reason is now in the code, and it is computed once for `escalation_rungs` and `stepped`
- Self-sufficiency: `bin/loop_claude/cycle/judge.py:149` `run_judge`: the docstring lists all three quote checks and why no misquote but the evidence's is a failure, which answers the 3am question without opening anything else
- Self-sufficiency: `bin/loop_claude/routing.py:66` `stepped`: the docstring explains the import cycle that keeps the verdict type out of routing
Maintainability senior, decided by:
- Self-sufficiency: `bin/loop_claude/cycle/judge.py:149` `run_judge`: the docstring lists the three quote checks, what each is checked against and what a misquote does, where the base had one comment covering only the answer
- Self-sufficiency: `bin/loop_claude/routing.py:63` `stepped`: the docstring states the import cycle that keeps the verdict type out, and `escalation_cap` says why the cap counts from item_rung
- Shape: `bin/loop_claude/routing.py:11` `ESCALATION_CAP`: differs only in case from `escalation_cap` at `:55`, an int against a function returning a `Rung`, and its comment ends in a stray "yes,."
- Shape: `bin/loop_claude/routing.py:78` `taken_in`: `stepped` no longer reads top to bottom; it chases `taken_in`, `kept_rung`, `item_rung` and `picked`, and the off-ladder rule lives only in `kept_rung`'s docstring
Junior A, decided by:
- Self-sufficiency, Overall: `bin/loop_claude/cycle/judge.py:149` `run_judge`: the docstring lists the three quote checks, how strictly each is checked and what a misquote does, where the base made the reader work out why grounded and verbatim differ
- Self-sufficiency: `bin/loop_claude/routing.py:55` `escalation_cap`: the limit is its own named function whose docstring says why it counts from item_rung, and `stepped` names `counted_out` and `escalates`
+13 -1
View File
@@ -206,7 +206,7 @@ A model off the ladder routes nothing: claude runs it at the effort its settings
The worker delegates a task only when it needs just the task's conclusion: a search or read across many files, or work independent enough to run in parallel. Edits, and whatever it must check line by line, stay in its own thread, since it would read a delegated diff again to check it. Measured over archived worker transcripts, subagent reports cost a median 8% of the worker's peak context, and `sed -n` and `grep` output made up 41% of its Bash results, so delegated searches save the most context. It scores each delegated task's R and C as for an item, and `bin/subagent-rung` turns them into the Agent tool's model and effort, counted from the project's rung. The rule names both, because the Agent tool sets a subagent's effort only when instructions ask. `CLAUDE.md` states the rule, since it holds in every session. A task that proves too hard for the worker's rung goes one rung up instead: `routing: a worker hands a task one rung up`.
The worker changes rung only as the chunk starts, after every second unfinished verdict, or when the judge reports a taken-in item whose R + C picks a higher rung, because a model switch inside a session discards its prompt cache. loop-claude routes on the first row of the next release in `todo.md`'s table that waits on no question. It reads neither an order the plan states in prose nor the task, so a chunk may start on another item's rung: a higher one costs money, and a lower one holds until the judge reports the item the worker took in, after the first cycle. `todo.md` item 136 ends this, routing on the item the worker says it takes.
Because a model switch inside a session discards its prompt cache, the worker changes rung only as the chunk starts, after every second unfinished verdict, when the judge reports a taken-in item whose R + C picks a higher rung, or when the judge finds the escalation trigger a worker cites in asking for a higher rung (`routing: a worker whose run ends asking for a higher rung`). loop-claude routes on the first row of the next release in `todo.md`'s table that waits on no question. It reads neither an order the plan states in prose nor the task, so a chunk may start on another item's rung: a higher one costs money, and a lower one holds until the judge reports the item the worker took in, after the first cycle. `todo.md` item 136 ends this, routing on the item the worker says it takes.
Valid while the index ranks these models, and while no newer model joins the catalog.
@@ -220,6 +220,18 @@ loop-claude's prompt names the rungs a trigger opens, so the worker computes no
Valid while the ladder orders its rungs by competence.
## routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, resumes one rung up next cycle, up to two rungs above the rung its item picks
2026-10-15, lilleman asked for it. Serves goals 1.1 and 1.4: when the hard part is the rest of the chunk, handing it to subagents piece by piece leaves the worker, which reads and checks each of their diffs, on a rung that proved too low. The worker climbs only once its run records a trigger.
loop-claude's prompt offers the step beside the escalated subagent of `routing: a worker hands a task one rung up`, as long as the limit leaves a rung that subagent may take without the human's yes. The worker then ends its run asking loop-claude for a higher rung and naming the trigger. The judge quotes an earlier passage that shows the trigger fired, such as a review round's commit title or the worker's report of the failing test, never the request itself. The trigger must have fired since any earlier such request, so one trigger buys one step. The judge sees the worker's text and tool calls but no tool results, so the quote shows that the run recorded the trigger, not that a tool's output bears it out. loop-claude checks the quote word for word, as it checks a quoted answer, since stitched fragments can make up a trigger. Without such a quote, the rung holds. A misquote is no failure, since holding the rung is safe and three failures stop an unattended loop. The next prompt names the worker's rung, so the worker learns whether the step went through.
The step is one rung up from the rung the worker would otherwise hold: its own, or the rung an item the judge saw it take in picks, whichever is higher. It resets the count of unfinished verdicts, so the step every second one takes counts afresh; when both steps fall on one verdict, the worker steps once. It stops at the limit escalated subagents stop at, two rungs above the rung the chunk started on. Past that limit the rung holds: loop-claude takes no yes from the human between cycles, so a worker there asks the human for an escalated subagent instead. The judge keeps its rung, as `routing: on a model judge-replay covered` says.
The archive quotes the judge's passage under `Escalation:` in the unfinished verdict, beside any step-up it causes. `bin/routing_figures/` counts that verdict as unfinished, not as an escalation by trigger, which it counts only from subagent prompts.
Valid while the worker reports in its transcript the triggers that fired.
## routing: on a model judge-replay covered, the judge runs on the rung its chunk's item picks, never below the project's rung
2026-10-15, lilleman's plan, settled by the replay below. Serves goals 1.2 and 1.4, under the ladder and bands of `routing: one ladder`.
+17
View File
@@ -401,3 +401,20 @@ Read most (opens and searches):
- `bin/loop_claude/test_loop_claude.py` 2
- `skills/comprehension-panel/SKILL.md` 2
- `skills/comprehension-panel/seat-prompts.md` 2
## Reading cost, chunk dbd1e7538163, finished 2026-10-15T19:40:18Z
146584 new input tokens before the first commit.
Read most (opens and searches):
- `README.md` 5
- `skills/larv-review/SKILL.md` 4
- `todo.md` 4
- `bin/judge_replay/main.py` 3
- `bin/loop_claude/cycle/retry.py` 3
- `docs/decisions.md` 3
- `CHANGELOG.md` 2
- `bin/claude_code/entries.py` 2
- `bin/failure.py` 2
- `bin/judge_replay/test_judge_replay.py` 2
+1 -10
View File
@@ -26,7 +26,6 @@ Release 0.3.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |
|---|---|---|---|---|---|---|---|---|---|---|
| 143 | 0.1.0 | | **Resume a worker one rung up in the next cycle when its run ends asking for a higher rung and the judge finds the escalation trigger it cites in the transcript.** | 5 | 5 | 7 | 6 | 7 | 1.1, 1.4 | 19.6 |
| 136 | 0.1.0 | decision | **Route a chunk on the item its worker says it takes, in place of the next release's first row.** | 2 | 3 | 4 | 4 | 5 | 1.4 | 14.8 |
| 150 | 0.2.0 | decision | **Make each README goal `maintain` or `achieve`, give each achieve goal a signal, a counter-signal and a one-line reason, and have a maintain goal veto an item that breaks it.** | 3 | 3 | 4 | 7 | 8 | 1.2 | 25.9 |
| 148 | 0.2.0 | decision | **Rank items by V / C, where C is 2^S and V sums each cited achieve goal's sum-reciprocal weight times how far the item moves it, and drop A.** | 5 | 5 | 6 | 7 | 8 | 1.1, 1.2 | 23.6 |
@@ -90,14 +89,6 @@ Release 0.3.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
## Details
### 143. Resume a worker one rung up in the next cycle when its run ends asking for a higher rung and the judge finds the escalation trigger it cites in the transcript.
The human asked for this on 2026-10-15. A worker may end its run asking for a higher rung, citing an escalation trigger of `CLAUDE.md` → "Working with the codebase". The judge looks for that trigger in the transcript. Where it finds it, the next cycle resumes the session one rung up; otherwise the rung holds. The worker never steps down within a chunk, and goes at most two rungs above the rung it started the chunk on unless the human agrees.
This revises `docs/decisions.md` → `routing: one ladder`, where the worker steps up only after every second unfinished verdict or when the judge reports a taken-in item that picks a higher rung. This item adds a third way.
The comprehension seats found two places it touches hard to read. Word the new check so that `bin/loop_claude/cycle/judge.py`'s `JUDGE_RULES` gets easier to read. `bin/loop_claude/routing.py`'s `kept` and `stepped` use different floors, and `stepped` does two jobs; make them easier to read where the new step joins them.
### 136. Route a chunk on the item its worker says it takes, in place of the next release's first row.
loop-claude routes on the first row of the next release, while the worker takes the item the plan's prose orders first, or a prerequisite, as the chunk that shipped item 88, a prerequisite, took item 88 in place of the first row. A first row of higher R + C pays for a dearer rung all chunk; one of lower R + C holds until the judge reports the item taken in. The judge reports the item the worker takes only after the first cycle, so routing the first cycle right needs the item before the worker starts.
@@ -106,7 +97,7 @@ Once this lands, the judge follows the item the worker takes too, since `judge_r
### 150. Make each README goal `maintain` or `achieve`, give each achieve goal a signal, a counter-signal and a one-line reason, and have a maintain goal veto an item that breaks it.
`rules/goals-and-audience.md` states the form, with a GOOD and a BANNED example of each. The Audience says whose value the goals serve. A maintain goal is weighed by none of the scores: an item that breaks one is redesigned or dropped. A new spelling of a goal line is something the human reads and writes, so a panel picks it, as `CLAUDE.md` → "Bench a syntax" says.
`rules/goals-and-audience.md` states the form, with a GOOD and a BANNED example of each. The Audience says whose value the goals serve. A maintain goal is weighed by none of the scores: an item that breaks one is redesigned or dropped. A new spelling of a goal line is something the human reads and writes, so a panel picks it, as `CLAUDE.md` → "Working loop", "Bench a syntax with the people who write it" says.
### 148. Rank items by V / C, where C is 2^S and V sums each cited achieve goal's sum-reciprocal weight times how far the item moves it, and drop A.