Merge pull request 'Accept an escalation quote only from transcript entries after the worker's last step-up, so a later trigger steps the worker again' (#321) from item-180 into main
This commit was merged in pull request #321.
This commit is contained in:
+1
-1
@@ -6,7 +6,7 @@
|
||||
- Each reviewer under `agents/` now writes its full report to a file that larv-review names in the reviewer's prompt, in a folder per chunk under `~/.local/state/larv-review/`. It returns only its verdict and one line per finding, so a review takes less of the worker's context. The worker reads a finding's full entry before it fixes or answers a 🔴 or 🟡. Each reviewer now holds Write. `rule-hook` refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`; no override waives this rule. Each chunk's folder stays after the merge, so `stability-replay` can replay a round that reads an earlier round's report. It replays on a copy of the folder.
|
||||
- `loop-claude` asks an interrupt's question on its own terminal and resumes the worker headless with your one-line answer, so the loop no longer waits for you to leave a claude session. Type `/talk` instead to open the worker's session in claude, as every interrupt did before, to read the diff or discuss first. A release ready to cut opens that session at once. Pressing Enter on an empty answer stops the loop, exit 2. Keys typed before the question shows are dropped, and a pasted answer keeps only its first line; type `/talk` for a longer one. The judge counts that answer as yours, like one typed in claude: the chunk stays open until the answer is recorded, where a later chunk could need it.
|
||||
- `todo.md` may mark a renamed release in a `## Renamed` section, one line per rename, such as `- 0.2.1 renamed to 0.3.0`. `chunk-preflight` and `loop-claude` then skip that release when they pick the next one, and larv-review's scoring-run trigger calls no run for it. Without the mark, a release named since the last tag whose items all moved to other releases reads as finished: `chunk-preflight` says it is ready to cut and the loop stops for you, and the trigger calls a scoring run. `rank_todo.py` fails on a `## Renamed` line of another shape, and on an item filed under a release it marks. Before you move a machine to this release, add the line to the `todo.md` of every project that renamed a release since its last tag, or its loop stops for that release's cut.
|
||||
- When a worker's run ends with a question and a request for a higher rung, `loop-claude` now steps the worker one rung up as the judge interrupts, or as it answers the question with a quote from `README.md` or another file it may quote. The judge must quote a passage showing that the escalation trigger the run cites fired. Before, the step was lost for an answered question, and the worker carried on from the answer on its old rung. The step has the same limits as the step on an unfinished verdict, and no later ruling of that cycle carries a second escalation. The archive's interrupt and quoted-answer outcomes now name the worker's rung.
|
||||
- When a worker's run ends with a question and a request for a higher rung, `loop-claude` now steps the worker one rung up as the judge interrupts, or as it answers the question with a quote from `README.md` or another file it may quote. The judge must quote a passage showing that the escalation trigger the run cites fired. Before, the step was lost for an answered question, and the worker carried on from the answer on its old rung. The step has the same limits as the step on an unfinished verdict. An escalation trigger that fired before the worker's last step-up doesn't step it again, so one trigger buys one step. A trigger that fires after the step steps the worker again, in the same cycle or a later one. A chunk left open across the upgrade to this release starts with no step-up on record, so a trigger that already stepped its worker may step it once more. The archive's interrupt and quoted-answer outcomes now name the worker's rung.
|
||||
- A scoring run due because a release's items ran out costs four more seats: it is done twice on one head. `larv-review agreement` compares how often Mid A and the inherited architect vote alike in paired rulings with how often their overall scores move the same way between the two runs. After 10 such pairs it gives a verdict, and on `Assumption fails:` larv-review asks you whether to keep the paired ruling.
|
||||
- `workflow-bench/bench.py` gives each run a fixed list of allowed tools. Before, it skipped every permission check. A run may launch agents, run `/security-review`, edit its clone (not the clone's `.git/` or `.claude/`) and its report, and read its clone and the copy of `~/.claude` it benches. For git it gets one command, `workflow-bench/review_git.py`, and no bare `git`. The wrapper runs `log`, `diff`, `show`, `status`, `ls-files`, `add` and `commit -m` on the run's clone, whatever the working directory. It allows only the options it lists for each, and refuses an argument that names a path outside the clone. git runs with no global or system config, hook, pager or editor. A run loads no settings or MCP servers from `~/.claude` or from the case's clone, so their hooks and extra directories do not apply to it. Every other call is denied. Expect a benched review that needs another git command, or to run a project's code or tests, to say in its report that it could not.
|
||||
- Every goal in `README.md` → Goals is marked `**Maintain:**`, a state to hold, or `**Achieve:**`, a direction to pursue. Each achieve goal has a signal, a counter-signal and a one-line reason under it. A change that breaks a maintain goal is redesigned or dropped, whatever it scores: `rules/todo-items.md` says so, and larv-review's goal alignment step no longer merges such a PR. `rules/goals-and-audience.md` states the form for every project's README. Expect larv-review to stop a PR on a maintain goal where it asked you before.
|
||||
|
||||
@@ -91,5 +91,6 @@ def rule(case: Case, rung: Rung, claude_dir: pathlib.Path, effects: Effects) ->
|
||||
return verdict_kind(Failure(records.reason))
|
||||
if isinstance(entries, Failure):
|
||||
return verdict_kind(entries)
|
||||
verdict: Verdict | Failure | Limited = first_reply(lambda: judge_once(case.root, case.task, premises, records, entries, rung, effects))
|
||||
# 0 for the route's mark, as before any step-up: the replay compares verdict kinds, and a mark changes only which escalation quotes count.
|
||||
verdict: Verdict | Failure | Limited = first_reply(lambda: judge_once(case.root, case.task, premises, records, entries, 0, rung, effects))
|
||||
return verdict if isinstance(verdict, Limited) else verdict_kind(verdict)
|
||||
|
||||
@@ -138,6 +138,8 @@ class Main(unittest.TestCase):
|
||||
self.assertTrue(out.endswith("returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort\n"))
|
||||
for said in ("The task an agent was given:\nDo it.", "1. Ship.", "From ~/.claude/CLAUDE.md:\nBe kind."):
|
||||
self.assertIn(said, self.prompts[0])
|
||||
# The replay shows the judge the whole transcript, with no step-up marked in it.
|
||||
self.assertNotIn("[step]", self.prompts[0])
|
||||
|
||||
def test_stops_at_the_projects_rung_and_names_none_when_it_misses(self):
|
||||
code, out, _ = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted and rung == REFERENCE, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
|
||||
@@ -8,6 +8,7 @@ from claude_code.reply import claude_reply, Limited
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.project.records import HUMAN_RULE_NAMES
|
||||
from claude_code.entries import human_text, loop_text
|
||||
from todo_table import LEVELS
|
||||
from markdown_text import listed
|
||||
from model_routing.route import on_its_scale
|
||||
@@ -56,6 +57,8 @@ FINISHED_RULES = (
|
||||
"continue: an open PR, a review not yet run, CI not yet reported, an answer not yet recorded."
|
||||
)
|
||||
JUDGE_RULES = "\n\n".join((INTERRUPT_RULES, QUOTE_RULES, FINISHED_RULES))
|
||||
# The line judge_prompt shows the judge where an escalation last stepped the worker up, which Reply.escalation's description points at. Only the judge reads it, so it is no [loop] line.
|
||||
STEP_LINE = "[step] loop-claude stepped the agent up a rung here, so the escalation triggers that fired before this line are spent."
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -66,7 +69,7 @@ class Reply:
|
||||
# Routes the next cycle's judge and reviewers, as bundled routes the worker.
|
||||
bundled_escape_and_impact: int = dataclasses.field(metadata={"description": "The highest E + I the agent states for a todo.md item it took into this chunk; 0 when it states none."})
|
||||
# Routes the worker one rung up, as the judge interrupts or quotes an answer, or on the next cycle: docs/decisions.md → `routing: a worker whose run ends asking for a higher rung`.
|
||||
escalation: str = dataclasses.field(metadata={"description": "Empty unless the chunk is unfinished, interrupt is true or answer quotes a passage, and the agent's run ends asking loop-claude for a higher rung, citing an escalation trigger. Then: a verbatim quote, without elisions, of an earlier transcript passage showing the trigger fired after any earlier such request, such as a review round's commit title or the agent's report of the finding or failing test. Never the request itself."})
|
||||
escalation: str = dataclasses.field(metadata={"description": "Empty unless the chunk is unfinished, interrupt is true or answer quotes a passage, and the agent's run ends asking loop-claude for a higher rung, citing an escalation trigger. Then: a verbatim quote, without elisions, of the transcript passage where the trigger fired, such as a review round's commit title or the agent's report of the finding or failing test. Where a [step] line says loop-claude stepped the agent up a rung, quote only the agent's own words after it, and leave this empty where the trigger fired only before it. Never a later mention of the trigger, and never the request itself."})
|
||||
evidence: str = dataclasses.field(metadata={"description": "A verbatim quote of the one transcript passage the verdict rests on, without elisions: what triggers the interrupt; for a quoted answer, the agent's question; else what shows the chunk finished or the work remaining."})
|
||||
finished: bool = dataclasses.field(metadata={"description": "Whether the chunk is finished."})
|
||||
interrupt: bool = dataclasses.field(metadata={"description": "Whether the human must now be interrupted."})
|
||||
@@ -100,6 +103,8 @@ class Interrupt:
|
||||
# The bundled L the judge saw, 0 for none, as on Unfinished: an escalation steps up from the rung that item picks.
|
||||
bundled: int = 0
|
||||
release_ready: bool = False
|
||||
# How many transcript entries the judge read, which judge_once stamps. An escalation the verdict carries moves the route's mark there; one on a verdict never stamped, None, steps nothing.
|
||||
entries_read: int | None = None
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -111,6 +116,7 @@ class QuotedAnswer:
|
||||
# As on Interrupt.
|
||||
escalation: str = ""
|
||||
bundled: int = 0
|
||||
entries_read: int | None = None
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -127,11 +133,28 @@ class Unfinished:
|
||||
escalation: str = ""
|
||||
# The bundled E + I the judge saw, 0 for none.
|
||||
bundled_escape_and_impact: int = 0
|
||||
# As on Interrupt.
|
||||
entries_read: int | None = None
|
||||
|
||||
|
||||
Verdict = Interrupt | QuotedAnswer | Finished | Unfinished
|
||||
|
||||
|
||||
class Grounds(typing.NamedTuple):
|
||||
"""What run_judge checks the judge's quotes against, word for word, since stitched fragments can say what their source does not."""
|
||||
# The transcript as the judge saw it. The evidence must quote it, its fragments in order: a misquote is Rejected, which judge retries.
|
||||
shown: str
|
||||
# The agent's own lines after the mark, all of them before any step-up. loop-claude's prompts and the human's lines are left out, since they repeat earlier passages, such as the judge's evidence or the agent's question. An escalation must quote both this and shown: a misquote, or a quote of a passage before the mark, drops the escalation, so the worker's rung holds.
|
||||
since_step: str
|
||||
# README.md's premises and the records. A quoted answer must quote them: a misquote puts the question to the human.
|
||||
quotable: str
|
||||
|
||||
|
||||
def escalation_mark(verdict: Interrupt | QuotedAnswer | Unfinished) -> int | None:
|
||||
"""Where the verdict's escalation would move the route's mark, if it steps the worker; None where it carries none."""
|
||||
return verdict.entries_read if verdict.escalation else None
|
||||
|
||||
|
||||
def verdict_from(output: object) -> Verdict | Failure:
|
||||
if not isinstance(output, dict):
|
||||
return Failure("the judge returned no reply")
|
||||
@@ -155,14 +178,15 @@ def verdict_from(output: object) -> Verdict | Failure:
|
||||
return Unfinished(output["evidence"], bundled, output["escalation"], bundled_escape_and_impact)
|
||||
|
||||
|
||||
def judge(root: str, task: str, premises: str, records: str, entries: list[dict], rung: Rung, effects: Effects, keep: typing.Callable[[Rejected], Stop | None]) -> Verdict | Stop:
|
||||
return retried("judge", lambda: judge_once(root, task, premises, records, entries, rung, effects), effects, lambda failure: keep(failure) if isinstance(failure, Rejected) else None)
|
||||
def judge(root: str, task: str, premises: str, records: str, entries: list[dict], escalated_at: int, rung: Rung, effects: Effects, keep: typing.Callable[[Rejected], Stop | None]) -> Verdict | Stop:
|
||||
return retried("judge", lambda: judge_once(root, task, premises, records, entries, escalated_at, rung, effects), effects, lambda failure: keep(failure) if isinstance(failure, Rejected) else None)
|
||||
|
||||
|
||||
def judge_once(root: str, task: str, premises: str, records: str, entries: list[dict], rung: Rung, effects: Effects) -> Verdict | Failure | Limited:
|
||||
"""One judge run on the transcript's entries, as judge retries it; judge-replay replays it."""
|
||||
prompt, shown = judge_prompt(task, premises, records, render(entries, typed_answer))
|
||||
return run_judge(root, judge_args(rung), prompt, shown, f"{premises}\n\n{records}", effects)
|
||||
def judge_once(root: str, task: str, premises: str, records: str, entries: list[dict], escalated_at: int, rung: Rung, effects: Effects) -> Verdict | Failure | Limited:
|
||||
"""One judge run on the transcript's entries, as judge retries it; judge-replay replays it. escalated_at is the route's mark: an escalation quote counts only from the entries after it."""
|
||||
prompt, grounds = judge_prompt(task, premises, records, entries, escalated_at)
|
||||
verdict = run_judge(root, judge_args(rung), prompt, grounds, effects)
|
||||
return dataclasses.replace(verdict, entries_read=len(entries)) if isinstance(verdict, Interrupt | QuotedAnswer | Unfinished) else verdict
|
||||
|
||||
|
||||
def judge_args(rung: Rung) -> list[str]:
|
||||
@@ -175,13 +199,8 @@ def judge_args(rung: Rung) -> list[str]:
|
||||
]
|
||||
|
||||
|
||||
def run_judge(root: str, args: list[str], prompt: str, shown: str, quotable: str, effects: Effects) -> Verdict | Failure | Limited:
|
||||
"""One judge run. Each quote in its verdict is checked word for word, since stitched fragments can say what their source does not:
|
||||
- the evidence, against shown (the transcript as the judge saw it), its fragments in order: a misquote is Rejected, which judge retries;
|
||||
- a quoted answer, against quotable: a misquote puts the question to the human;
|
||||
- an escalation, against shown: a misquote drops it, so the worker's rung holds.
|
||||
quotes_checked checks the last two. Neither misquote is a Failure, which would otherwise exit an unattended loop.
|
||||
"""
|
||||
def run_judge(root: str, args: list[str], prompt: str, grounds: Grounds, effects: Effects) -> Verdict | Failure | Limited:
|
||||
"""One judge run, each quote in its verdict checked against grounds as Grounds says. quotes_checked checks the quoted answer and the escalation: neither misquote is a Failure, which would otherwise exit an unattended loop."""
|
||||
reply = claude_reply(effects.claude, args, root, prompt, "the judge")
|
||||
if isinstance(reply, Failure | Limited):
|
||||
return reply
|
||||
@@ -189,22 +208,24 @@ def run_judge(root: str, args: list[str], prompt: str, shown: str, quotable: str
|
||||
replied = json.dumps(reply, ensure_ascii=False)
|
||||
if isinstance(verdict, Failure):
|
||||
return Rejected(verdict.reason, replied)
|
||||
if not grounded(verdict.evidence, shown):
|
||||
if not grounded(verdict.evidence, grounds.shown):
|
||||
return Rejected(f"the judge quoted {verdict.evidence!r}, which is not in the transcript" + (f", for the question: {verdict.question}" if isinstance(verdict, Interrupt | QuotedAnswer) else ""), replied)
|
||||
return quotes_checked(verdict, shown, quotable)
|
||||
return quotes_checked(verdict, grounds)
|
||||
|
||||
|
||||
def quotes_checked(verdict: Verdict, shown: str, quotable: str) -> Verdict:
|
||||
"""The verdict with its quoted answer and its escalation checked, as run_judge says; a quoted answer put to the human keeps the escalation it carried."""
|
||||
if isinstance(verdict, QuotedAnswer) and not verbatim(verdict.answer, quotable):
|
||||
def quotes_checked(verdict: Verdict, grounds: Grounds) -> Verdict:
|
||||
"""The verdict with its quoted answer and its escalation checked, as Grounds says; a quoted answer put to the human keeps the escalation it carried."""
|
||||
if isinstance(verdict, QuotedAnswer) and not verbatim(verdict.answer, grounds.quotable):
|
||||
verdict = Interrupt(verdict.evidence, verdict.question, verdict.escalation, verdict.bundled)
|
||||
if isinstance(verdict, Interrupt | QuotedAnswer | Unfinished) and verdict.escalation and not verbatim(verdict.escalation, shown):
|
||||
if isinstance(verdict, Interrupt | QuotedAnswer | Unfinished) and verdict.escalation and not (verbatim(verdict.escalation, grounds.since_step) and verbatim(verdict.escalation, grounds.shown)):
|
||||
return dataclasses.replace(verdict, escalation="")
|
||||
return verdict
|
||||
|
||||
|
||||
def judge_prompt(task: str, premises: str, records: str, rendered_transcript: str) -> tuple[str, str]:
|
||||
"""The prompt, and the part of the transcript it shows, which the judge's evidence must quote."""
|
||||
def judge_prompt(task: str, premises: str, records: str, entries: list[dict], escalated_at: int) -> tuple[str, Grounds]:
|
||||
"""The prompt, and what the judge's quotes are checked against. Where escalated_at, the route's mark, is above 0, the prompt shows STEP_LINE after that many entries."""
|
||||
transcript = "\n\n".join(part for part in (render(entries[:escalated_at], typed_answer), STEP_LINE if escalated_at else "", render(entries[escalated_at:], typed_answer)) if part)
|
||||
agents_own = [entry for entry in entries[escalated_at:] if human_text(entry) is None and loop_text(entry) is None]
|
||||
head = (
|
||||
f"From README.md:\n{premises}\n\n"
|
||||
f"The records: the project's working rules and decisions, then the human's rules for every project:\n{records or 'none'}\n\n"
|
||||
@@ -213,8 +234,8 @@ def judge_prompt(task: str, premises: str, records: str, rendered_transcript: st
|
||||
)
|
||||
room = max(0, JUDGE_PROMPT_CHARS - len(head) - len(JUDGE_RULES) - 2)
|
||||
# The tail, where an interrupt or a finish lands; the session's transcript grows cycle over cycle.
|
||||
shown = rendered_transcript[max(0, len(rendered_transcript) - room):]
|
||||
return f"{head}{shown}\n\n{JUDGE_RULES}", shown
|
||||
shown = transcript[max(0, len(transcript) - room):]
|
||||
return f"{head}{shown}\n\n{JUDGE_RULES}", Grounds(shown, render(agents_own, typed_answer), f"{premises}\n\n{records}")
|
||||
|
||||
|
||||
def grounded(evidence: str, rendered_transcript: str) -> bool:
|
||||
|
||||
@@ -85,17 +85,26 @@ def rung_named(rung: Rung, route: Route, anchor: Rung) -> str:
|
||||
|
||||
|
||||
def escalation(route: Route, anchor: Rung | None) -> str:
|
||||
"""The prompt's escalation sentences, the marker's among them. Empty off the ladder or on its top rung. Where escalation_rungs opens none, the worker escalates only with the human's yes.
|
||||
Else they name the rungs escalation_rungs opens to a subagent, and offer to resume the worker one rung up for the rest of the chunk, up to the highest of those rungs."""
|
||||
"""The prompt's escalation sentences, the marker's among them, each led by a space. Empty off the ladder or on its top rung. Where escalation_rungs opens none, the worker escalates only with the human's yes.
|
||||
Else they name the rungs escalation_rungs opens to a subagent, and offer to resume the worker one rung up for the rest of the chunk, up to the highest of those rungs.
|
||||
|
||||
For a worker on Opus 5.5 at high, on an Opus 5.5 project, routed by item 3 of L 4, they read: ` When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort xhigh`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort max`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number. Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort max`.`
|
||||
"""
|
||||
if anchor is None or route.rung not in LADDER[:-1]:
|
||||
return ""
|
||||
marked = f" Open the subagent's prompt with `{ESCALATION_MARKER} <n>:`, n the trigger's number."
|
||||
marked = f"Open the subagent's prompt with `{ESCALATION_MARKER} <n>:`, n the trigger's number."
|
||||
rungs = [f"`{rung.agent_args()}`" for rung in escalation_rungs(anchor, route)]
|
||||
if not rungs:
|
||||
return f" An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `{climbed(route.rung, 1).agent_args()}`, one rung above yours, only once the human agrees; loop-claude will not resume you higher for a trigger either, so ask the human for that yes.{marked}"
|
||||
return " " + " ".join((
|
||||
f"An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `{climbed(route.rung, 1).agent_args()}`, one rung above yours, only once the human agrees; loop-claude will not resume you higher for a trigger either, so ask the human for that yes.",
|
||||
marked,
|
||||
))
|
||||
further = f", and on each further trigger one rung higher, up to {rungs[-1]}" if len(rungs) > 1 else ""
|
||||
resumed_higher = f" Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last such request: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to {rungs[-1]}."
|
||||
return f" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to {rungs[0]}, one rung above yours{further}; above that, ask the human first.{marked}{resumed_higher}"
|
||||
return " " + " ".join((
|
||||
f"When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to {rungs[0]}, one rung above yours{further}; above that, ask the human first.",
|
||||
marked,
|
||||
f"Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to {rungs[-1]}.",
|
||||
))
|
||||
|
||||
|
||||
def reruns(route: Route, anchor: Rung | None) -> str:
|
||||
@@ -149,7 +158,7 @@ def typed_answer(prompt: str) -> Typed | None:
|
||||
|
||||
def stepped_to(route: Route, anchor: Rung | None, stepped: bool) -> str:
|
||||
"""The sentences naming the worker's new rung, as resumed has them, where the ruling that brought the answer stepped the worker up; empty otherwise."""
|
||||
return f" Your request for a higher rung went through: loop-claude runs you on {route.rung}.{rungs_held(route, anchor)} A further request for a higher rung steps you up no more in this cycle." if stepped else ""
|
||||
return f" Your request for a higher rung went through: loop-claude runs you on {route.rung}. The escalation triggers that fired before this step are spent.{rungs_held(route, anchor)}" if stepped else ""
|
||||
|
||||
|
||||
def past_the_cap(question: str, answer: str, cap: int) -> str:
|
||||
|
||||
@@ -6,8 +6,8 @@ from loop_claude.cycle.states import Active, Archived, Closing, Ended, Event, Hu
|
||||
from transcript_archive.write import archive
|
||||
from claude_code.session import delete_session, Session, session_open, transcript_file
|
||||
from loop_claude.effects import Effects
|
||||
from loop_claude.cycle.judge import escalation_mark, Finished, Interrupt, judge, QuotedAnswer, Rejected, Unfinished
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Interrupt, judge, QuotedAnswer, Rejected, Unfinished, Verdict
|
||||
from loop_claude.routing import judge_rung, stepped_by_ruling
|
||||
from transcript_archive import outcomes
|
||||
from loop_claude.paths import Paths
|
||||
@@ -43,8 +43,6 @@ class Cycle:
|
||||
# Only an interrupt's or a quoted answer's escalation changes the route within a cycle. No transition reads the route or the fields below, so they stay out of the states.
|
||||
# The route before that ruling stepped it up, for ask and keep_answer to archive and for worker_start to tell the worker; None unless the latest event is that ruling, or the answer the human typed to it.
|
||||
stepped_from: Route | None = None
|
||||
# Whether an interrupt or a quoted answer of this cycle took an escalation, after which no ruling of the cycle carries one.
|
||||
escalation_spent: bool = False
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -73,10 +71,10 @@ def escalated(cycle: Cycle, event: Event) -> Cycle:
|
||||
if isinstance(event, HumanTyped):
|
||||
return cycle
|
||||
cycle = dataclasses.replace(cycle, stepped_from=None)
|
||||
if not (isinstance(event, Ruled) and isinstance(event.verdict, Interrupt | QuotedAnswer) and event.verdict.escalation):
|
||||
if not (isinstance(event, Ruled) and isinstance(event.verdict, Interrupt | QuotedAnswer) and cycle.anchor):
|
||||
return cycle
|
||||
route = stepped_by_ruling(cycle.anchor, cycle.route, event.verdict.bundled) if cycle.anchor else cycle.route
|
||||
return dataclasses.replace(cycle, route=route, stepped_from=cycle.route if route != cycle.route else None, escalation_spent=True)
|
||||
route = stepped_by_ruling(cycle.anchor, cycle.route, event.verdict.bundled, escalation_mark(event.verdict))
|
||||
return dataclasses.replace(cycle, route=route, stepped_from=cycle.route if route != cycle.route else None)
|
||||
|
||||
|
||||
def stopped(cycle: Cycle, stop: Stop, why: str) -> Stop:
|
||||
@@ -149,15 +147,8 @@ def rule(cycle: Cycle, transcript_path: str) -> Ruled | Stop:
|
||||
entries = read_transcript(transcript_path)
|
||||
if isinstance(entries, Failure):
|
||||
return Stop(1, entries.reason)
|
||||
verdict = judge(cycle.root, cycle.task, cycle.premises, cycle.records, entries, judge_of(cycle), cycle.effects, lambda rejected: keep_rejection(cycle, transcript_path, rejected))
|
||||
return verdict if isinstance(verdict, Stop) else Ruled(unspent(verdict, cycle.escalation_spent))
|
||||
|
||||
|
||||
def unspent(verdict: Verdict, spent: bool) -> Verdict:
|
||||
"""The verdict without its escalation once an interrupt or a quoted answer of the cycle took one. The judge may quote a different passage for the trigger that already stepped the route, and no comparison of quotes tells the two apart, so one trigger buys one step. It lives here, not in judge.quotes_checked, since it reads the cycle, and judge-replay replays the judge alone."""
|
||||
if spent and isinstance(verdict, Interrupt | QuotedAnswer | Unfinished) and verdict.escalation:
|
||||
return dataclasses.replace(verdict, escalation="")
|
||||
return verdict
|
||||
verdict = judge(cycle.root, cycle.task, cycle.premises, cycle.records, entries, cycle.route.escalated_at, judge_of(cycle), cycle.effects, lambda rejected: keep_rejection(cycle, transcript_path, rejected))
|
||||
return verdict if isinstance(verdict, Stop) else Ruled(verdict)
|
||||
|
||||
|
||||
def keep_rejection(cycle: Cycle, transcript_path: str, rejected: Rejected) -> Stop | None:
|
||||
|
||||
@@ -89,12 +89,13 @@ scale counts as none. Every second unfinished verdict on a chunk steps it up one
|
||||
higher rung, once the judge quotes the transcript showing that the escalation trigger it named
|
||||
fired (~/.claude/CLAUDE.md → "Working with the codebase" lists them). An interrupt or a quoted answer
|
||||
with such a quote steps the rung as it comes in, before the human is asked or the worker resumes
|
||||
(routing.stepped_by_ruling), and no later ruling of that cycle carries a second escalation. Each step
|
||||
on an escalation stops two rungs above the rung the chunk's item picks, and the worker's prompts
|
||||
(routing.stepped_by_ruling). Each such step marks how many transcript entries the session held,
|
||||
and from then on the judge must quote the worker's own lines after that mark, where it reads a
|
||||
[step] line. Each step on an escalation stops two rungs above the rung the chunk's item picks, and the worker's prompts
|
||||
offer it only while the worker is below that limit. Each finished or unfinished verdict, interrupt,
|
||||
quoted answer, stop and step-up in the archive names the worker's rung, with the item, L and E + I
|
||||
that routed the chunk, and any taken-in E + I that picks a higher review rung, which is archived as
|
||||
a step-up too. A rerun resumes an open chunk on the latest
|
||||
that routed the chunk, any taken-in E + I that picks a higher review rung (archived as a step-up
|
||||
too), and the mark. A rerun resumes an open chunk on the latest
|
||||
route named there, and counts its unfinished verdicts afresh. A rung no longer on the ladder, or
|
||||
below the one its item's L picks, is dropped, and the chunk is routed afresh from todo.md. So is a
|
||||
chunk whose archived line names an R + C, as loop-claude wrote before it routed on L. A todo.md or
|
||||
@@ -172,8 +173,8 @@ from loop_claude.loop.reload import decode_handed, exec_if_edited, fingerprint,
|
||||
from chunk_pick import default_plan, DefaultPlan
|
||||
from git_repo import default_ref
|
||||
from loop_claude.effects import Effects
|
||||
from loop_claude.cycle.judge import escalation_mark, Finished, Unfinished
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Unfinished
|
||||
from loop_claude.routing import first_route, kept, routed_item, stepped
|
||||
from loop_claude.project.model_choice import get_or_ask_models
|
||||
from todo_table import Item
|
||||
@@ -255,7 +256,7 @@ def routed(loop_run: LoopRun, anchor: Rung | None, archived: typing.Callable[[],
|
||||
def after_verdict(loop_run: LoopRun, anchor: Rung | None, verdict: Finished | Unfinished) -> LoopRun:
|
||||
if isinstance(verdict, Finished):
|
||||
return dataclasses.replace(loop_run, route=None, unfinished_evidence="")
|
||||
route = stepped(anchor, loop_run.route, verdict.bundled, bool(verdict.escalation)).taking_in(verdict.bundled_escape_and_impact) if anchor and loop_run.route else loop_run.route
|
||||
route = stepped(anchor, loop_run.route, verdict.bundled, escalation_mark(verdict)).taking_in(verdict.bundled_escape_and_impact) if anchor and loop_run.route else loop_run.route
|
||||
return dataclasses.replace(loop_run, route=route, unfinished_evidence=verdict.evidence)
|
||||
|
||||
|
||||
|
||||
+12
-10
@@ -61,27 +61,29 @@ def escalation_cap(anchor: Rung, route: Route) -> Rung:
|
||||
return climbed(item_rung(anchor, route), ESCALATION_CAP)
|
||||
|
||||
|
||||
def stepped(anchor: Rung, route: Route, bundled: int, escalated: bool) -> Route:
|
||||
def stepped(anchor: Rung, route: Route, bundled: int, escalated_at: int | None) -> Route:
|
||||
"""The route after one more unfinished verdict. Its rung never falls below taken_in's. It steps up one rung on every second unfinished verdict, or on an escalation the judge found while the rung is below escalation_cap; where both fall on one verdict, it steps once.
|
||||
|
||||
bundled and escalated are the verdict's fields, not the verdict, since loop_claude.cycle.judge imports this module through prompts.
|
||||
bundled is the highest L the judge saw the worker take in, 0 for none. escalated_at is how many transcript entries the judge read where the verdict carries an escalation, None where it carries none. A step moves the route's mark, Route.escalated_at, to escalated_at; a step on the count of unfinished verdicts alone keeps the mark. They are passed in place of the verdict, since loop_claude.cycle.judge imports this module through prompts.
|
||||
"""
|
||||
rung = taken_in(anchor, route, bundled)
|
||||
unfinished = route.unfinished + 1
|
||||
if unfinished >= UNFINISHED_PER_STEP or (escalated and below_cap(anchor, route, rung)):
|
||||
return stepped_from(route, rung)
|
||||
if unfinished >= UNFINISHED_PER_STEP or (escalated_at is not None and below_cap(anchor, route, rung)):
|
||||
return one_rung_up(route, rung, route.escalated_at if escalated_at is None else escalated_at)
|
||||
return dataclasses.replace(route, rung=rung, unfinished=unfinished)
|
||||
|
||||
|
||||
def stepped_by_ruling(anchor: Rung, route: Route, bundled: int) -> Route:
|
||||
"""The route after the judge's interrupt or quoted answer carried an escalation: taken_in's rung, one rung up while that is below escalation_cap. Neither is an unfinished verdict, so the count of those restarts only with a step."""
|
||||
def stepped_by_ruling(anchor: Rung, route: Route, bundled: int, escalated_at: int | None) -> Route:
|
||||
"""The route after the judge's interrupt or quoted answer, its arguments as stepped's: unchanged without an escalation; with one, taken_in's rung, one rung up while that is below escalation_cap. Neither is an unfinished verdict, so the count of those restarts only with a step."""
|
||||
if escalated_at is None:
|
||||
return route
|
||||
rung = taken_in(anchor, route, bundled)
|
||||
return stepped_from(route, rung) if below_cap(anchor, route, rung) else dataclasses.replace(route, rung=rung)
|
||||
return one_rung_up(route, rung, escalated_at) if below_cap(anchor, route, rung) else dataclasses.replace(route, rung=rung)
|
||||
|
||||
|
||||
def stepped_from(route: Route, rung: Rung) -> Route:
|
||||
"""The route one rung above rung, with the count of unfinished verdicts restarted."""
|
||||
return dataclasses.replace(route, rung=stepped_up(rung), unfinished=0)
|
||||
def one_rung_up(route: Route, rung: Rung, escalated_at: int) -> Route:
|
||||
"""The route one rung above rung, with the count of unfinished verdicts restarted and the mark at escalated_at."""
|
||||
return dataclasses.replace(route, rung=stepped_up(rung), unfinished=0, escalated_at=escalated_at)
|
||||
|
||||
|
||||
def below_cap(anchor: Rung, route: Route, rung: Rung) -> bool:
|
||||
|
||||
@@ -460,8 +460,8 @@ class Cycles(unittest.TestCase):
|
||||
("claude-opus-5-5", "high"),
|
||||
])
|
||||
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at high effort, routed by item 4's L of 4 and E + I of 5\n\n<UTC> Stepped up\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at high effort, routed by item 4's L of 4 and E + I of 5\n\n<UTC> Stepped up\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5, escalated at transcript entry 4\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5, escalated at transcript entry 4\n", outcomes)
|
||||
|
||||
def test_steps_the_worker_up_once_as_the_judge_interrupts_when_the_run_also_cites_an_escalation_trigger(self):
|
||||
checkout = Checkout(self)
|
||||
@@ -477,9 +477,9 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual(run.roles(), ["worker", "judge", "interrupt", "judge", "worker", "judge"])
|
||||
self.assertEqual([(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] != "judge"], [("claude-opus-5-5", "high"), ("claude-opus-5-5", "xhigh"), ("claude-opus-5-5", "xhigh")])
|
||||
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
|
||||
self.assertIn("<UTC> Stepped up\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5\n", "".join(outcomes))
|
||||
self.assertIn("<UTC> Stepped up\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5, escalated at transcript entry 4\n", "".join(outcomes))
|
||||
self.assertNotIn("at max effort", "".join(outcomes))
|
||||
self.assertIn("Verdict: interrupt\n\nA or B?\n\nEvidence:\n> Shall I use A or B?\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at high effort, routed by item 4's L of 4 and E + I of 5\n\n<UTC> Stepped up\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5\n", "".join(outcomes))
|
||||
self.assertIn("Verdict: interrupt\n\nA or B?\n\nEvidence:\n> Shall I use A or B?\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at high effort, routed by item 4's L of 4 and E + I of 5\n\n<UTC> Stepped up\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5, escalated at transcript entry 4\n", "".join(outcomes))
|
||||
|
||||
def test_runs_a_cheap_items_worker_below_the_projects_rung_with_its_reviewers_and_judge_on_it(self):
|
||||
checkout = Checkout(self)
|
||||
|
||||
@@ -113,10 +113,11 @@ def judged(verdict, evidence="working", answer="", escalation="", release_ready=
|
||||
class Worker:
|
||||
"""A worker claude: writes a line to the session's transcript, then streams a result."""
|
||||
|
||||
def __init__(self, home, prompted=False):
|
||||
def __init__(self, home, prompted=False, text="working"):
|
||||
self.home = home
|
||||
# Whether it records its -p prompt in the transcript first, as claude does.
|
||||
self.prompted = prompted
|
||||
self.text = text
|
||||
|
||||
def __call__(self, args):
|
||||
flag = "--session-id" if "--session-id" in args else "--resume"
|
||||
@@ -127,7 +128,7 @@ class Worker:
|
||||
handle.write(json.dumps({"type": "custom-title"}) + "\n" + json.dumps({"type": "queue-operation", "timestamp": str(uuid7())}) + "\n")
|
||||
if self.prompted:
|
||||
handle.write(json.dumps({**human(args[args.index("-p") + 1]), "promptSource": "sdk"}) + "\n")
|
||||
handle.write(json.dumps(assistant({"type": "text", "text": "working"})) + "\n")
|
||||
handle.write(json.dumps(assistant({"type": "text", "text": self.text})) + "\n")
|
||||
return Streamed(0)
|
||||
|
||||
|
||||
@@ -213,7 +214,7 @@ class Cycles(unittest.TestCase):
|
||||
def test_a_fresh_worker_reads_the_preflights_facts_and_a_resumed_one_does_not(self):
|
||||
worker = Worker(self.paths.claude_code_transcripts.parent)
|
||||
fake = FakeEffects(streams=[worker, worker], claude=[judged("unfinished"), judged("finished")])
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working"))
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working", entries_read=3))
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake, unfinished_evidence="working")).verdict, Finished("working"))
|
||||
fresh, resumed = (args[args.index("-p") + 1] for args in fake.made("stream_claude"))
|
||||
self.assertIn("- A preflight fact.", fresh)
|
||||
@@ -239,9 +240,9 @@ class Cycles(unittest.TestCase):
|
||||
|
||||
def test_an_unfinished_chunks_transcript_is_archived_and_the_next_cycle_resumes_its_session_with_the_judges_evidence(self):
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)] * 2, claude=[judged("unfinished")] * 2)
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working"))
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working", entries_read=3))
|
||||
self.assertTrue(session.transcript_file(self.session))
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake, unfinished_evidence="working")).verdict, Unfinished("working"))
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake, unfinished_evidence="working")).verdict, Unfinished("working", entries_read=4))
|
||||
workers = fake.made("stream_claude")
|
||||
self.assertIn("--session-id", workers[0])
|
||||
self.assertIn("--resume", workers[1])
|
||||
@@ -292,7 +293,7 @@ class Cycles(unittest.TestCase):
|
||||
(folder / "custom-title.json").write_text("first")
|
||||
(folder.parent / f"{self.session.id}.orphaned-1.jsonl").write_text("o\n")
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)] * 3, claude=[judged("unfinished"), judged("finished"), judged("unfinished")])
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working"))
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working", entries_read=3))
|
||||
[chunk] = self.archive.iterdir()
|
||||
kept = chunk / "session" / str(self.session.id)
|
||||
self.assertEqual((chunk / "session" / f"{self.session.id}.orphaned-1.jsonl").read_text(), "o\n")
|
||||
@@ -308,7 +309,7 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual(stat.S_IMODE((kept / "subagents" / "agent-a.jsonl").stat().st_mode), 0o600)
|
||||
folder.mkdir()
|
||||
(folder / "custom-title.json").write_text("third")
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working"))
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working", entries_read=3))
|
||||
[next_chunk] = set(self.archive.iterdir()) - {chunk}
|
||||
self.assertEqual(sorted(path.read_text() for path in (next_chunk / "session" / str(self.session.id)).glob("custom-title.json*")), ["third"])
|
||||
|
||||
@@ -328,7 +329,7 @@ class Cycles(unittest.TestCase):
|
||||
return 0
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("interrupt"), judged("unfinished")], inputs=["/talk"], on_terminal_claude=[answer], git=[ran()])
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working"))
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working", entries_read=4))
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: interrupt\n\nWhich one?\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n", "Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: unfinished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n"])
|
||||
self.assertIn(str(self.session.id), fake.made("claude_on_terminal")[0])
|
||||
judge_prompts = [stdin for args, stdin in fake.made("claude")]
|
||||
@@ -356,13 +357,13 @@ class Cycles(unittest.TestCase):
|
||||
cycle = dataclasses.replace(self.cycle(fake, records="From AGENTS.md:\nUse A."), anchor=Routing.OPUS, route=Route(Routing.SONNET, 179, 2, 5))
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
concluded = run.run_cycle(cycle)
|
||||
self.assertEqual(concluded.route, Route(Routing.OPUS, 179, 2, 5))
|
||||
self.assertEqual(concluded.route, Route(Routing.OPUS, 179, 2, 5, escalated_at=3))
|
||||
first, resumed = fake.made("stream_claude")
|
||||
self.assertEqual([args[args.index("--effort") + 1] for args in (first, resumed)], ["high", "medium"])
|
||||
self.assertTrue(resumed[resumed.index("-p") + 1].endswith(f"Continue the task. Your request for a higher rung went through: loop-claude runs you on {Routing.OPUS}.{prompts.rungs_held(concluded.route, Routing.OPUS)} A further request for a higher rung steps you up no more in this cycle."))
|
||||
self.assertTrue(resumed[resumed.index("-p") + 1].endswith(f"Continue the task. Your request for a higher rung went through: loop-claude runs you on {Routing.OPUS}. The escalation triggers that fired before this step are spent.{prompts.rungs_held(concluded.route, Routing.OPUS)}"))
|
||||
self.assertNotIn("went through", first[first.index("-p") + 1])
|
||||
answered = f"Verdict: quoted answer\n\nWhich one?\n\n{prompts.QUOTE_SOURCES} says:\n> Use A.\n\nEvidence:\n> working\n\nEscalation:\n> working\n\nJudge: judge-model\nWorker: {Route(Routing.SONNET, 179, 2, 5)}\n"
|
||||
stepped = f"Stepped up\n\nWorker: {Route(Routing.OPUS, 179, 2, 5)}\n"
|
||||
stepped = f"Stepped up\n\nWorker: {Route(Routing.OPUS, 179, 2, 5, escalated_at=3)}\n"
|
||||
kept = "".join(self.archived_outcomes())
|
||||
self.assertIn(f"Z {answered}\n20", kept)
|
||||
self.assertIn(f"Z {stepped}", kept)
|
||||
@@ -375,13 +376,43 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual(run.run_cycle(cycle), Stop(2, "the human is needed to answer the question above, but nothing was typed"))
|
||||
kept = "".join(self.archived_outcomes())
|
||||
self.assertIn(f"Evidence:\n> working\n\nEscalation:\n> working\n\nJudge: judge-model\nWorker: {Route(Routing.SONNET, 179, 2, 5)}\n", kept)
|
||||
self.assertIn(f"Stepped up\n\nWorker: {Route(Routing.OPUS, 179, 2, 5)}\n", kept)
|
||||
self.assertIn(f"Stepped up\n\nWorker: {Route(Routing.OPUS, 179, 2, 5, escalated_at=6)}\n", kept)
|
||||
|
||||
def test_a_ruling_steps_the_route_only_on_an_escalation_from_the_rung_its_bundled_l_picks(self):
|
||||
cycle = dataclasses.replace(self.cycle(FakeEffects()), anchor=Routing.OPUS, route=Route(Routing.OPUS, 3, 3, 4))
|
||||
for ruling in (Interrupt("e", "q", entries_read=3), QuotedAnswer("e", "q", "A.", entries_read=3)):
|
||||
with self.subTest(ruling=ruling):
|
||||
self.assertEqual(run.escalated(cycle, states.Ruled(ruling)), cycle)
|
||||
stepped = run.escalated(cycle, states.Ruled(Interrupt("e", "q", "round 2: 🔴", 4, entries_read=3)))
|
||||
self.assertEqual((stepped.route, stepped.stepped_from), (Route(Routing.XHIGH, 3, 3, 4, escalated_at=3), cycle.route))
|
||||
|
||||
def test_an_escalation_after_an_interrupt_stepped_the_worker_steps_it_again_in_the_same_cycle(self):
|
||||
streams = [Worker(self.paths.claude_code_transcripts.parent, text="round 2: 🔴"), Worker(self.paths.claude_code_transcripts.parent), Worker(self.paths.claude_code_transcripts.parent)]
|
||||
rulings = [judged("interrupt", evidence="round 2: 🔴", escalation="round 2: 🔴"), judged("quoted", answer="Use A.", escalation="working"), judged("finished")]
|
||||
fake = FakeEffects(streams=streams, claude=rulings, inputs=["Use B."])
|
||||
cycle = dataclasses.replace(self.cycle(fake, records="From AGENTS.md:\nUse A."), anchor=Routing.OPUS, route=Route(Routing.SONNET, 179, 2, 5))
|
||||
with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
|
||||
concluded = run.run_cycle(cycle)
|
||||
self.assertEqual(concluded.route, Route(Routing.HIGH, 179, 2, 5, escalated_at=4))
|
||||
self.assertEqual([args[args.index("--effort") + 1] for args in fake.made("stream_claude")], ["high", "medium", "high"])
|
||||
|
||||
def test_an_escalation_quoted_from_before_the_workers_last_step_up_steps_nothing_and_a_later_one_reaches_the_verdict(self):
|
||||
for escalation, verdict in (("round 2: 🔴", Unfinished("working", entries_read=4)), ("working", Unfinished("working", 0, "working", entries_read=4))):
|
||||
with self.subTest(escalation=escalation):
|
||||
self.session = session.session_named(f"A proj {escalation} loop-claude", self.paths.claude_code_transcripts)
|
||||
streams = [Worker(self.paths.claude_code_transcripts.parent, text="round 2: 🔴"), Worker(self.paths.claude_code_transcripts.parent)]
|
||||
rulings = [judged("interrupt", evidence="round 2: 🔴", escalation="round 2: 🔴"), judged("unfinished", escalation=escalation)]
|
||||
fake = FakeEffects(streams=streams, claude=rulings, inputs=["Use B."])
|
||||
cycle = dataclasses.replace(self.cycle(fake), anchor=Routing.OPUS, route=Route(Routing.SONNET, 179, 2, 5))
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
concluded = run.run_cycle(cycle)
|
||||
self.assertEqual(concluded, run.Concluded(verdict, Route(Routing.OPUS, 179, 2, 5, escalated_at=3)))
|
||||
|
||||
def test_an_answer_typed_at_the_terminal_resumes_the_worker_headless_and_the_judge_reads_it_as_the_humans(self):
|
||||
worker = Worker(self.paths.claude_code_transcripts.parent, prompted=True)
|
||||
fake = FakeEffects(streams=[worker, worker], claude=[judged("interrupt"), judged("unfinished")], inputs=["Use B."])
|
||||
with contextlib.redirect_stderr(io.StringIO()) as said:
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working"))
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working", entries_read=6))
|
||||
self.assertIn("\nWhich one?\n", said.getvalue())
|
||||
self.assertEqual(fake.made("claude_on_terminal"), [])
|
||||
resumed = fake.made("stream_claude")[1]
|
||||
@@ -399,7 +430,7 @@ class Cycles(unittest.TestCase):
|
||||
cycle = dataclasses.replace(self.cycle(fake), anchor=Routing.OPUS, route=Route(Routing.SONNET, 179, 2, 5))
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
concluded = run.run_cycle(cycle)
|
||||
self.assertEqual(concluded.route, Route(Routing.OPUS, 179, 2, 5))
|
||||
self.assertEqual(concluded.route, Route(Routing.OPUS, 179, 2, 5, escalated_at=3))
|
||||
first, resumed = fake.made("stream_claude")
|
||||
self.assertEqual([args[args.index("--effort") + 1] for args in (first, resumed)], ["high", "medium"])
|
||||
typed = resumed[resumed.index("-p") + 1]
|
||||
@@ -522,7 +553,7 @@ class Limits(unittest.TestCase):
|
||||
def test_the_judge_reports_a_limit_whatever_its_exit_code(self):
|
||||
reply = json.dumps({"type": "result", "api_error_status": 429, "is_error": True, "result": "You've hit your session limit · resets 7:40pm (Europe/Stockholm)"})
|
||||
fake = FakeEffects(claude=[ran(reply, code=1, stderr="limited")])
|
||||
self.assertEqual(judge.run_judge("/r", [], "", "", "", fake.effects()), Limited(None, "You've hit your session limit · resets 7:40pm (Europe/Stockholm)"))
|
||||
self.assertEqual(judge.run_judge("/r", [], "", judge.Grounds("", "", ""), fake.effects()), Limited(None, "You've hit your session limit · resets 7:40pm (Europe/Stockholm)"))
|
||||
|
||||
def test_reads_only_a_rejecting_events_whole_seconds(self):
|
||||
for info, want in [({"status": "rejected", "resetsAt": NOW}, NOW), ({"status": "allowed", "resetsAt": NOW}, None), ({"status": "rejected", "resetsAt": True}, None), ({"status": "rejected"}, None), ("rejected", None)]:
|
||||
@@ -741,40 +772,47 @@ class Routing(unittest.TestCase):
|
||||
self.assertEqual(routing.first_route(self.OPUS, None), Route(self.OPUS))
|
||||
|
||||
def test_steps_up_a_rung_after_every_second_unfinished_verdict(self):
|
||||
once = routing.stepped(self.OPUS, Route(self.OPUS, 3, 3, 4), 0, False)
|
||||
once = routing.stepped(self.OPUS, Route(self.OPUS, 3, 3, 4), 0, None)
|
||||
self.assertEqual(once, Route(self.OPUS, 3, 3, 4, unfinished=1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, once, 0, False), Route(self.HIGH, 3, 3, 4))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(LADDER[-1], unfinished=1), 0, False).rung, LADDER[-1])
|
||||
self.assertEqual(routing.stepped(self.OPUS, once, 0, None), Route(self.HIGH, 3, 3, 4))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(LADDER[-1], unfinished=1), 0, None).rung, LADDER[-1])
|
||||
|
||||
def test_steps_up_to_the_rung_a_bundled_items_l_picks_and_never_down(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS), 4, False).rung, self.HIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 4, False).rung, self.XHIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS, unfinished=1), 4, False).rung, self.XHIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS), 4, None).rung, self.HIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 4, None).rung, self.XHIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS, unfinished=1), 4, None).rung, self.XHIGH)
|
||||
|
||||
def test_steps_up_from_a_rung_below_the_projects_one_rung_at_a_time(self):
|
||||
low = Route(Rung("claude-haiku-5-5", "max"), 3, 1, 3, unfinished=1)
|
||||
self.assertEqual(routing.stepped(self.OPUS, low, 0, False).rung, self.SONNET)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 1, 3), 0, False).rung, low.rung)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 1, 3), 2, False).rung, self.SONNET)
|
||||
self.assertEqual(routing.stepped(self.OPUS, low, 0, None).rung, self.SONNET)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 1, 3), 0, None).rung, low.rung)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 1, 3), 2, None).rung, self.SONNET)
|
||||
|
||||
def test_steps_up_a_rung_on_an_escalation_the_judge_found_up_to_two_above_the_items_rung(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.HIGH, 3, 4, 5), 0, True), Route(self.XHIGH, 3, 4, 5))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH, 3, 4, 5), 0, True), Route(self.MAX, 3, 4, 5))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.MAX, 3, 4, 5), 0, True), Route(self.MAX, 3, 4, 5, unfinished=1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1), 0, True), Route(self.XHIGH, 3, 4, 5))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 0, True), Route(self.XHIGH, unfinished=1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.SONNET, 3, 2, 4), 4, True), Route(self.HIGH, 3, 2, 4, unfinished=1))
|
||||
def test_steps_up_a_rung_on_an_escalation_the_judge_found_up_to_two_above_the_items_rung_marking_the_transcript_entries_it_read(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.HIGH, 3, 4, 5), 0, 7), Route(self.XHIGH, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH, 3, 4, 5, escalated_at=7), 0, 9), Route(self.MAX, 3, 4, 5, escalated_at=9))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.MAX, 3, 4, 5, escalated_at=9), 0, 12), Route(self.MAX, 3, 4, 5, unfinished=1, escalated_at=9))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1), 0, 7), Route(self.XHIGH, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 0, 7), Route(self.XHIGH, unfinished=1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.SONNET, 3, 2, 4), 4, 7), Route(self.HIGH, 3, 2, 4, unfinished=1))
|
||||
|
||||
def test_keeps_the_mark_through_a_step_no_escalation_took(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS, 3, 3, 4, unfinished=1, escalated_at=5), 0, None), Route(self.HIGH, 3, 3, 4, escalated_at=5))
|
||||
|
||||
def test_steps_up_a_rung_as_the_judge_interrupts_up_to_two_above_the_items_rung_without_counting_an_unfinished_verdict(self):
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1), 0), Route(self.XHIGH, 3, 4, 5))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.XHIGH, 3, 4, 5), 0), Route(self.MAX, 3, 4, 5))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.MAX, 3, 4, 5, unfinished=1), 0), Route(self.MAX, 3, 4, 5, unfinished=1))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.SONNET, 3, 2, 4), 0), Route(self.OPUS, 3, 2, 4))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1), 0, 7), Route(self.XHIGH, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.XHIGH, 3, 4, 5, escalated_at=7), 0, 9), Route(self.MAX, 3, 4, 5, escalated_at=9))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.MAX, 3, 4, 5, unfinished=1, escalated_at=9), 0, 12), Route(self.MAX, 3, 4, 5, unfinished=1, escalated_at=9))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.SONNET, 3, 2, 4), 0, 7), Route(self.OPUS, 3, 2, 4, escalated_at=7))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1, escalated_at=2), 4, None), Route(self.HIGH, 3, 4, 5, unfinished=1, escalated_at=2))
|
||||
|
||||
def test_raises_the_worker_to_a_taken_in_items_rung_above_the_escalation_cap_without_a_step_as_the_judge_interrupts(self):
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.OPUS, 3, 1, 3, escalated_at=2), 4, 7), Route(self.HIGH, 3, 1, 3, escalated_at=2))
|
||||
|
||||
def test_steps_up_from_the_rung_a_taken_in_item_picks_as_the_judge_interrupts(self):
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.OPUS, 3, 3, 4), 4), Route(self.XHIGH, 3, 3, 4))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.XHIGH, 3, 4, 5), 4), Route(self.MAX, 3, 4, 5))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.MAX, 3, 4, 5, unfinished=1), 4), Route(self.MAX, 3, 4, 5, unfinished=1))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.OPUS, 3, 3, 4), 4, 7), Route(self.XHIGH, 3, 3, 4, escalated_at=7))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.XHIGH, 3, 4, 5), 4, 7), Route(self.MAX, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.MAX, 3, 4, 5, unfinished=1), 4, 7), Route(self.MAX, 3, 4, 5, unfinished=1))
|
||||
|
||||
def test_keeps_an_e_plus_i_taken_in_only_where_it_picks_a_higher_review_rung(self):
|
||||
taken = Route(self.OPUS, 3, 3, 4).taking_in(6)
|
||||
@@ -789,8 +827,8 @@ class Routing(unittest.TestCase):
|
||||
self.assertEqual(Route(self.OPUS).taking_in(0), Route(self.OPUS))
|
||||
|
||||
def test_a_rung_an_edit_took_off_the_ladder_counts_as_the_projects(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 0, False).rung, self.OPUS)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 2, False).rung, self.OPUS)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 0, None).rung, self.OPUS)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 2, None).rung, self.OPUS)
|
||||
|
||||
def test_runs_the_judge_on_the_rung_its_chunks_items_e_plus_i_picks_never_below_the_projects(self):
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.OPUS, 3, 3, 5)), self.HIGH)
|
||||
@@ -839,6 +877,10 @@ class Routing(unittest.TestCase):
|
||||
self.assertEqual(route_from(str(unrouted)), unrouted)
|
||||
self.assertIsNone(route_from("claude-opus-5-5 at high effort, taking in an E + I of 9"))
|
||||
self.assertIsNone(route_from("claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 5, taking in an E + I of 5"))
|
||||
marked = Route(self.HIGH, 3, 4, 5, escalated_at=12, escape_and_impact_taken_in=7)
|
||||
self.assertEqual(str(marked), "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 5, taking in an E + I of 7, escalated at transcript entry 12")
|
||||
self.assertEqual(route_from(str(marked)), marked)
|
||||
self.assertEqual(route_from(str(Route(self.HIGH, escalated_at=3))), Route(self.HIGH, escalated_at=3))
|
||||
|
||||
def test_reads_no_route_from_a_line_whose_levels_are_off_their_scales_or_an_r_plus_c_route_wrote(self):
|
||||
for line in ("claude-opus-5-5 at high effort, routed by item 3's L of 5 and E + I of 5", "claude-opus-5-5 at high effort, routed by item 3's L of 0 and E + I of 5", "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 9", "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 1", "claude-opus-5-5 at high effort, routed by item 3's R + C of 11"):
|
||||
@@ -905,6 +947,11 @@ class Steps(unittest.TestCase):
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Finished("x")), dataclasses.replace(run_with, route=None, unfinished_evidence=""))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Unfinished("new", 4)), dataclasses.replace(run_with, route=Route(Rung("claude-opus-5-5", "high"), unfinished=1), unfinished_evidence="new"))
|
||||
|
||||
def test_an_unfinished_verdicts_escalation_marks_the_transcript_entries_the_judge_read(self):
|
||||
run_with = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=Route(self.OPUS, 3, 3, 4, escalated_at=2))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Unfinished("new", 0, "round 2: 🔴", entries_read=9)).route, Route(Rung("claude-opus-5-5", "high"), 3, 3, 4, escalated_at=9))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Unfinished("new", entries_read=9)).route, Route(self.OPUS, 3, 3, 4, unfinished=1, escalated_at=2))
|
||||
|
||||
def test_archives_a_step_up_as_it_happens_so_a_rerun_resumes_on_it(self):
|
||||
with tempfile.TemporaryDirectory() as home:
|
||||
opened = session.session_named("A proj loop-claude", pathlib.Path(home, "projects"))
|
||||
@@ -1087,11 +1134,11 @@ class WorkerArgs(unittest.TestCase):
|
||||
self.assertIn("hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort max`, one rung above yours;", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 5), opus))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("m")), None))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("claude-opus-5-5", "low"), 3, 3, 4), opus))
|
||||
self.assertIn("before creating a new one.\n\nWhen an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number. Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last such request: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`. A reviewer", prompts.fresh("Do it.", "## Goals", Route(opus), opus, self.FACTS))
|
||||
self.assertIn("before creating a new one.\n\nWhen an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number. Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`. A reviewer", prompts.fresh("Do it.", "## Goals", Route(opus), opus, self.FACTS))
|
||||
|
||||
def test_offers_the_worker_a_higher_rung_for_the_rest_of_the_chunk_up_to_the_escalation_cap(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
offer = " Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last such request: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`."
|
||||
offer = " Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`."
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus, self.FACTS))
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus))
|
||||
self.assertNotIn("Where the hard part", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 3, 4), opus))
|
||||
@@ -1163,11 +1210,35 @@ class Records(unittest.TestCase):
|
||||
|
||||
class JudgePrompt(unittest.TestCase):
|
||||
def test_shows_the_transcripts_tail_and_returns_what_it_showed(self):
|
||||
prompt, shown = judge.judge_prompt("Do it.", "## Goals", "", "old " + "x" * judge.JUDGE_PROMPT_CHARS + " new")
|
||||
self.assertTrue(shown.endswith(" new"))
|
||||
self.assertNotIn("old", shown)
|
||||
self.assertIn(shown, prompt)
|
||||
entries = [assistant({"type": "text", "text": "old " + "x" * judge.JUDGE_PROMPT_CHARS}), assistant({"type": "text", "text": "new"})]
|
||||
prompt, grounds = judge.judge_prompt("Do it.", "## Goals", "", entries, 0)
|
||||
self.assertTrue(grounds.shown.endswith("\n\nnew"))
|
||||
self.assertNotIn("old", grounds.shown)
|
||||
self.assertIn(grounds.shown, prompt)
|
||||
self.assertLessEqual(len(prompt), judge.JUDGE_PROMPT_CHARS)
|
||||
self.assertEqual(grounds.since_step, "old " + "x" * judge.JUDGE_PROMPT_CHARS + "\n\nnew")
|
||||
self.assertEqual(grounds.quotable, "## Goals\n\n")
|
||||
|
||||
def test_shows_the_judge_where_the_worker_last_stepped_up_and_checks_an_escalation_only_after_it(self):
|
||||
entries = [assistant({"type": "text", "text": "round 2: 🔴"}), assistant({"type": "text", "text": "round 3: 🔴"})]
|
||||
prompt, grounds = judge.judge_prompt("Do it.", "## Goals", "", entries, 1)
|
||||
self.assertRegex(prompt, "round 2: 🔴\n\n\\[step\\] [^\n]+\n\nround 3: 🔴\n\n")
|
||||
self.assertEqual(grounds.since_step, "round 3: 🔴")
|
||||
prompt, grounds = judge.judge_prompt("Do it.", "## Goals", "", entries, 0)
|
||||
self.assertNotIn("stepped the agent up", prompt)
|
||||
self.assertEqual(grounds.since_step, "round 2: 🔴\n\nround 3: 🔴")
|
||||
|
||||
def test_drops_an_escalation_quote_from_after_the_step_up_that_the_tail_it_shows_cuts_off(self):
|
||||
entries = [assistant({"type": "text", "text": "round 2: 🔴"}), assistant({"type": "text", "text": "old " + "x" * judge.JUDGE_PROMPT_CHARS}), assistant({"type": "text", "text": "new"})]
|
||||
_, grounds = judge.judge_prompt("Do it.", "## Goals", "", entries, 1)
|
||||
self.assertEqual(judge.quotes_checked(Unfinished("e", 0, "old x", entries_read=3), grounds), Unfinished("e", entries_read=3))
|
||||
self.assertEqual(judge.quotes_checked(Unfinished("e", 0, "new", entries_read=3), grounds), Unfinished("e", 0, "new", entries_read=3))
|
||||
|
||||
def test_shows_a_typed_answer_before_the_step_up_as_the_humans(self):
|
||||
typed = prompts.typed("Which one?", "Use B.", Route(Rung("worker-model")), None, False)
|
||||
entries = [{**human(typed), "promptSource": "sdk"}, assistant({"type": "text", "text": "round 3: 🔴"})]
|
||||
_, grounds = judge.judge_prompt("Do it.", "## Goals", "", entries, 1)
|
||||
self.assertIn("\n\n[human] Use B.\n\n[step] ", grounds.shown)
|
||||
|
||||
|
||||
class Grounded(unittest.TestCase):
|
||||
@@ -1241,9 +1312,9 @@ class Verdict(unittest.TestCase):
|
||||
|
||||
def test_drops_an_escalation_quote_the_transcript_does_not_hold_word_for_word_from_an_interrupt_or_an_unfinished_verdict(self):
|
||||
shown = "The agent: round 2: 🔴. Which one?"
|
||||
self.assertEqual(judge.quotes_checked(Interrupt("e", "q", "round 2: 🔴"), shown, ""), Interrupt("e", "q", "round 2: 🔴"))
|
||||
self.assertEqual(judge.quotes_checked(Interrupt("e", "q", "round 2: … 🔴"), shown, ""), Interrupt("e", "q"))
|
||||
self.assertEqual(judge.quotes_checked(Unfinished("e", 0, "round 3: 🔴"), shown, ""), Unfinished("e"))
|
||||
self.assertEqual(judge.quotes_checked(Interrupt("e", "q", "round 2: 🔴"), judge.Grounds(shown, shown, "")), Interrupt("e", "q", "round 2: 🔴"))
|
||||
self.assertEqual(judge.quotes_checked(Interrupt("e", "q", "round 2: … 🔴"), judge.Grounds(shown, shown, "")), Interrupt("e", "q"))
|
||||
self.assertEqual(judge.quotes_checked(Unfinished("e", 0, "round 3: 🔴"), judge.Grounds(shown, shown, "")), Unfinished("e"))
|
||||
|
||||
def test_takes_an_escalation_quote_and_a_bundled_l_on_a_quoted_answer(self):
|
||||
reply = {"answer": "A.", "bundled": 4, "bundled_escape_and_impact": 0, "escalation": "round 2: 🔴", "interrupt": False, "evidence": "e", "finished": False, "question": "Which one?", "release_ready": False}
|
||||
@@ -1252,21 +1323,35 @@ class Verdict(unittest.TestCase):
|
||||
|
||||
def test_drops_an_escalation_quote_the_transcript_does_not_hold_word_for_word_from_a_quoted_answer(self):
|
||||
shown = "The agent: round 2: 🔴. Which one?"
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "A.", "round 2: 🔴"), shown, "A."), QuotedAnswer("e", "q", "A.", "round 2: 🔴"))
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "A.", "round 2: … 🔴"), shown, "A."), QuotedAnswer("e", "q", "A."))
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "A.", "round 2: 🔴"), judge.Grounds(shown, shown, "A.")), QuotedAnswer("e", "q", "A.", "round 2: 🔴"))
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "A.", "round 2: … 🔴"), judge.Grounds(shown, shown, "A.")), QuotedAnswer("e", "q", "A."))
|
||||
|
||||
def test_puts_a_quoted_answer_the_records_do_not_hold_to_the_human_with_the_escalation_it_carried(self):
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "not there", "round 2: 🔴", 4), "The agent: round 2: 🔴.", "A."), Interrupt("e", "q", "round 2: 🔴", 4))
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "not there", "round 2: 🔴", 4), judge.Grounds("The agent: round 2: 🔴.", "The agent: round 2: 🔴.", "A.")), Interrupt("e", "q", "round 2: 🔴", 4))
|
||||
|
||||
def test_puts_a_quoted_answer_the_records_do_not_hold_word_for_word_to_the_human(self):
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "not there"), "shown", "the records"), Interrupt("e", "q"))
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "the records"), "shown", "From the records: the records"), QuotedAnswer("e", "q", "the records"))
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "not there"), judge.Grounds("shown", "shown", "the records")), Interrupt("e", "q"))
|
||||
self.assertEqual(judge.quotes_checked(QuotedAnswer("e", "q", "the records"), judge.Grounds("shown", "shown", "From the records: the records")), QuotedAnswer("e", "q", "the records"))
|
||||
|
||||
def test_drops_an_escalation_from_every_ruling_of_a_cycle_once_an_interrupt_took_one(self):
|
||||
self.assertEqual(run.unspent(Unfinished("e", 0, "round 2: 🔴"), True), Unfinished("e"))
|
||||
self.assertEqual(run.unspent(Interrupt("e", "q", "another passage"), True), Interrupt("e", "q"))
|
||||
self.assertEqual(run.unspent(QuotedAnswer("e", "q", "A.", "another passage"), True), QuotedAnswer("e", "q", "A."))
|
||||
self.assertEqual(run.unspent(Unfinished("e", 0, "round 2: 🔴"), False), Unfinished("e", 0, "round 2: 🔴"))
|
||||
def test_drops_an_escalation_quote_found_only_in_the_transcript_entries_before_the_workers_last_step_up(self):
|
||||
entries = [assistant({"type": "text", "text": "round 2: 🔴"}), assistant({"type": "text", "text": "round 3: 🔴"}), assistant({"type": "text", "text": "working"})]
|
||||
def judged_on(mark, escalation):
|
||||
return judge.judge_once("/r", "Do it.", "## Goals", "", entries, mark, Rung("judge-model"), FakeEffects(claude=[judged("unfinished", escalation=escalation)]).effects())
|
||||
self.assertEqual(judged_on(0, "round 2: 🔴"), Unfinished("working", 0, "round 2: 🔴", entries_read=3))
|
||||
self.assertEqual(judged_on(1, "round 2: 🔴"), Unfinished("working", entries_read=3))
|
||||
self.assertEqual(judged_on(1, "round 3: 🔴"), Unfinished("working", 0, "round 3: 🔴", entries_read=3))
|
||||
self.assertEqual(judged_on(1, "round 2: 🔴\n\nround 3: 🔴"), Unfinished("working", entries_read=3))
|
||||
self.assertEqual(judged_on(3, "round 3: 🔴"), Unfinished("working", entries_read=3))
|
||||
|
||||
def test_drops_an_escalation_quote_that_only_loop_claudes_prompts_or_the_humans_lines_repeat_after_the_step_up(self):
|
||||
entries = [assistant({"type": "text", "text": "round 2: 🔴"}), {**human("as the work remaining: round 2: 🔴"), "promptSource": "sdk"}, human("round 2: 🔴 again"), assistant({"type": "text", "text": "working"})]
|
||||
verdict = judge.judge_once("/r", "Do it.", "## Goals", "", entries, 1, Rung("judge-model"), FakeEffects(claude=[judged("unfinished", escalation="round 2: 🔴")]).effects())
|
||||
self.assertEqual(verdict, Unfinished("working", entries_read=4))
|
||||
|
||||
def test_takes_evidence_from_before_the_workers_last_step_up(self):
|
||||
entries = [assistant({"type": "text", "text": "round 2: 🔴"}), assistant({"type": "text", "text": "working"})]
|
||||
verdict = judge.judge_once("/r", "Do it.", "## Goals", "", entries, 1, Rung("judge-model"), FakeEffects(claude=[judged("unfinished", evidence="round 2: 🔴")]).effects())
|
||||
self.assertEqual(verdict, Unfinished("round 2: 🔴", entries_read=2))
|
||||
|
||||
|
||||
class LoopRuns(unittest.TestCase):
|
||||
|
||||
@@ -8,17 +8,24 @@ from model_routing.ladder import REVIEW_STEPS, Rung
|
||||
|
||||
# How a worker opens the prompt of a subagent it escalates a task to, `Escalation trigger 2: …`: docs/decisions.md → `routing: a worker hands a task one rung up`.
|
||||
ESCALATION_MARKER = "Escalation trigger"
|
||||
# An archived worker line: the rung, then the item and the levels that routed it, if any, then the E + I taken in, if any. A line an older loop-claude wrote names an R + C in place of L and E + I: archived_worker reads only its rung, for the routing figures; with no levels, route_from returns None and the chunk routes afresh.
|
||||
ROUTED = re.compile(r"(?P<rung>.+?)(?:, routed by item (?P<item>[0-9]+)'s (?:L of (?P<likelihood>[0-9]+) and E \+ I of (?P<escape_and_impact>[0-9]+)|.+?))?(?:, taking in an E \+ I of (?P<escape_and_impact_taken_in>[0-9]+))?")
|
||||
# An archived worker line. A line an older loop-claude wrote names an R + C in place of L and E + I: archived_worker reads only its rung, for the routing figures; with no levels, route_from returns None and the chunk routes afresh.
|
||||
# Each group reads one fragment of Route.__str__. A reworded fragment leaves the line unreadable, so the chunk routes afresh, its mark back at 0.
|
||||
ROUTED = re.compile(r"""
|
||||
(?P<rung>.+?) # the rung, lazy so the groups below take what follows it
|
||||
(?:,\ routed\ by\ item\ (?P<item>[0-9]+)'s\ (?:L\ of\ (?P<likelihood>[0-9]+)\ and\ E\ \+\ I\ of\ (?P<escape_and_impact>[0-9]+)|.+?))? # routed: `, routed by item 3's L of 4 and E + I of 5`, or an older line's R + C
|
||||
(?:,\ taking\ in\ an\ E\ \+\ I\ of\ (?P<escape_and_impact_taken_in>[0-9]+))? # taken_in: `, taking in an E + I of 6`
|
||||
(?:,\ escalated\ at\ transcript\ entry\ (?P<escalated_at>[0-9]+))? # escalated: `, escalated at transcript entry 12`
|
||||
""", re.VERBOSE)
|
||||
|
||||
|
||||
class ArchivedWorker(typing.NamedTuple):
|
||||
"""What an archived worker line names: the worker's rung, the item that routed it, its L and E + I, and the E + I taken in; 0 for each it does not name."""
|
||||
"""What an archived worker line names: the worker's rung, the item that routed it, its L and E + I, the E + I taken in and the escalation's mark; 0 for each it does not name."""
|
||||
rung: Rung
|
||||
item: int
|
||||
likelihood: int
|
||||
escape_and_impact: int
|
||||
escape_and_impact_taken_in: int
|
||||
escalated_at: int
|
||||
|
||||
|
||||
HIGHEST_ESCAPE_AND_IMPACT = LEVELS["E"] + LEVELS["I"]
|
||||
@@ -55,6 +62,8 @@ class Route:
|
||||
unfinished: int = 0
|
||||
# The highest E + I the judge saw the worker take in, kept only where it picks a higher review step than the item's; 0 otherwise.
|
||||
escape_and_impact_taken_in: int = 0
|
||||
# The mark: how many transcript entries the session held when an escalation last stepped the worker up, 0 before any. The judge's escalation quote counts only from the entries after it, so a passage that already bought a step cannot buy another; the session's transcript only grows, so the mark holds across cycles.
|
||||
escalated_at: int = 0
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Raises on a programming error only, so nothing catches it: route_from and the reload decoder check levels_fit before they build a route.
|
||||
@@ -78,7 +87,8 @@ class Route:
|
||||
"""What the archive records as each verdict's worker; ROUTED reads it back."""
|
||||
routed = f", routed by item {self.item}'s L of {self.likelihood} and E + I of {self.escape_and_impact}" if self.item else ""
|
||||
taken_in = f", taking in an E + I of {self.escape_and_impact_taken_in}" if self.escape_and_impact_taken_in else ""
|
||||
return f"{self.rung}{routed}{taken_in}"
|
||||
escalated = f", escalated at transcript entry {self.escalated_at}" if self.escalated_at else ""
|
||||
return f"{self.rung}{routed}{taken_in}{escalated}"
|
||||
|
||||
|
||||
def archived_worker(worker: str) -> ArchivedWorker | None:
|
||||
@@ -87,7 +97,7 @@ def archived_worker(worker: str) -> ArchivedWorker | None:
|
||||
rung = Rung.parse(found["rung"]) if found else None
|
||||
if not (found and rung):
|
||||
return None
|
||||
return ArchivedWorker(rung, *(int(found[name] or 0) for name in ("item", "likelihood", "escape_and_impact", "escape_and_impact_taken_in")))
|
||||
return ArchivedWorker(rung, *(int(found[name] or 0) for name in ("item", "likelihood", "escape_and_impact", "escape_and_impact_taken_in", "escalated_at")))
|
||||
|
||||
|
||||
def route_from(worker: str) -> Route | None:
|
||||
@@ -95,4 +105,4 @@ def route_from(worker: str) -> Route | None:
|
||||
archived = archived_worker(worker)
|
||||
if archived is None or not levels_fit(archived.item, archived.likelihood, archived.escape_and_impact, archived.escape_and_impact_taken_in):
|
||||
return None
|
||||
return Route(archived.rung, archived.item, archived.likelihood, archived.escape_and_impact, escape_and_impact_taken_in=archived.escape_and_impact_taken_in)
|
||||
return Route(archived.rung, archived.item, archived.likelihood, archived.escape_and_impact, escape_and_impact_taken_in=archived.escape_and_impact_taken_in, escalated_at=archived.escalated_at)
|
||||
|
||||
@@ -2207,3 +2207,53 @@ Inherited architect, decided by:
|
||||
- Locality, Shape: `bin/chunk_preflight/choices.py:24` `planned`: each pick variant carries its release, so the messages read straight off it, where 6e0fda4 paired a recomputed `earliest` with a string by hand
|
||||
- Shape: `bin/chunk_preflight/gathered.py:1` `gathered`: data only, with the choices in `choices.py`, though `facts.py` imports `PULLS_LIST` from the decision module
|
||||
- Shape: `bin/chunk_pick.py:118` `unreleased_changes`: with `committed_since_release` and one `version`, the names say what they count
|
||||
|
||||
## 2026-10-18T12:16:18Z, PR #321 at 862137b, against 0462980
|
||||
|
||||
Ruling: same
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Mid A | same | worse | better | better | same |
|
||||
| Inherited architect | same | worse | better | better | better |
|
||||
| Maintainability senior | same | better | better | better | better |
|
||||
| Junior A | same | worse | same | better | same |
|
||||
|
||||
Mid A, decided by:
|
||||
|
||||
- Locality: `bin/loop_claude/cycle/judge.py:153` `escalation_mark`: "one trigger buys one step" was one flag and one function in `run.py`'s `unspent`; now a mark across cycles read through `Route.escalated_at`, `entries_read` stamped late by `judge_once`, None versus 0, `STEP_LINE`, `Grounds.since_step` and the routing steps, over three modules
|
||||
- Locality: `bin/loop_claude/cycle/judge.py:216` `quotes_checked`: turns a `QuotedAnswer` into an `Interrupt` without `entries_read`, correct only because `judge_once` stamps after `run_judge`, which nothing here says
|
||||
- Locality: `bin/model_routing/route.py:12` `ROUTED`: the mark survives only by round-tripping through `Route.__str__`, so rewording one fragment resets it to 0 at a distance
|
||||
- Shape, Self-sufficiency: `bin/model_routing/route.py:13` `ROUTED`: a verbose regex with each group naming the `__str__` fragment it reads, in place of one dense line
|
||||
- Shape: `bin/loop_claude/cycle/judge.py:143` `Grounds`: one named tuple whose fields document their own checks, in place of loose strings and a docstring list
|
||||
- Shape: `bin/loop_claude/routing.py:84` `one_rung_up`: no longer shares its name with `Cycle.stepped_from`, which means the route before the step
|
||||
- Self-sufficiency: `bin/loop_claude/cycle/prompts.py:87` `escalation`: the docstring shows the full sentences it renders for a concrete rung
|
||||
|
||||
Inherited architect, decided by:
|
||||
|
||||
- Locality: `bin/loop_claude/cycle/judge.py:185` `judge_once`: "one trigger buys one step" runs through five units in four modules and outlives the run via the archive line, with three "nothing" values for one idea: 0 for no mark, None for no escalation, None for never stamped
|
||||
- Locality: `bin/loop_claude/cycle/judge.py:219` `quotes_checked`: builds an `Interrupt` without `entries_read`, safe only because the stamp comes later; swapped, the escalation is silently dropped
|
||||
- Shape, Self-sufficiency: `bin/loop_claude/cycle/judge.py:143` `Grounds`: each checked text is a named field with its rule and penalty beside it, so the judge-rejection path reads easier
|
||||
- Shape: `bin/loop_claude/routing.py:84` `one_rung_up`: the name says what it does, and its `escalated_at` parameter shows it moves the mark, where `stepped_from` clashed with `Cycle.stepped_from`
|
||||
- Locality, Shape: `bin/loop_claude/cycle/run.py:69` `escalated`: `Cycle` lost the cycle-scoped `escalation_spent` and `rule` no longer filters through `unspent`, so `run.py` reads on its own
|
||||
- Self-sufficiency: `bin/model_routing/route.py:13` `ROUTED`: each group labelled by the fragment it reads, with a warning that a reword resets the mark
|
||||
- Self-sufficiency: `bin/loop_claude/cycle/prompts.py:88` `escalation`: the docstring shows the rendered text for one concrete route
|
||||
- Self-sufficiency: `bin/loop_claude/cycle/judge.py:61` `STEP_LINE`: defined beside the slicing that inserts it, with the entries-only-grow assumption written on `Route.escalated_at`
|
||||
|
||||
Maintainability senior, decided by:
|
||||
|
||||
- Locality: `bin/model_routing/route.py:66` `Route.escalated_at`: one persisted mark that holds across cycles replaces a cycle-scoped flag and a filter in `rule` that rewrote the judge's verdict outside the judge
|
||||
- Locality: `bin/loop_claude/routing.py:76` `stepped_by_ruling`: "no escalation, no step" now lives in routing, so `run.py`'s `escalated` reads straighter; offset in part by `judge_once` stamping `entries_read` after `quotes_checked` rebuilds an `Interrupt`, an order that matters
|
||||
- Self-sufficiency: `bin/loop_claude/cycle/judge.py:143` `Grounds`: each checked text a named field with its own comment, in place of two positional strings and a docstring list
|
||||
- Self-sufficiency: `bin/model_routing/route.py:13` `ROUTED`: each group commented with the `Route.__str__` fragment it reads back
|
||||
- Self-sufficiency: `bin/loop_claude/cycle/prompts.py:87` `escalation`: the docstring quotes the sentences it produces, and the body joins whole parts in place of space-led fragments
|
||||
- Shape: `bin/loop_claude/routing.py:84` `one_rung_up`: renamed from `stepped_from`, which meant the opposite of `Cycle.stepped_from` beside it
|
||||
|
||||
Junior A, decided by:
|
||||
|
||||
- Locality: `bin/model_routing/route.py:66` `Route.escalated_at`: "one trigger buys one step" moved from one file's cycle-local flag and `unspent` filter to a mark threaded through six files
|
||||
- Locality: `bin/loop_claude/cycle/judge.py:189` `judge_once`: stamps `entries_read` after the run, an unstamped None silently steps nothing, and "none" is 0 on the route but None in `stepped` and `stepped_by_ruling`
|
||||
- Self-sufficiency: `bin/model_routing/route.py:13` `ROUTED`: a verbose pattern whose groups name the fragments they read, in place of one dense line
|
||||
- Self-sufficiency, Shape: `bin/loop_claude/cycle/judge.py:143` `Grounds`: each field documents what must quote it and what a misquote does, though `judge` and `judge_once` grow to nine positional parameters
|
||||
- Shape: `bin/loop_claude/routing.py:84` `one_rung_up`: removes the name it shared with `Cycle.stepped_from`, which meant the opposite
|
||||
- Self-sufficiency: `bin/loop_claude/cycle/prompts.py:87` `escalation`: a worked example of the exact sentences, as its neighbours already had
|
||||
|
||||
+6
-4
@@ -310,15 +310,17 @@ Valid while the ladder orders its rungs by competence.
|
||||
|
||||
2026-10-15, lilleman asked for it. Serves goals 1.1 and 1.4: when the hard part is the rest of the chunk, handing it to subagents piece by piece leaves the worker, which reads and checks each of their diffs, on a rung that proved too low. The worker climbs only once its run records a trigger.
|
||||
|
||||
loop-claude's prompt offers the step beside the escalated subagent of `routing: a worker hands a task one rung up`, as long as the limit leaves a rung that subagent may take without the human's yes. The worker then ends its run asking loop-claude for a higher rung and naming the trigger. The judge quotes an earlier passage that shows the trigger fired, such as a review round's commit title or the worker's report of the failing test, never the request itself. The trigger must have fired since any earlier such request, so one trigger buys one step. The judge sees the worker's text and tool calls but no tool results, so the quote shows that the run recorded the trigger, not that a tool's output bears it out. loop-claude checks the quote word for word, as it checks a quoted answer, since stitched fragments can make up a trigger. Without such a quote, the rung holds. A misquote is no failure, since holding the rung is safe and three failures stop an unattended loop. The next cycle's prompt names the worker's rung, so the worker learns whether the step went through. When a quoted answer, or an interrupt the human answers on loop-claude's terminal, steps the worker up, the prompt that resumes it with the answer names the new rung, so the worker learns of the step sooner.
|
||||
loop-claude's prompt offers the step beside the escalated subagent of `routing: a worker hands a task one rung up`, as long as the limit leaves a rung that subagent may take without the human's yes. The worker then ends its run asking loop-claude for a higher rung and naming the trigger. The judge quotes the earlier passage where the trigger fired, such as a review round's commit title or the worker's report of the failing test, never a later mention of it and never the request itself. The quoted passage must sit after the worker's last step-up, which loop-claude checks, so one passage buys one step. The judge sees the worker's text and tool calls but no tool results, so the quote shows that the run recorded the trigger, not that a tool's output bears it out. loop-claude checks the quote word for word, as it checks a quoted answer, since stitched fragments can make up a trigger. Without such a quote, the rung holds. A misquote is no failure, since holding the rung is safe and three failures stop an unattended loop. The next cycle's prompt names the worker's rung, so the worker learns whether the step went through. When a quoted answer, or an interrupt the human answers on loop-claude's terminal, steps the worker up, the prompt that resumes it with the answer names the new rung, so the worker learns of the step sooner.
|
||||
|
||||
A run that asks for a higher rung may also ask a question. The judge then interrupts, or quotes the answer from README.md or the records, and either ruling carries the escalation quote too, checked the same way. loop-claude steps the rung as the ruling comes in, before the human is asked or the worker resumes with the quoted answer, since the work after the answer is work the stepped rung should do. An interrupt or a quoted answer is no unfinished verdict, so it adds to no count. The judge may quote a different passage for the trigger that already stepped the rung, and loop-claude cannot tell that from a new trigger. So once an interrupt or a quoted answer has taken an escalation, no later ruling of that cycle carries one. A trigger that fires after the answer steps nothing in that cycle.
|
||||
A run that asks for a higher rung may also ask a question. The judge then interrupts, or quotes the answer from README.md or the records, and either ruling carries the escalation quote too, checked the same way. loop-claude steps the rung as the ruling comes in, before the human is asked or the worker resumes with the quoted answer, since the work after the answer is work the stepped rung should do. An interrupt or a quoted answer is no unfinished verdict, so it adds to no count.
|
||||
|
||||
The judge may quote a different passage for the trigger that already stepped the rung, and loop-claude cannot tell from the quote alone whether the trigger is new. So it goes by where the quoted passage sits in the transcript. When an escalation steps the worker, the route marks how many transcript entries the session held: the mark. loop-claude then accepts an escalation quote only from the worker's own lines after the mark, never from its own prompts or the human's lines, which repeat earlier passages. The judge reads a `[step]` line at the mark saying the triggers before it are spent, so it quotes a passage the check accepts. The session's transcript only grows, so the mark holds across cycles. The trigger that bought a step, and any rewording of it before the mark, cannot buy another. A trigger that fires after the step steps the worker again, in the same cycle or a later one. One risk remains: a worker on the stepped rung may restate an old trigger after the mark, and the judge may quote that. The cap two rungs above the rung the item picks limits what this costs. Before the mark, once an interrupt or a quoted answer had taken an escalation, no later ruling of that cycle carried one, so a trigger that fired after the answer was lost. lilleman chose the mark on 2026-10-18 over two alternatives: lifting that rule when a worker run ended would have let one trigger buy two steps, and telling the worker its request was dropped would still have lost the step.
|
||||
|
||||
The step is one rung up from the rung the worker would otherwise hold: its own, or the rung an item the judge saw it take in picks, whichever is higher. It resets the count of unfinished verdicts, so the step every second one takes counts afresh; when both steps fall on one verdict, the worker steps once. It stops at the limit escalated subagents stop at, two rungs above the rung the chunk started on. Past that limit the rung holds: loop-claude takes no yes from the human between cycles, so a worker there asks the human for an escalated subagent instead. The judge keeps its rung, as `routing: on a model judge-replay covered` says.
|
||||
|
||||
The archive quotes the judge's passage under `Escalation:` in the unfinished verdict, the interrupt or the quoted answer. The step an unfinished verdict causes is archived after it. The step an interrupt or a quoted answer causes is archived with it, at once: the interrupt or answer names the rung the worker ran on until then, and a step-up archived beside it names the rung it runs on from then. So the routing figures charge each stretch to its own rung, and a loop that dies while the human answers resumes on the stepped rung. `bin/routing_figures/` counts none of these steps as an escalation by trigger, which it counts only from subagent prompts.
|
||||
The archive quotes the judge's passage under `Escalation:` in the unfinished verdict, the interrupt or the quoted answer. The step an unfinished verdict causes is archived after it. The step an interrupt or a quoted answer causes is archived with it, at once: the interrupt or answer names the rung the worker ran on until then, and a step-up archived beside it names the rung it runs on from then. So the routing figures charge each stretch to its own rung, and a loop that dies while the human answers resumes on the stepped rung. Each archived worker line names the mark once there is one, so a resumed loop keeps it. `bin/routing_figures/` counts none of these steps as an escalation by trigger, which it counts only from subagent prompts.
|
||||
|
||||
Valid while the worker reports in its transcript the triggers that fired.
|
||||
Valid while the worker reports in its transcript the triggers that fired, and while a chunk's session transcript only grows.
|
||||
|
||||
## routing: on a model judge-replay covered, the judge runs on the rung a review would get from the highest E + I among its chunk's items, never below the project's rung
|
||||
|
||||
|
||||
@@ -661,3 +661,7 @@ Read most (opens and searches):
|
||||
## Reading cost, chunk e7a12c46be6e, finished 2026-10-18T09:10:49Z
|
||||
|
||||
115559 new input tokens before the first commit.
|
||||
|
||||
## Reading cost, chunk b897b24d5e70, finished 2026-10-18T11:15:04Z
|
||||
|
||||
87043 new input tokens before the first commit.
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 180 | 0.2.0 | | | **Accept an escalation quote only from transcript entries after the worker's last step-up, so a trigger that fires after an interrupt or a quoted answer stepped the worker steps it again.** | 3 | 2 | 3 | 6 | 5 | 1.1 by 4 | 2.1 |
|
||||
| 159 | 0.2.0 | | decision | **Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.** | 4 | 2 | 3 | 7 | 6 | 1.4 by 7 | 1.9 |
|
||||
| 189 | 0.2.0 | | | **State one achieve goal for the health of a project's code in `~/.claude`, which every project's ranking weighs, most while its comprehension panel's overall is under 7.0, unless its README voids it.** | 4 | 2 | 3 | 7 | 6 | 1.1 by 6 | 1.7 |
|
||||
| 29 | 0.3.0 | | | **Make the contributor a standing member of every project's audience, unless its `## Audience` says maintainability does not matter.** | 2 | 2 | 3 | 5 | 2 | 1.1 by 5, 1.4 by 5 | 16.7 |
|
||||
@@ -92,19 +91,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 180. Accept an escalation quote only from transcript entries after the worker's last step-up, so a trigger that fires after an interrupt or a quoted answer stepped the worker steps it again.
|
||||
|
||||
`docs/decisions.md` → `routing: a worker whose run ends asking for a higher rung…` says a trigger that fires after the human's answer waits for the next cycle. Today it doesn't step the worker: once an interrupt or a quoted answer has taken an escalation, `unspent` in `bin/loop_claude/cycle/run.py` drops the escalation of every later ruling of the cycle. The dropped request stays in the transcript, so in the next cycle the judge finds no trigger that fired since the worker's last request. A quoted answer resumes the worker headless up to three times a cycle, and so does an answer the human types at loop-claude's terminal, so the loss is reachable with no one watching. `unspent` exists because loop-claude cannot tell from the judge's quotes alone whether a trigger already bought a step. Instead, identify a trigger by where it sits in the transcript:
|
||||
|
||||
- **The mark.** When an escalation steps the worker, store on the `Route` how many transcript entries the session held at that moment. The session's transcript only grows, so the mark holds across cycles.
|
||||
- **The check.** `quotes_checked` in `bin/loop_claude/cycle/judge.py` checks an escalation quote against the transcript rendered from the entries after the mark, and drops a quote found only before it, as it drops a misquote. The trigger that bought the last step, and any rewording of it before the mark, can no longer buy another.
|
||||
- **No per-cycle guard.** Remove `escalation_spent` and `unspent`: an interrupt, a quoted answer or an unfinished verdict, in the same cycle or a later one, may each carry an escalation the check accepts.
|
||||
- **The judge and the worker.** The judge's schema asks for the passage where the trigger fired, not a later mention of it, and drops "after any earlier such request", which the check now enforces. The worker's prompt asks for a trigger that fired since its last step-up, in place of "since your last such request" in `bin/loop_claude/cycle/prompts.py` → `escalation`. The prompt that resumes a stepped worker says the triggers before its step are spent.
|
||||
- **The archive.** The archived step-up records the mark, so a loop that dies and resumes keeps it.
|
||||
- **The decision.** Revise `routing: a worker whose run ends asking for a higher rung…`: replace "no later ruling of that cycle carries one" with the mark rule, and record the risk that remains. A worker on the stepped rung may restate an old trigger after the mark, and the cap two rungs above the rung its item picks bounds that.
|
||||
|
||||
lilleman chose this design on 2026-10-18 over clearing `escalation_spent` when a worker run ends, which lets one trigger buy two steps, and over telling the worker its request was dropped, which keeps the loss. Found by the stability review of PR #307.
|
||||
|
||||
### 159. Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.
|
||||
|
||||
lilleman asked, on 2026-10-16, for the prose pass to run on the worker's rung or lower, and the same day, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. Today `bin/loop_claude/cycle/prompts.py`'s `review_launches` keeps every reviewer at or above the project's rung whatever the item's E + I, so a docs-only chunk such as item 140's, routed to Haiku 5.5, runs its reviewers on Opus 5.5.
|
||||
@@ -327,7 +313,7 @@ Found by the principles audit of PR #242. `bin/loop_claude/test_black_box.py:113
|
||||
|
||||
### 137. Archive a chunk's route as a record loop-claude reads back field by field, in place of the `Worker:` prose `ROUTED` parses, and name `stepped` and the judge's `bundled` by what they do.
|
||||
|
||||
PR #262's paired ruling found the project reads worse: a rerun resumes on its rung only by parsing `Rung.__str__`'s and `Route.__str__`'s wording back with `bin/loop_claude/routing.py`'s `ROUTED`, a string contract over four files, where a reword silently routes afresh. Write the route as a structured line, such as JSON after `Worker:`, that `transcript_archive` reads into fields. `stepped` both raises to a bundled item's rung and counts towards a one-rung step; `Reply.bundled` names neither L nor highest. Make `bin/loop_claude/loop/main.py`'s `run_loop` easier to read where it threads the route.
|
||||
PR #262's paired ruling found the project reads worse: a rerun resumes on its rung only by parsing `Rung.__str__`'s and `Route.__str__`'s wording back with `bin/loop_claude/routing.py`'s `ROUTED`, a string contract over four files, where a reword silently routes afresh. Write the route as a structured line, such as JSON after `Worker:`, that `transcript_archive` reads into fields. `stepped` both raises to a bundled item's rung and counts towards a one-rung step; `Reply.bundled` names neither L nor highest. Make `bin/loop_claude/loop/main.py`'s `run_loop` easier to read where it threads the route. The structured line must keep the escalation's mark, archived today as `escalated at transcript entry N`: without it, a trigger that already stepped the worker could step it again.
|
||||
|
||||
### 66. Hold systems-architect's method thresholds only while the comprehension panel finds the flagged code hard.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user