Merge pull request 'Archive the route as a JSON record loop-claude reads back field by field, and name the stepping functions and the judge's taken-in levels by what they do' (#357) from item-137 into main
This commit was merged in pull request #357.
This commit is contained in:
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- loop-claude now archives the route on each `Worker:` line as a JSON record, such as `{"model": "claude-opus-5-5", "effort": "high", "item": 3, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 12}`, and reads it back field by field. The line used to be a sentence, such as `claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 5`, which loop-claude parsed back word for word: rewording the sentence in the code would have made every archived line unreadable, and a rerun would have routed the open chunk afresh. A rerun and the routing figures still read the sentences archived since loop-claude routed on L. A rerun that finds a `Worker:` line it can't read now says so on stderr. The judge's reply now names the levels the worker took in `highest_likelihood_taken_in` and `highest_escape_and_impact_taken_in`, in place of `bundled` and `bundled_escape_and_impact`.
|
||||
- A chunk now takes every unblocked item of its release that fires the same review passes, whatever its purpose: one PR title no longer has to state what the items achieve together. An item with R above 5 ships alone. `chunk-preflight` says so for such an item, and never lists one as an item that may join another.
|
||||
- A branch whose name is not UTF-8 now shows no branch in the status line, in place of a name with replacement characters.
|
||||
- A new technical principle, "About seven things per level", sets soft limits on what a reader takes in at once: the parameters and branches of a function, the methods of a class, the files and folders in a folder, the sections of a document and the components of a system. Aim for four or five of the things that act on each other, such as parameters. Things found by name, such as files named by locale, may run far past seven. Crossing a limit asks you to group the things or record why they stay. The systems-architect reviewer now reads its limits from this principle. It rates a unit that crosses a limit with no recorded reason a 🟡, where it used to be a 🔴, and no longer asks for the limits as merge-gate rules.
|
||||
|
||||
@@ -34,13 +34,14 @@ from project_key import checkout_of
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from rung_replay.search import first_reply, replayed
|
||||
from loop_claude.cycle.judge import judge_once, Verdict
|
||||
from loop_claude.cycle.judge import judge_once
|
||||
from claude_code.reply import Limited
|
||||
from model_routing.ladder import lowest_worker_rung, project_rung, Rung
|
||||
from loop_claude.project.premises import read_premises
|
||||
from loop_claude.project.records import read_records
|
||||
from claude_code.transcript import read_transcript
|
||||
from loop_claude.stop import Stop
|
||||
from loop_claude.verdicts import Verdict
|
||||
|
||||
TOOL = "judge-replay"
|
||||
USAGE = f"{TOOL} <model>, a model on the routing ladder"
|
||||
|
||||
@@ -6,7 +6,7 @@ import typing
|
||||
|
||||
from transcript_archive.outcomes import ANSWERED, FINISHED, first_verdict, INTERRUPTED, task_of, UNFINISHED
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict
|
||||
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict
|
||||
from rung_replay.search import Replay
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINIS
|
||||
from judge_replay.replay import Case, case_from, report, reproduced, sample, verdict_kind
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished
|
||||
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished
|
||||
from judge_replay import main
|
||||
from rung_replay.search import Miss, Replay, Tried
|
||||
from model_routing.ladder import Rung
|
||||
@@ -122,7 +122,7 @@ class Main(unittest.TestCase):
|
||||
|
||||
@staticmethod
|
||||
def verdict(interrupt, evidence):
|
||||
output = {"answer": "", "bundled": 0, "bundled_escape_and_impact": 0, "escalation": "", "evidence": evidence, "finished": not interrupt, "interrupt": interrupt, "question": "Which?" if interrupt else "", "release_ready": False}
|
||||
output = {"answer": "", "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "evidence": evidence, "finished": not interrupt, "interrupt": interrupt, "question": "Which?" if interrupt else "", "release_ready": False}
|
||||
return {"type": "result", "structured_output": output}
|
||||
|
||||
def test_replays_the_models_projects_on_the_reference_then_from_the_projects_rung_down_and_prints_the_lowest_that_returns_all(self):
|
||||
|
||||
@@ -7,6 +7,7 @@ from loop_claude.cycle.prompts import ANSWER_DESTINATIONS, ANSWERS_RECORDED, typ
|
||||
from claude_code.reply import claude_reply, Limited
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict
|
||||
from claude_code.entries import human_text, loop_text
|
||||
from todo_table import LEVELS
|
||||
from model_routing.route import on_its_scale
|
||||
@@ -65,9 +66,9 @@ STEP_LINE = "[step] loop-claude stepped the worker up a rung here, so the escala
|
||||
class Reply:
|
||||
answer: str = dataclasses.field(metadata={"description": "With interrupt false, the passage of README.md or the records that answers the worker's question, quoted verbatim, without elisions. Empty otherwise."})
|
||||
# Routes the next cycle's worker. Counting the item the chunk was routed on is harmless: loop-claude never steps the worker down.
|
||||
bundled: int = dataclasses.field(metadata={"description": "The highest L the worker states for a todo.md item it took into this chunk; 0 when it states none."})
|
||||
# Routes the next cycle's judge and reviewers, as bundled routes the worker.
|
||||
bundled_escape_and_impact: int = dataclasses.field(metadata={"description": "The highest E + I the worker states for a todo.md item it took into this chunk; 0 when it states none."})
|
||||
highest_likelihood_taken_in: int = dataclasses.field(metadata={"description": "The highest L the worker states for a todo.md item it took into this chunk; 0 when it states none."})
|
||||
# Routes the next cycle's judge and reviewers, as highest_likelihood_taken_in routes the worker.
|
||||
highest_escape_and_impact_taken_in: int = dataclasses.field(metadata={"description": "The highest E + I the worker states for a todo.md item it took into this chunk; 0 when it states none."})
|
||||
# Routes the worker one rung up, as the judge interrupts or quotes an answer, or on the next cycle: docs/decisions.md → `routing: a worker whose run ends asking for a higher rung`.
|
||||
escalation: str = dataclasses.field(metadata={"description": "Empty unless the chunk is unfinished, interrupt is true or answer quotes a passage, and the worker's run ends asking loop-claude for a higher rung, citing an escalation trigger. Then: a verbatim quote, without elisions, of the transcript passage where the trigger fired, such as a review round's commit title or the worker's report of the finding or failing test. Where a [step] line says loop-claude stepped the worker up a rung, quote only the worker's own words after it, and leave this empty where the trigger fired only before it. Never a later mention of the trigger, and never the request itself."})
|
||||
evidence: str = dataclasses.field(metadata={"description": "A verbatim quote of the one transcript passage the verdict rests on, without elisions: what triggers the interrupt; for a quoted answer, the worker's question; else what shows the chunk finished or the work remaining."})
|
||||
@@ -94,50 +95,6 @@ class Rejected(Failure):
|
||||
reply: str
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Interrupt:
|
||||
evidence: str
|
||||
question: str
|
||||
# The judge's quote showing that the worker's cited escalation trigger fired; empty for none.
|
||||
escalation: str = ""
|
||||
# The bundled L the judge saw, 0 for none, as on Unfinished: an escalation steps up from the rung that item picks.
|
||||
bundled: int = 0
|
||||
release_ready: bool = False
|
||||
# How many transcript entries the judge read, which judge_once stamps. An escalation the verdict carries moves the route's mark there; one on a verdict never stamped, None, steps nothing.
|
||||
entries_read: int | None = None
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class QuotedAnswer:
|
||||
"""The worker asked a question that README.md or the records answer. The answer field quotes the passage."""
|
||||
evidence: str
|
||||
question: str
|
||||
answer: str
|
||||
# As on Interrupt.
|
||||
escalation: str = ""
|
||||
bundled: int = 0
|
||||
entries_read: int | None = None
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Finished:
|
||||
evidence: str
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Unfinished:
|
||||
evidence: str
|
||||
# The bundled L the judge saw, 0 for none.
|
||||
bundled: int = 0
|
||||
# The judge's quote showing that the worker's cited escalation trigger fired; empty for none.
|
||||
escalation: str = ""
|
||||
# The bundled E + I the judge saw, 0 for none.
|
||||
bundled_escape_and_impact: int = 0
|
||||
# As on Interrupt.
|
||||
entries_read: int | None = None
|
||||
|
||||
|
||||
Verdict = Interrupt | QuotedAnswer | Finished | Unfinished
|
||||
|
||||
|
||||
class Grounds(typing.NamedTuple):
|
||||
@@ -150,11 +107,6 @@ class Grounds(typing.NamedTuple):
|
||||
quotable: str
|
||||
|
||||
|
||||
def escalation_mark(verdict: Interrupt | QuotedAnswer | Unfinished) -> int | None:
|
||||
"""Where the verdict's escalation would move the route's mark, if it steps the worker; None where it carries none."""
|
||||
return verdict.entries_read if verdict.escalation else None
|
||||
|
||||
|
||||
def verdict_from(output: object) -> Verdict | Failure:
|
||||
if not isinstance(output, dict):
|
||||
return Failure("the judge returned no reply")
|
||||
@@ -162,20 +114,20 @@ def verdict_from(output: object) -> Verdict | Failure:
|
||||
# type(), not isinstance(): JSON true is no integer.
|
||||
if type(output.get(name)) is not kind:
|
||||
return Failure(f"the judge's reply has no {name} of type {JSON_TYPES[kind]}")
|
||||
# A bundled level off its scale counts as none: holding the rung is safe, and three failures would stop an unattended loop.
|
||||
bundled = output["bundled"] if 0 <= output["bundled"] <= HIGHEST_L else 0
|
||||
# An L taken in off its scale counts as none: holding the rung is safe, and three failures would stop an unattended loop.
|
||||
likelihood = output["highest_likelihood_taken_in"] if 0 <= output["highest_likelihood_taken_in"] <= HIGHEST_L else 0
|
||||
if output["interrupt"]:
|
||||
if not output["question"].strip():
|
||||
return Failure("the judge interrupted without a question")
|
||||
return Interrupt(output["evidence"], output["question"], output["escalation"], bundled, output["release_ready"])
|
||||
return Interrupt(output["evidence"], output["question"], escalation=output["escalation"], highest_likelihood_taken_in=likelihood, release_ready=output["release_ready"])
|
||||
if output["answer"].strip():
|
||||
if not output["question"].strip():
|
||||
return Failure("the judge quoted an answer without a question")
|
||||
return QuotedAnswer(output["evidence"], output["question"], output["answer"], output["escalation"], bundled)
|
||||
return QuotedAnswer(output["evidence"], output["question"], output["answer"], escalation=output["escalation"], highest_likelihood_taken_in=likelihood)
|
||||
if output["finished"]:
|
||||
return Finished(output["evidence"])
|
||||
bundled_escape_and_impact = output["bundled_escape_and_impact"] if on_its_scale(output["bundled_escape_and_impact"]) else 0
|
||||
return Unfinished(output["evidence"], bundled, output["escalation"], bundled_escape_and_impact)
|
||||
escape_and_impact = output["highest_escape_and_impact_taken_in"] if on_its_scale(output["highest_escape_and_impact_taken_in"]) else 0
|
||||
return Unfinished(output["evidence"], escalation=output["escalation"], highest_likelihood_taken_in=likelihood, highest_escape_and_impact_taken_in=escape_and_impact)
|
||||
|
||||
|
||||
def judge(root: str, task: str, premises: str, records: str, entries: list[dict], escalated_at: int, rung: Rung, effects: Effects, keep: typing.Callable[[Rejected], Stop | None]) -> Verdict | Stop:
|
||||
@@ -216,7 +168,7 @@ def run_judge(root: str, args: list[str], prompt: str, grounds: Grounds, effects
|
||||
def quotes_checked(verdict: Verdict, grounds: Grounds) -> Verdict:
|
||||
"""The verdict with its quoted answer and its escalation checked, as Grounds says; a quoted answer put to the human keeps the escalation it carried."""
|
||||
if isinstance(verdict, QuotedAnswer) and not verbatim(verdict.answer, grounds.quotable):
|
||||
verdict = Interrupt(verdict.evidence, verdict.question, verdict.escalation, verdict.bundled)
|
||||
verdict = Interrupt(verdict.evidence, verdict.question, escalation=verdict.escalation, highest_likelihood_taken_in=verdict.highest_likelihood_taken_in)
|
||||
if isinstance(verdict, Interrupt | QuotedAnswer | Unfinished) and verdict.escalation and not (verbatim(verdict.escalation, grounds.since_step) and verbatim(verdict.escalation, grounds.shown)):
|
||||
return dataclasses.replace(verdict, escalation="")
|
||||
return verdict
|
||||
|
||||
@@ -3,12 +3,13 @@ import pathlib
|
||||
import typing
|
||||
|
||||
from loop_claude.cycle.states import Active, Archived, Closing, Ended, Event, HumanTalked, HumanTyped, Interrupting, Judging, QUOTED_ANSWERS_PER_CYCLE, Ruled, State, TalkChosen, Talking, transition, TypedAnswer, WorkerRan, Working
|
||||
from loop_claude.routing import after_ruling, judge_rung
|
||||
from transcript_archive.write import archive
|
||||
from claude_code.session import delete_session, Session, session_open, transcript_file
|
||||
from loop_claude.effects import Effects
|
||||
from loop_claude.cycle.judge import escalation_mark, Finished, Interrupt, judge, QuotedAnswer, Rejected, Unfinished
|
||||
from failure import Failure
|
||||
from loop_claude.routing import judge_rung, stepped_by_ruling
|
||||
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished
|
||||
from loop_claude.cycle.judge import judge, Rejected
|
||||
from transcript_archive import outcomes
|
||||
from loop_claude.paths import Paths
|
||||
from loop_claude.cycle import prompts
|
||||
@@ -73,13 +74,13 @@ def escalated(cycle: Cycle, event: Event) -> Cycle:
|
||||
cycle = dataclasses.replace(cycle, stepped_from=None)
|
||||
if not (isinstance(event, Ruled) and isinstance(event.verdict, Interrupt | QuotedAnswer) and cycle.anchor):
|
||||
return cycle
|
||||
route = stepped_by_ruling(cycle.anchor, cycle.route, event.verdict.bundled, escalation_mark(event.verdict))
|
||||
route = after_ruling(cycle.anchor, cycle.route, event.verdict)
|
||||
return dataclasses.replace(cycle, route=route, stepped_from=cycle.route if route != cycle.route else None)
|
||||
|
||||
|
||||
def stopped(cycle: Cycle, stop: Stop, why: str) -> Stop:
|
||||
path = transcript_file(cycle.session)
|
||||
failure = archive(cycle.project_archive, path, cycle.task, outcomes.stopped(str(cycle.route), why), cycle.effects.now()) if path else None
|
||||
failure = archive(cycle.project_archive, path, cycle.task, outcomes.stopped(cycle.route.record(), why), cycle.effects.now()) if path else None
|
||||
# A failed archive exits 1 whatever the stop; when the stop was this archive failing, its reason is said once.
|
||||
return Stop(1, f"{failure.reason}\n{stop.reason}".rstrip()) if failure and failure.reason != stop.reason else stop
|
||||
|
||||
@@ -137,7 +138,7 @@ def keep_answer(cycle: Cycle, answered: QuotedAnswer) -> Stop | None:
|
||||
if not path:
|
||||
return None
|
||||
# As in ask: the answer names the rung the worker ran on until then, and a step-up archived after it names the rung it runs on from then.
|
||||
quoted = outcomes.answered(answered.question, prompts.QUOTE_SOURCES, answered.answer, answered.evidence, answered.escalation, str(judge_of(cycle)), str(cycle.stepped_from or cycle.route))
|
||||
quoted = outcomes.answered(answered.question, prompts.QUOTE_SOURCES, answered.answer, answered.evidence, answered.escalation, str(judge_of(cycle)), (cycle.stepped_from or cycle.route).record())
|
||||
if failure := archive_stepped(cycle, path, quoted):
|
||||
return Stop(1, failure.reason)
|
||||
return None
|
||||
@@ -162,7 +163,7 @@ def ask(cycle: Cycle, transcript_path: str, ruling: Interrupt | QuotedAnswer) ->
|
||||
question = question_for(ruling)
|
||||
# Archived now, so the transcript the question was asked on is kept with it; the human's answer then grows the transcript.
|
||||
# The interrupt names the rung the worker ran on before it; a step-up archived after it names the stepped rung, so the figures charge each stretch to its own rung.
|
||||
interrupted = outcomes.interrupted(question, ruling.evidence, str(judge_of(cycle)), ruling.escalation, str(cycle.stepped_from or cycle.route))
|
||||
interrupted = outcomes.interrupted(question, ruling.evidence, str(judge_of(cycle)), ruling.escalation, (cycle.stepped_from or cycle.route).record())
|
||||
if failure := archive_stepped(cycle, transcript_path, interrupted):
|
||||
return Stop(1, failure.reason)
|
||||
# Only the human may cut a release, so a release ready to cut opens claude's session at once.
|
||||
@@ -175,7 +176,7 @@ def ask(cycle: Cycle, transcript_path: str, ruling: Interrupt | QuotedAnswer) ->
|
||||
|
||||
def archive_stepped(cycle: Cycle, transcript_path: str, outcome: str) -> Failure | None:
|
||||
"""Archives the outcome of a ruling, then the step-up it caused when it stepped the route. The first failure is returned."""
|
||||
for kept in (outcome, outcomes.stepped_up(str(cycle.route))) if cycle.stepped_from else (outcome,):
|
||||
for kept in (outcome, outcomes.stepped_up(cycle.route.record())) if cycle.stepped_from else (outcome,):
|
||||
if failure := archive(cycle.project_archive, transcript_path, cycle.task, kept, cycle.effects.now()):
|
||||
return failure
|
||||
return None
|
||||
@@ -188,7 +189,7 @@ def question_for(ruling: Interrupt | QuotedAnswer) -> str:
|
||||
|
||||
|
||||
def close(cycle: Cycle, transcript_path: str, verdict: Finished | Unfinished) -> Archived | Stop:
|
||||
outcome = outcomes.finished(str(cycle.route), verdict.evidence, str(judge_of(cycle))) if isinstance(verdict, Finished) else outcomes.unfinished(str(cycle.route), verdict.evidence, str(judge_of(cycle)), verdict.escalation)
|
||||
outcome = outcomes.finished(cycle.route.record(), verdict.evidence, str(judge_of(cycle))) if isinstance(verdict, Finished) else outcomes.unfinished(cycle.route.record(), verdict.evidence, str(judge_of(cycle)), verdict.escalation)
|
||||
if failure := archive(cycle.project_archive, transcript_path, cycle.task, outcome, cycle.effects.now()):
|
||||
return Stop(1, failure.reason)
|
||||
if isinstance(verdict, Finished) and (failure := delete_session(cycle.session)):
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import dataclasses
|
||||
|
||||
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict
|
||||
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict
|
||||
|
||||
# A backstop for what the judge misses, such as a reworded repeat, so the worker cannot loop unattended on answers quoted from README.md or the records.
|
||||
QUOTED_ANSWERS_PER_CYCLE = 3
|
||||
|
||||
@@ -94,24 +94,26 @@ The judge reports the highest L and the highest E + I the worker states for an i
|
||||
next cycle steps the worker up to the rung that L picks, never down, and the reviewers to the rung
|
||||
picked by the higher of two: that E + I, or the routed item's (Route.reviewed_on in
|
||||
bin/model_routing/route.py). A level off its scale counts as none. Every second unfinished verdict
|
||||
on a chunk steps it up one rung (routing.stepped). So does an unfinished verdict on a worker that
|
||||
ended its run asking for a higher rung, once the judge quotes the transcript showing that the
|
||||
escalation trigger it named fired (~/.claude/CLAUDE.md → "Working with the codebase" lists them). An
|
||||
interrupt or a quoted answer with such a quote steps the rung as it comes in, before the human is
|
||||
asked or the worker resumes (routing.stepped_by_ruling). Each such step marks how many transcript
|
||||
entries the session held, and from then on the judge must quote the worker's own lines after that
|
||||
mark, where it reads a [step] line. Each step on an escalation stops two rungs above the rung the
|
||||
chunk's item picks, and the worker's prompts offer it only while the worker is below that limit.
|
||||
on a chunk steps it up one rung (routing.after_unfinished). So does an unfinished verdict on a
|
||||
worker that ended its run asking for a higher rung, once the judge quotes the transcript showing
|
||||
that the escalation trigger it named fired (~/.claude/CLAUDE.md → "Working with the codebase" lists
|
||||
them). An interrupt or a quoted answer with such a quote steps the rung as it comes in, before the
|
||||
human is asked or the worker resumes (routing.after_ruling). Each such step marks how many
|
||||
transcript entries the session held, and from then on the judge must quote the worker's own lines
|
||||
after that mark, where it reads a [step] line. Each step on an escalation stops two rungs above the
|
||||
rung the chunk's item picks, and the worker's prompts offer it only while the worker is below that
|
||||
limit.
|
||||
|
||||
Routing: the archive and reruns
|
||||
|
||||
Each finished or unfinished verdict, interrupt, quoted answer, stop and step-up in the archive names
|
||||
the worker's rung, with the item, L and E + I that routed the chunk, any taken-in E + I that picks a
|
||||
higher review rung (archived as a step-up too), and the mark. A rerun resumes an open chunk on the
|
||||
latest route named there, and counts its unfinished verdicts afresh. A rung no longer on the ladder,
|
||||
or below the one its item's L picks, is dropped, and the chunk is routed afresh from todo.md. So is
|
||||
a chunk whose archived line names an R + C, as loop-claude wrote before it routed on L. A todo.md or
|
||||
an archive that cannot be read or parsed is said on stderr, and the chunk is routed without it.
|
||||
Each finished or unfinished verdict, interrupt, quoted answer, stop and step-up in the archive ends
|
||||
with a Worker: line, the route's JSON record (Route.record in bin/model_routing/route.py). A rerun
|
||||
resumes an open chunk on the latest route named there, and counts its unfinished verdicts afresh.
|
||||
It also reads a Worker: line an older loop-claude wrote as a sentence, such as "claude-opus-5-5 at
|
||||
high effort, routed by item 3's L of 4 and E + I of 5". A rung no longer on the ladder, or below the
|
||||
one its item's L picks, is dropped, and the chunk is routed afresh from todo.md. So is a chunk whose
|
||||
archived line names an R + C, as loop-claude wrote before it routed on L. A todo.md or an archive
|
||||
that cannot be read or parsed is said on stderr, and the chunk is routed without it.
|
||||
|
||||
Routing: the judge's rung
|
||||
|
||||
@@ -181,18 +183,18 @@ import pathlib
|
||||
import sys
|
||||
import typing
|
||||
|
||||
from loop_claude.routing import after_unfinished, first_route, kept, routed_item
|
||||
from transcript_archive.write import archive
|
||||
from transcript_archive.location import archive_root, project_archive
|
||||
from loop_claude.loop.routing_check import check_routing
|
||||
from loop_claude.project.checkout import checkout_root
|
||||
from loop_claude.cycle.run import Cycle, run_cycle
|
||||
from loop_claude.cycle.run import Concluded, Cycle, run_cycle
|
||||
from loop_claude.loop.reload import decode_handed, exec_if_edited, fingerprint, LoopRun, RELOAD_FLAG
|
||||
from chunk_pick import default_plan, DefaultPlan
|
||||
from git_repo import default_ref
|
||||
from loop_claude.effects import Effects
|
||||
from loop_claude.cycle.judge import escalation_mark, Finished, Unfinished
|
||||
from failure import Failure
|
||||
from loop_claude.routing import first_route, kept, routed_item, stepped
|
||||
from loop_claude.verdicts import Finished
|
||||
from loop_claude.project.model_choice import get_or_ask_models
|
||||
from todo_table import Item
|
||||
from transcript_archive.read import latest_worker
|
||||
@@ -246,43 +248,44 @@ def run_loop(argv: list[str], env: typing.Mapping[str, str], paths: Paths, launc
|
||||
# After the reads above, so a loop that stops on one makes no worktree.
|
||||
facts, item = chunk_start(session, launcher.with_name("chunk-preflight"), loop_run.root, effects)
|
||||
anchor = project_rung(models.worker)
|
||||
loop_run = routed(loop_run, anchor, lambda: archived_route(archive_folder, session), item)
|
||||
route = loop_run.route or Route(Rung(models.worker))
|
||||
route = routed(loop_run.route, anchor, models.worker, lambda: archived_route(archive_folder, session), item)
|
||||
print(f"\n=== cycle {loop_run.cycle_number}, worker on {route.rung} ===", flush=True)
|
||||
concluded = run_cycle(Cycle(anchor=anchor, effects=effects, facts=facts, judge_model=models.judge, paths=paths, premises=premises, project_archive=archive_folder, records=records, root=loop_run.root, route=route, session=session, task=loop_run.task, unfinished_evidence=loop_run.unfinished_evidence))
|
||||
if isinstance(concluded, Stop):
|
||||
return concluded
|
||||
# The route the cycle ended on, which a step-up at an interrupt or a quoted answer may have raised and archived, is the one the verdict steps from.
|
||||
ended = dataclasses.replace(loop_run, route=concluded.route if anchor else None)
|
||||
before, loop_run = ended.route, after_verdict(ended, anchor, concluded.verdict)
|
||||
if stop := archive_step(archive_folder, session, loop_run, before, effects):
|
||||
loop_run = after_verdict(loop_run, anchor, concluded)
|
||||
if stop := archive_step(archive_folder, session, loop_run.task, concluded.route, loop_run.route, effects):
|
||||
return stop
|
||||
if stop := check_routing(archive_root(paths.home, env), anchor or Rung(models.worker), paths, effects):
|
||||
return stop
|
||||
|
||||
|
||||
def routed(loop_run: LoopRun, anchor: Rung | None, archived: typing.Callable[[], Route | None], item: typing.Callable[[], Item | None]) -> LoopRun:
|
||||
"""The loop run with its chunk's route, or None when the project's model is off the ladder. A chunk keeps its route, or on a rerun the one its archive last names,
|
||||
while that is at or above the rung its item's L picks from anchor, the project's rung; else the plan item routes it afresh."""
|
||||
def routed(route: Route | None, anchor: Rung | None, model: str, archived: typing.Callable[[], Route | None], item: typing.Callable[[], Item | None]) -> Route:
|
||||
"""The route the cycle runs on. anchor is the project's rung, and model the project's model.
|
||||
|
||||
Where anchor is None, the model is off the ladder, and the worker runs on it. Otherwise the chunk keeps route, or on a rerun the route its archive last names, while that route's rung is at or above the one its item's L picks from anchor. Else the plan item routes the chunk afresh.
|
||||
"""
|
||||
if anchor is None:
|
||||
return dataclasses.replace(loop_run, route=None)
|
||||
route = loop_run.route or archived()
|
||||
return dataclasses.replace(loop_run, route=route if route and kept(route, anchor) else first_route(anchor, item()))
|
||||
return Route(Rung(model))
|
||||
route = route or archived()
|
||||
return route if route and kept(route, anchor) else first_route(anchor, item())
|
||||
|
||||
|
||||
def after_verdict(loop_run: LoopRun, anchor: Rung | None, verdict: Finished | Unfinished) -> LoopRun:
|
||||
def after_verdict(loop_run: LoopRun, anchor: Rung | None, concluded: Concluded) -> LoopRun:
|
||||
"""The loop run the next cycle starts from. A finished chunk drops its route. An unfinished one takes the route after_unfinished makes of the route its cycle ended on, which an interrupt's or a quoted answer's escalation may have raised; off the ladder, it keeps that route."""
|
||||
verdict = concluded.verdict
|
||||
if isinstance(verdict, Finished):
|
||||
return dataclasses.replace(loop_run, route=None, unfinished_evidence="")
|
||||
route = stepped(anchor, loop_run.route, verdict.bundled, escalation_mark(verdict)).taking_in(verdict.bundled_escape_and_impact) if anchor and loop_run.route else loop_run.route
|
||||
route = after_unfinished(anchor, concluded.route, verdict) if anchor else concluded.route
|
||||
return dataclasses.replace(loop_run, route=route, unfinished_evidence=verdict.evidence)
|
||||
|
||||
|
||||
def archive_step(archive_folder: pathlib.Path, session: Session, loop_run: LoopRun, before: Route | None, effects: Effects) -> Stop | None:
|
||||
"""Archives the rung an unfinished verdict stepped the chunk up to, or the higher E + I it took in; the verdict itself names the route its cycle ran on."""
|
||||
if not (before and loop_run.route) or str(loop_run.route) == str(before):
|
||||
def archive_step(archive_folder: pathlib.Path, session: Session, task: str, before: Route, after: Route | None, effects: Effects) -> Stop | None:
|
||||
"""Archives the route after an unfinished verdict as a step-up, where its record differs from the route before it: a higher rung, or a higher E + I taken in. The verdict itself names the route before, which its cycle ran on."""
|
||||
if after is None or after.record() == before.record():
|
||||
return None
|
||||
path = transcript_file(session)
|
||||
failure = archive(archive_folder, path, loop_run.task, outcomes.stepped_up(str(loop_run.route)), effects.now()) if path else None
|
||||
failure = archive(archive_folder, path, task, outcomes.stepped_up(after.record()), effects.now()) if path else None
|
||||
return Stop(1, failure.reason) if failure else None
|
||||
|
||||
|
||||
@@ -293,7 +296,10 @@ def archived_route(archive_folder: pathlib.Path, session: Session) -> Route | No
|
||||
if isinstance(worker, Failure):
|
||||
print(f"routing the chunk afresh: {worker.reason}", file=sys.stderr)
|
||||
return None
|
||||
return route_from(worker) if worker else None
|
||||
route = route_from(worker) if worker else None
|
||||
if worker and route is None:
|
||||
print(f"routing the chunk afresh: the archive's latest Worker: line names no route loop-claude can read: {worker}", file=sys.stderr)
|
||||
return route
|
||||
|
||||
|
||||
def chunk_start(session: Session, preflight: pathlib.Path, root: str, effects: Effects) -> tuple[str, typing.Callable[[], Item | None]]:
|
||||
|
||||
@@ -22,7 +22,7 @@ class LoopRun:
|
||||
root: str
|
||||
task: str
|
||||
unfinished_evidence: str
|
||||
# None until the chunk is routed, and where the project's model is off the ladder.
|
||||
# Set by each unfinished verdict; None before the chunk's first one, and once it finishes.
|
||||
route: Route | None = None
|
||||
|
||||
|
||||
|
||||
+20
-16
@@ -3,6 +3,7 @@ import dataclasses
|
||||
|
||||
from chunk_pick import chunk_item, LastCut, Next
|
||||
from model_routing.ladder import climbed, highest, LADDER, position, project_rung, review_floor, Rung, stepped_up, worker_rung
|
||||
from loop_claude.verdicts import escalation_mark, Interrupt, QuotedAnswer, Unfinished
|
||||
from failure import Failure
|
||||
from todo_table import Item, items, NoItemsTable
|
||||
from model_routing.route import Route
|
||||
@@ -63,24 +64,27 @@ def escalation_cap(anchor: Rung, route: Route) -> Rung:
|
||||
return climbed(item_rung(anchor, route), ESCALATION_CAP)
|
||||
|
||||
|
||||
def stepped(anchor: Rung, route: Route, bundled: int, escalated_at: int | None) -> Route:
|
||||
"""The route after one more unfinished verdict. Its rung never falls below taken_in's. It steps up one rung on every second unfinished verdict, or on an escalation the judge found while the rung is below escalation_cap; where both fall on one verdict, it steps once.
|
||||
def after_unfinished(anchor: Rung, route: Route, verdict: Unfinished) -> Route:
|
||||
"""The route the next cycle runs on after an unfinished verdict.
|
||||
|
||||
bundled is the highest L the judge saw the worker take in, 0 for none. escalated_at is how many transcript entries the judge read where the verdict carries an escalation, None where it carries none. A step moves the route's mark, Route.escalated_at, to escalated_at; a step on the count of unfinished verdicts alone keeps the mark. They are passed in place of the verdict, since loop_claude.cycle.judge imports this module through prompts.
|
||||
Its rung never falls. It first rises to the rung that the verdict's highest L taken in picks, where that is higher. From there it steps one rung up on every second unfinished verdict, or on an escalation while below escalation_cap; where both fall on one verdict, it steps once. A step on an escalation moves the route's mark, Route.escalated_at, to where the judge stopped reading, so the trigger it quoted buys no second step; a step on the count of unfinished verdicts alone keeps the mark. Last, the route takes in the verdict's highest E + I, as Route.taking_in does.
|
||||
"""
|
||||
rung = taken_in(anchor, route, bundled)
|
||||
unfinished = route.unfinished + 1
|
||||
if unfinished >= UNFINISHED_PER_STEP or (escalated_at is not None and below_cap(anchor, route, rung)):
|
||||
return one_rung_up(route, rung, route.escalated_at if escalated_at is None else escalated_at)
|
||||
return dataclasses.replace(route, rung=rung, unfinished=unfinished)
|
||||
rung = raised(anchor, route, verdict.highest_likelihood_taken_in)
|
||||
mark = escalation_mark(verdict)
|
||||
if route.unfinished + 1 >= UNFINISHED_PER_STEP or (mark is not None and below_cap(anchor, route, rung)):
|
||||
after = one_rung_up(route, rung, route.escalated_at if mark is None else mark)
|
||||
else:
|
||||
after = dataclasses.replace(route, rung=rung, unfinished=route.unfinished + 1)
|
||||
return after.taking_in(verdict.highest_escape_and_impact_taken_in)
|
||||
|
||||
|
||||
def stepped_by_ruling(anchor: Rung, route: Route, bundled: int, escalated_at: int | None) -> Route:
|
||||
"""The route after the judge's interrupt or quoted answer, its arguments as stepped's: unchanged without an escalation; with one, taken_in's rung, one rung up while that is below escalation_cap. Neither is an unfinished verdict, so the count of those restarts only with a step."""
|
||||
if escalated_at is None:
|
||||
def after_ruling(anchor: Rung, route: Route, verdict: Interrupt | QuotedAnswer) -> Route:
|
||||
"""The route after the judge's interrupt or quoted answer: unchanged without an escalation. With one, raised to the rung the highest L taken in picks, and one rung up from there while below escalation_cap, moving the mark as after_unfinished does. Neither is an unfinished verdict, so the count of those restarts only with a step."""
|
||||
mark = escalation_mark(verdict)
|
||||
if mark is None:
|
||||
return route
|
||||
rung = taken_in(anchor, route, bundled)
|
||||
return one_rung_up(route, rung, escalated_at) if below_cap(anchor, route, rung) else dataclasses.replace(route, rung=rung)
|
||||
rung = raised(anchor, route, verdict.highest_likelihood_taken_in)
|
||||
return one_rung_up(route, rung, mark) if below_cap(anchor, route, rung) else dataclasses.replace(route, rung=rung)
|
||||
|
||||
|
||||
def one_rung_up(route: Route, rung: Rung, escalated_at: int) -> Route:
|
||||
@@ -93,9 +97,9 @@ def below_cap(anchor: Rung, route: Route, rung: Rung) -> bool:
|
||||
return position(rung) < position(escalation_cap(anchor, route))
|
||||
|
||||
|
||||
def taken_in(anchor: Rung, route: Route, bundled: int) -> Rung:
|
||||
"""kept_rung, raised to the rung bundled picks from anchor; bundled is the highest L the judge saw the worker take into the chunk, 0 for none."""
|
||||
return highest(kept_rung(anchor, route), worker_rung(anchor, bundled)) if bundled else kept_rung(anchor, route)
|
||||
def raised(anchor: Rung, route: Route, likelihood_taken_in: int) -> Rung:
|
||||
"""kept_rung, raised to the rung that likelihood_taken_in picks from anchor. likelihood_taken_in is the highest L the worker took into the chunk, or 0 for none."""
|
||||
return highest(kept_rung(anchor, route), worker_rung(anchor, likelihood_taken_in)) if likelihood_taken_in else kept_rung(anchor, route)
|
||||
|
||||
|
||||
def judge_rung(judge_model: str, route: Route) -> Rung:
|
||||
|
||||
@@ -21,6 +21,10 @@ from test_env import isolated_env
|
||||
|
||||
BIN = pathlib.Path(__file__).resolve().parents[1]
|
||||
TASK = "Do the next chunk."
|
||||
# How the archive records a worker that runs off the ladder.
|
||||
OFF_LADDER = '{"model": "worker-model"}'
|
||||
# How a reload hands that worker's route.
|
||||
OFF_LADDER_ROUTE = {"rung": {"model": "worker-model", "effort": ""}, "item": 0, "likelihood": 0, "escape_and_impact": 0, "unfinished": 0, "escape_and_impact_taken_in": 0, "escalated_at": 0}
|
||||
|
||||
|
||||
# Plays claude from FAKE_CLAUDE_SCENARIO, one step per call, logging each call; a step for another role,
|
||||
@@ -129,8 +133,8 @@ def worker(**step):
|
||||
return {"role": "worker", **step}
|
||||
|
||||
|
||||
def judge(interrupt=False, finished=False, evidence="working", question="", bundled=0, bundled_escape_and_impact=0, escalation="", release_ready=False, **step):
|
||||
verdict = {"answer": "", "bundled": bundled, "bundled_escape_and_impact": bundled_escape_and_impact, "escalation": escalation, "evidence": evidence, "finished": finished, "interrupt": interrupt, "question": question, "release_ready": release_ready}
|
||||
def judge(interrupt=False, finished=False, evidence="working", question="", highest_likelihood_taken_in=0, highest_escape_and_impact_taken_in=0, escalation="", release_ready=False, **step):
|
||||
verdict = {"answer": "", "highest_likelihood_taken_in": highest_likelihood_taken_in, "highest_escape_and_impact_taken_in": highest_escape_and_impact_taken_in, "escalation": escalation, "evidence": evidence, "finished": finished, "interrupt": interrupt, "question": question, "release_ready": release_ready}
|
||||
return {"role": "judge", "verdict": verdict, **step}
|
||||
|
||||
|
||||
@@ -386,9 +390,9 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual([path.name for path in checkout.transcripts()[0].parent.iterdir()], [checkout.transcripts()[0].name])
|
||||
self.assertIn("=== cycle 3, worker on worker-model ===", run.stdout)
|
||||
copies = {verdict.removeprefix(f"Task: {TASK}\n\n"): copy for copy, verdict in checkout.archived().items()}
|
||||
self.assertEqual(sorted(copies), ["<UTC> Stopped: Ctrl-C\n\nWorker: worker-model\n", "<UTC> Verdict: finished\n\nEvidence:\n> landed-it\n\nJudge: judge-model\nWorker: worker-model\n", "<UTC> Verdict: unfinished\n\nEvidence:\n> started\n\nJudge: judge-model\nWorker: worker-model\n"])
|
||||
self.assertIn('"text": "landed-it"', copies["<UTC> Verdict: finished\n\nEvidence:\n> landed-it\n\nJudge: judge-model\nWorker: worker-model\n"])
|
||||
self.assertIn('"text": "next chunk"', copies["<UTC> Stopped: Ctrl-C\n\nWorker: worker-model\n"])
|
||||
self.assertEqual(sorted(copies), [f"<UTC> Stopped: Ctrl-C\n\nWorker: {OFF_LADDER}\n", f"<UTC> Verdict: finished\n\nEvidence:\n> landed-it\n\nJudge: judge-model\nWorker: {OFF_LADDER}\n", f"<UTC> Verdict: unfinished\n\nEvidence:\n> started\n\nJudge: judge-model\nWorker: {OFF_LADDER}\n"])
|
||||
self.assertIn('"text": "landed-it"', copies[f"<UTC> Verdict: finished\n\nEvidence:\n> landed-it\n\nJudge: judge-model\nWorker: {OFF_LADDER}\n"])
|
||||
self.assertIn('"text": "next chunk"', copies[f"<UTC> Stopped: Ctrl-C\n\nWorker: {OFF_LADDER}\n"])
|
||||
self.assertEqual(len(list(checkout.archive().glob(f"proj/*/session/{session}.orphaned-1-ab.jsonl"))), 1)
|
||||
|
||||
def test_routes_the_worker_by_its_items_l_and_steps_it_up_within_the_chunk(self):
|
||||
@@ -398,7 +402,7 @@ class Cycles(unittest.TestCase):
|
||||
run = checkout.run([
|
||||
worker(say="one"), judge(evidence="one"),
|
||||
worker(say="two"), judge(evidence="two"),
|
||||
worker(say="three"), judge(evidence="three", bundled=4),
|
||||
worker(say="three"), judge(evidence="three", highest_likelihood_taken_in=4),
|
||||
worker(say="four"), judge(finished=True, evidence="four"),
|
||||
worker(say="next chunk"),
|
||||
])
|
||||
@@ -414,8 +418,8 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual({(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] == "judge"}, {("judge-model", None)})
|
||||
self.assertIn("loop-claude runs you on claude-sonnet-5-5 at high effort, routed by item 4's L of 2. Name each todo.md item this chunk takes, with its L and E + I, whether or not it is item 4.", argument(workers[0], "-p"))
|
||||
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> three\n\nJudge: judge-model\nWorker: claude-opus-5-5 at medium effort, routed by item 4's L of 2 and E + I of 3\n\n<UTC> Stepped up\n\nWorker: claude-opus-5-5 at high effort, routed by item 4's L of 2 and E + I of 3\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: finished\n\nEvidence:\n> four\n\nJudge: judge-model\nWorker: claude-opus-5-5 at high effort, routed by item 4's L of 2 and E + I of 3\n", outcomes)
|
||||
self.assertIn('<UTC> Verdict: unfinished\n\nEvidence:\n> three\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "medium", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n\n<UTC> Stepped up\n\nWorker: {"model": "claude-opus-5-5", "effort": "high", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n', outcomes)
|
||||
self.assertIn('<UTC> Verdict: finished\n\nEvidence:\n> four\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "high", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n', outcomes)
|
||||
|
||||
def test_routes_on_and_states_the_same_item_from_the_fetched_default_branch_whatever_the_checkout_holds(self):
|
||||
checkout = Checkout(self)
|
||||
@@ -460,8 +464,8 @@ class Cycles(unittest.TestCase):
|
||||
("claude-opus-5-5", "high"),
|
||||
])
|
||||
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at high effort, routed by item 4's L of 4 and E + I of 5\n\n<UTC> Stepped up\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5, escalated at transcript entry 4\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5, escalated at transcript entry 4\n", outcomes)
|
||||
self.assertIn('<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "high", "item": 4, "likelihood": 4, "escape_and_impact": 5}\n\n<UTC> Stepped up\n\nWorker: {"model": "claude-opus-5-5", "effort": "xhigh", "item": 4, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 4}\n', outcomes)
|
||||
self.assertIn('<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "xhigh", "item": 4, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 4}\n', outcomes)
|
||||
|
||||
def test_steps_the_worker_up_once_as_the_judge_interrupts_when_the_run_also_cites_an_escalation_trigger(self):
|
||||
checkout = Checkout(self)
|
||||
@@ -477,9 +481,9 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual(run.roles(), ["worker", "judge", "interrupt", "judge", "worker", "judge"])
|
||||
self.assertEqual([(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] != "judge"], [("claude-opus-5-5", "high"), ("claude-opus-5-5", "xhigh"), ("claude-opus-5-5", "xhigh")])
|
||||
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
|
||||
self.assertIn("<UTC> Stepped up\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5, escalated at transcript entry 4\n", "".join(outcomes))
|
||||
self.assertIn('<UTC> Stepped up\n\nWorker: {"model": "claude-opus-5-5", "effort": "xhigh", "item": 4, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 4}\n', "".join(outcomes))
|
||||
self.assertNotIn("at max effort", "".join(outcomes))
|
||||
self.assertIn("Verdict: interrupt\n\nA or B?\n\nEvidence:\n> Shall I use A or B?\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at high effort, routed by item 4's L of 4 and E + I of 5\n\n<UTC> Stepped up\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5, escalated at transcript entry 4\n", "".join(outcomes))
|
||||
self.assertIn('Verdict: interrupt\n\nA or B?\n\nEvidence:\n> Shall I use A or B?\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "high", "item": 4, "likelihood": 4, "escape_and_impact": 5}\n\n<UTC> Stepped up\n\nWorker: {"model": "claude-opus-5-5", "effort": "xhigh", "item": 4, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 4}\n', "".join(outcomes))
|
||||
|
||||
def test_runs_a_cheap_items_worker_below_the_projects_rung_with_its_reviewers_and_judge_on_it(self):
|
||||
checkout = Checkout(self)
|
||||
@@ -491,7 +495,7 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual((argument(first, "--model"), argument(first, "--effort")), ("claude-sonnet-5-5", "high"))
|
||||
ruling = next(call for call in run.calls if call["role"] == "judge")
|
||||
self.assertEqual((argument(ruling, "--model"), argument(ruling, "--effort")), ("claude-opus-5-5", "medium"))
|
||||
self.assertIn("Judge: claude-opus-5-5 at medium effort\nWorker: claude-sonnet-5-5 at high effort", "".join(checkout.archived().values()))
|
||||
self.assertIn('Judge: claude-opus-5-5 at medium effort\nWorker: {"model": "claude-sonnet-5-5", "effort": "high", ', "".join(checkout.archived().values()))
|
||||
self.assertIn("Launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.", argument(first, "-p"))
|
||||
|
||||
def test_routes_the_judge_and_the_reviewers_on_the_highest_e_plus_i_the_worker_takes_in(self):
|
||||
@@ -499,7 +503,7 @@ class Cycles(unittest.TestCase):
|
||||
checkout.env["LOOP_CLAUDE_WORKER_MODEL"] = checkout.env["LOOP_CLAUDE_JUDGE_MODEL"] = "claude-opus-5-5"
|
||||
checkout.commit_plan(CHEAP)
|
||||
run = checkout.run([
|
||||
worker(say="one"), judge(evidence="one", bundled_escape_and_impact=6),
|
||||
worker(say="one"), judge(evidence="one", highest_escape_and_impact_taken_in=6),
|
||||
worker(say="two"), judge(finished=True, evidence="two"),
|
||||
worker(say="next chunk"),
|
||||
])
|
||||
@@ -509,8 +513,8 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual({argument(call, "--effort") for call in workers}, {"high"})
|
||||
self.assertIn("Launch each reviewer with the Agent tool set to model opus, effort xhigh, the rung picked by an item of E + I 6 taken into the chunk.", argument(workers[1], "-p"))
|
||||
outcomes = "".join(checkout.archived().values())
|
||||
self.assertIn("<UTC> Stepped up\n\nWorker: claude-sonnet-5-5 at high effort, routed by item 4's L of 2 and E + I of 3, taking in an E + I of 6\n", outcomes)
|
||||
self.assertIn("Judge: claude-opus-5-5 at xhigh effort\nWorker: claude-sonnet-5-5 at high effort, routed by item 4's L of 2 and E + I of 3, taking in an E + I of 6\n", outcomes)
|
||||
self.assertIn('<UTC> Stepped up\n\nWorker: {"model": "claude-sonnet-5-5", "effort": "high", "item": 4, "likelihood": 2, "escape_and_impact": 3, "escape_and_impact_taken_in": 6}\n', outcomes)
|
||||
self.assertIn('Judge: claude-opus-5-5 at xhigh effort\nWorker: {"model": "claude-sonnet-5-5", "effort": "high", "item": 4, "likelihood": 2, "escape_and_impact": 3, "escape_and_impact_taken_in": 6}\n', outcomes)
|
||||
|
||||
def test_a_rerun_resumes_an_open_chunk_on_the_rung_it_last_ran(self):
|
||||
checkout = Checkout(self)
|
||||
@@ -647,7 +651,7 @@ class Interrupt(unittest.TestCase):
|
||||
self.assertEqual(run.stderr.strip().splitlines()[-2:], ["A or B?", "the human is needed to answer the question above, but stdin or stdout is not a terminal; rerun loop-claude on a terminal"])
|
||||
[(copy, verdict)] = checkout.archived().items()
|
||||
self.assertEqual(copy, (checkout.transcripts()[0]).read_text())
|
||||
self.assertEqual(verdict, f"Task: {TASK}\n\n<UTC> Verdict: interrupt\n\nA or B?\n\nEvidence:\n> Shall I use A or B?\n\nJudge: judge-model\nWorker: worker-model\n\n<UTC> Stopped: the human is needed to answer the question above, but stdin or stdout is not a terminal; rerun loop-claude on a terminal\n\nWorker: worker-model\n")
|
||||
self.assertEqual(verdict, f"Task: {TASK}\n\n<UTC> Verdict: interrupt\n\nA or B?\n\nEvidence:\n> Shall I use A or B?\n\nJudge: judge-model\nWorker: {OFF_LADDER}\n\n<UTC> Stopped: the human is needed to answer the question above, but stdin or stdout is not a terminal; rerun loop-claude on a terminal\n\nWorker: {OFF_LADDER}\n")
|
||||
self.assertEqual(stat.S_IMODE((checkout.temp / "deep").stat().st_mode), 0o700)
|
||||
self.assertIn(f"transcript: {checkout.archive() / 'proj'}/", run.stderr)
|
||||
session = argument(run.calls[0], "--session-id")
|
||||
@@ -857,7 +861,7 @@ class Reload(unittest.TestCase):
|
||||
self.assertEqual(run.roles(), ["worker", "judge"] * 5)
|
||||
self.assertIn("keeping the running script", run.stderr)
|
||||
reloads = [json.loads(argv[1]) for argv in run.starts if argv[:1] == ["--reloaded"]]
|
||||
self.assertEqual(reloads, [{"cycle_number": cycle, "root": str(checkout.root), "task": TASK, "unfinished_evidence": "working", "route": None} for cycle in (2, 3, 4)])
|
||||
self.assertEqual(reloads, [{"cycle_number": cycle, "root": str(checkout.root), "task": TASK, "unfinished_evidence": "working", "route": OFF_LADDER_ROUTE} for cycle in (2, 3, 4)])
|
||||
self.assertIn("=== cycle 5,", run.stdout)
|
||||
|
||||
outside = checkout.temp / "outside"
|
||||
|
||||
+163
-127
@@ -23,7 +23,7 @@ from chunk_pick import DefaultPlan, LastCut
|
||||
from loop_claude.effects import Effects, PYTHON_TIMEOUT, run_python
|
||||
from claude_code import entries, reply, session
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished, verdict_from
|
||||
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished
|
||||
from loop_claude.cycle import interrupt, judge, prompts, retry, run, states, worker
|
||||
from model_routing import ladder
|
||||
from model_routing.ladder import LADDER, Rung
|
||||
@@ -32,9 +32,17 @@ from loop_claude.loop import main, reload, routing_check
|
||||
from transcript_archive import outcomes, write
|
||||
from loop_claude import paths, routing
|
||||
from transcript_archive.location import project_archive
|
||||
from model_routing.route import Route, route_from
|
||||
from model_routing.route import Record, Route, route_from
|
||||
from loop_claude.stop import Stop
|
||||
from utc_stamp import utc
|
||||
from loop_claude.cycle.judge import verdict_from
|
||||
|
||||
# How the archive records a worker that runs off the ladder.
|
||||
OFF_LADDER = '{"model": "worker-model"}'
|
||||
# How it records routes by item 179's L of 2 and E + I of 5, on Routing's rungs.
|
||||
SONNET_179 = '{"model": "claude-sonnet-5-5", "effort": "high", "item": 179, "likelihood": 2, "escape_and_impact": 5}'
|
||||
OPUS_179_AT_3 = '{"model": "claude-opus-5-5", "effort": "medium", "item": 179, "likelihood": 2, "escape_and_impact": 5, "escalated_at": 3}'
|
||||
OPUS_179_AT_6 = '{"model": "claude-opus-5-5", "effort": "medium", "item": 179, "likelihood": 2, "escape_and_impact": 5, "escalated_at": 6}'
|
||||
|
||||
CWD = pathlib.Path("/c")
|
||||
NOW = 1790000000
|
||||
@@ -108,7 +116,7 @@ def ran(stdout="", code=0, stderr=""):
|
||||
|
||||
|
||||
def judged(verdict, evidence="working", answer="", escalation="", release_ready=False):
|
||||
reply = {"answer": answer, "bundled": 0, "bundled_escape_and_impact": 0, "escalation": escalation, "evidence": evidence, "finished": verdict == "finished", "interrupt": verdict == "interrupt", "question": "Which one?" if verdict in ("interrupt", "quoted") else "", "release_ready": release_ready}
|
||||
reply = {"answer": answer, "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": escalation, "evidence": evidence, "finished": verdict == "finished", "interrupt": verdict == "interrupt", "question": "Which one?" if verdict in ("interrupt", "quoted") else "", "release_ready": release_ready}
|
||||
return ran(json.dumps({"structured_output": reply}))
|
||||
|
||||
|
||||
@@ -210,7 +218,7 @@ class Cycles(unittest.TestCase):
|
||||
def test_a_finished_chunk_is_archived_and_its_session_deleted(self):
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("finished")])
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Finished("working"))
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: finished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n"])
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: finished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: {OFF_LADDER}\n"])
|
||||
self.assertIsNone(session.transcript_file(self.session))
|
||||
|
||||
def test_a_fresh_worker_reads_the_preflights_facts_and_a_resumed_one_does_not(self):
|
||||
@@ -249,14 +257,14 @@ class Cycles(unittest.TestCase):
|
||||
self.assertIn("--session-id", workers[0])
|
||||
self.assertIn("--resume", workers[1])
|
||||
self.assertIn("as the work remaining:\n\n> working\n\nThe chunk is finished once", workers[1][workers[1].index("-p") + 1])
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: unfinished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n"] * 2)
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: unfinished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: {OFF_LADDER}\n"] * 2)
|
||||
|
||||
def test_a_cycle_stopped_by_a_failure_or_ctrl_c_archives_what_its_session_holds(self):
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[ran(code=1)] * 3)
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).code, 1)
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[KeyboardInterrupt()])
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)), Stop(130, ""))
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Stopped: Ctrl-C\n\nWorker: worker-model\n", "Task: Do it.\n\n2026-09-21T14:13:20Z Stopped: the judge failed three times running: the judge exited 1\n\nWorker: worker-model\n"])
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Stopped: Ctrl-C\n\nWorker: {OFF_LADDER}\n", f"Task: Do it.\n\n2026-09-21T14:13:20Z Stopped: the judge failed three times running: the judge exited 1\n\nWorker: {OFF_LADDER}\n"])
|
||||
|
||||
def test_each_judge_reply_the_loop_rejects_is_archived_with_why_beside_the_verdict_that_follows(self):
|
||||
unfilled = ran(json.dumps({"structured_output": {"interrupt": False}}))
|
||||
@@ -268,7 +276,7 @@ class Cycles(unittest.TestCase):
|
||||
"Task: Do it.\n\n"
|
||||
f"{stamp} Judge reply rejected: the judge's reply has no answer of type string\n\nReply:\n> {unfilled.stdout}\n\nJudge: judge-model\n\n"
|
||||
f"{stamp} Judge reply rejected: the judge quoted 'never said', which is not in the transcript\n\nReply:\n> {misquoted.stdout}\n\nJudge: judge-model\n\n"
|
||||
f"{stamp} Verdict: finished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n"])
|
||||
f"{stamp} Verdict: finished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: {OFF_LADDER}\n"])
|
||||
|
||||
def test_stops_when_a_rejected_reply_cannot_be_archived(self):
|
||||
self.archive.parent.mkdir(parents=True)
|
||||
@@ -332,7 +340,7 @@ class Cycles(unittest.TestCase):
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("interrupt"), judged("unfinished")], inputs=["/talk"], on_terminal_claude=[answer], git=[""])
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Unfinished("working", entries_read=4))
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: interrupt\n\nWhich one?\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n", "Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: unfinished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n"])
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: interrupt\n\nWhich one?\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: {OFF_LADDER}\n", f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: unfinished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: {OFF_LADDER}\n"])
|
||||
self.assertIn(str(self.session.id), fake.made("claude_on_terminal")[0])
|
||||
judge_prompts = [stdin for args, stdin in fake.made("claude")]
|
||||
self.assertEqual(len(judge_prompts), 2)
|
||||
@@ -351,8 +359,8 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual(len(fake.made("stream_claude")), 4)
|
||||
self.assertEqual(printed.getvalue().count("=== README.md's Goals and Audience, AGENTS.md, docs/decisions.md, ~/.claude/CLAUDE.md, ~/.claude/rules/ or ~/.claude/skills/larv-review/standards/ answered: Which one? ===\nUse *A*."), 3)
|
||||
self.assertIn("Which one?\n\nREADME.md's Goals and Audience, AGENTS.md, docs/decisions.md, ~/.claude/CLAUDE.md, ~/.claude/rules/ or ~/.claude/skills/larv-review/standards/ says:\nUse *A*.\n\nThose files have answered 3 questions this cycle", said.getvalue())
|
||||
quoted = "2026-09-21T14:13:20Z Verdict: quoted answer\n\nWhich one?\n\nREADME.md's Goals and Audience, AGENTS.md, docs/decisions.md, ~/.claude/CLAUDE.md, ~/.claude/rules/ or ~/.claude/skills/larv-review/standards/ says:\n> Use *A*.\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n"
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: interrupt\n\nWhich one?\n\nREADME.md's Goals and Audience, AGENTS.md, docs/decisions.md, ~/.claude/CLAUDE.md, ~/.claude/rules/ or ~/.claude/skills/larv-review/standards/ says:\nUse *A*.\n\nThose files have answered 3 questions this cycle, the most allowed, so this one comes to you.\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n\n2026-09-21T14:13:20Z Stopped: the human is needed to answer the question above, but nothing was typed\n\nWorker: worker-model\n", *[f"Task: Do it.\n\n{quoted}"] * 3])
|
||||
quoted = f"2026-09-21T14:13:20Z Verdict: quoted answer\n\nWhich one?\n\nREADME.md's Goals and Audience, AGENTS.md, docs/decisions.md, ~/.claude/CLAUDE.md, ~/.claude/rules/ or ~/.claude/skills/larv-review/standards/ says:\n> Use *A*.\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: {OFF_LADDER}\n"
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: interrupt\n\nWhich one?\n\nREADME.md's Goals and Audience, AGENTS.md, docs/decisions.md, ~/.claude/CLAUDE.md, ~/.claude/rules/ or ~/.claude/skills/larv-review/standards/ says:\nUse *A*.\n\nThose files have answered 3 questions this cycle, the most allowed, so this one comes to you.\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: {OFF_LADDER}\n\n2026-09-21T14:13:20Z Stopped: the human is needed to answer the question above, but nothing was typed\n\nWorker: {OFF_LADDER}\n", *[f"Task: Do it.\n\n{quoted}"] * 3])
|
||||
|
||||
def test_a_quoted_answer_that_cites_an_escalation_trigger_steps_the_worker_up_before_it_resumes_and_archives_the_step(self):
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)] * 2, claude=[judged("quoted", answer="Use A.", escalation="working"), judged("finished")])
|
||||
@@ -364,8 +372,8 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual([args[args.index("--effort") + 1] for args in (first, resumed)], ["high", "medium"])
|
||||
self.assertTrue(resumed[resumed.index("-p") + 1].endswith(f"Continue the task. Your request for a higher rung went through: loop-claude runs you on {Routing.OPUS}. The escalation triggers that fired before this step are spent.{prompts.rungs_held(concluded.route, Routing.OPUS)}"))
|
||||
self.assertNotIn("went through", first[first.index("-p") + 1])
|
||||
answered = f"Verdict: quoted answer\n\nWhich one?\n\n{prompts.QUOTE_SOURCES} says:\n> Use A.\n\nEvidence:\n> working\n\nEscalation:\n> working\n\nJudge: judge-model\nWorker: {Route(Routing.SONNET, 179, 2, 5)}\n"
|
||||
stepped = f"Stepped up\n\nWorker: {Route(Routing.OPUS, 179, 2, 5, escalated_at=3)}\n"
|
||||
answered = f"Verdict: quoted answer\n\nWhich one?\n\n{prompts.QUOTE_SOURCES} says:\n> Use A.\n\nEvidence:\n> working\n\nEscalation:\n> working\n\nJudge: judge-model\nWorker: {SONNET_179}\n"
|
||||
stepped = f"Stepped up\n\nWorker: {OPUS_179_AT_3}\n"
|
||||
kept = "".join(self.archived_outcomes())
|
||||
self.assertIn(f"Z {answered}\n20", kept)
|
||||
self.assertIn(f"Z {stepped}", kept)
|
||||
@@ -377,10 +385,10 @@ class Cycles(unittest.TestCase):
|
||||
with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertEqual(run.run_cycle(cycle), Stop(2, "the human is needed to answer the question above, but nothing was typed"))
|
||||
kept = "".join(self.archived_outcomes())
|
||||
self.assertIn(f"Evidence:\n> working\n\nEscalation:\n> working\n\nJudge: judge-model\nWorker: {Route(Routing.SONNET, 179, 2, 5)}\n", kept)
|
||||
self.assertIn(f"Stepped up\n\nWorker: {Route(Routing.OPUS, 179, 2, 5, escalated_at=6)}\n", kept)
|
||||
self.assertIn(f"Evidence:\n> working\n\nEscalation:\n> working\n\nJudge: judge-model\nWorker: {SONNET_179}\n", kept)
|
||||
self.assertIn(f"Stepped up\n\nWorker: {OPUS_179_AT_6}\n", kept)
|
||||
|
||||
def test_a_ruling_steps_the_route_only_on_an_escalation_from_the_rung_its_bundled_l_picks(self):
|
||||
def test_a_ruling_steps_the_route_only_on_an_escalation_from_the_rung_its_taken_in_l_picks(self):
|
||||
cycle = dataclasses.replace(self.cycle(FakeEffects()), anchor=Routing.OPUS, route=Route(Routing.OPUS, 3, 3, 4))
|
||||
for ruling in (Interrupt("e", "q", entries_read=3), QuotedAnswer("e", "q", "A.", entries_read=3)):
|
||||
with self.subTest(ruling=ruling):
|
||||
@@ -495,7 +503,7 @@ class Cycles(unittest.TestCase):
|
||||
stop = run.run_cycle(self.cycle(fake))
|
||||
self.assertEqual(stop, Stop(1, f"could not delete {folder}: Permission denied; delete every file and folder matching {folder}*, then rerun loop-claude; a rerun before that resumes the finished chunk's session"))
|
||||
self.assertTrue(session.session_open(self.session))
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: finished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n\n2026-09-21T14:13:20Z Stopped: {stop.reason}\n\nWorker: worker-model\n"])
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: finished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: {OFF_LADDER}\n\n2026-09-21T14:13:20Z Stopped: {stop.reason}\n\nWorker: {OFF_LADDER}\n"])
|
||||
|
||||
def test_stops_when_the_worker_leaves_no_transcript(self):
|
||||
fake = FakeEffects(streams=[Streamed(0)])
|
||||
@@ -736,6 +744,16 @@ TODO = """# todo
|
||||
"""
|
||||
|
||||
|
||||
def unfinished(likelihood=0, escalated_at=None):
|
||||
"""An unfinished verdict: the worker took in an item of that L, 0 for none, and the judge found an escalation after reading escalated_at transcript entries, None for none."""
|
||||
return Unfinished("e", escalation="round 2: 🔴" if escalated_at is not None else "", highest_likelihood_taken_in=likelihood, entries_read=escalated_at)
|
||||
|
||||
|
||||
def interrupted(likelihood=0, escalated_at=None):
|
||||
"""An interrupt, its arguments as unfinished's."""
|
||||
return Interrupt("e", "q", escalation="round 2: 🔴" if escalated_at is not None else "", highest_likelihood_taken_in=likelihood, entries_read=escalated_at)
|
||||
|
||||
|
||||
class Routing(unittest.TestCase):
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
HIGH = Rung("claude-opus-5-5", "high")
|
||||
@@ -774,47 +792,47 @@ class Routing(unittest.TestCase):
|
||||
self.assertEqual(routing.first_route(self.OPUS, None), Route(self.OPUS))
|
||||
|
||||
def test_steps_up_a_rung_after_every_second_unfinished_verdict(self):
|
||||
once = routing.stepped(self.OPUS, Route(self.OPUS, 3, 3, 4), 0, None)
|
||||
once = routing.after_unfinished(self.OPUS, Route(self.OPUS, 3, 3, 4), unfinished())
|
||||
self.assertEqual(once, Route(self.OPUS, 3, 3, 4, unfinished=1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, once, 0, None), Route(self.HIGH, 3, 3, 4))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(LADDER[-1], unfinished=1), 0, None).rung, LADDER[-1])
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, once, unfinished()), Route(self.HIGH, 3, 3, 4))
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(LADDER[-1], unfinished=1), unfinished()).rung, LADDER[-1])
|
||||
|
||||
def test_steps_up_to_the_rung_a_bundled_items_l_picks_and_never_down(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS), 4, None).rung, self.HIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 4, None).rung, self.XHIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS, unfinished=1), 4, None).rung, self.XHIGH)
|
||||
def test_steps_up_to_the_rung_a_taken_in_items_l_picks_and_never_down(self):
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(self.OPUS), unfinished(4)).rung, self.HIGH)
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(self.XHIGH), unfinished(4)).rung, self.XHIGH)
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(self.OPUS, unfinished=1), unfinished(4)).rung, self.XHIGH)
|
||||
|
||||
def test_steps_up_from_a_rung_below_the_projects_one_rung_at_a_time(self):
|
||||
low = Route(self.SONNET, 3, 1, 3, unfinished=1)
|
||||
self.assertEqual(routing.stepped(self.OPUS, low, 0, None).rung, self.OPUS)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 1, 3), 0, None).rung, low.rung)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 1, 3), 3, None).rung, self.OPUS)
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, low, unfinished()).rung, self.OPUS)
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(low.rung, 3, 1, 3), unfinished()).rung, low.rung)
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(low.rung, 3, 1, 3), unfinished(3)).rung, self.OPUS)
|
||||
|
||||
def test_steps_up_a_rung_on_an_escalation_the_judge_found_up_to_two_above_the_items_rung_marking_the_transcript_entries_it_read(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.HIGH, 3, 4, 5), 0, 7), Route(self.XHIGH, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH, 3, 4, 5, escalated_at=7), 0, 9), Route(self.MAX, 3, 4, 5, escalated_at=9))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.MAX, 3, 4, 5, escalated_at=9), 0, 12), Route(self.MAX, 3, 4, 5, unfinished=1, escalated_at=9))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1), 0, 7), Route(self.XHIGH, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 0, 7), Route(self.XHIGH, unfinished=1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.SONNET, 3, 2, 4), 4, 7), Route(self.HIGH, 3, 2, 4, unfinished=1))
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(self.HIGH, 3, 4, 5), unfinished(escalated_at=7)), Route(self.XHIGH, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(self.XHIGH, 3, 4, 5, escalated_at=7), unfinished(escalated_at=9)), Route(self.MAX, 3, 4, 5, escalated_at=9))
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(self.MAX, 3, 4, 5, escalated_at=9), unfinished(escalated_at=12)), Route(self.MAX, 3, 4, 5, unfinished=1, escalated_at=9))
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1), unfinished(escalated_at=7)), Route(self.XHIGH, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(self.XHIGH), unfinished(escalated_at=7)), Route(self.XHIGH, unfinished=1))
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(self.SONNET, 3, 2, 4), unfinished(4, escalated_at=7)), Route(self.HIGH, 3, 2, 4, unfinished=1))
|
||||
|
||||
def test_keeps_the_mark_through_a_step_no_escalation_took(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS, 3, 3, 4, unfinished=1, escalated_at=5), 0, None), Route(self.HIGH, 3, 3, 4, escalated_at=5))
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(self.OPUS, 3, 3, 4, unfinished=1, escalated_at=5), unfinished()), Route(self.HIGH, 3, 3, 4, escalated_at=5))
|
||||
|
||||
def test_steps_up_a_rung_as_the_judge_interrupts_up_to_two_above_the_items_rung_without_counting_an_unfinished_verdict(self):
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1), 0, 7), Route(self.XHIGH, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.XHIGH, 3, 4, 5, escalated_at=7), 0, 9), Route(self.MAX, 3, 4, 5, escalated_at=9))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.MAX, 3, 4, 5, unfinished=1, escalated_at=9), 0, 12), Route(self.MAX, 3, 4, 5, unfinished=1, escalated_at=9))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.SONNET, 3, 2, 4), 0, 7), Route(self.OPUS, 3, 2, 4, escalated_at=7))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1, escalated_at=2), 4, None), Route(self.HIGH, 3, 4, 5, unfinished=1, escalated_at=2))
|
||||
self.assertEqual(routing.after_ruling(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1), interrupted(escalated_at=7)), Route(self.XHIGH, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.after_ruling(self.OPUS, Route(self.XHIGH, 3, 4, 5, escalated_at=7), interrupted(escalated_at=9)), Route(self.MAX, 3, 4, 5, escalated_at=9))
|
||||
self.assertEqual(routing.after_ruling(self.OPUS, Route(self.MAX, 3, 4, 5, unfinished=1, escalated_at=9), interrupted(escalated_at=12)), Route(self.MAX, 3, 4, 5, unfinished=1, escalated_at=9))
|
||||
self.assertEqual(routing.after_ruling(self.OPUS, Route(self.SONNET, 3, 2, 4), interrupted(escalated_at=7)), Route(self.OPUS, 3, 2, 4, escalated_at=7))
|
||||
self.assertEqual(routing.after_ruling(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1, escalated_at=2), interrupted(4)), Route(self.HIGH, 3, 4, 5, unfinished=1, escalated_at=2))
|
||||
|
||||
def test_raises_the_worker_to_a_taken_in_items_rung_above_the_escalation_cap_without_a_step_as_the_judge_interrupts(self):
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.OPUS, 3, 1, 3, escalated_at=2), 4, 7), Route(self.HIGH, 3, 1, 3, escalated_at=2))
|
||||
self.assertEqual(routing.after_ruling(self.OPUS, Route(self.OPUS, 3, 1, 3, escalated_at=2), interrupted(4, escalated_at=7)), Route(self.HIGH, 3, 1, 3, escalated_at=2))
|
||||
|
||||
def test_steps_up_from_the_rung_a_taken_in_item_picks_as_the_judge_interrupts(self):
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.OPUS, 3, 3, 4), 4, 7), Route(self.XHIGH, 3, 3, 4, escalated_at=7))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.XHIGH, 3, 4, 5), 4, 7), Route(self.MAX, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.stepped_by_ruling(self.OPUS, Route(self.MAX, 3, 4, 5, unfinished=1), 4, 7), Route(self.MAX, 3, 4, 5, unfinished=1))
|
||||
self.assertEqual(routing.after_ruling(self.OPUS, Route(self.OPUS, 3, 3, 4), interrupted(4, escalated_at=7)), Route(self.XHIGH, 3, 3, 4, escalated_at=7))
|
||||
self.assertEqual(routing.after_ruling(self.OPUS, Route(self.XHIGH, 3, 4, 5), interrupted(4, escalated_at=7)), Route(self.MAX, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.after_ruling(self.OPUS, Route(self.MAX, 3, 4, 5, unfinished=1), interrupted(4, escalated_at=7)), Route(self.MAX, 3, 4, 5, unfinished=1))
|
||||
|
||||
def test_keeps_an_e_plus_i_taken_in_only_where_it_picks_a_higher_review_rung(self):
|
||||
taken = Route(self.OPUS, 3, 3, 4).taking_in(6)
|
||||
@@ -829,8 +847,8 @@ class Routing(unittest.TestCase):
|
||||
self.assertEqual(Route(self.OPUS).taking_in(0), Route(self.OPUS))
|
||||
|
||||
def test_a_rung_an_edit_took_off_the_ladder_counts_as_the_projects(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 0, None).rung, self.OPUS)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 2, None).rung, self.OPUS)
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(Rung("claude-opus-5-5", "low")), unfinished()).rung, self.OPUS)
|
||||
self.assertEqual(routing.after_unfinished(self.OPUS, Route(Rung("claude-opus-5-5", "low")), unfinished(2)).rung, self.OPUS)
|
||||
|
||||
def test_runs_the_judge_on_the_rung_its_chunks_items_e_plus_i_picks_never_below_the_projects(self):
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.OPUS, 3, 3, 5)), self.HIGH)
|
||||
@@ -864,46 +882,47 @@ class Routing(unittest.TestCase):
|
||||
without_i = TODO.replace("| E | I | R |", "| E | R |").replace("|---|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|---|").replace("| 4 | 3 | 5 | 10 |", "| 4 | 3 | 10 |").replace("| 4 | 2 | 3 | 7 |", "| 4 | 2 | 7 |").replace("| 1 | 1 | 1 | 1 |", "| 1 | 1 | 1 |")
|
||||
self.assertIsNone(routing.routed_item(without_i, NO_CUT))
|
||||
|
||||
def test_archives_the_route_in_a_line_it_reads_back(self):
|
||||
def test_archives_the_route_as_a_json_record_it_reads_back_field_by_field(self):
|
||||
route = Route(self.HIGH, 3, 4, 5)
|
||||
self.assertEqual(str(route), "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 5")
|
||||
self.assertEqual(route_from(str(route)), route)
|
||||
self.assertEqual(route_from(str(Route(self.HIGH))), Route(self.HIGH))
|
||||
self.assertEqual(str(Route(Rung("worker-model"))), "worker-model")
|
||||
self.assertIsNone(route_from("worker-model"))
|
||||
taken = Route(self.HIGH, 3, 4, 5, escape_and_impact_taken_in=7)
|
||||
self.assertEqual(str(taken), "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 5, taking in an E + I of 7")
|
||||
self.assertEqual(route_from(str(taken)), taken)
|
||||
unrouted = Route(self.HIGH, escape_and_impact_taken_in=6)
|
||||
self.assertEqual(str(unrouted), "claude-opus-5-5 at high effort, taking in an E + I of 6")
|
||||
self.assertEqual(route_from(str(unrouted)), unrouted)
|
||||
self.assertIsNone(route_from("claude-opus-5-5 at high effort, taking in an E + I of 9"))
|
||||
self.assertIsNone(route_from("claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 5, taking in an E + I of 5"))
|
||||
marked = Route(self.HIGH, 3, 4, 5, escalated_at=12, escape_and_impact_taken_in=7)
|
||||
self.assertEqual(str(marked), "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 5, taking in an E + I of 7, escalated at transcript entry 12")
|
||||
self.assertEqual(route_from(str(marked)), marked)
|
||||
self.assertEqual(route_from(str(Route(self.HIGH, escalated_at=3))), Route(self.HIGH, escalated_at=3))
|
||||
self.assertEqual(route.record(), '{"model": "claude-opus-5-5", "effort": "high", "item": 3, "likelihood": 4, "escape_and_impact": 5}')
|
||||
self.assertEqual(route_from(route.record()), route)
|
||||
self.assertEqual(Route(Rung("worker-model")).record(), '{"model": "worker-model"}')
|
||||
self.assertEqual(route_from('{"model": "worker-model"}'), Route(Rung("worker-model")))
|
||||
marked = Route(self.HIGH, 3, 4, 5, escape_and_impact_taken_in=7, escalated_at=12)
|
||||
self.assertEqual(marked.record(), '{"model": "claude-opus-5-5", "effort": "high", "item": 3, "likelihood": 4, "escape_and_impact": 5, "escape_and_impact_taken_in": 7, "escalated_at": 12}')
|
||||
self.assertEqual(route_from(marked.record()), marked)
|
||||
self.assertEqual(route_from(Route(self.HIGH, escalated_at=3).record()), Route(self.HIGH, escalated_at=3))
|
||||
|
||||
def test_reads_no_route_from_a_line_whose_levels_are_off_their_scales_or_an_r_plus_c_route_wrote(self):
|
||||
for line in ("claude-opus-5-5 at high effort, routed by item 3's L of 5 and E + I of 5", "claude-opus-5-5 at high effort, routed by item 3's L of 0 and E + I of 5", "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 9", "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 1", "claude-opus-5-5 at high effort, routed by item 3's R + C of 11"):
|
||||
def test_leaves_the_count_of_unfinished_verdicts_out_of_the_record_so_a_rerun_restarts_it(self):
|
||||
self.assertEqual(route_from(Route(self.HIGH, 3, 4, 5, unfinished=1).record()), Route(self.HIGH, 3, 4, 5))
|
||||
|
||||
def test_reads_no_route_from_a_record_of_unknown_or_mistyped_fields_or_levels_off_their_scales(self):
|
||||
for record in ('{"effort": "high"}', '{"model": "claude-opus-5-5", "effort": "high", "unfinished": 1}', '{"model": "claude-opus-5-5", "item": "3"}', '{"model": "claude-opus-5-5", "effort": 3}', '{"model": 5}', '{"model": "claude-opus-5-5", "effort": "high", "escalated_at": -3}', '{"model": "claude-opus-5-5", "item": true, "likelihood": 4, "escape_and_impact": 5}', '{"model": "claude-opus-5-5", "effort": "high", "item": 3, "likelihood": 5, "escape_and_impact": 5}', '{"model": "claude-opus-5-5", "effort": "high", "escape_and_impact_taken_in": 9}', '["claude-opus-5-5"]', '{"model": '):
|
||||
with self.subTest(record=record):
|
||||
self.assertIsNone(route_from(record))
|
||||
|
||||
def test_reads_the_route_from_a_line_loop_claude_wrote_before_it_archived_json(self):
|
||||
self.assertEqual(route_from("claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 5"), Route(self.HIGH, 3, 4, 5))
|
||||
self.assertEqual(route_from("claude-opus-5-5 at high effort"), Route(self.HIGH))
|
||||
self.assertEqual(route_from("claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 5, taking in an E + I of 7, escalated at transcript entry 12"), Route(self.HIGH, 3, 4, 5, escape_and_impact_taken_in=7, escalated_at=12))
|
||||
self.assertEqual(route_from("claude-opus-5-5 at high effort, taking in an E + I of 6"), Route(self.HIGH, escape_and_impact_taken_in=6))
|
||||
for line in ("worker-model", "claude-opus-5-5 at high effort, routed by item 3's L of 5 and E + I of 5", "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 5, taking in an E + I of 5", "claude-opus-5-5 at high effort, routed by item 3's R + C of 11"):
|
||||
with self.subTest(line=line):
|
||||
self.assertIsNone(route_from(line))
|
||||
|
||||
def test_keeps_a_chunks_route_or_its_archived_one_at_or_above_the_rung_its_item_picks_and_else_routes_afresh(self):
|
||||
def run_with(route=None):
|
||||
return main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=route)
|
||||
item = routing.routed_item(TODO, NO_CUT)
|
||||
archived = Route(self.XHIGH, 3, 4, 5)
|
||||
def never():
|
||||
raise AssertionError("not asked")
|
||||
self.assertIsNone(main.routed(run_with(archived), None, never, never).route)
|
||||
self.assertEqual(main.routed(run_with(archived), self.OPUS, never, never).route, archived)
|
||||
self.assertEqual(main.routed(run_with(), self.OPUS, lambda: archived, never).route, archived)
|
||||
self.assertEqual(main.routed(run_with(), self.OPUS, lambda: None, lambda: item).route, Route(self.HIGH, 3, 4, 5))
|
||||
self.assertEqual(main.routed(run_with(Route(self.SONNET)), self.OPUS, never, lambda: None).route, Route(self.OPUS))
|
||||
self.assertEqual(main.routed(archived, None, "worker-model", never, never), Route(Rung("worker-model")))
|
||||
self.assertEqual(main.routed(archived, self.OPUS, "claude-opus-5-5", never, never), archived)
|
||||
self.assertEqual(main.routed(None, self.OPUS, "claude-opus-5-5", lambda: archived, never), archived)
|
||||
self.assertEqual(main.routed(None, self.OPUS, "claude-opus-5-5", lambda: None, lambda: item), Route(self.HIGH, 3, 4, 5))
|
||||
self.assertEqual(main.routed(Route(self.SONNET), self.OPUS, "claude-opus-5-5", never, lambda: None), Route(self.OPUS))
|
||||
cheap = Route(self.SONNET, 4, 2, 3)
|
||||
self.assertEqual(main.routed(run_with(cheap), self.OPUS, never, never).route, cheap)
|
||||
self.assertEqual(main.routed(run_with(Route(Rung("claude-haiku-5-5", "max"), 4, 2, 3)), self.OPUS, never, lambda: None).route, Route(self.OPUS))
|
||||
self.assertEqual(main.routed(cheap, self.OPUS, "claude-opus-5-5", never, never), cheap)
|
||||
self.assertEqual(main.routed(Route(Rung("claude-haiku-5-5", "max"), 4, 2, 3), self.OPUS, "claude-opus-5-5", never, lambda: None), Route(self.OPUS))
|
||||
|
||||
|
||||
class RouteSources(unittest.TestCase):
|
||||
@@ -922,15 +941,23 @@ class RouteSources(unittest.TestCase):
|
||||
with tempfile.TemporaryDirectory() as home:
|
||||
opened = session.session_named("A proj loop-claude", pathlib.Path(home, "projects"))
|
||||
archive = pathlib.Path(home, "archive")
|
||||
self.assertIsNone(main.archived_route(archive, opened))
|
||||
transcript = pathlib.Path(home, "projects", "-fake", f"{opened.id}.jsonl")
|
||||
transcript.parent.mkdir(parents=True)
|
||||
transcript.write_text(json.dumps({"type": "queue-operation", "timestamp": "1"}) + "\n")
|
||||
self.assertIsNone(main.archived_route(archive, opened))
|
||||
route = Route(Rung("claude-opus-5-5", "high"), 3, 4, 5)
|
||||
with contextlib.redirect_stderr(io.StringIO()) as stderr:
|
||||
self.assertIsNone(main.archived_route(archive, opened))
|
||||
transcript.parent.mkdir(parents=True)
|
||||
transcript.write_text(json.dumps({"type": "queue-operation", "timestamp": "1"}) + "\n")
|
||||
self.assertIsNone(main.archived_route(archive, opened))
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(route), "working", "j", ""), NOW))
|
||||
self.assertEqual(main.archived_route(archive, opened), route)
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(route.record(), "working", "j", ""), NOW))
|
||||
with contextlib.redirect_stderr(stderr):
|
||||
self.assertEqual(main.archived_route(archive, opened), route)
|
||||
self.assertEqual(stderr.getvalue(), "")
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.stepped_up(Record('{"model": "claude-opus-5-5", "rung": 3}')), NOW + 1))
|
||||
with contextlib.redirect_stderr(io.StringIO()) as stderr:
|
||||
self.assertIsNone(main.archived_route(archive, opened))
|
||||
self.assertEqual(stderr.getvalue(), 'routing the chunk afresh: the archive\'s latest Worker: line names no route loop-claude can read: {"model": "claude-opus-5-5", "rung": 3}\n')
|
||||
(chunk,) = archive.iterdir()
|
||||
chunk.chmod(0)
|
||||
try:
|
||||
@@ -946,13 +973,23 @@ class Steps(unittest.TestCase):
|
||||
|
||||
def test_a_finished_verdict_clears_the_route_and_the_evidence(self):
|
||||
run_with = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="old", route=Route(self.OPUS))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Finished("x")), dataclasses.replace(run_with, route=None, unfinished_evidence=""))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Unfinished("new", 4)), dataclasses.replace(run_with, route=Route(Rung("claude-opus-5-5", "high"), unfinished=1), unfinished_evidence="new"))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, run.Concluded(Finished("x"), run_with.route)), dataclasses.replace(run_with, route=None, unfinished_evidence=""))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, run.Concluded(Unfinished("new", 4), run_with.route)), dataclasses.replace(run_with, route=Route(Rung("claude-opus-5-5", "high"), unfinished=1), unfinished_evidence="new"))
|
||||
|
||||
def test_steps_from_the_route_its_cycle_ended_on_which_a_ruling_may_have_raised(self):
|
||||
run_with = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=Route(self.OPUS, 3, 3, 4))
|
||||
raised = Route(Rung("claude-opus-5-5", "high"), 3, 3, 4, escalated_at=5)
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, run.Concluded(Unfinished("new"), raised)).route, dataclasses.replace(raised, unfinished=1))
|
||||
|
||||
def test_keeps_the_route_its_cycle_ended_on_off_the_ladder(self):
|
||||
off_ladder = Route(Rung("worker-model"))
|
||||
run_with = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="")
|
||||
self.assertEqual(main.after_verdict(run_with, None, run.Concluded(Unfinished("new", 4, "round 2: 🔴", entries_read=9), off_ladder)).route, off_ladder)
|
||||
|
||||
def test_an_unfinished_verdicts_escalation_marks_the_transcript_entries_the_judge_read(self):
|
||||
run_with = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=Route(self.OPUS, 3, 3, 4, escalated_at=2))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Unfinished("new", 0, "round 2: 🔴", entries_read=9)).route, Route(Rung("claude-opus-5-5", "high"), 3, 3, 4, escalated_at=9))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Unfinished("new", entries_read=9)).route, Route(self.OPUS, 3, 3, 4, unfinished=1, escalated_at=2))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, run.Concluded(Unfinished("new", 0, "round 2: 🔴", entries_read=9), run_with.route)).route, Route(Rung("claude-opus-5-5", "high"), 3, 3, 4, escalated_at=9))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, run.Concluded(Unfinished("new", entries_read=9), run_with.route)).route, Route(self.OPUS, 3, 3, 4, unfinished=1, escalated_at=2))
|
||||
|
||||
def test_archives_a_step_up_as_it_happens_so_a_rerun_resumes_on_it(self):
|
||||
with tempfile.TemporaryDirectory() as home:
|
||||
@@ -963,18 +1000,17 @@ class Steps(unittest.TestCase):
|
||||
transcript.write_text(json.dumps({"type": "queue-operation", "timestamp": "1"}) + "\n")
|
||||
before = Route(self.OPUS, 3, 3, 4)
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(before), "working", "j", ""), NOW))
|
||||
after = main.after_verdict(main.LoopRun(cycle_number=1, root="/r", task="Do it.", unfinished_evidence="", route=before), self.OPUS, Unfinished("working", 4))
|
||||
self.assertIsNone(main.archive_step(archive, opened, after, before, FakeEffects().effects()))
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(before.record(), "working", "j", ""), NOW))
|
||||
after = main.after_verdict(main.LoopRun(cycle_number=1, root="/r", task="Do it.", unfinished_evidence=""), self.OPUS, run.Concluded(Unfinished("working", 4), before))
|
||||
self.assertIsNone(main.archive_step(archive, opened, "Do it.", before, after.route, FakeEffects().effects()))
|
||||
self.assertEqual(main.archived_route(archive, opened), Route(Rung("claude-opus-5-5", "high"), 3, 3, 4))
|
||||
self.assertIsNone(main.archive_step(archive, opened, after, after.route, FakeEffects().effects()))
|
||||
self.assertIsNone(main.archive_step(archive, opened, after, None, FakeEffects().effects()))
|
||||
finished = main.after_verdict(after, self.OPUS, Finished("done"))
|
||||
self.assertIsNone(main.archive_step(archive, opened, finished, after.route, FakeEffects().effects()))
|
||||
self.assertIsNone(main.archive_step(archive, opened, "Do it.", after.route, after.route, FakeEffects().effects()))
|
||||
finished = main.after_verdict(after, self.OPUS, run.Concluded(Finished("done"), after.route))
|
||||
self.assertIsNone(main.archive_step(archive, opened, "Do it.", after.route, finished.route, FakeEffects().effects()))
|
||||
self.assertEqual(main.archived_route(archive, opened), Route(Rung("claude-opus-5-5", "high"), 3, 3, 4))
|
||||
blocked = pathlib.Path(home, "blocked")
|
||||
blocked.write_text("")
|
||||
stop = main.archive_step(blocked, opened, after, before, FakeEffects().effects())
|
||||
stop = main.archive_step(blocked, opened, "Do it.", before, after.route, FakeEffects().effects())
|
||||
self.assertEqual((stop.code, stop.reason.split(":")[0]), (1, "could not archive the transcript"))
|
||||
|
||||
def test_archives_a_higher_e_plus_i_taken_in_so_a_rerun_keeps_it(self):
|
||||
@@ -986,9 +1022,9 @@ class Steps(unittest.TestCase):
|
||||
transcript.write_text(json.dumps({"type": "queue-operation", "timestamp": "1"}) + "\n")
|
||||
before = Route(self.OPUS, 3, 3, 4)
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(before), "working", "j", ""), NOW))
|
||||
after = main.after_verdict(main.LoopRun(cycle_number=1, root="/r", task="Do it.", unfinished_evidence="", route=before), self.OPUS, Unfinished("working", 0, bundled_escape_and_impact=6))
|
||||
self.assertIsNone(main.archive_step(archive, opened, after, before, FakeEffects().effects()))
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(before.record(), "working", "j", ""), NOW))
|
||||
after = main.after_verdict(main.LoopRun(cycle_number=1, root="/r", task="Do it.", unfinished_evidence=""), self.OPUS, run.Concluded(Unfinished("working", highest_escape_and_impact_taken_in=6), before))
|
||||
self.assertIsNone(main.archive_step(archive, opened, "Do it.", before, after.route, FakeEffects().effects()))
|
||||
self.assertEqual(main.archived_route(archive, opened), Route(self.OPUS, 3, 3, 4, escape_and_impact_taken_in=6))
|
||||
|
||||
|
||||
@@ -1297,48 +1333,48 @@ class Grounded(unittest.TestCase):
|
||||
|
||||
class Verdict(unittest.TestCase):
|
||||
def test_needs_every_field_the_schema_requires(self):
|
||||
self.assertEqual(verdict_from({"bundled": 0, "bundled_escape_and_impact": 0, "escalation": "", "answer": "", "interrupt": False, "evidence": "e", "finished": True, "question": "", "release_ready": False}), Finished("e"))
|
||||
self.assertEqual(verdict_from({"bundled": 0, "bundled_escape_and_impact": 0, "escalation": "", "answer": "", "interrupt": False, "evidence": "e", "finished": False, "question": "ignored", "release_ready": False}), Unfinished("e"))
|
||||
self.assertEqual(verdict_from({"bundled": 0, "bundled_escape_and_impact": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "e", "finished": True, "question": "Which one?", "release_ready": False}), Interrupt("e", "Which one?"))
|
||||
self.assertEqual(verdict_from({"bundled": 0, "bundled_escape_and_impact": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "e", "finished": False, "question": "Cut 0.2.0?", "release_ready": True}), Interrupt("e", "Cut 0.2.0?", release_ready=True))
|
||||
self.assertEqual(verdict_from({"bundled": 0, "bundled_escape_and_impact": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "e", "finished": False, "question": "Cut 0.2.0?"}), Failure("the judge's reply has no release_ready of type boolean"))
|
||||
self.assertEqual(verdict_from({"bundled": 0, "bundled_escape_and_impact": 0, "escalation": "", "answer": "A.", "interrupt": False, "evidence": "e", "finished": False, "question": "Which one?", "release_ready": False}), QuotedAnswer("e", "Which one?", "A."))
|
||||
self.assertEqual(verdict_from({"bundled": 0, "bundled_escape_and_impact": 0, "escalation": "", "answer": "A.", "interrupt": True, "evidence": "e", "finished": False, "question": "Which one?", "release_ready": False}), Interrupt("e", "Which one?"))
|
||||
self.assertEqual(verdict_from({"bundled": 0, "bundled_escape_and_impact": 0, "escalation": "", "answer": "A.", "interrupt": False, "evidence": "e", "finished": True, "question": "", "release_ready": False}), Failure("the judge quoted an answer without a question"))
|
||||
self.assertEqual(verdict_from({"bundled": 0, "bundled_escape_and_impact": 0, "escalation": "", "answer": "", "interrupt": "no", "evidence": "", "finished": True, "question": "", "release_ready": False}), Failure("the judge's reply has no interrupt of type boolean"))
|
||||
self.assertEqual(verdict_from({"bundled": 0, "bundled_escape_and_impact": 0, "escalation": "", "answer": "", "interrupt": False, "evidence": None, "finished": True, "question": "", "release_ready": False}), Failure("the judge's reply has no evidence of type string"))
|
||||
self.assertEqual(verdict_from({"bundled": 0, "bundled_escape_and_impact": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "x", "finished": False, "question": " ", "release_ready": False}), Failure("the judge interrupted without a question"))
|
||||
self.assertEqual(verdict_from({"highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": False, "evidence": "e", "finished": True, "question": "", "release_ready": False}), Finished("e"))
|
||||
self.assertEqual(verdict_from({"highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": False, "evidence": "e", "finished": False, "question": "ignored", "release_ready": False}), Unfinished("e"))
|
||||
self.assertEqual(verdict_from({"highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "e", "finished": True, "question": "Which one?", "release_ready": False}), Interrupt("e", "Which one?"))
|
||||
self.assertEqual(verdict_from({"highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "e", "finished": False, "question": "Cut 0.2.0?", "release_ready": True}), Interrupt("e", "Cut 0.2.0?", release_ready=True))
|
||||
self.assertEqual(verdict_from({"highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "e", "finished": False, "question": "Cut 0.2.0?"}), Failure("the judge's reply has no release_ready of type boolean"))
|
||||
self.assertEqual(verdict_from({"highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "A.", "interrupt": False, "evidence": "e", "finished": False, "question": "Which one?", "release_ready": False}), QuotedAnswer("e", "Which one?", "A."))
|
||||
self.assertEqual(verdict_from({"highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "A.", "interrupt": True, "evidence": "e", "finished": False, "question": "Which one?", "release_ready": False}), Interrupt("e", "Which one?"))
|
||||
self.assertEqual(verdict_from({"highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "A.", "interrupt": False, "evidence": "e", "finished": True, "question": "", "release_ready": False}), Failure("the judge quoted an answer without a question"))
|
||||
self.assertEqual(verdict_from({"highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": "no", "evidence": "", "finished": True, "question": "", "release_ready": False}), Failure("the judge's reply has no interrupt of type boolean"))
|
||||
self.assertEqual(verdict_from({"highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": False, "evidence": None, "finished": True, "question": "", "release_ready": False}), Failure("the judge's reply has no evidence of type string"))
|
||||
self.assertEqual(verdict_from({"highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "x", "finished": False, "question": " ", "release_ready": False}), Failure("the judge interrupted without a question"))
|
||||
self.assertEqual(verdict_from(None), Failure("the judge returned no reply"))
|
||||
|
||||
def test_takes_a_bundled_l_on_its_scale_and_counts_any_other_as_none(self):
|
||||
reply = {"answer": "", "bundled_escape_and_impact": 0, "escalation": "", "interrupt": False, "evidence": "e", "finished": False, "question": "", "release_ready": False}
|
||||
self.assertEqual(verdict_from({**reply, "bundled": 4}), Unfinished("e", 4))
|
||||
self.assertEqual(verdict_from({**reply, "bundled": True}), Failure("the judge's reply has no bundled of type integer"))
|
||||
for bundled in (5, 13, -1):
|
||||
with self.subTest(bundled=bundled):
|
||||
self.assertEqual(verdict_from({**reply, "bundled": bundled}), Unfinished("e", 0))
|
||||
def test_takes_the_highest_l_taken_in_on_its_scale_and_counts_any_other_as_none(self):
|
||||
reply = {"answer": "", "highest_escape_and_impact_taken_in": 0, "escalation": "", "interrupt": False, "evidence": "e", "finished": False, "question": "", "release_ready": False}
|
||||
self.assertEqual(verdict_from({**reply, "highest_likelihood_taken_in": 4}), Unfinished("e", 4))
|
||||
self.assertEqual(verdict_from({**reply, "highest_likelihood_taken_in": True}), Failure("the judge's reply has no highest_likelihood_taken_in of type integer"))
|
||||
for highest_likelihood_taken_in in (5, 13, -1):
|
||||
with self.subTest(highest_likelihood_taken_in=highest_likelihood_taken_in):
|
||||
self.assertEqual(verdict_from({**reply, "highest_likelihood_taken_in": highest_likelihood_taken_in}), Unfinished("e", 0))
|
||||
|
||||
|
||||
def test_takes_a_bundled_e_plus_i_on_its_scale_and_counts_any_other_as_none(self):
|
||||
reply = {"answer": "", "bundled": 0, "escalation": "", "interrupt": False, "evidence": "e", "finished": False, "question": "", "release_ready": False}
|
||||
self.assertEqual(verdict_from({**reply, "bundled_escape_and_impact": 6}), Unfinished("e", 0, bundled_escape_and_impact=6))
|
||||
self.assertEqual(verdict_from({**reply, "bundled_escape_and_impact": 2}), Unfinished("e", 0, bundled_escape_and_impact=2))
|
||||
self.assertEqual(verdict_from({**reply, "bundled_escape_and_impact": False}), Failure("the judge's reply has no bundled_escape_and_impact of type integer"))
|
||||
for bundled in (1, 9, -1):
|
||||
with self.subTest(bundled=bundled):
|
||||
self.assertEqual(verdict_from({**reply, "bundled_escape_and_impact": bundled}), Unfinished("e", 0))
|
||||
def test_takes_the_highest_e_plus_i_taken_in_on_its_scale_and_counts_any_other_as_none(self):
|
||||
reply = {"answer": "", "highest_likelihood_taken_in": 0, "escalation": "", "interrupt": False, "evidence": "e", "finished": False, "question": "", "release_ready": False}
|
||||
self.assertEqual(verdict_from({**reply, "highest_escape_and_impact_taken_in": 6}), Unfinished("e", 0, highest_escape_and_impact_taken_in=6))
|
||||
self.assertEqual(verdict_from({**reply, "highest_escape_and_impact_taken_in": 2}), Unfinished("e", 0, highest_escape_and_impact_taken_in=2))
|
||||
self.assertEqual(verdict_from({**reply, "highest_escape_and_impact_taken_in": False}), Failure("the judge's reply has no highest_escape_and_impact_taken_in of type integer"))
|
||||
for escape_and_impact in (1, 9, -1):
|
||||
with self.subTest(escape_and_impact=escape_and_impact):
|
||||
self.assertEqual(verdict_from({**reply, "highest_escape_and_impact_taken_in": escape_and_impact}), Unfinished("e", 0))
|
||||
|
||||
def test_takes_an_escalation_quote_on_an_unfinished_verdict(self):
|
||||
reply = {"answer": "", "bundled": 0, "bundled_escape_and_impact": 0, "escalation": "round 2: 🔴", "interrupt": False, "evidence": "e", "finished": False, "question": "", "release_ready": False}
|
||||
reply = {"answer": "", "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "round 2: 🔴", "interrupt": False, "evidence": "e", "finished": False, "question": "", "release_ready": False}
|
||||
self.assertEqual(verdict_from(reply), Unfinished("e", 0, "round 2: 🔴"))
|
||||
self.assertEqual(verdict_from({**reply, "finished": True}), Finished("e"))
|
||||
self.assertEqual(verdict_from({**reply, "escalation": None}), Failure("the judge's reply has no escalation of type string"))
|
||||
|
||||
def test_takes_an_escalation_quote_on_an_interrupt(self):
|
||||
reply = {"answer": "", "bundled": 0, "bundled_escape_and_impact": 0, "escalation": "round 2: 🔴", "interrupt": True, "evidence": "e", "finished": False, "question": "Which one?", "release_ready": False}
|
||||
reply = {"answer": "", "highest_likelihood_taken_in": 0, "highest_escape_and_impact_taken_in": 0, "escalation": "round 2: 🔴", "interrupt": True, "evidence": "e", "finished": False, "question": "Which one?", "release_ready": False}
|
||||
self.assertEqual(verdict_from(reply), Interrupt("e", "Which one?", "round 2: 🔴"))
|
||||
self.assertEqual(verdict_from({**reply, "bundled": 4}), Interrupt("e", "Which one?", "round 2: 🔴", 4))
|
||||
self.assertEqual(verdict_from({**reply, "bundled": 5}), Interrupt("e", "Which one?", "round 2: 🔴"))
|
||||
self.assertEqual(verdict_from({**reply, "highest_likelihood_taken_in": 4}), Interrupt("e", "Which one?", "round 2: 🔴", 4))
|
||||
self.assertEqual(verdict_from({**reply, "highest_likelihood_taken_in": 5}), Interrupt("e", "Which one?", "round 2: 🔴"))
|
||||
|
||||
def test_drops_an_escalation_quote_the_transcript_does_not_hold_word_for_word_from_an_interrupt_or_an_unfinished_verdict(self):
|
||||
shown = "The agent: round 2: 🔴. Which one?"
|
||||
@@ -1346,10 +1382,10 @@ class Verdict(unittest.TestCase):
|
||||
self.assertEqual(judge.quotes_checked(Interrupt("e", "q", "round 2: … 🔴"), judge.Grounds(shown, shown, "")), Interrupt("e", "q"))
|
||||
self.assertEqual(judge.quotes_checked(Unfinished("e", 0, "round 3: 🔴"), judge.Grounds(shown, shown, "")), Unfinished("e"))
|
||||
|
||||
def test_takes_an_escalation_quote_and_a_bundled_l_on_a_quoted_answer(self):
|
||||
reply = {"answer": "A.", "bundled": 4, "bundled_escape_and_impact": 0, "escalation": "round 2: 🔴", "interrupt": False, "evidence": "e", "finished": False, "question": "Which one?", "release_ready": False}
|
||||
def test_takes_an_escalation_quote_and_the_highest_l_taken_in_on_a_quoted_answer(self):
|
||||
reply = {"answer": "A.", "highest_likelihood_taken_in": 4, "highest_escape_and_impact_taken_in": 0, "escalation": "round 2: 🔴", "interrupt": False, "evidence": "e", "finished": False, "question": "Which one?", "release_ready": False}
|
||||
self.assertEqual(verdict_from(reply), QuotedAnswer("e", "Which one?", "A.", "round 2: 🔴", 4))
|
||||
self.assertEqual(verdict_from({**reply, "bundled": 5}), QuotedAnswer("e", "Which one?", "A.", "round 2: 🔴"))
|
||||
self.assertEqual(verdict_from({**reply, "highest_likelihood_taken_in": 5}), QuotedAnswer("e", "Which one?", "A.", "round 2: 🔴"))
|
||||
|
||||
def test_drops_an_escalation_quote_the_transcript_does_not_hold_word_for_word_from_a_quoted_answer(self):
|
||||
shown = "The agent: round 2: 🔴. Which one?"
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
"""The verdicts cycle.judge.verdict_from builds from the judge's reply. They sit outside cycle/, since routing reads them and cycle imports routing."""
|
||||
import dataclasses
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Interrupt:
|
||||
evidence: str
|
||||
question: str
|
||||
# The judge's quote showing that the worker's cited escalation trigger fired; empty for none.
|
||||
escalation: str = ""
|
||||
# The highest L the judge saw the worker take into the chunk, 0 for none: an escalation steps up from the rung that L picks.
|
||||
highest_likelihood_taken_in: int = 0
|
||||
release_ready: bool = False
|
||||
# How many transcript entries the judge read, which judge_once stamps. An escalation the verdict carries moves the route's mark there. None means judge_once never stamped it, and then its escalation doesn't step the worker.
|
||||
entries_read: int | None = None
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class QuotedAnswer:
|
||||
"""The worker asked a question that README.md or the records answer. The answer field quotes the passage."""
|
||||
evidence: str
|
||||
question: str
|
||||
answer: str
|
||||
# As on Interrupt.
|
||||
escalation: str = ""
|
||||
highest_likelihood_taken_in: int = 0
|
||||
entries_read: int | None = None
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Finished:
|
||||
evidence: str
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Unfinished:
|
||||
evidence: str
|
||||
# As on Interrupt; the worker also rises to the rung that L picks.
|
||||
highest_likelihood_taken_in: int = 0
|
||||
# As on Interrupt.
|
||||
escalation: str = ""
|
||||
# The highest E + I the judge saw the worker take into the chunk, 0 for none.
|
||||
highest_escape_and_impact_taken_in: int = 0
|
||||
# As on Interrupt.
|
||||
entries_read: int | None = None
|
||||
|
||||
|
||||
Verdict = Interrupt | QuotedAnswer | Finished | Unfinished
|
||||
|
||||
|
||||
def escalation_mark(verdict: Interrupt | QuotedAnswer | Unfinished) -> int | None:
|
||||
"""Where the verdict's escalation would move the route's mark, if it steps the worker; None where it carries none."""
|
||||
return verdict.entries_read if verdict.escalation else None
|
||||
@@ -36,7 +36,7 @@ class Rung:
|
||||
return Rung(model, effort)
|
||||
return Rung(family.model, effort) if effort else family
|
||||
|
||||
# The one spelling of a rung. It is persisted, so changing it routes every stored chunk afresh: parse no longer reads their rungs.
|
||||
# The one spelling of a rung. loop-claude hands it to rule-hook in environment variables, and archive lines written before route.Route.record hold it, so parse must keep reading it.
|
||||
def __str__(self) -> str:
|
||||
return f"{self.model} at {self.effort} effort" if self.effort else self.model
|
||||
|
||||
|
||||
+40
-38
@@ -1,5 +1,6 @@
|
||||
"""A chunk's route: its worker's rung and the plan item that picked it, in the words the archive records each verdict's worker in."""
|
||||
"""A chunk's route: its worker's rung and the plan item that picked it. The archive keeps the route as a record beside each verdict."""
|
||||
import dataclasses
|
||||
import json
|
||||
import re
|
||||
import typing
|
||||
|
||||
@@ -8,24 +9,10 @@ from model_routing.ladder import REVIEW_STEPS, Rung
|
||||
|
||||
# How a worker opens the prompt of a subagent it escalates a task to, `Escalation trigger 2: …`: docs/decisions.md → `routing: a worker hands a task one rung up`.
|
||||
ESCALATION_MARKER = "Escalation trigger"
|
||||
# An archived worker line. A line an older loop-claude wrote names an R + C in place of L and E + I: archived_worker reads only its rung, for the routing figures; with no levels, route_from returns None and the chunk routes afresh.
|
||||
# Each group reads one fragment of Route.__str__. A reworded fragment leaves the line unreadable, so the chunk routes afresh, its mark back at 0.
|
||||
ROUTED = re.compile(r"""
|
||||
(?P<rung>.+?) # the rung, lazy so the groups below take what follows it
|
||||
(?:,\ routed\ by\ item\ (?P<item>[0-9]+)'s\ (?:L\ of\ (?P<likelihood>[0-9]+)\ and\ E\ \+\ I\ of\ (?P<escape_and_impact>[0-9]+)|.+?))? # routed: `, routed by item 3's L of 4 and E + I of 5`, or an older line's R + C
|
||||
(?:,\ taking\ in\ an\ E\ \+\ I\ of\ (?P<escape_and_impact_taken_in>[0-9]+))? # taken_in: `, taking in an E + I of 6`
|
||||
(?:,\ escalated\ at\ transcript\ entry\ (?P<escalated_at>[0-9]+))? # escalated: `, escalated at transcript entry 12`
|
||||
""", re.VERBOSE)
|
||||
|
||||
|
||||
class ArchivedWorker(typing.NamedTuple):
|
||||
"""What an archived worker line names: the worker's rung, the item that routed it, its L and E + I, the E + I taken in and the escalation's mark; 0 for each it does not name."""
|
||||
rung: Rung
|
||||
item: int
|
||||
likelihood: int
|
||||
escape_and_impact: int
|
||||
escape_and_impact_taken_in: int
|
||||
escalated_at: int
|
||||
# What Route.record writes. The outcome writers take only this as the worker, so mypy refuses a str(route) there.
|
||||
Record = typing.NewType("Record", str)
|
||||
# A worker line loop-claude wrote before it archived the route as a record. Nothing writes this format any more, so its wording never changes. route_from reads it for the chunks the routing figures still count, and for a chunk a rerun resumes across the change.
|
||||
LEGACY_WORKER = re.compile(r"(?P<rung>.+?)(?:, routed by item (?P<item>[0-9]+)'s L of (?P<likelihood>[0-9]+) and E \+ I of (?P<escape_and_impact>[0-9]+))?(?:, taking in an E \+ I of (?P<escape_and_impact_taken_in>[0-9]+))?(?:, escalated at transcript entry (?P<escalated_at>[0-9]+))?")
|
||||
|
||||
|
||||
HIGHEST_ESCAPE_AND_IMPACT = LEVELS["E"] + LEVELS["I"]
|
||||
@@ -53,6 +40,7 @@ class Route:
|
||||
|
||||
unfinished counts the unfinished verdicts towards the next one-rung step; a reload hands it over with the rung.
|
||||
"""
|
||||
# Each field below but rung and unfinished is a key of the archive's records, so renaming one makes the records already archived unreadable, and their chunks route afresh.
|
||||
# The fields a loop-claude from before L and E + I replaced R + C hands on reload; the reloaded script drops such a route, and the chunk routes afresh.
|
||||
RETIRED: typing.ClassVar[frozenset[str]] = frozenset({"risk_and_complexity"})
|
||||
rung: Rung
|
||||
@@ -66,13 +54,13 @@ class Route:
|
||||
escalated_at: int = 0
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Raises on a programming error only, so nothing catches it: route_from and the reload decoder check levels_fit before they build a route.
|
||||
# Raises on a programming error only, so nothing catches it: route_from and the reload decoder check fits before they build a route.
|
||||
if not levels_fit(self.item, self.likelihood, self.escape_and_impact, self.escape_and_impact_taken_in):
|
||||
raise ValueError(f"a route by item {self.item} cannot hold an L of {self.likelihood}, an E + I of {self.escape_and_impact} and an E + I taken in of {self.escape_and_impact_taken_in}")
|
||||
|
||||
@staticmethod
|
||||
def fits(fields: typing.Mapping[str, typing.Any]) -> bool:
|
||||
"""Whether a route may be built from these fields, as the reload decoder asks of a handed one."""
|
||||
"""Whether a route may be built from these fields."""
|
||||
return levels_fit(fields.get("item", 0), fields.get("likelihood", 0), fields.get("escape_and_impact", 0), fields.get("escape_and_impact_taken_in", 0))
|
||||
|
||||
def taking_in(self, escape_and_impact: int) -> "Route":
|
||||
@@ -83,26 +71,40 @@ class Route:
|
||||
"""The E + I that picks the reviewers' and the judge's rung: the routed item's or the highest taken in, whichever is higher; 0 for neither."""
|
||||
return max(self.escape_and_impact, self.escape_and_impact_taken_in)
|
||||
|
||||
def __str__(self) -> str:
|
||||
"""What the archive records as each verdict's worker; ROUTED reads it back."""
|
||||
routed = f", routed by item {self.item}'s L of {self.likelihood} and E + I of {self.escape_and_impact}" if self.item else ""
|
||||
taken_in = f", taking in an E + I of {self.escape_and_impact_taken_in}" if self.escape_and_impact_taken_in else ""
|
||||
escalated = f", escalated at transcript entry {self.escalated_at}" if self.escalated_at else ""
|
||||
return f"{self.rung}{routed}{taken_in}{escalated}"
|
||||
def record(self) -> Record:
|
||||
"""What the archive records as each verdict's worker: a JSON object of the rung's model and effort and the RECORD_FIELDS, leaving out each that is 0 or empty; route_from reads it back."""
|
||||
fields = {"model": self.rung.model, "effort": self.rung.effort, **{name: getattr(self, name) for name in RECORD_FIELDS}}
|
||||
return Record(json.dumps({name: value for name, value in fields.items() if value}))
|
||||
|
||||
|
||||
def archived_worker(worker: str) -> ArchivedWorker | None:
|
||||
"""What an archived worker line names; None for a model run off the ladder."""
|
||||
found = ROUTED.fullmatch(worker)
|
||||
rung = Rung.parse(found["rung"]) if found else None
|
||||
if not (found and rung):
|
||||
return None
|
||||
return ArchivedWorker(rung, *(int(found[name] or 0) for name in ("item", "likelihood", "escape_and_impact", "escape_and_impact_taken_in", "escalated_at")))
|
||||
# The Route fields a record holds beside the rung's model and effort. A human reads the record in the archive, so it is flatter than the JSON a reload hands over (loop_claude/loop/reload.py) and leaves out zeros. It leaves out unfinished, so a rerun restarts that count.
|
||||
RECORD_FIELDS = tuple(field.name for field in dataclasses.fields(Route) if field.name not in ("rung", "unfinished"))
|
||||
|
||||
|
||||
def route_from(worker: str) -> Route | None:
|
||||
"""The route an archived worker line names; None for a model run off the ladder, or a route by anything but levels on their scales."""
|
||||
archived = archived_worker(worker)
|
||||
if archived is None or not levels_fit(archived.item, archived.likelihood, archived.escape_and_impact, archived.escape_and_impact_taken_in):
|
||||
"""The route an archived worker line names, whether a record or a line written before records. None where it names no model, holds fields Route.record never writes, or holds levels off their scales; and for an older line, where it names a model without an effort."""
|
||||
fields = record_fields(worker) if worker.startswith("{") else legacy_fields(worker)
|
||||
if fields is None or not Route.fits(fields):
|
||||
return None
|
||||
return Route(archived.rung, archived.item, archived.likelihood, archived.escape_and_impact, escape_and_impact_taken_in=archived.escape_and_impact_taken_in, escalated_at=archived.escalated_at)
|
||||
return Route(Rung(fields["model"], fields.get("effort", "")), **{name: fields.get(name, 0) for name in RECORD_FIELDS})
|
||||
|
||||
|
||||
def record_fields(record: str) -> dict[str, typing.Any] | None:
|
||||
"""The fields of a record Route.record wrote; None for any other text."""
|
||||
try:
|
||||
fields = json.loads(record)
|
||||
except ValueError:
|
||||
return None
|
||||
if not (isinstance(fields, dict) and isinstance(fields.get("model"), str) and isinstance(fields.get("effort", ""), str) and fields.keys() <= {"model", "effort", *RECORD_FIELDS}):
|
||||
return None
|
||||
# type(), not isinstance(): JSON true is no integer.
|
||||
return fields if all(type(fields.get(name, 0)) is int and fields.get(name, 0) >= 0 for name in RECORD_FIELDS) else None
|
||||
|
||||
|
||||
def legacy_fields(line: str) -> dict[str, typing.Any] | None:
|
||||
"""The fields of a LEGACY_WORKER line, as record_fields gives them; None for any other text."""
|
||||
found = LEGACY_WORKER.fullmatch(line)
|
||||
rung = Rung.parse(found["rung"]) if found else None
|
||||
if not (found and rung):
|
||||
return None
|
||||
return {"model": rung.model, "effort": rung.effort, **{name: int(value or 0) for name, value in found.groupdict().items() if name != "rung"}}
|
||||
|
||||
@@ -5,9 +5,9 @@ import re
|
||||
import statistics
|
||||
import typing
|
||||
|
||||
from model_routing.route import archived_worker, ESCALATION_MARKER
|
||||
from claude_code.entries import assistant_parts, parts, tool_call, usage_count
|
||||
from git_command import commits
|
||||
from model_routing.route import ESCALATION_MARKER, route_from
|
||||
from model_routing.ladder import position, Rung
|
||||
from round_titles import round_counts
|
||||
from shell_command import simple_commands
|
||||
@@ -91,7 +91,7 @@ def measured(chunk: Chunk, home: str) -> ChunkFigures:
|
||||
reports[kind] = reports.get(kind, Reports(0, 0, 0)) + Reports(1, len(reported.get(subagent.launched_by, "")), sum(map(len, results(subagent.entries).values())))
|
||||
for _, tools, spent in spend(subagent.entries):
|
||||
add(tokens, chunk.project, kind, rung, tools, spent)
|
||||
start = archived_worker(cycles[0][1]) if cycles else None
|
||||
start = route_from(cycles[0][1]) if cycles else None
|
||||
return ChunkFigures(
|
||||
tokens=tokens,
|
||||
start=start.rung if start and start.item else None,
|
||||
@@ -130,11 +130,11 @@ def add(tokens: dict[str, collections.Counter[str]], project: str, kind: str, ru
|
||||
|
||||
|
||||
def rung_at(cycles: list[tuple[str, str]], timestamp: str) -> Rung | str:
|
||||
"""The rung of the cycle running at timestamp: the worker named by the first outcome stamped at or after it, or else by the last outcome. A worker off the ladder is returned as its text."""
|
||||
"""The rung of the cycle running at timestamp: the worker named by the first outcome stamped at or after it, or else by the last outcome. A worker line route_from cannot read is returned as its text, and "unrouted" where no outcome names a worker."""
|
||||
second = timestamp[:19] + "Z"
|
||||
found = next((rung for stamp, rung in cycles if stamp >= second), cycles[-1][1] if cycles else "unrouted")
|
||||
archived = archived_worker(found)
|
||||
return archived.rung if archived else found
|
||||
route = route_from(found)
|
||||
return route.rung if route else found
|
||||
|
||||
|
||||
def subagent_rung(entries: list[dict], arguments: dict, worker: Rung | str) -> Rung:
|
||||
|
||||
@@ -15,10 +15,11 @@ from routing_figures.measure import calls, Chunk, ChunkFigures, due, measured, r
|
||||
from failure import Failure
|
||||
from model_routing.ladder import LADDER, Rung
|
||||
from transcript_archive import outcomes, write
|
||||
from model_routing.route import Route
|
||||
|
||||
LOW = Rung("claude-opus-5-5", "medium")
|
||||
HIGH = Rung("claude-opus-5-5", "high")
|
||||
LOW_ROUTE = f"{LOW}, routed by item 7's L of 3 and E + I of 4"
|
||||
LOW_ROUTE = Route(LOW, 7, 3, 4).record()
|
||||
|
||||
|
||||
def message(id_, at, *parts, model="claude-opus-5-5", tokens=(1, 10, 100)):
|
||||
@@ -49,7 +50,7 @@ class Measured(unittest.TestCase):
|
||||
message("m2", "2026-10-14T10:05:00.000Z", {"type": "text", "text": "done"}),
|
||||
message("m3", "2026-10-14T11:00:00.000Z", {"type": "text", "text": "on"}),
|
||||
]
|
||||
kept = stamped("2026-10-14T10:30:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j", "")) + stamped("2026-10-14T10:30:00Z", outcomes.stepped_up(str(HIGH))) + stamped("2026-10-14T12:00:00Z", outcomes.finished(str(HIGH), "merged", "j"))
|
||||
kept = stamped("2026-10-14T10:30:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j", "")) + stamped("2026-10-14T10:30:00Z", outcomes.stepped_up(Route(HIGH).record())) + stamped("2026-10-14T12:00:00Z", outcomes.finished(Route(HIGH).record(), "merged", "j"))
|
||||
found = measured(Chunk("log", [kept], worker, []), "/h")
|
||||
self.assertEqual(found.tokens["project"], {"log": 333})
|
||||
self.assertEqual(found.tokens["subagent type"], {"worker": 333})
|
||||
@@ -84,22 +85,22 @@ class Measured(unittest.TestCase):
|
||||
worker = [commit("m1", "Answer stability round 1: 1 🔴, 0 🟡"), commit("m2", "Answer stability round 2: 0 🔴, 0 🟡"), commit("m3", "Answer stability round 2: 0 🔴, 0 🟡"), commit("m4", "Answer prose round 1: 0 🔴, 1 🟡")]
|
||||
self.assertEqual(measured(Chunk("log", [], worker, []), "/h").rounds_past_first, 1)
|
||||
|
||||
def test_reads_the_rung_of_a_route_an_older_loop_claude_archived_by_r_plus_c(self):
|
||||
older = f"{LOW}, routed by item 7's R + C of 9"
|
||||
def test_reads_the_route_of_a_line_loop_claude_wrote_before_it_archived_json(self):
|
||||
older = f"{LOW}, routed by item 7's L of 3 and E + I of 4"
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.finished(older, "m", "j"))
|
||||
found = measured(Chunk("log", [kept], [message("m1", "2026-10-14T09:00:00Z")], []), "/h")
|
||||
self.assertEqual(found.start, LOW)
|
||||
self.assertEqual(found.tokens["rung"], {str(LOW): 111})
|
||||
|
||||
def test_starts_no_route_for_a_chunk_no_item_routed(self):
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.finished(str(LOW), "m", "j"))
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.finished(Route(LOW).record(), "m", "j"))
|
||||
found = measured(Chunk("log", [kept], [message("m1", "2026-10-14T09:00:00Z")], []), "/h")
|
||||
self.assertIsNone(found.start)
|
||||
self.assertEqual(found.tokens["rung"], {str(LOW): 111})
|
||||
|
||||
|
||||
def test_puts_a_subagent_on_the_model_it_ran_at_the_effort_of_the_cycle_that_launched_it(self):
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j", "")) + stamped("2026-10-14T11:00:00Z", outcomes.finished(str(HIGH), "m", "j"))
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j", "")) + stamped("2026-10-14T11:00:00Z", outcomes.finished(Route(HIGH).record(), "m", "j"))
|
||||
worker = [
|
||||
message("m1", "2026-10-14T10:00:00.900Z"),
|
||||
message("m2", "2026-10-14T10:30:00Z", call("Agent", "a1", subagent_type="Explore", model="sonnet")),
|
||||
@@ -211,7 +212,7 @@ class Archive(unittest.TestCase):
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(kept / "log", str(transcript), "t", outcomes.unfinished(LOW_ROUTE, "e", "j", ""), 1790000000))
|
||||
transcript.write_text(json.dumps(first) + "\n" + json.dumps(message("m2", "2099-01-01T00:00:00Z")) + "\n")
|
||||
self.assertIsNone(write.archive(kept / "log", str(transcript), "t", outcomes.finished(str(HIGH), "m", "j"), 4070908800))
|
||||
self.assertIsNone(write.archive(kept / "log", str(transcript), "t", outcomes.finished(Route(HIGH).record(), "m", "j"), 4070908800))
|
||||
(found,) = archive.measure_archive(kept, "/h", "").chunks
|
||||
self.assertEqual((found.start, found.unfinished), (LOW, 1))
|
||||
self.assertEqual(found.tokens["rung"], {str(LOW): 111, str(HIGH): 111})
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import re
|
||||
|
||||
from markdown_text import quoted
|
||||
from model_routing.route import Record
|
||||
from utc_stamp import STAMPED
|
||||
|
||||
TASK = "Task: "
|
||||
@@ -28,23 +29,23 @@ def escalation_block(escalation: str) -> str:
|
||||
return f"Escalation:\n{quoted(escalation)}\n\n" if escalation else ""
|
||||
|
||||
|
||||
def finished(worker: str, evidence: str, judge: str) -> str:
|
||||
def finished(worker: Record, evidence: str, judge: str) -> str:
|
||||
return f"{FINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{JUDGE}{judge}\n{WORKER}{worker}"
|
||||
|
||||
|
||||
def unfinished(worker: str, evidence: str, judge: str, escalation: str) -> str:
|
||||
def unfinished(worker: Record, evidence: str, judge: str, escalation: str) -> str:
|
||||
"""escalation is the judge's quote showing that the worker's cited escalation trigger fired; empty for none."""
|
||||
escalated = escalation_block(escalation)
|
||||
return f"{UNFINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{JUDGE}{judge}\n{WORKER}{worker}"
|
||||
|
||||
|
||||
def interrupted(question: str, evidence: str, judge: str, escalation: str, worker: str) -> str:
|
||||
"""escalation is the judge's quote showing that the worker's cited escalation trigger fired; empty for none. worker is the rung the worker ran on until the interrupt."""
|
||||
def interrupted(question: str, evidence: str, judge: str, escalation: str, worker: Record) -> str:
|
||||
"""escalation is the judge's quote showing that the worker's cited escalation trigger fired; empty for none. worker is the record of the route the worker ran on until the interrupt."""
|
||||
escalated = escalation_block(escalation)
|
||||
return f"{INTERRUPTED}\n\n{question}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{JUDGE}{judge}\n{WORKER}{worker}"
|
||||
|
||||
|
||||
def answered(question: str, sources: str, answer: str, evidence: str, escalation: str, judge: str, worker: str) -> str:
|
||||
def answered(question: str, sources: str, answer: str, evidence: str, escalation: str, judge: str, worker: Record) -> str:
|
||||
"""escalation and worker are as on interrupted."""
|
||||
escalated = escalation_block(escalation)
|
||||
return f"{ANSWERED}\n\n{question}\n\n{sources} says:\n{quoted(answer)}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{JUDGE}{judge}\n{WORKER}{worker}"
|
||||
@@ -55,12 +56,12 @@ def rejected(why: str, reply: str, judge: str) -> str:
|
||||
return f"{REJECTED}: {why}\n\nReply:\n{quoted(reply)}\n\n{JUDGE}{judge}"
|
||||
|
||||
|
||||
def stepped_up(worker: str) -> str:
|
||||
def stepped_up(worker: Record) -> str:
|
||||
"""The rung the next cycle steps up to, archived as it steps, so a rerun after any exit resumes on it."""
|
||||
return f"Stepped up\n\n{WORKER}{worker}"
|
||||
|
||||
|
||||
def stopped(worker: str, why: str) -> str:
|
||||
def stopped(worker: Record, why: str) -> str:
|
||||
return f"Stopped: {why}\n\n{WORKER}{worker}"
|
||||
|
||||
|
||||
|
||||
@@ -2516,3 +2516,32 @@ Inherited architect, decided by:
|
||||
- Self-sufficiency: `bin/larv_review/main.py:34` "larv-review --help": the help no longer states when work has landed, and points to `plan.py` instead
|
||||
- Locality: `bin/score_outcomes/main.py:134` `read_outcomes`: whether a commit joins the chunk above depends on the previous iteration's state, with a positional `Chunk` tuple
|
||||
- Locality: `bin/todo_table.py:137` `NoItemsTable`: every caller must test it before `Failure`, and swapping the checks silently changes the message
|
||||
|
||||
## 2026-10-22T20:32:06Z, PR #357 at 3f8e175, against 64023b5
|
||||
|
||||
Ruling: better
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Mid A | better | better | better | same | better |
|
||||
| Inherited architect | better | better | better | better | better |
|
||||
|
||||
Mid A, decided by:
|
||||
|
||||
- Locality, Shape: `bin/loop_claude/routing.py:67` `after_unfinished`: takes the verdict itself and owns the whole route after an unfinished verdict, where `stepped` took loose ints, explained an import-cycle workaround and left `taking_in` to the caller
|
||||
- Locality, Navigation: `bin/loop_claude/loop/main.py:251` `run_loop`: `routed` returns a Route and `after_verdict` takes `Concluded`, so the off-ladder None and the `ended`/`before` juggling are gone and `archive_step` compares two records in one pass
|
||||
- Navigation, Shape: `bin/loop_claude/verdicts.py:1` "verdicts": the verdict types left the long `cycle/judge.py` for a small module whose docstring says why it sits outside cycle/
|
||||
- Shape: `bin/loop_claude/cycle/judge.py:69` `Reply.highest_likelihood_taken_in`: `bundled` gave no hint it carries an L; the new name does, and the verdicts are built by keyword
|
||||
- Navigation, Shape: `bin/chunk_pick.py:50` `ReadyToCut`: no longer clashes with the judge's `Finished`
|
||||
- Shape: `bin/chunk_pick.py:109` `next_release`: replaces the five-argument `standing` and is shared with larv_review's `landed_towards`; `Upcoming.is_earliest` needs its comment
|
||||
- Locality: `bin/model_routing/route.py:74` `Route.record`: a typed JSON record replaces the prose that `ROUTED` parsed back, at the cost of a second, labelled read path
|
||||
|
||||
Inherited architect, decided by:
|
||||
|
||||
- Locality, Shape: `bin/loop_claude/routing.py:67` `after_unfinished`: takes the Unfinished verdict and does all the next route's work, the E + I intake included, where `stepped` took loose ints because of an import cycle
|
||||
- Locality: `bin/loop_claude/loop/main.py:274` `after_verdict`: one call replaces the base's order-sensitive `ended` copy and `before` unpacking, and `archive_step` names both routes outright
|
||||
- Shape: `bin/loop_claude/cycle/judge.py:122` `verdict_from`: verdicts built by keyword where positional calls met differing field orders, and `bundled` renamed to what it holds
|
||||
- Navigation, Shape: `bin/chunk_pick.py:50` `ReadyToCut`: ends the two-`Finished` clash, and `next_release` splits the five-argument `standing`
|
||||
- Self-sufficiency, Locality: `bin/model_routing/route.py:13` `Record`: the archived worker line is a typed JSON record the outcome writers alone accept, with a warning that renaming a field orphans records; the legacy regex is unannotated and R + C lines fall through `rung_at` as text without a docstring saying so
|
||||
- Navigation, Self-sufficiency: `bin/loop_claude/verdicts.py:1` "verdicts": its own module, with field comments stating what `entries_read=None` means
|
||||
- Self-sufficiency: `bin/todo_table.py:205` `ships_alone`: the R above 5 rule is one named function, where the base held it only in the charter
|
||||
|
||||
@@ -791,3 +791,7 @@ Read most (opens and searches):
|
||||
## Reading cost, chunk fc21489c7990, finished 2026-10-22T18:57:09Z
|
||||
|
||||
55808 new input tokens before the first commit.
|
||||
|
||||
## Reading cost, chunk 1034c157d8d2, finished 2026-10-22T19:29:13Z
|
||||
|
||||
100386 new input tokens before the first commit.
|
||||
|
||||
@@ -6,13 +6,12 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 218`
|
||||
`Next ID = 219`
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 137 | 0.4.0 | | | **Archive a chunk's route as a record loop-claude reads back field by field, in place of the `Worker:` prose `ROUTED` parses, and name `stepped` and the judge's `bundled` by what they do.** | 4 | 2 | 3 | 7 | 4 | code health by 9, 1.4 by 9 | 32.9 |
|
||||
| 211 | 0.4.0 | | | **State each routing rule once, in the docstring of the code that enforces it, and cut `loop-claude --help`'s routing paragraph to what the human running the loop needs.** | 2 | 2 | 1 | 3 | 4 | code health by 9, 1.4 by 9 | 32.9 |
|
||||
| 212 | 0.4.0 | | | **Split `review_launches` into a choice of each pass's rung, made in `bin/model_routing` beside `ladder.review_rungs`, and whole prompt sentences that render that choice.** | 2 | 2 | 3 | 5 | 4 | code health by 9 | 26.9 |
|
||||
| 214 | 0.4.0 | | | **Have rule-hook refuse a round commit past round 3 of its pass, unless an override quotes the human's answer, so a program holds the round cap.** | 2 | 1 | 3 | 4 | 3 | code health by 3, 1.1 by 3 | 16.6 |
|
||||
@@ -28,6 +27,7 @@
|
||||
| 167 | 0.4.0 | | | **Refuse a `/security-review` or `/claude-security` Skill call below the project's rung, as rule-hook refuses such an Agent call.** | 3 | 2 | 3 | 6 | 2 | 1.1 by 4 | 5.6 |
|
||||
| 6 | 0.4.0 | | decision | **Move what `loop-claude` writes out of `~/.claude.json`, and ask the human about each write that must stay.** | 3 | 3 | 3 | 7 | 3 | 1.1 by 5 | 5.3 |
|
||||
| 1 | 0.5.0 | | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 1 | 2 | 1 | 2 | 1 | | 0.0 |
|
||||
| 218 | 0.5.0 | 170 | | **Drop `LEGACY_WORKER` from `bin/model_routing/route.py`, so `route_from` reads only the JSON record.** | 1 | 2 | 1 | 2 | 1 | code health by 2 | 14.4 |
|
||||
| 123 | 0.5.0 | | | **Have `bin/comprehension_history.py` parse each history entry once into seats, votes and places, so `latest_places`, `worse_places` and `worse_votes` neither depend on table order nor split a label to find a seat.** | 2 | 1 | 2 | 3 | 5 | code health by 5, 1.4 by 5 | 11.0 |
|
||||
| 139 | 0.5.0 | | | **Split `bin/routing_figures/measure.py` by concern, and keep the figures' measured cycle and each raised call as fields loop-claude reads back, not as the prose it renders.** | 3 | 1 | 2 | 4 | 5 | code health by 5, 1.4 by 5 | 11.0 |
|
||||
| 158 | 0.5.0 | | decision | **Measure the fixed cost of a chunk from loop-claude's archive, in tokens and hours against its items' S, and set F in `Score = 100 * V / (2**S + F)` to it.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 5, 1.4 by 5 | 10.3 |
|
||||
@@ -76,10 +76,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 137. Archive a chunk's route as a record loop-claude reads back field by field, in place of the `Worker:` prose `ROUTED` parses, and name `stepped` and the judge's `bundled` by what they do.
|
||||
|
||||
PR #262's paired ruling found the project reads worse: a rerun resumes on its rung only by parsing `Rung.__str__`'s and `Route.__str__`'s wording back with `bin/loop_claude/routing.py`'s `ROUTED`, a string contract over four files, where a reword silently routes afresh. Write the route as a structured line, such as JSON after `Worker:`, that `transcript_archive` reads into fields. `stepped` both raises to a bundled item's rung and counts towards a one-rung step, and release 0.3.0's Maintainability senior named `bin/loop_claude/routing.py:64` `stepped` for "seven one-line rung functions each defined by the others, four inputs held together, and an import cycle steering the interface" (rerun). `Reply.bundled` names neither L nor highest. Release 0.3.0's scoring runs found the same contract (first run: PR #324 at c713910; rerun: PR #324 rerun at c713910): `bin/model_routing/route.py:13` `ROUTED`, "parsed back by regexes in four places, so rewording one string silently changes routing" (Junior A, first run), and `:86` `Route.__str__`, "parsed back by ROUTED, routing_figures, reading_cost, score_outcomes and judge_replay" (inherited architect, rerun). That lowers Locality, the lowest dimension. Make `ROUTED`, `Route.__str__`, `stepped`, `Reply.bundled` and `bin/loop_claude/loop/main.py`'s `run_loop` easier to read. The structured line must keep the escalation's mark, archived today as `escalated at transcript entry N`: without it, a trigger that already stepped the worker could step it again.
|
||||
|
||||
### 211. State each routing rule once, in the docstring of the code that enforces it, and cut `loop-claude --help`'s routing paragraph to what the human running the loop needs.
|
||||
|
||||
Release 0.3.0's scoring runs (PR #324 at c713910) named `bin/loop_claude/loop/main.py:1` "loop-claude --help" at two seats in both runs: "a 160-line spec whose routing paragraph carries about 15 rules, re-describing what routing.py, ladder.py and retry.py enforce" (Maintainability senior), and "its routing paragraphs packing every rule into one block that routed and after_verdict need open" (Junior A). Mid A named `:259` `after_verdict`: "the 164-line docstring is the only full map of routing, and after_verdict chains it all in one expression". This lowers Locality: to change `routed` or `after_verdict`, a reader opens the help for rules the code enforces. Split `after_verdict`'s one expression into named steps. Make "loop-claude --help" and `after_verdict` easier to read. Item 184 rewraps two of the help's lines.
|
||||
@@ -98,7 +94,7 @@ The architecture pass of PR #296 found one rung under two names: `bin/model_rout
|
||||
|
||||
### 210. Hold the escalation mark in one module that states that one trigger buys one step, so a reader changes it without opening four files.
|
||||
|
||||
Release 0.3.0's scoring runs found the mark spread out (first run: PR #324 at c713910; rerun: PR #324 rerun at c713910). `bin/model_routing/route.py:66` `Route.escalated_at`: "the escalation mark lives in six units across four files" (Junior A, first run), "one concept spread over five files" (Mid A, rerun). `bin/loop_claude/cycle/judge.py:153` `escalation_mark`: "no single place states that one trigger buys one step" (Maintainability senior, rerun). `bin/loop_claude/cycle/run.py:45` `Cycle.stepped_from` and `:69` `escalated`: "state outside the state machine, cleared on every event but HumanTyped" (Mid A, rerun). `bin/loop_claude/cycle/judge.py:216` `quotes_checked` checks the escalation against `Grounds.since_step` beside the quoted answer: "three quote checks follow different rules" (Junior A, rerun). This lowers Locality: changing the mark means opening `route.py`, `judge.py`, `run.py` and `routing.py` together. Move the mark's stamping, `STEP_LINE`, the since-step grounds and their check, and the step's state into one module, and make `stepped_from` part of the cycle's state machine. Make `Route.escalated_at`, `escalation_mark`, `Cycle.stepped_from`, `escalated` and `quotes_checked` easier to read. Item 137 also archives the escalation's mark, so whichever of the two ships second keeps that mark.
|
||||
Release 0.3.0's scoring runs found the mark spread out (first run: PR #324 at c713910; rerun: PR #324 rerun at c713910). `bin/model_routing/route.py:66` `Route.escalated_at`: "the escalation mark lives in six units across four files" (Junior A, first run), "one concept spread over five files" (Mid A, rerun). `bin/loop_claude/cycle/judge.py:153` `escalation_mark`: "no single place states that one trigger buys one step" (Maintainability senior, rerun). `bin/loop_claude/cycle/run.py:45` `Cycle.stepped_from` and `:69` `escalated`: "state outside the state machine, cleared on every event but HumanTyped" (Mid A, rerun). `bin/loop_claude/cycle/judge.py:216` `quotes_checked` checks the escalation against `Grounds.since_step` beside the quoted answer: "three quote checks follow different rules" (Junior A, rerun). This lowers Locality: changing the mark means opening `route.py`, `judge.py`, `run.py` and `routing.py` together. Move the mark's stamping, `STEP_LINE`, the since-step grounds and their check, and the step's state into one module, and make `stepped_from` part of the cycle's state machine. Make `Route.escalated_at`, `escalation_mark`, `Cycle.stepped_from`, `escalated` and `quotes_checked` easier to read. The archive's `Worker:` record holds the mark as `escalated_at`, so a rerun keeps it; the move keeps that field. `escalation_mark` now lives in `bin/loop_claude/verdicts.py`.
|
||||
|
||||
### 190. Rewrite the `docs/decisions.md` entries `todo.md: one table …`, `loop-claude: a judge decides …` and `loop-claude: before it opens claude …` as the current state, so a reader no longer replays superseded paragraphs to learn what holds.
|
||||
|
||||
@@ -144,6 +140,10 @@ It lives under `~/.claude`, which `.gitignore` keeps from the remote, so a lost
|
||||
|
||||
The README line covers the bench's transcripts in `workflow-bench/logs/` too (goal 3).
|
||||
|
||||
### 218. Drop `LEGACY_WORKER` from `bin/model_routing/route.py`, so `route_from` reads only the JSON record.
|
||||
|
||||
PR #357 archived the route as a JSON record, and kept `LEGACY_WORKER` to read the `Worker:` lines an older loop-claude wrote as sentences. The routing figures still count chunks whose archive holds such lines: 51 when PR #357 shipped, every chunk finished after the reset of 2026-10-21T22:16:41Z. Item 170 changes the routing steps, after which `AGENTS.md` has a `Reset` appended to `routing-calls-raised.md`, and the figures then count only chunks that hold records. So this item waits for item 170. Before starting, check that `routing-calls-raised.md` holds a `Reset` stamped after PR #357's merge; where none does, block this item on the next item that changes the ladder or its steps instead. Delete `LEGACY_WORKER`, `legacy_fields`, the tests that read sentences (in `bin/loop_claude/test_loop_claude.py` and `bin/routing_figures/test_routing_figures.py`), and the sentence on them in "loop-claude --help". Reword `route_from`'s docstring, and the comment at `Rung.__str__` in `bin/model_routing/ladder.py`, which gives the old lines as a reason `parse` must keep reading that spelling.
|
||||
|
||||
### 123. Have `bin/comprehension_history.py` parse each history entry once into seats, votes and places, so `latest_places`, `worse_places` and `worse_votes` neither depend on table order nor split a label to find a seat.
|
||||
|
||||
Three seats of PR #257's scoring run named `bin/comprehension_history.py:106` `latest_places`; the logic it rests on is `:114` `worse_places` and `:126` `worse_votes`. They regex the markdown the module writes. `worse_votes` keeps a seat's first row through `setdefault`, because the votes table comes first. `worse_places` recovers the seat with `label.split(",")[0]`. `Entry` still holds scoring seats' places for rulings recorded before PR #255. Release 0.3.0's seats named near-synonyms left from before PR #255: `:150` `scoring_runs`, "near-synonyms scored and scoring_runs differ only on a ruling from before PR #255" (Maintainability senior), `:130` `newest_scored`, "four near-identical names over two concepts", and `:159` `scored`. Make these functions easier to read.
|
||||
@@ -344,7 +344,7 @@ The key names the project's session, archive folder and models entry. `project_k
|
||||
|
||||
### 155. Lower the reading cost of the files a chunk reads most before its first commit.
|
||||
|
||||
`bin/reading-cost` on 2026-10-22: the median of the last 5 chunks is 81,812 new input tokens before their first commit, up from 71,637 for the 5 before them. The files read most over the last 5 were `skills/larv-review/SKILL.md`, 3 times, and `rules/todo-items.md`, 2 times. The chunk that ships this item runs `bin/reading-cost --reset`.
|
||||
`bin/reading-cost` on 2026-10-22: the median of the last 5 chunks is 100,386 new input tokens before their first commit, up from 73,910 for the 5 before them. The files read most over the last 5 were `skills/larv-review/SKILL.md`, 3 times, and `rules/todo-items.md`, 2 times. The chunk that ships this item runs `bin/reading-cost --reset`.
|
||||
|
||||
### 51. Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user