Archive each judge run's prompt and reply beside its verdict, and replay judge-replay on the archived prompt #568

Merged
lilleman merged 9 commits from item-303 into main 2026-11-17 01:37:43 +01:00
17 changed files with 398 additions and 74 deletions
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- loop-claude's archive keeps each judge run's prompt and json reply in a judge/ folder beside its chunk's transcripts, and names the run on a `Judge run:` line of the outcome it produced; judge-replay replays a verdict on its archived prompt where the archive holds one.
- A plan kept as a Jira epic takes no changelog bullet: `todo-item finish` there needs neither `--changelog` nor `--no-changelog`, and refuses `--changelog`, since the issues' history is the changelog.
## 0.11.0
+15
View File
@@ -0,0 +1,15 @@
from claude_code.entries import Entry
from loop_claude import escalation_mark
from failure import Failure
from loop_claude.cycle.judge import Grounds, JUDGE_RULES, TASK_HEADING, TRANSCRIPT_HEADING
from claude_code.transcript import human_answers
from loop_claude.cycle.prompts import typed_answer
def grounds_of(prompt: str, entries: list[Entry]) -> Grounds | Failure:
"""What the quotes of a judge run on prompt are checked against, as judge_prompt would have made it for the transcript's entries: the premises and the records before the task are quotable, the transcript the judge was shown comes before the rules. The prompt is split on the headings judge_prompt writes, at the first of each; a Failure where it lacks one. Rules of an earlier wording stay part of the transcript, a slack that lets an evidence quote them pass."""
before_task, task_heading, after_task = prompt.partition(f"\n\n{TASK_HEADING}")
_, transcript_heading, shown = after_task.partition(TRANSCRIPT_HEADING)
if not (task_heading and transcript_heading):
return Failure(f"the archived prompt lacks the heading {TASK_HEADING.strip()!r} or {TRANSCRIPT_HEADING.strip()!r}, which judge_prompt writes")
return Grounds(shown.removesuffix(f"\n\n{JUDGE_RULES}"), escalation_mark.after_mark(entries, 0), before_task, tuple(human_answers(entries, typed_answer)))
+25 -10
View File
@@ -4,13 +4,16 @@ usage: judge-replay <model>
<model> is a project model on the routing ladder this setup ships, such as claude-opus-5-5. The tool replays the
verdicts archived for the projects that ~/.claude/loop-claude-models.json runs on it: the latest
15 of each kind, finished, unfinished, interrupt and quoted answer. Each transcript copy in the
archive is what the judge read for the verdict beside it, so the replay hands the judge that copy,
with the task, premises and records each project holds now.
15 of each kind, finished, unfinished, interrupt and quoted answer. Where the archive kept the judge's own
prompt (docs/decisions.md → `loop-claude: the archive keeps the raw transcript and each judge run`), the
replay hands the judge that prompt, which holds what the judge read then. For a verdict from before the
archive kept it, the replay builds the prompt from the transcript copy beside the verdict, with the task,
premises and records each project holds now.
It first replays every verdict on the reference judge: the model at xhigh effort, the rung the
judge ran on before loop-claude routed it. A verdict the reference judge returns another kind on
may rest on records or judge rules that have changed since; either way no rung is held to it. It
may rest, where its prompt was rebuilt, on records or judge rules that have changed since; where the archived prompt was
replayed, the judge's own variance is the cause. Either way no rung is held to it. It
then replays the rest on each rung above the model's cheapest that a chunk's E + I raises a
review to, except the reference judge's rung. Then it replays them on each candidate rung,
highest first: from the model's cheapest rung down to the lowest rung a worker routes to. It stops
@@ -25,7 +28,7 @@ keeps the judge to its own model's rungs; a rung above them that misses no inter
evidence to let the judge climb onto it.
Like the loop, it tries a judge three times running before it counts the reply as failed. A
verdict whose project or transcript cannot be read replays as failed too. Exit 1, saying why, on
verdict whose project, transcript or archived judge run cannot be read replays as failed too. Exit 1, saying why, on
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
or a usage limit.
"""
@@ -35,12 +38,14 @@ import typing
from transcript_archive.location import archive_root, transcript_of
from transcript_archive.read import archived_outcomes
from rung_replay.command import arguments_or_exit, down_from, failed, limited_exit, projects_on
from loop_claude.cycle.judge import Brief, judge_once
from loop_claude.cycle.judge import Brief, judge_archived, judge_once
from judge_replay.replay import Case, case_from, Kind, report, reproduced, sample, verdict_kind
from transcript_archive.project_key import checkout_of
from loop_effects.effects import Effects
from failure import Failure
from rung_replay.search import first_reply, replayed
from judge_replay.archived_prompt import grounds_of
from transcript_archive.judge_run import JudgeRun
from model_routing.ladder import Ladder, REVIEW_STEPS, Rung
from claude_code.reply import Limited
from model_routing.ladder_file import read_ladder, SHIPPED
@@ -107,13 +112,23 @@ def archived_cases(model: str, root: pathlib.Path, home: pathlib.Path) -> list[C
def rule(case: Case, rung: Rung, home: pathlib.Path, effects: Effects) -> Kind | Limited:
premises, records, entries = read_premises(case.root, home), read_records(case.root, home, home / ".claude"), read_transcript(str(case.transcript))
entries = read_transcript(str(case.transcript))
if isinstance(entries, Failure):
return verdict_kind(entries)
if case.judge_run:
kept = JudgeRun.read(case.judge_run)
if isinstance(kept, Failure):
return verdict_kind(kept)
grounds = grounds_of(kept.prompt, entries)
if isinstance(grounds, Failure):
return verdict_kind(grounds)
verdict: Verdict | Failure | Limited = first_reply(lambda: judge_archived(case.root, kept.prompt, grounds, len(entries), rung, effects))
return verdict if isinstance(verdict, Limited) else verdict_kind(verdict)
premises, records = read_premises(case.root, home), read_records(case.root, home, home / ".claude")
if isinstance(premises, Failure):
return verdict_kind(premises)
if isinstance(records, Failure):
return verdict_kind(records)
if isinstance(entries, Failure):
return verdict_kind(entries)
# 0 for the route's mark, as before any step-up: the replay compares verdict kinds, and a mark changes only which escalation quotes count.
verdict: Verdict | Failure | Limited = first_reply(lambda: judge_once(case.root, Brief(case.task, premises, records, entries, 0), rung, effects))
verdict = first_reply(lambda: judge_once(case.root, Brief(case.task, premises, records, entries, 0), rung, effects))
return verdict if isinstance(verdict, Limited) else verdict_kind(verdict)
+6 -3
View File
@@ -4,7 +4,7 @@ import dataclasses
import pathlib
import typing
from transcript_archive.outcomes import ANSWERED, FINISHED, first_verdict, INTERRUPTED, task_of, UNFINISHED
from transcript_archive.outcomes import ANSWERED, FINISHED, first_verdict, INTERRUPTED, judge_run_at, task_of, UNFINISHED
from failure import Failure
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict
from rung_replay.search import Replay
@@ -22,6 +22,8 @@ class Case:
transcript: pathlib.Path
at: str
archived_kind: Kind
# The judge run the archive kept beside the verdict, which holds the prompt the judge read; None where it kept none.
judge_run: pathlib.Path | None = None
def case_from(root: str, outcomes: str, transcript: pathlib.Path) -> Case | None:
@@ -32,7 +34,8 @@ def case_from(root: str, outcomes: str, transcript: pathlib.Path) -> Case | None
task, verdict = task_of(outcomes), first_verdict(outcomes)
if task is None or verdict is None:
return None
return Case(root, task, transcript, *verdict)
run = judge_run_at(outcomes, verdict[0])
return Case(root, task, transcript, *verdict, transcript.parent / run if run else None)
def sample(cases: list[Case], per_kind: int) -> list[Case]:
@@ -88,4 +91,4 @@ def report(model: str, reference: Rung, cases: list[Case], replay: Replay[Case,
def missed_lines(missed: list[tuple[Case, Kind]]) -> list[str]:
return [f" {found.root} {found.at}: archived {found.archived_kind}, replayed {kind}" for found, kind in missed]
return [f" {found.root} {found.at}: archived {found.archived_kind}, replayed {kind} on {'its archived prompt' if found.judge_run else 'a prompt rebuilt from today\'s records'}" for found, kind in missed]
+114 -3
View File
@@ -10,11 +10,14 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINISHED
from claude_code.entries import AssistantTurn, HumanTurn, LoopPrompt, Text
from judge_replay.replay import Case, case_from, report, reproduced, sample, verdict_kind
from project_config.test_project_config import configured
from loop_effects.effects import Effects
from failure import Failure
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished
from judge_replay.archived_prompt import grounds_of
from loop_claude.cycle.judge import judge_archived, JUDGE_RULES, Rejected, TASK_HEADING, TRANSCRIPT_HEADING
from judge_replay import main
from rung_replay.search import Miss, Replay, Tried
from model_routing.ladder import Rung
@@ -41,6 +44,11 @@ class CaseFrom(unittest.TestCase):
found = case_from("/code/p", f"Task: t\n\n2026-10-14T10:00:00Z {verdict}\n\nmore\n", TRANSCRIPT)
self.assertEqual(found.archived_kind, verdict)
def test_names_the_judge_run_that_the_first_verdict_records_beside_it(self):
outcomes = f"Task: t\n\n2026-10-14T10:00:00Z Verdict: unfinished\n\nJudge: j\nJudge run: judge/{'a' * 64}.json\nWorker: x\n\n2026-10-14T11:00:00Z Verdict: finished\n\nJudge: j\nJudge run: judge/{'b' * 64}.json\nWorker: x\n"
self.assertEqual(case_from("/code/p", outcomes, TRANSCRIPT).judge_run, TRANSCRIPT.parent / f"judge/{'a' * 64}.json")
self.assertIsNone(case_from("/code/p", "Task: t\n\n2026-10-14T10:00:00Z Verdict: finished\n\nJudge: j\nWorker: x\n", TRANSCRIPT).judge_run)
def test_skips_a_copy_with_no_verdict(self):
self.assertIsNone(case_from("/code/p", "Task: t\n\n2026-10-14T10:00:00Z Stepped up\n\nWorker: y\n", TRANSCRIPT))
self.assertIsNone(case_from("/code/p", "not an outcomes file", TRANSCRIPT))
@@ -68,16 +76,21 @@ class Reproduced(unittest.TestCase):
class Report(unittest.TestCase):
def test_names_whether_a_missed_verdict_was_replayed_on_its_archived_prompt(self):
kept = Case("/code/p", "Do it.", TRANSCRIPT, "2026-10-14T10:00:00Z", FINISHED, pathlib.Path("/archive/p/c/judge/run.json"))
lines = report("claude-opus-5-5", REFERENCE, [kept], Replay([UNFINISHED], [], [])).split("\n")
self.assertIn(" /code/p 2026-10-14T10:00:00Z: archived Verdict: finished, replayed Verdict: unfinished on its archived prompt", lines)
def test_counts_the_verdicts_and_names_each_miss_and_the_lowest_rung(self):
cases = [case("2026-10-14T10:00:00Z", FINISHED), case("2026-10-14T10:00:01Z", INTERRUPTED)]
replay = Replay([UNFINISHED, INTERRUPTED], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(cases[1], INTERRUPTED, FINISHED)])], MIDDLE)
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, replay).splitlines(), [
"Replayed 2 verdicts of claude-opus-5-5 projects: 1 Verdict: finished, 1 Verdict: interrupt.",
"The reference judge, claude-opus-5-5 at xhigh effort, reproduced 1. It returned another kind on:",
" /code/p 2026-10-14T10:00:00Z: archived Verdict: finished, replayed Verdict: unfinished",
" /code/p 2026-10-14T10:00:00Z: archived Verdict: finished, replayed Verdict: unfinished on a prompt rebuilt from today's records",
"claude-sonnet-5-5 at high effort missed 0.",
"claude-haiku-5-5 at xhigh effort missed 1:",
" /code/p 2026-10-14T10:00:01Z: archived Verdict: interrupt, replayed Verdict: finished",
" /code/p 2026-10-14T10:00:01Z: archived Verdict: interrupt, replayed Verdict: finished on a prompt rebuilt from today's records",
"Lowest candidate that, like each candidate above it, returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort",
])
missed_all = report("claude-opus-5-5", REFERENCE, cases, Replay([FINISHED, INTERRUPTED], [Tried(MIDDLE, [Miss(cases[1], INTERRUPTED, FINISHED)])], None)).splitlines()
@@ -90,7 +103,7 @@ class Report(unittest.TestCase):
climbs = [Tried(Rung("claude-fable-5-2", "low"), [Miss(cases[1], INTERRUPTED, FINISHED)]), Tried(Rung("claude-fable-5-2", "high"), [])]
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, Replay([FINISHED, INTERRUPTED], [Tried(MIDDLE, [])], MIDDLE, climbs)).splitlines()[2:5], [
"claude-fable-5-2 at low effort, a rung a review climbs to, missed 1:",
" /code/p 2026-10-14T10:00:01Z: archived Verdict: interrupt, replayed Verdict: finished",
" /code/p 2026-10-14T10:00:01Z: archived Verdict: interrupt, replayed Verdict: finished on a prompt rebuilt from today's records",
"claude-fable-5-2 at high effort, a rung a review climbs to, missed 0.",
])
@@ -161,6 +174,30 @@ class Main(unittest.TestCase):
# The replay shows the judge the whole transcript, with no step-up marked in it.
self.assertNotIn("[step]", self.prompts[0])
def test_hands_the_judge_the_prompt_the_archive_kept_and_rebuilds_the_prompt_of_a_verdict_with_none(self):
kept = self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1"
prompt = "From the goals file, OLD premises\n\nThe records: none\n\nThe task the worker was given:\nDo it.\n\nWhat the worker did and said, what the human typed on [human] lines and what loop-claude told it on [loop] lines, latest last:\nsaid Verdict: finished\n\nOLD rules"
(kept / "judge").mkdir()
(kept / "judge" / "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.json").write_text(json.dumps({"prompt": prompt, "reply": {"type": "result"}}))
(kept / "t0.outcomes.md").write_text("Task: Do it.\n\n2026-10-14T10:00:00Z Verdict: finished\n\nJudge: j\nJudge run: judge/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.json\nWorker: x\n")
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
self.assertEqual(code, 0, err)
self.assertIn("The reference judge, claude-opus-5-5 at xhigh effort, reproduced 2.", out)
# The first replay, on the finished verdict, reads the archived prompt; the second, on the interrupt, is rebuilt from the records of today.
self.assertEqual(self.prompts[0], prompt)
self.assertEqual(self.calls[0][2], self.calls[1][2])
self.assertTrue(self.calls[0][2].endswith("/code/proj"))
self.assertIn("The task the worker was given:\nDo it.", self.prompts[1])
self.assertIn("1. Ship.", self.prompts[1])
def test_a_verdict_whose_archived_judge_run_cannot_be_read_replays_as_failed(self):
kept = self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1"
(kept / "t0.outcomes.md").write_text("Task: Do it.\n\n2026-10-14T10:00:00Z Verdict: finished\n\nJudge: j\nJudge run: judge/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.json\nWorker: x\n")
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
self.assertEqual(code, 0, err)
self.assertIn("The reference judge, claude-opus-5-5 at xhigh effort, reproduced 1. It returned another kind on:", out)
self.assertIn("could not read the archived judge run", out)
def test_tries_no_rung_below_the_lowest_a_worker_routes_to_where_every_rung_returns_every_verdict(self):
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
self.assertEqual(code, 0, err)
@@ -226,6 +263,80 @@ class Main(unittest.TestCase):
self.assertEqual(ran.returncode, 0)
self.assertIn("usage: judge-replay <model>", ran.stdout)
def test_exits_1_on_a_usage_limit_met_on_an_archived_prompt(self):
kept = self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1"
(kept / "judge").mkdir()
(kept / "judge" / "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.json").write_text(json.dumps({"prompt": "P", "reply": {}}))
(kept / "t0.outcomes.md").write_text("Task: Do it.\n\n2026-10-14T10:00:00Z Verdict: finished\n\nJudge: j\nJudge run: judge/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.json\nWorker: x\n")
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: {"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
self.assertEqual((code, "usage limit: You've hit your limit" in err), (1, True))
class JudgeArchivedRuns(unittest.TestCase):
"""judge_archived on a prompt as judge_prompt writes it: the premises, the task, the transcript the judge was shown, and the rules."""
PROMPT = f"HEAD: Ship fast.\n\n{TASK_HEADING}Do it.\n\n{TRANSCRIPT_HEADING}Worker chose option B.\n[human] Use Postgres.\n\n{JUDGE_RULES}"
ENTRIES = [AssistantTurn(parts=(Text("Worker chose option B."),)), HumanTurn(text="Use Postgres."), LoopPrompt(text="Keep going.")]
def judged(self, prompt=PROMPT, **fields):
"""What judge_archived returns on prompt where the judge's structured output is fields over an unfinished verdict, and the cwd claude ran in."""
output = {"answer": "", "escalation": "", "evidence": "Worker chose option B.", "finished": False, "interrupt": False, "question": "", "answered_before": "", "left_open": "", "reversed": "", "release_ready": False, **fields}
cwds = []
def claude(args, cwd, prompt):
cwds.append(cwd)
return subprocess.CompletedProcess(args, 0, json.dumps({"type": "result", "structured_output": output, "total_cost_usd": 0.25}), "")
effects = Effects(claude=claude, claude_on_terminal=unused, discard_typed=unused, dm=unused, execv=unused, git=unused, input=unused, local_time=unused, memory=unused, now=unused, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused, tmux=unused, tmux_on_terminal=unused)
grounds = grounds_of(prompt, self.ENTRIES)
return (grounds if isinstance(grounds, Failure) else judge_archived("/code/p", prompt, grounds, len(self.ENTRIES), REFERENCE, effects)), cwds
def test_runs_in_the_projects_folder_and_stamps_the_verdict_with_the_entries_read(self):
verdict, cwds = self.judged()
self.assertEqual((cwds, verdict.entries_read), (["/code/p"], 3))
def test_checks_the_evidence_against_the_transcript_the_judge_was_shown_and_not_the_premises(self):
self.assertIsInstance(self.judged(evidence="Ship fast.")[0], Rejected)
self.assertIsInstance(self.judged(evidence="Worker chose option B.")[0], Unfinished)
def test_keeps_the_cost_and_the_run_of_a_rejected_reply(self):
rejected, _ = self.judged(evidence="Never said.")
self.assertEqual((rejected.cost, rejected.run.prompt), (0.25, self.PROMPT))
def test_keeps_the_cost_of_a_reply_rejected_for_its_shape(self):
rejected, _ = self.judged(finished="yes")
self.assertEqual((type(rejected), rejected.cost), (Rejected, 0.25))
def test_checks_neither_the_task_nor_the_rules_as_the_transcript_or_the_premises(self):
self.assertIsInstance(self.judged(evidence=JUDGE_RULES)[0], Rejected)
self.assertIsInstance(self.judged(answer="Do it.", question="Which?")[0], Interrupt)
def test_splits_the_prompt_at_the_first_transcript_heading(self):
quoting = self.PROMPT.replace("Worker chose", f"Quoted: {TRANSCRIPT_HEADING}Worker chose", 1)
self.assertIsInstance(self.judged(quoting, evidence="Quoted:")[0], Unfinished)
def test_splits_the_prompt_at_the_first_task_heading(self):
quoting = self.PROMPT.replace("Do it.", f"Quote:\n\n{TASK_HEADING}Ship slowly.", 1)
self.assertIsInstance(self.judged(quoting, answer="Ship fast.", question="Which?")[0], QuotedAnswer)
self.assertIsInstance(self.judged(quoting, answer="Quote:", question="Which?")[0], Interrupt)
def test_fails_before_the_judge_runs_on_a_prompt_without_a_heading(self):
for prompt in ("No headings at all.", f"{TASK_HEADING}Do it.", f"\n\n{TASK_HEADING}Do it.\n\nNo transcript heading."):
with self.subTest(prompt=prompt):
failure, cwds = self.judged(prompt)
self.assertEqual((type(failure), cwds), (Failure, []))
self.assertIn("lacks the heading", failure.reason)
def test_checks_a_quoted_answer_against_the_premises(self):
self.assertIsInstance(self.judged(answer="Ship fast.", question="Which?")[0], QuotedAnswer)
self.assertIsInstance(self.judged(answer="Use Postgres.", question="Which?")[0], Interrupt)
def test_keeps_an_escalation_that_the_worker_wrote_after_the_start_of_the_transcript(self):
self.assertEqual(self.judged(escalation="Worker chose option B.")[0].escalation, "Worker chose option B.")
self.assertEqual(self.judged(escalation="Use Postgres.")[0].escalation, "")
def test_names_the_typed_answer_that_holds_a_reversal(self):
self.assertEqual(self.judged(reversed="Use Postgres.")[0].judged.reversed, "Use Postgres.")
if __name__ == "__main__":
unittest.main()
+24 -8
View File
@@ -11,6 +11,7 @@ from loop_claude import escalation_mark
from failure import Failure
from loop_claude.verdicts import Finished, Interrupt, Judged, QuotedAnswer, Unfinished, Verdict
from claude_code.transcript import human_answers
from transcript_archive.judge_run import JudgeRun
from loop_claude.quotes import plain, verbatim
from loop_claude.cycle.retry import retried
from model_routing.ladder import Rung
@@ -64,6 +65,10 @@ FINISHED_RULES = (
"continue: an open PR, a review not yet run, CI not yet reported, an answer not yet recorded."
)
JUDGE_RULES = "\n\n".join((INTERRUPT_RULES, QUOTE_RULES, FINISHED_RULES))
# Open the task and the transcript in the prompt; the premises and the records come before the task, the rules after the transcript.
# judge_replay.archived_prompt splits each archived prompt on them, so rewording one makes every run archived before it replay as failed.
TASK_HEADING = "The task the worker was given:\n"
TRANSCRIPT_HEADING = "What the worker did and said, what the human typed on [human] lines and what loop-claude told it on [loop] lines, latest last:\n"
@dataclasses.dataclass(frozen=True)
@@ -94,10 +99,15 @@ JUDGE_SCHEMA = {
@dataclasses.dataclass(frozen=True)
class Rejected(Failure):
"""A judge reply loop-claude rejected: reason says why, reply holds the reply as json, and cost is what the run cost in dollars, 0 where its reply records none."""
reply: str
"""A judge reply loop-claude rejected: reason says why, run holds the prompt and the reply, and cost is what the run cost in dollars, 0 where its reply records none."""
run: JudgeRun
cost: float = 0.0
@property
def reply(self) -> str:
"""The reply as json."""
return json.dumps(self.run.reply, ensure_ascii=False)
class Grounds(typing.NamedTuple):
"""The texts quotes_checked checks the judge's quotes against. A quote must appear in its text as written, since a quote reworded or stitched together can say what its source does not; only the evidence may join passages with an ellipsis, each passage in order."""
@@ -155,6 +165,12 @@ def judge_once(root: str, brief: Brief, rung: Rung, effects: Effects) -> Verdict
return verdict if isinstance(verdict, Failure | Limited) else escalation_mark.stamped(verdict, len(brief.entries))
def judge_archived(root: str, prompt: str, grounds: Grounds, entries_read: int, rung: Rung, effects: Effects) -> Verdict | Failure | Limited:
"""One judge run on a prompt the archive kept from an earlier run, as judge-replay replays it: the judge reads what it read then, and its quotes are checked against grounds, which judge_replay.archived_prompt makes from that prompt in place of the records of today."""
verdict = run_judge(root, judge_args(rung), prompt, grounds, effects)
return verdict if isinstance(verdict, Failure | Limited) else escalation_mark.stamped(verdict, entries_read)
def judge_args(rung: Rung) -> list[str]:
# --json-schema has claude check its reply against the schema and return it as the json result's structured_output.
# No tools, so the verdict rests on the transcript grounded() checks it against.
@@ -172,14 +188,14 @@ def run_judge(root: str, args: list[str], prompt: str, grounds: Grounds, effects
return reply
output = structured_output(reply)
verdict = verdict_from(output)
replied = json.dumps(reply, ensure_ascii=False)
run = JudgeRun(prompt, reply)
cost = cost_of(reply)
if isinstance(verdict, Failure):
return Rejected(verdict.reason, replied, cost)
return Rejected(verdict.reason, run, cost)
checked = quotes_checked(verdict, grounds)
if isinstance(checked, Failure):
return Rejected(checked.reason, replied, cost)
return dataclasses.replace(checked, judged=Judged(cost, tokens_of(reply.get("usage")), reversal_of(output, grounds.typed)))
return Rejected(checked.reason, run, cost)
return dataclasses.replace(checked, judged=Judged(cost, tokens_of(reply.get("usage")), reversal_of(output, grounds.typed), run))
def reversal_of(output: object, typed: tuple[str, ...]) -> str:
@@ -255,8 +271,8 @@ def judge_prompt(brief: Brief) -> tuple[str, Grounds]:
head = (
f"From the goals file, {brief.premises}\n\n"
f"The records: the project's working rules and decisions, then the human's rules for every project:\n{brief.records or 'none'}\n\n"
f"The task the worker was given:\n{brief.task}\n\n"
"What the worker did and said, what the human typed on [human] lines and what loop-claude told it on [loop] lines, latest last:\n"
f"{TASK_HEADING}{brief.task}\n\n"
f"{TRANSCRIPT_HEADING}"
)
room = max(0, JUDGE_PROMPT_CHARS - len(head) - len(JUDGE_RULES) - 2)
transcript = escalation_mark.transcript(brief.entries, brief.escalated_at, typed_answer)
+12 -11
View File
@@ -14,6 +14,7 @@ from failure import Failure
from loop_claude.verdicts import Finished, Interrupt, Judged, QuotedAnswer, Unfinished, Verdict
from loop_claude.cycle.interrupt import interrupt, Talked, WorkerSession
from loop_claude.routing import judge_rung, next_route, step_up, StepUp
from transcript_archive.judge_run import JudgeRun
from model_routing.ladder import Ladder, Rung
from transcript_archive.read import last_spent
from transcript_archive import outcomes
@@ -98,9 +99,9 @@ def archived_stop(project_archive: pathlib.Path, session: Session, task: str, ro
return Stop(1, f"{failure.reason}\n{stop.reason}".rstrip()) if failure and failure.reason != stop.reason else stop
def archived(project_archive: pathlib.Path, transcript_path: str, task: str, outcome: str, effects: Effects) -> Failure | None:
def archived(project_archive: pathlib.Path, transcript_path: str, task: str, outcome: str, effects: Effects, judge_run: JudgeRun | None = None) -> Failure | None:
"""transcript_archive.write.archive, stamped now and saying its notes through effects. A cycle and close.py, which has no Cycle, both archive through it."""
return archive(project_archive, transcript_path, task, outcome, effects.now(), lambda text: say(text, effects))
return archive(project_archive, transcript_path, task, outcome, effects.now(), lambda text: say(text, effects), judge_run)
def judge_of(cycle: Cycle) -> Rung:
@@ -167,7 +168,7 @@ def rule(cycle: Cycle, transcript_path: str, quoted_answers: int) -> Ruled | Sto
if isinstance(annotation, Failure):
return Stop(1, annotation.reason)
stepped = ruling_step_up(cycle, verdict)
if failure := archive_stepped(cycle, transcript_path, ruling_outcome(cycle, verdict, quoted_answers, annotation), stepped):
if failure := archive_stepped(cycle, transcript_path, ruling_outcome(cycle, verdict, quoted_answers, annotation), stepped, verdict.judged.run):
return Stop(1, failure.reason)
return Ruled(verdict, stepped.after if stepped else None)
@@ -187,13 +188,13 @@ def annotation_of(cycle: Cycle, transcript_path: str, entries: list[Entry], judg
def ruling_outcome(cycle: Cycle, ruling: Interrupt | QuotedAnswer, quoted_answers: int, annotation: outcomes.Annotation) -> str:
"""The archive's outcome for a ruling: interrupted where it goes to the human, with the question put to them, and else answered from the premises or the records."""
if isinstance(ruling, QuotedAnswer) and not to_the_human(ruling, quoted_answers):
return outcomes.answered(question=ruling.question, sources=prompts.QUOTE_SOURCES, answer=ruling.answer, evidence=ruling.evidence, escalation=ruling.escalation, judge=str(judge_of(cycle)), worker=cycle.route.worker_line(), judge_cost=ruling.judged.cost, annotation=annotation)
return outcomes.interrupted(question_for(ruling), ruling.evidence, str(judge_of(cycle)), ruling.escalation, cycle.route.worker_line(), ruling.judged.cost, annotation)
return outcomes.answered(question=ruling.question, sources=prompts.QUOTE_SOURCES, answer=ruling.answer, evidence=ruling.evidence, escalation=ruling.escalation, judge=str(judge_of(cycle)), worker=cycle.route.worker_line(), judge_cost=ruling.judged.cost, annotation=annotation, judge_run=ruling.judged.run)
return outcomes.interrupted(question_for(ruling), ruling.evidence, str(judge_of(cycle)), ruling.escalation, cycle.route.worker_line(), judge_cost=ruling.judged.cost, annotation=annotation, judge_run=ruling.judged.run)
def keep_rejection(cycle: Cycle, transcript_path: str, rejected: Rejected) -> Stop | None:
"""Archives a judge reply loop-claude rejected, so the archive shows what the judge returned before a retry or an exit 1."""
if unkept := archived(cycle.project_archive, transcript_path, cycle.task, outcomes.rejected(rejected.reason, rejected.reply, str(judge_of(cycle)), rejected.cost), cycle.effects):
if unkept := archived(cycle.project_archive, transcript_path, cycle.task, outcomes.rejected(rejected.reason, rejected.reply, str(judge_of(cycle)), judge_cost=rejected.cost, judge_run=rejected.run), cycle.effects, rejected.run):
return Stop(1, unkept.reason)
return None
@@ -211,9 +212,9 @@ def ask(cycle: Cycle, transcript_path: str, ruling: Interrupt | QuotedAnswer) ->
return HumanTyped(TypedAnswer(ruling.question, answer))
def archive_stepped(cycle: Cycle, transcript_path: str, outcome: str, step_up: StepUp | None) -> Failure | None:
"""Archives the outcome of a ruling, then the step-up it made, if it made one. The first failure is returned."""
if failure := archived(cycle.project_archive, transcript_path, cycle.task, outcome, cycle.effects):
def archive_stepped(cycle: Cycle, transcript_path: str, outcome: str, step_up: StepUp | None, judge_run: JudgeRun | None) -> Failure | None:
"""Archives the outcome of a ruling and the judge run that gave it, then the step-up it made, if it made one. The first failure is returned."""
if failure := archived(cycle.project_archive, transcript_path, cycle.task, outcome, cycle.effects, judge_run):
return failure
return archived_step(cycle.project_archive, transcript_path, cycle.task, step_up, cycle.effects)
@@ -236,8 +237,8 @@ def close(cycle: Cycle, transcript_path: str, verdict: Finished | Unfinished) ->
annotation = entries if isinstance(entries, Failure) else annotation_of(cycle, transcript_path, entries, verdict.judged)
if isinstance(annotation, Failure):
return Stop(1, annotation.reason)
outcome = outcomes.finished(cycle.route.worker_line(), verdict.evidence, str(judge_of(cycle)), verdict.judged.cost, annotation) if isinstance(verdict, Finished) else outcomes.unfinished(cycle.route.worker_line(), verdict.evidence, str(judge_of(cycle)), verdict.escalation, verdict.judged.cost, annotation)
if failure := archived(cycle.project_archive, transcript_path, cycle.task, outcome, cycle.effects):
outcome = outcomes.finished(cycle.route.worker_line(), verdict.evidence, str(judge_of(cycle)), judge_cost=verdict.judged.cost, annotation=annotation, judge_run=verdict.judged.run) if isinstance(verdict, Finished) else outcomes.unfinished(cycle.route.worker_line(), verdict.evidence, str(judge_of(cycle)), verdict.escalation, judge_cost=verdict.judged.cost, annotation=annotation, judge_run=verdict.judged.run)
if failure := archived(cycle.project_archive, transcript_path, cycle.task, outcome, cycle.effects, verdict.judged.run):
return Stop(1, failure.reason)
if isinstance(verdict, Finished) and (failure := delete_session(cycle.session)):
return Stop(1, failure.reason)
+3 -1
View File
@@ -342,7 +342,9 @@ class Checkout:
for archived in (copy, kept):
self.test.assertEqual(stat.S_IMODE(archived.stat().st_mode), 0o600)
stamped = re.sub(r"^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ ", "<UTC> ", kept.read_text(), flags=re.MULTILINE)
outcomes[copy.read_text()] = re.sub(r"^Config: .*$", "Config: <config>", stamped, flags=re.MULTILINE)
configured = re.sub(r"^Config: .*$", "Config: <config>", stamped, flags=re.MULTILINE)
# The judge's run is named by the hash of its prompt and reply, which test_loop_claude.py checks.
outcomes[copy.read_text()] = re.sub(r"^Judge run: judge/[0-9a-f]{64}\.json\n", "", configured, flags=re.MULTILINE)
return outcomes
+32 -1
View File
@@ -174,6 +174,9 @@ def ran(stdout="", code=0, stderr=""):
return subprocess.CompletedProcess([], code, stdout, stderr)
JUDGE_RUN_LINE = re.compile(r"^Judge run: judge/[0-9a-f]{64}\.json\n", re.MULTILINE)
def judged(verdict, evidence="working", answer="", escalation="", release_ready=False, cost=None, reversed=""):
reply = {"answer": answer, "escalation": escalation, "evidence": evidence, "finished": verdict == "finished", "interrupt": verdict == "interrupt", "question": "Which one?" if verdict in ("interrupt", "quoted") else "", "answered_before": "", "left_open": "", "release_ready": release_ready, "reversed": reversed}
return ran(json.dumps({"structured_output": reply, **({} if cost is None else {"total_cost_usd": cost})}))
@@ -289,7 +292,8 @@ class Cycles(unittest.TestCase):
return run.Cycle(config=CONFIG, window=WINDOW, effects=fake.effects(), facts=facts, judge_model="judge-model", ladder=LADDER, premises="## Goals", project=None, project_archive=self.archive, records=records, root=str(self.home), route=Route(Rung("worker-model")), session=self.session, task="Do it.", plan=plan_of(""), unfinished_evidence=unfinished_evidence)
def archived_outcomes(self):
return sorted(path.read_text() for path in self.archive.glob("*/*.outcomes.md"))
"""Each outcomes file's text without its `Judge run:` lines, which name a file by the hash of a prompt and its reply: the tests of the judge's runs check them."""
return sorted(JUDGE_RUN_LINE.sub("", path.read_text()) for path in self.archive.glob("*/*.outcomes.md"))
def test_a_finished_chunk_is_archived_and_its_session_deleted(self):
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("finished")])
@@ -383,6 +387,32 @@ Worker: {OFF_LADDER}
f"{stamp} Judge reply rejected: the judge quoted 'never said', which is not in the transcript\n\nReply:\n> {misquoted.stdout}\n\nJudge: judge-model\n\n"
f"{stamp} Verdict: finished\n\nEvidence:\n> working\n\nConfig: {CONFIG}\nJudge: judge-model\nWorker: {OFF_LADDER}\n"])
def test_archives_each_judge_runs_prompt_and_reply_beside_the_rejection_interrupt_and_verdict_it_gave(self):
rejected, interrupted, finished = ran(json.dumps({"structured_output": {"interrupt": False}})), judged("interrupt", cost=0.25), judged("finished")
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)] * 2, claude=[rejected, interrupted, finished], inputs=["Use B."])
with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Finished("working"))
prompts = [stdin for args, stdin in fake.made("claude")[:3]]
[chunk] = self.archive.iterdir()
kept = "\n".join(path.read_text() for path in chunk.glob("*.outcomes.md"))
named = re.findall(r"^Judge run: (judge/[0-9a-f]{64}\.json)$", kept, re.MULTILINE)
self.assertEqual(len(named), 3, kept)
runs = [json.loads((chunk / name).read_text()) for name in named]
replies = [json.loads(reply.stdout) for reply in (rejected, interrupted, finished)]
self.assertEqual(sorted((found["prompt"], json.dumps(found["reply"])) for found in runs), sorted((prompt, json.dumps(reply)) for prompt, reply in zip(prompts, replies)))
self.assertEqual(sorted(path.name for path in (chunk / "judge").iterdir()), sorted(name.removeprefix("judge/") for name in named))
def test_stops_when_a_judge_run_cannot_be_archived(self):
self.archive.mkdir(parents=True)
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("finished")])
def replace_file(path, data):
return Failure("disk full") if "/judge/" in str(path) else real_replace_file(path, data)
real_replace_file = write.replace_file
with unittest.mock.patch("transcript_archive.write.replace_file", replace_file):
stop = run.run_cycle(self.cycle(fake))
self.assertEqual(stop.code, 1)
self.assertRegex(stop.reason, r"^could not archive the transcript: .*/judge/[0-9a-f]{64}\.json: disk full$")
def test_archives_what_each_judge_run_cost_on_an_interrupt_a_quoted_answer_and_a_rejected_misquote(self):
for first, then, inputs in (
(judged("interrupt", cost=0.25), judged("finished"), ["Use B."]),
@@ -396,6 +426,7 @@ Worker: {OFF_LADDER}
run.run_cycle(dataclasses.replace(self.cycle(fake, records="From AGENTS.md:\nUse A."), project_archive=pathlib.Path(archive)))
kept = "".join(path.read_text() for path in pathlib.Path(archive).glob("*/*.outcomes.md"))
self.assertEqual(kept.count("Judge cost: 0.2500\n"), 1, kept)
self.assertIn("Judge cost: 0.2500\nJudge run: judge/", kept)
def test_stops_with_exit_1_where_a_subagents_transcript_cannot_be_read_for_the_spend(self):
for verdict in ("interrupt", "finished"):
+3
View File
@@ -1,6 +1,7 @@
"""The verdicts cycle.judge.verdict_from builds from the judge's reply; cycle.run.rule then stamps every verdict but Finished with the items the chunk takes. They sit outside cycle/, since routing reads them and cycle imports routing."""
import dataclasses
from transcript_archive.judge_run import JudgeRun
from claude_code.tokens import Tokens
@@ -22,6 +23,8 @@ class Judged:
tokens: Tokens = Tokens()
# The human's whole typed answer that holds the judge's quote of a reversal of a choice the worker took on its own recommendation; empty for none, and where no typed answer holds the quote.
reversed: str = ""
# The prompt it read and its reply, which the archive keeps beside the verdict. Left out of comparisons and reprs, so that equal verdicts stay equal whatever prompt they came from.
run: JudgeRun | None = dataclasses.field(default=None, compare=False, repr=False)
@dataclasses.dataclass(frozen=True)
+3
View File
@@ -3,6 +3,7 @@
<archive root>/<project key>/<chunk>/<sha256>.jsonl
<archive root>/<project key>/<chunk>/<sha256>.outcomes.md
<archive root>/<project key>/<chunk>/session/...
<archive root>/<project key>/<chunk>/judge/<sha256>.json
- The archive root is $LOOP_CLAUDE_TRANSCRIPTS, or ~/.claude/loop-claude-transcripts (location.archive_root).
- The project key is the checkout's path relative to ~/code, or to / outside it (project_key.project_key).
@@ -18,6 +19,8 @@
Code keeps in a folder named after the session, beside the transcript. A file there is replaced only by a version that
grew from it. A file Claude Code rewrote whole lands beside its copy as <name>.<hash of its content>
(write.copy_for).
- judge/ holds each judge run an outcome names on its `Judge run:` line: the prompt the judge read and claude's
json reply, named by the hash of that content (judge_run.JudgeRun). judge-replay reads the prompt back.
write.archive writes the layout, at every interrupt and as every cycle ends. read.py and the tools that
measure finished chunks read it.
+35
View File
@@ -0,0 +1,35 @@
import dataclasses
import hashlib
import json
import pathlib
from failure import Failure
# The folder of a chunk's judge runs, beside its transcript copies.
FOLDER = "judge"
@dataclasses.dataclass(frozen=True)
class JudgeRun:
"""The prompt a judge run read and claude's json reply to it. The judge has no tools and one turn, so these two are its whole transcript."""
prompt: str
reply: dict
def content(self) -> bytes:
return json.dumps({"prompt": self.prompt, "reply": self.reply}, ensure_ascii=False, indent=1).encode()
@property
def file(self) -> str:
"""Where the archive keeps the run, relative to its chunk's folder. The name is the hash of the content, so a run archived twice is kept once."""
return f"{FOLDER}/{hashlib.sha256(self.content()).hexdigest()}.json"
@staticmethod
def read(path: pathlib.Path) -> "JudgeRun | Failure":
try:
kept = json.loads(path.read_text(encoding="utf-8"))
prompt, reply = kept["prompt"], kept["reply"]
except (OSError, ValueError, KeyError, TypeError) as failure:
return Failure(f"could not read the archived judge run {path}: {failure}")
if not (isinstance(prompt, str) and isinstance(reply, dict)):
return Failure(f"could not read the archived judge run {path}: its prompt is no string or its reply no object")
return JudgeRun(prompt, reply)
+26 -13
View File
@@ -1,6 +1,7 @@
import re
import typing
from transcript_archive.judge_run import FOLDER, JudgeRun
from model_routing.prices import PRICES
from text_files.markdown_text import quoted
from text_files.utc_stamp import STAMPED
@@ -29,6 +30,9 @@ JUDGE = "Judge: "
# What the judge's run cost, in dollars, on the line after JUDGE; absent where the verdict recorded none.
JUDGE_COST = "Judge cost: "
JUDGE_COST_AT = re.compile(rf"^{re.escape(JUDGE)}(.+)\n{re.escape(JUDGE_COST)}([0-9.]+)$", re.MULTILINE)
# The judge run the verdict came from, as a path in the chunk's folder (judge_run.JudgeRun.file), on the line after the judge's rung and cost; absent where the archive kept none.
JUDGE_RUN = "Judge run: "
JUDGE_RUN_AT = re.compile(rf"^{re.escape(JUDGE_RUN)}({FOLDER}/[0-9a-f]{{64}}\.json)$", re.MULTILINE)
# The claude-config release and commit that ran the cycle, on the line before JUDGE, such as `Config: v0.8.0 (<sha>)`.
@@ -84,9 +88,18 @@ def spent_stamps(outcomes: str) -> list[str]:
return [stamp for stamp, outcome in each_outcome(outcomes) if f"\n{SPENT}" in outcome]
def judge_lines(judge: str, judge_cost: float) -> str:
"""The judge's rung, and what its run cost where that is known."""
return f"{JUDGE}{judge}" + (f"\n{JUDGE_COST}{judge_cost:.4f}" if judge_cost else "")
def judge_lines(judge: str, judge_cost: float, judge_run: JudgeRun | None) -> str:
"""The judge's rung, what its run cost where that is known, and the archived run where there is one."""
return f"{JUDGE}{judge}" + (f"\n{JUDGE_COST}{judge_cost:.4f}" if judge_cost else "") + (f"\n{JUDGE_RUN}{judge_run.file}" if judge_run else "")
def judge_run_at(outcomes: str, stamp: str) -> str | None:
"""The path of the judge run that the outcome stamped stamp names; None where it names none."""
for found, outcome in each_outcome(outcomes):
# The last such line: an interrupt's question, which comes first, may hold one.
if found == stamp and (named := JUDGE_RUN_AT.findall(outcome)):
return named[-1]
return None
def judge_costs(outcomes: str) -> list[tuple[str, float]]:
@@ -98,31 +111,31 @@ def escalation_block(escalation: str) -> str:
return f"Escalation:\n{quoted(escalation)}\n\n" if escalation else ""
def finished(worker: WorkerLine, evidence: str, judge: str, judge_cost: float = 0.0, annotation: Annotation = Annotation()) -> str:
return f"{FINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{annotation_lines(annotation)}{judge_lines(judge, judge_cost)}\n{WORKER}{worker}"
def finished(worker: WorkerLine, evidence: str, judge: str, *, judge_cost: float = 0.0, annotation: Annotation = Annotation(), judge_run: JudgeRun | None = None) -> str:
return f"{FINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{annotation_lines(annotation)}{judge_lines(judge, judge_cost, judge_run)}\n{WORKER}{worker}"
def unfinished(worker: WorkerLine, evidence: str, judge: str, escalation: str, judge_cost: float = 0.0, annotation: Annotation = Annotation()) -> str:
def unfinished(worker: WorkerLine, evidence: str, judge: str, escalation: str, *, judge_cost: float = 0.0, annotation: Annotation = Annotation(), judge_run: JudgeRun | None = None) -> str:
"""escalation is the judge's quote showing that the worker's cited escalation trigger fired; empty for none."""
escalated = escalation_block(escalation)
return f"{UNFINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{annotation_lines(annotation)}{judge_lines(judge, judge_cost)}\n{WORKER}{worker}"
return f"{UNFINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{annotation_lines(annotation)}{judge_lines(judge, judge_cost, judge_run)}\n{WORKER}{worker}"
def interrupted(question: str, evidence: str, judge: str, escalation: str, worker: WorkerLine, judge_cost: float = 0.0, annotation: Annotation = Annotation()) -> str:
def interrupted(question: str, evidence: str, judge: str, escalation: str, worker: WorkerLine, *, judge_cost: float = 0.0, annotation: Annotation = Annotation(), judge_run: JudgeRun | None = None) -> str:
"""escalation is the judge's quote showing that the worker's cited escalation trigger fired; empty for none. worker is the line naming the route the worker ran on until the interrupt."""
escalated = escalation_block(escalation)
return f"{INTERRUPTED}\n\n{question}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{annotation_lines(annotation)}{judge_lines(judge, judge_cost)}\n{WORKER}{worker}"
return f"{INTERRUPTED}\n\n{question}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{annotation_lines(annotation)}{judge_lines(judge, judge_cost, judge_run)}\n{WORKER}{worker}"
def answered(*, question: str, sources: str, answer: str, evidence: str, escalation: str, judge: str, worker: WorkerLine, judge_cost: float = 0.0, annotation: Annotation = Annotation()) -> str:
def answered(*, question: str, sources: str, answer: str, evidence: str, escalation: str, judge: str, worker: WorkerLine, judge_cost: float = 0.0, annotation: Annotation = Annotation(), judge_run: JudgeRun | None = None) -> str:
"""escalation and worker are as on interrupted. Every parameter is keyword-only, since a swap of two str ones would still type-check."""
escalated = escalation_block(escalation)
return f"{ANSWERED}\n\n{question}\n\n{sources} says:\n{quoted(answer)}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{annotation_lines(annotation)}{judge_lines(judge, judge_cost)}\n{WORKER}{worker}"
return f"{ANSWERED}\n\n{question}\n\n{sources} says:\n{quoted(answer)}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{annotation_lines(annotation)}{judge_lines(judge, judge_cost, judge_run)}\n{WORKER}{worker}"
def rejected(why: str, reply: str, judge: str, judge_cost: float = 0.0) -> str:
def rejected(why: str, reply: str, judge: str, *, judge_cost: float = 0.0, judge_run: JudgeRun | None = None) -> str:
"""A judge reply loop-claude rejected, with the reason and the reply as json."""
return f"{REJECTED}: {why}\n\nReply:\n{quoted(reply)}\n\n{judge_lines(judge, judge_cost)}"
return f"{REJECTED}: {why}\n\nReply:\n{quoted(reply)}\n\n{judge_lines(judge, judge_cost, judge_run)}"
def stepped_up(worker: WorkerLine) -> str:
@@ -1,6 +1,7 @@
import contextlib
import hashlib
import io
import json
import pathlib
import sys
import tempfile
@@ -10,6 +11,7 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from transcript_archive.project_key import checkout_of, project_key
from failure import Failure
from transcript_archive.judge_run import JudgeRun
from transcript_archive import outcomes, read, write
from transcript_archive.location import project_archive
from text_files.utc_stamp import utc
@@ -31,6 +33,81 @@ class Archive(unittest.TestCase):
self.assertEqual(kept_outcomes.read_text(), "Task: Do it.\n\n2026-09-21T14:13:20Z Stopped: Ctrl-C\n\nWorker: m\n")
class JudgeRuns(unittest.TestCase):
judged = JudgeRun("The prompt.", {"structured_output": {"finished": True}, "total_cost_usd": 0.5})
def test_a_run_is_kept_once_in_the_chunk_folder_under_the_hash_of_its_content(self):
with tempfile.TemporaryDirectory() as root:
kept = pathlib.Path(root, "archive")
transcript = pathlib.Path(root, "t.jsonl")
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n')
with contextlib.redirect_stderr(io.StringIO()):
for offset in (0, 1):
outcome = outcomes.finished("w", "merged", "j", judge_run=self.judged)
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcome, NOW + offset, judge_run=self.judged))
(folder,) = kept.glob("*/")
(file,) = folder.glob("judge/*")
self.assertEqual(file, folder / self.judged.file)
self.assertEqual(file.name, hashlib.sha256(file.read_bytes()).hexdigest() + ".json")
self.assertEqual(json.loads(file.read_text()), {"prompt": "The prompt.", "reply": {"structured_output": {"finished": True}, "total_cost_usd": 0.5}})
self.assertEqual(JudgeRun.read(file), self.judged)
def test_names_the_run_on_the_line_after_the_judge_lines_and_reads_the_name_back(self):
outcome = outcomes.finished("w", "e", "j", judge_cost=0.5, judge_run=self.judged)
self.assertEqual(outcome, f"Verdict: finished\n\nEvidence:\n> e\n\nJudge: j\nJudge cost: 0.5000\nJudge run: {self.judged.file}\nWorker: w")
self.assertEqual(outcomes.rejected("why", "{}", "j", judge_run=self.judged), f"Judge reply rejected: why\n\nReply:\n> {{}}\n\nJudge: j\nJudge run: {self.judged.file}")
self.assertEqual(outcomes.workers(f"{utc(NOW)} {outcome}\n"), [(utc(NOW), "w")])
self.assertEqual(outcomes.judge_costs(f"{utc(NOW)} {outcome}\n"), [("j", 0.5)])
kept = f"Task: Do it.\n\n{utc(NOW)} {outcomes.stopped('w', 'Ctrl-C')}\n\n{utc(NOW + 1)} {outcome}\n\n{utc(NOW + 2)} {outcomes.finished('w', 'e', 'j')}\n"
self.assertEqual(outcomes.judge_run_at(kept, utc(NOW + 1)), self.judged.file)
self.assertIsNone(outcomes.judge_run_at(kept, utc(NOW + 2)))
def test_keeps_a_run_as_readable_utf_8_json_indented_by_one_space(self):
run = JudgeRun("Prompt é.", {"a": 1})
self.assertEqual(run.content(), b'{\n "prompt": "Prompt \xc3\xa9.",\n "reply": {\n "a": 1\n }\n}')
def test_reads_a_file_that_is_no_run_as_a_failure(self):
with tempfile.TemporaryDirectory() as root:
for text in ("not json", '{"prompt": "P"}', '{"prompt": 1, "reply": {}}', '{"prompt": "P", "reply": []}'):
with self.subTest(text=text):
path = pathlib.Path(root, "run.json")
path.write_text(text)
self.assertRegex(JudgeRun.read(path).reason, f"^could not read the archived judge run {path}: ")
def test_reads_no_path_that_leaves_the_judge_folder_from_an_outcome(self):
outcome = f"{utc(NOW)} Verdict: finished\n\nJudge run: judge/../../secret.json\n"
self.assertIsNone(outcomes.judge_run_at(outcome, utc(NOW)))
def test_names_the_run_on_each_kind_of_outcome_that_has_a_judge(self):
named = f"Judge: j\nJudge run: {self.judged.file}\n"
kept = [
outcomes.unfinished("w", "e", "j", "", judge_run=self.judged),
outcomes.interrupted("Q?", "e", "j", "", "w", judge_run=self.judged),
outcomes.answered(question="Q?", sources="S", answer="a", evidence="e", escalation="", judge="j", worker="w", judge_run=self.judged),
]
self.assertEqual([named in outcome for outcome in kept], [True, True, True])
def test_names_the_folder_it_cannot_make_in_the_failure_to_keep_a_run(self):
with tempfile.TemporaryDirectory() as root:
(pathlib.Path(root) / "judge").write_text("a file where the folder should be")
failure = write.keep_judge_run(pathlib.Path(root), self.judged)
self.assertRegex(failure.reason, f"^could not archive the transcript: {root}/judge: ")
self.assertNotIn("None", failure.reason)
def test_writes_no_outcome_where_the_run_cannot_be_kept(self):
with tempfile.TemporaryDirectory() as root:
kept = pathlib.Path(root, "archive")
transcript = pathlib.Path(root, "t.jsonl")
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n')
with contextlib.redirect_stderr(io.StringIO()):
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.stopped("w", "x"), NOW))
(folder,) = kept.glob("*/")
(folder / "judge").write_text("a file where the folder should be")
failure = write.archive(kept, str(transcript), "Do it.", outcomes.finished("w", "e", "j", judge_run=self.judged), NOW + 1, judge_run=self.judged)
self.assertIsInstance(failure, Failure)
self.assertNotIn("Verdict: finished", next(folder.glob("*.outcomes.md")).read_text())
class CopyFor(unittest.TestCase):
def test_skips_a_version_already_kept_beside_its_copy(self):
with tempfile.TemporaryDirectory() as temp:
+18 -2
View File
@@ -5,14 +5,15 @@ import typing
from transcript_archive.location import chunk_folder, outcomes_of
from failure import Failure
from transcript_archive.judge_run import JudgeRun
from text_files.file_writes import make_private, replace_file
from claude_code.session import session_files
from transcript_archive.outcomes import TASK
from text_files.utc_stamp import utc
def archive(project_archive: pathlib.Path, path: str, task: str, outcome: str, at: float, note: typing.Callable[[str], None] = lambda text: None) -> Failure | None:
"""Keeps the transcript at path, its outcomes file and its session's files in project_archive, in the layout this package's docstring gives; the failure that stopped it, or None. The outcome is stamped and appended to the outcomes file, which opens with the task only when the copy is new. Where a new copy lands, note is told where."""
def archive(project_archive: pathlib.Path, path: str, task: str, outcome: str, at: float, note: typing.Callable[[str], None] = lambda text: None, judge_run: JudgeRun | None = None) -> Failure | None:
"""Keeps the transcript at path, its outcomes file and its session's files in project_archive, in the layout this package's docstring gives; the failure that stopped it, or None. The outcome is stamped and appended to the outcomes file, which opens with the task only when the copy is new. Where a new copy lands, note is told where. A judge_run the outcome names is kept first, so the outcome never names a run the archive lacks."""
try:
with open(path, "rb") as handle:
raw = handle.read()
@@ -23,6 +24,8 @@ def archive(project_archive: pathlib.Path, path: str, task: str, outcome: str, a
for sibling in session_files(path):
if sibling != path and (unkept := mirror(pathlib.Path(sibling), folder / "session" / os.path.basename(sibling))):
return unkept
if judge_run and (unkept := keep_judge_run(folder, judge_run)):
return unkept
copy = folder / f"{hashlib.sha256(raw).hexdigest()}.jsonl"
outcomes = outcomes_of(copy)
stamped = f"{utc(at)} {outcome}"
@@ -36,6 +39,19 @@ def archive(project_archive: pathlib.Path, path: str, task: str, outcome: str, a
return None
def keep_judge_run(folder: pathlib.Path, run: JudgeRun) -> Failure | None:
kept = folder / run.file
if kept.exists():
return None
try:
make_private(kept.parent)
except OSError as failure:
return unarchived(failure.filename, failure.strerror)
if unwritten := replace_file(kept, run.content()):
return unarchived(kept, unwritten.reason)
return None
def unarchived(where: object, why: str | None) -> Failure:
return Failure(f"could not archive the transcript: {where}: {why}")
+3 -3
View File
@@ -182,16 +182,16 @@ Valid while `~/.claude` is a checkout of this repo.
Valid while claude takes `--session-id` and resumes by it, and the human runs loop-claude where tmux is installed.
## loop-claude: the archive keeps the raw transcript, and a failed archive stops the loop
## loop-claude: the archive keeps the raw transcript and each judge run, and a failed archive stops the loop
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. Valid while transcripts stay small enough to copy whole.
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. The judge's runs are archived too, as `judge/<sha256>.json` in the chunk's folder: the prompt it read and its json reply. Together they are its whole transcript, since it runs with no tools and one turn. A run is kept whole and deduplicated by hash, and the outcome of the verdict, interrupt or rejected reply it produced names it on a `Judge run:` line. Keeping it whole costs little: a prompt is at most `JUDGE_PROMPT_CHARS`, 400,000 characters. Measured 2026-11-17, the archive's 911 judge lines come to at most 0.36 GB of prompts, against 3.5 GB in its 1,211 transcript copies, the largest 23.4 MB. The judge is still priced from its `Judge cost:` line, which is claude's own total for the run, and the Spent block counts the live reply's tokens; the archived reply keeps its `usage` for a later reader, and nothing reads it back. `judge-replay` replays a verdict on its archived prompt where there is one. Valid while transcripts, and the judge's prompts, stay small enough to copy whole.
## loop-claude: each verdict records the claude-config release and commit that ran it, the raw tokens each thread spent by model and kind since the previous outcome that recorded a spend, and the human's typed answer the judge quotes as reversing a choice the worker took on its own recommendation
2026-11-12, lilleman, todo item 305. Serves goals 1, 1.2 and 1.4. A change to the setup can be judged only by comparing chunks run before it with chunks run after, so each verdict, interrupt and quoted answer in loop-claude's archive records three things beside its judge lines.
- `Config: <release> (<commit>)`: `git describe` and the commit of `~/.claude`, so outcomes group by the setup that produced them. Where git can't tell, it records `unknown` and why, on one line.
- A `Spent since` block: the raw tokens each thread spent, by model and by kind (input, cache write 5m, cache write 1h, cache read, output), then the list prices in force, from `bin/model_routing/prices.py`. The threads are the worker's main thread, each subagent by its agent type and description, and the judge run that ruled. Each message counts once, by its id. The stretch runs from the previous outcome that recorded a spend, so what a `/talk` or a stopped run spent between rulings is counted once. It keeps tokens, since prices change. The prices recorded beside them give what the stretch cost when it ran. Claude Code doesn't report thinking apart from output, so it is left in output. The judge's tokens come from its json reply's `usage`, until item 303 archives the judge's own transcript.
- A `Spent since` block: the raw tokens each thread spent, by model and by kind (input, cache write 5m, cache write 1h, cache read, output), then the list prices in force, from `bin/model_routing/prices.py`. The threads are the worker's main thread, each subagent by its agent type and description, and the judge run that ruled. Each message counts once, by its id. The stretch runs from the previous outcome that recorded a spend, so what a `/talk` or a stopped run spent between rulings is counted once. It keeps tokens, since prices change. The prices recorded beside them give what the stretch cost when it ran. Claude Code doesn't report thinking apart from output, so it is left in output. The judge's tokens come from its json reply's `usage`, which the archived judge run holds too.
- A `Reversed:` quote: the human's typed answer that overrides a choice the worker took on its own recommendation and recorded. The judge quotes the latest such `[human]` line; loop-claude keeps the whole typed answer that holds the quote verbatim, and drops a quote no typed answer holds. `score-outcomes` counts the distinct answers per finished chunk. That count is goal 1.2's counter-signal, and the 2026-11-08 entry of `loop-claude: a judge decides each interrupt…` is valid only while it stays low. It counts only chunks that record a Config line, since earlier judges didn't look for reversals.
Valid while score-outcomes reads the reversals from the archive.
+1 -19
View File
@@ -12,9 +12,8 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 303 | 0.12.0 | | defect | **Archive each judge run's prompt and json reply beside the verdict it gave, so judge cost, replays and backups cover the judge as they cover the worker.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3, 3 by 2 | 4.2 |
| 306 | 0.12.0 | | | **Set a strategy that keeps loop-claude's archive and larv-review's review folders backed up and small over the years, compressing older entries without losing any.** | 3 | 2 | 4 | 7 | 5 | 3 by 5 | 3.9 |
| 304 | 0.12.0 | 303 | defect | **Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 2, 3 by 2 | 3.8 |
| 304 | 0.12.0 | | defect | **Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 2, 3 by 2 | 3.8 |
| 387 | 0.12.0 | | | **Bind a project's rung to its ladder in one value, so the routing functions and `Cycle` take one argument for both.** | 3 | 1 | 3 | 5 | 5 | code health by 3 | 2.9 |
| 338 | 0.12.0 | | owner | **Script each step that 0.9.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.3 |
| 288 | 0.12.0 | | | **Point the two reading-cost decisions at code health and goal 1.4's counter-signal, in place of the goal 1.1 they cite.** | 1 | 2 | 1 | 2 | 1 | code health by 1, 1.4 by 1 | 2.2 |
@@ -76,23 +75,6 @@
## Details
### 303. Archive each judge run's prompt and json reply beside the verdict it gave, so judge cost, replays and backups cover the judge as they cover the worker.
Today no judge transcript reaches loop-claude's archive. `bin/claude_code/reply.py` runs every headless reply, the judge's included, with `--no-session-persistence`, so its runs stay out of the session picker; Claude Code then writes no session file for the archive to copy. No decision chose to leave the judge out: `loop-claude: the archive keeps the raw transcript` was written for the worker's `.jsonl`. Goal 3 asks for the transcripts of unattended runs to be backed up, and the judge is one. The gap also leaves the judge's spend measurable only from the `Judge cost:` lines, which older verdicts lack ($136 recorded over 2026-09-30 to 2026-11-11), and leaves `judge-replay` rebuilding each prompt from today's records: of the 39 verdicts it replayed on 2026-10-15, 8 may rest on records that had changed since.
The judge runs with no tools and one turn (`judge_args` in `bin/loop_claude/cycle/judge.py`), so its whole transcript is the prompt it read and the json reply. Recommended: keep `--no-session-persistence`, and have the archive write the prompt and the reply beside the outcomes entry of the verdict, interrupt or rejected reply they produced, deduplicated by content hash as the worker's copies are. A failed write stops the loop, as that decision says. Persisting the session and copying its file would also work, but fills the picker and needs a cleanup. The prompt repeats the worker's transcript, which the archive already holds: measure the size per cycle first, and if it outgrows "small enough to copy whole", keep the prompt without the transcript part and name the worker copy and entry range it held. Then let `judge-replay` read the archived prompt where one exists, and price the judge from the archived reply's usage. Update that decision entry to say the judge's runs are archived too.
Touches:
- `bin/loop_claude/cycle/judge.py`: `run_judge`, `judge_once`
- `bin/loop_claude/cycle/run.py`: `(module)` (small)
- `bin/transcript_archive/write.py`: `archive`, `copy_for`
- `bin/transcript_archive/location.py`
- `bin/transcript_archive/test_transcript_archive.py`
- `bin/judge_replay/main.py`: `archived_cases`, `rule`
- `bin/judge_replay/replay.py`: `case_from`
- `bin/judge_replay/test_judge_replay.py`
### 306. Set a strategy that keeps loop-claude's archive and larv-review's review folders backed up and small over the years, compressing older entries without losing any.
Measured on 2026-11-11: loop-claude's archive is 6.3 GB (12,721 files from October alone, 0.84 GB older than 30 days), and `~/.local/state/larv-review/` is 997 MB, 984 MB of it under `code/`, with no pruning: `skills/larv-review/SKILL.md` → "Reports" keeps every folder after the merge, because `review-replay` replays rounds from them. The largest archived worker transcript, 23.4 MB, gzips to 9.1 MB at level 6. The disk holds 65 GB free today.