Merge remote-tracking branch 'origin/main' into predict-touches
CI / tests (pull_request) Successful in 2m59s
CI / tests (pull_request) Successful in 2m59s
This commit was merged in pull request #570.
This commit is contained in:
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- loop-claude's archive keeps each judge run's prompt and json reply in a judge/ folder beside its chunk's transcripts, and names the run on a `Judge run:` line of the outcome it produced; judge-replay replays a verdict on its archived prompt where the archive holds one.
|
||||
- A plan kept as a Jira epic takes no changelog bullet: `todo-item finish` there needs neither `--changelog` nor `--no-changelog`, and refuses `--changelog`, since the issues' history is the changelog.
|
||||
|
||||
## 0.11.0
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
from claude_code.entries import Entry
|
||||
from loop_claude import escalation_mark
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Grounds, JUDGE_RULES, TASK_HEADING, TRANSCRIPT_HEADING
|
||||
from claude_code.transcript import human_answers
|
||||
from loop_claude.cycle.prompts import typed_answer
|
||||
|
||||
|
||||
def grounds_of(prompt: str, entries: list[Entry]) -> Grounds | Failure:
|
||||
"""What the quotes of a judge run on prompt are checked against, as judge_prompt would have made it for the transcript's entries: the premises and the records before the task are quotable, the transcript the judge was shown comes before the rules. The prompt is split on the headings judge_prompt writes, at the first of each; a Failure where it lacks one. Rules of an earlier wording stay part of the transcript, a slack that lets an evidence quote them pass."""
|
||||
before_task, task_heading, after_task = prompt.partition(f"\n\n{TASK_HEADING}")
|
||||
_, transcript_heading, shown = after_task.partition(TRANSCRIPT_HEADING)
|
||||
if not (task_heading and transcript_heading):
|
||||
return Failure(f"the archived prompt lacks the heading {TASK_HEADING.strip()!r} or {TRANSCRIPT_HEADING.strip()!r}, which judge_prompt writes")
|
||||
return Grounds(shown.removesuffix(f"\n\n{JUDGE_RULES}"), escalation_mark.after_mark(entries, 0), before_task, tuple(human_answers(entries, typed_answer)))
|
||||
+25
-10
@@ -4,13 +4,16 @@ usage: judge-replay <model>
|
||||
|
||||
<model> is a project model on the routing ladder this setup ships, such as claude-opus-5-5. The tool replays the
|
||||
verdicts archived for the projects that ~/.claude/loop-claude-models.json runs on it: the latest
|
||||
15 of each kind, finished, unfinished, interrupt and quoted answer. Each transcript copy in the
|
||||
archive is what the judge read for the verdict beside it, so the replay hands the judge that copy,
|
||||
with the task, premises and records each project holds now.
|
||||
15 of each kind, finished, unfinished, interrupt and quoted answer. Where the archive kept the judge's own
|
||||
prompt (docs/decisions.md → `loop-claude: the archive keeps the raw transcript and each judge run`), the
|
||||
replay hands the judge that prompt, which holds what the judge read then. For a verdict from before the
|
||||
archive kept it, the replay builds the prompt from the transcript copy beside the verdict, with the task,
|
||||
premises and records each project holds now.
|
||||
|
||||
It first replays every verdict on the reference judge: the model at xhigh effort, the rung the
|
||||
judge ran on before loop-claude routed it. A verdict the reference judge returns another kind on
|
||||
may rest on records or judge rules that have changed since; either way no rung is held to it. It
|
||||
may rest, where its prompt was rebuilt, on records or judge rules that have changed since; where the archived prompt was
|
||||
replayed, the judge's own variance is the cause. Either way no rung is held to it. It
|
||||
then replays the rest on each rung above the model's cheapest that a chunk's E + I raises a
|
||||
review to, except the reference judge's rung. Then it replays them on each candidate rung,
|
||||
highest first: from the model's cheapest rung down to the lowest rung a worker routes to. It stops
|
||||
@@ -25,7 +28,7 @@ keeps the judge to its own model's rungs; a rung above them that misses no inter
|
||||
evidence to let the judge climb onto it.
|
||||
|
||||
Like the loop, it tries a judge three times running before it counts the reply as failed. A
|
||||
verdict whose project or transcript cannot be read replays as failed too. Exit 1, saying why, on
|
||||
verdict whose project, transcript or archived judge run cannot be read replays as failed too. Exit 1, saying why, on
|
||||
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
|
||||
or a usage limit.
|
||||
"""
|
||||
@@ -35,12 +38,14 @@ import typing
|
||||
from transcript_archive.location import archive_root, transcript_of
|
||||
from transcript_archive.read import archived_outcomes
|
||||
from rung_replay.command import arguments_or_exit, down_from, failed, limited_exit, projects_on
|
||||
from loop_claude.cycle.judge import Brief, judge_once
|
||||
from loop_claude.cycle.judge import Brief, judge_archived, judge_once
|
||||
from judge_replay.replay import Case, case_from, Kind, report, reproduced, sample, verdict_kind
|
||||
from transcript_archive.project_key import checkout_of
|
||||
from loop_effects.effects import Effects
|
||||
from failure import Failure
|
||||
from rung_replay.search import first_reply, replayed
|
||||
from judge_replay.archived_prompt import grounds_of
|
||||
from transcript_archive.judge_run import JudgeRun
|
||||
from model_routing.ladder import Ladder, REVIEW_STEPS, Rung
|
||||
from claude_code.reply import Limited
|
||||
from model_routing.ladder_file import read_ladder, SHIPPED
|
||||
@@ -107,13 +112,23 @@ def archived_cases(model: str, root: pathlib.Path, home: pathlib.Path) -> list[C
|
||||
|
||||
|
||||
def rule(case: Case, rung: Rung, home: pathlib.Path, effects: Effects) -> Kind | Limited:
|
||||
premises, records, entries = read_premises(case.root, home), read_records(case.root, home, home / ".claude"), read_transcript(str(case.transcript))
|
||||
entries = read_transcript(str(case.transcript))
|
||||
if isinstance(entries, Failure):
|
||||
return verdict_kind(entries)
|
||||
if case.judge_run:
|
||||
kept = JudgeRun.read(case.judge_run)
|
||||
if isinstance(kept, Failure):
|
||||
return verdict_kind(kept)
|
||||
grounds = grounds_of(kept.prompt, entries)
|
||||
if isinstance(grounds, Failure):
|
||||
return verdict_kind(grounds)
|
||||
verdict: Verdict | Failure | Limited = first_reply(lambda: judge_archived(case.root, kept.prompt, grounds, len(entries), rung, effects))
|
||||
return verdict if isinstance(verdict, Limited) else verdict_kind(verdict)
|
||||
premises, records = read_premises(case.root, home), read_records(case.root, home, home / ".claude")
|
||||
if isinstance(premises, Failure):
|
||||
return verdict_kind(premises)
|
||||
if isinstance(records, Failure):
|
||||
return verdict_kind(records)
|
||||
if isinstance(entries, Failure):
|
||||
return verdict_kind(entries)
|
||||
# 0 for the route's mark, as before any step-up: the replay compares verdict kinds, and a mark changes only which escalation quotes count.
|
||||
verdict: Verdict | Failure | Limited = first_reply(lambda: judge_once(case.root, Brief(case.task, premises, records, entries, 0), rung, effects))
|
||||
verdict = first_reply(lambda: judge_once(case.root, Brief(case.task, premises, records, entries, 0), rung, effects))
|
||||
return verdict if isinstance(verdict, Limited) else verdict_kind(verdict)
|
||||
|
||||
@@ -4,7 +4,7 @@ import dataclasses
|
||||
import pathlib
|
||||
import typing
|
||||
|
||||
from transcript_archive.outcomes import ANSWERED, FINISHED, first_verdict, INTERRUPTED, task_of, UNFINISHED
|
||||
from transcript_archive.outcomes import ANSWERED, FINISHED, first_verdict, INTERRUPTED, judge_run_at, task_of, UNFINISHED
|
||||
from failure import Failure
|
||||
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict
|
||||
from rung_replay.search import Replay
|
||||
@@ -22,6 +22,8 @@ class Case:
|
||||
transcript: pathlib.Path
|
||||
at: str
|
||||
archived_kind: Kind
|
||||
# The judge run the archive kept beside the verdict, which holds the prompt the judge read; None where it kept none.
|
||||
judge_run: pathlib.Path | None = None
|
||||
|
||||
|
||||
def case_from(root: str, outcomes: str, transcript: pathlib.Path) -> Case | None:
|
||||
@@ -32,7 +34,8 @@ def case_from(root: str, outcomes: str, transcript: pathlib.Path) -> Case | None
|
||||
task, verdict = task_of(outcomes), first_verdict(outcomes)
|
||||
if task is None or verdict is None:
|
||||
return None
|
||||
return Case(root, task, transcript, *verdict)
|
||||
run = judge_run_at(outcomes, verdict[0])
|
||||
return Case(root, task, transcript, *verdict, transcript.parent / run if run else None)
|
||||
|
||||
|
||||
def sample(cases: list[Case], per_kind: int) -> list[Case]:
|
||||
@@ -88,4 +91,4 @@ def report(model: str, reference: Rung, cases: list[Case], replay: Replay[Case,
|
||||
|
||||
|
||||
def missed_lines(missed: list[tuple[Case, Kind]]) -> list[str]:
|
||||
return [f" {found.root} {found.at}: archived {found.archived_kind}, replayed {kind}" for found, kind in missed]
|
||||
return [f" {found.root} {found.at}: archived {found.archived_kind}, replayed {kind} on {'its archived prompt' if found.judge_run else 'a prompt rebuilt from today\'s records'}" for found, kind in missed]
|
||||
|
||||
@@ -10,11 +10,14 @@ import unittest
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINISHED
|
||||
from claude_code.entries import AssistantTurn, HumanTurn, LoopPrompt, Text
|
||||
from judge_replay.replay import Case, case_from, report, reproduced, sample, verdict_kind
|
||||
from project_config.test_project_config import configured
|
||||
from loop_effects.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished
|
||||
from judge_replay.archived_prompt import grounds_of
|
||||
from loop_claude.cycle.judge import judge_archived, JUDGE_RULES, Rejected, TASK_HEADING, TRANSCRIPT_HEADING
|
||||
from judge_replay import main
|
||||
from rung_replay.search import Miss, Replay, Tried
|
||||
from model_routing.ladder import Rung
|
||||
@@ -41,6 +44,11 @@ class CaseFrom(unittest.TestCase):
|
||||
found = case_from("/code/p", f"Task: t\n\n2026-10-14T10:00:00Z {verdict}\n\nmore\n", TRANSCRIPT)
|
||||
self.assertEqual(found.archived_kind, verdict)
|
||||
|
||||
def test_names_the_judge_run_that_the_first_verdict_records_beside_it(self):
|
||||
outcomes = f"Task: t\n\n2026-10-14T10:00:00Z Verdict: unfinished\n\nJudge: j\nJudge run: judge/{'a' * 64}.json\nWorker: x\n\n2026-10-14T11:00:00Z Verdict: finished\n\nJudge: j\nJudge run: judge/{'b' * 64}.json\nWorker: x\n"
|
||||
self.assertEqual(case_from("/code/p", outcomes, TRANSCRIPT).judge_run, TRANSCRIPT.parent / f"judge/{'a' * 64}.json")
|
||||
self.assertIsNone(case_from("/code/p", "Task: t\n\n2026-10-14T10:00:00Z Verdict: finished\n\nJudge: j\nWorker: x\n", TRANSCRIPT).judge_run)
|
||||
|
||||
def test_skips_a_copy_with_no_verdict(self):
|
||||
self.assertIsNone(case_from("/code/p", "Task: t\n\n2026-10-14T10:00:00Z Stepped up\n\nWorker: y\n", TRANSCRIPT))
|
||||
self.assertIsNone(case_from("/code/p", "not an outcomes file", TRANSCRIPT))
|
||||
@@ -68,16 +76,21 @@ class Reproduced(unittest.TestCase):
|
||||
|
||||
|
||||
class Report(unittest.TestCase):
|
||||
def test_names_whether_a_missed_verdict_was_replayed_on_its_archived_prompt(self):
|
||||
kept = Case("/code/p", "Do it.", TRANSCRIPT, "2026-10-14T10:00:00Z", FINISHED, pathlib.Path("/archive/p/c/judge/run.json"))
|
||||
lines = report("claude-opus-5-5", REFERENCE, [kept], Replay([UNFINISHED], [], [])).split("\n")
|
||||
self.assertIn(" /code/p 2026-10-14T10:00:00Z: archived Verdict: finished, replayed Verdict: unfinished on its archived prompt", lines)
|
||||
|
||||
def test_counts_the_verdicts_and_names_each_miss_and_the_lowest_rung(self):
|
||||
cases = [case("2026-10-14T10:00:00Z", FINISHED), case("2026-10-14T10:00:01Z", INTERRUPTED)]
|
||||
replay = Replay([UNFINISHED, INTERRUPTED], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(cases[1], INTERRUPTED, FINISHED)])], MIDDLE)
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, replay).splitlines(), [
|
||||
"Replayed 2 verdicts of claude-opus-5-5 projects: 1 Verdict: finished, 1 Verdict: interrupt.",
|
||||
"The reference judge, claude-opus-5-5 at xhigh effort, reproduced 1. It returned another kind on:",
|
||||
" /code/p 2026-10-14T10:00:00Z: archived Verdict: finished, replayed Verdict: unfinished",
|
||||
" /code/p 2026-10-14T10:00:00Z: archived Verdict: finished, replayed Verdict: unfinished on a prompt rebuilt from today's records",
|
||||
"claude-sonnet-5-5 at high effort missed 0.",
|
||||
"claude-haiku-5-5 at xhigh effort missed 1:",
|
||||
" /code/p 2026-10-14T10:00:01Z: archived Verdict: interrupt, replayed Verdict: finished",
|
||||
" /code/p 2026-10-14T10:00:01Z: archived Verdict: interrupt, replayed Verdict: finished on a prompt rebuilt from today's records",
|
||||
"Lowest candidate that, like each candidate above it, returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort",
|
||||
])
|
||||
missed_all = report("claude-opus-5-5", REFERENCE, cases, Replay([FINISHED, INTERRUPTED], [Tried(MIDDLE, [Miss(cases[1], INTERRUPTED, FINISHED)])], None)).splitlines()
|
||||
@@ -90,7 +103,7 @@ class Report(unittest.TestCase):
|
||||
climbs = [Tried(Rung("claude-fable-5-2", "low"), [Miss(cases[1], INTERRUPTED, FINISHED)]), Tried(Rung("claude-fable-5-2", "high"), [])]
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, Replay([FINISHED, INTERRUPTED], [Tried(MIDDLE, [])], MIDDLE, climbs)).splitlines()[2:5], [
|
||||
"claude-fable-5-2 at low effort, a rung a review climbs to, missed 1:",
|
||||
" /code/p 2026-10-14T10:00:01Z: archived Verdict: interrupt, replayed Verdict: finished",
|
||||
" /code/p 2026-10-14T10:00:01Z: archived Verdict: interrupt, replayed Verdict: finished on a prompt rebuilt from today's records",
|
||||
"claude-fable-5-2 at high effort, a rung a review climbs to, missed 0.",
|
||||
])
|
||||
|
||||
@@ -161,6 +174,30 @@ class Main(unittest.TestCase):
|
||||
# The replay shows the judge the whole transcript, with no step-up marked in it.
|
||||
self.assertNotIn("[step]", self.prompts[0])
|
||||
|
||||
def test_hands_the_judge_the_prompt_the_archive_kept_and_rebuilds_the_prompt_of_a_verdict_with_none(self):
|
||||
kept = self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1"
|
||||
prompt = "From the goals file, OLD premises\n\nThe records: none\n\nThe task the worker was given:\nDo it.\n\nWhat the worker did and said, what the human typed on [human] lines and what loop-claude told it on [loop] lines, latest last:\nsaid Verdict: finished\n\nOLD rules"
|
||||
(kept / "judge").mkdir()
|
||||
(kept / "judge" / "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.json").write_text(json.dumps({"prompt": prompt, "reply": {"type": "result"}}))
|
||||
(kept / "t0.outcomes.md").write_text("Task: Do it.\n\n2026-10-14T10:00:00Z Verdict: finished\n\nJudge: j\nJudge run: judge/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.json\nWorker: x\n")
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertIn("The reference judge, claude-opus-5-5 at xhigh effort, reproduced 2.", out)
|
||||
# The first replay, on the finished verdict, reads the archived prompt; the second, on the interrupt, is rebuilt from the records of today.
|
||||
self.assertEqual(self.prompts[0], prompt)
|
||||
self.assertEqual(self.calls[0][2], self.calls[1][2])
|
||||
self.assertTrue(self.calls[0][2].endswith("/code/proj"))
|
||||
self.assertIn("The task the worker was given:\nDo it.", self.prompts[1])
|
||||
self.assertIn("1. Ship.", self.prompts[1])
|
||||
|
||||
def test_a_verdict_whose_archived_judge_run_cannot_be_read_replays_as_failed(self):
|
||||
kept = self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1"
|
||||
(kept / "t0.outcomes.md").write_text("Task: Do it.\n\n2026-10-14T10:00:00Z Verdict: finished\n\nJudge: j\nJudge run: judge/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.json\nWorker: x\n")
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertIn("The reference judge, claude-opus-5-5 at xhigh effort, reproduced 1. It returned another kind on:", out)
|
||||
self.assertIn("could not read the archived judge run", out)
|
||||
|
||||
def test_tries_no_rung_below_the_lowest_a_worker_routes_to_where_every_rung_returns_every_verdict(self):
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
self.assertEqual(code, 0, err)
|
||||
@@ -226,6 +263,80 @@ class Main(unittest.TestCase):
|
||||
self.assertEqual(ran.returncode, 0)
|
||||
self.assertIn("usage: judge-replay <model>", ran.stdout)
|
||||
|
||||
def test_exits_1_on_a_usage_limit_met_on_an_archived_prompt(self):
|
||||
kept = self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1"
|
||||
(kept / "judge").mkdir()
|
||||
(kept / "judge" / "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.json").write_text(json.dumps({"prompt": "P", "reply": {}}))
|
||||
(kept / "t0.outcomes.md").write_text("Task: Do it.\n\n2026-10-14T10:00:00Z Verdict: finished\n\nJudge: j\nJudge run: judge/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.json\nWorker: x\n")
|
||||
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: {"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
|
||||
self.assertEqual((code, "usage limit: You've hit your limit" in err), (1, True))
|
||||
|
||||
|
||||
class JudgeArchivedRuns(unittest.TestCase):
|
||||
"""judge_archived on a prompt as judge_prompt writes it: the premises, the task, the transcript the judge was shown, and the rules."""
|
||||
PROMPT = f"HEAD: Ship fast.\n\n{TASK_HEADING}Do it.\n\n{TRANSCRIPT_HEADING}Worker chose option B.\n[human] Use Postgres.\n\n{JUDGE_RULES}"
|
||||
ENTRIES = [AssistantTurn(parts=(Text("Worker chose option B."),)), HumanTurn(text="Use Postgres."), LoopPrompt(text="Keep going.")]
|
||||
|
||||
def judged(self, prompt=PROMPT, **fields):
|
||||
"""What judge_archived returns on prompt where the judge's structured output is fields over an unfinished verdict, and the cwd claude ran in."""
|
||||
output = {"answer": "", "escalation": "", "evidence": "Worker chose option B.", "finished": False, "interrupt": False, "question": "", "answered_before": "", "left_open": "", "reversed": "", "release_ready": False, **fields}
|
||||
cwds = []
|
||||
|
||||
def claude(args, cwd, prompt):
|
||||
cwds.append(cwd)
|
||||
return subprocess.CompletedProcess(args, 0, json.dumps({"type": "result", "structured_output": output, "total_cost_usd": 0.25}), "")
|
||||
|
||||
effects = Effects(claude=claude, claude_on_terminal=unused, discard_typed=unused, dm=unused, execv=unused, git=unused, input=unused, local_time=unused, memory=unused, now=unused, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused, tmux=unused, tmux_on_terminal=unused)
|
||||
grounds = grounds_of(prompt, self.ENTRIES)
|
||||
return (grounds if isinstance(grounds, Failure) else judge_archived("/code/p", prompt, grounds, len(self.ENTRIES), REFERENCE, effects)), cwds
|
||||
|
||||
def test_runs_in_the_projects_folder_and_stamps_the_verdict_with_the_entries_read(self):
|
||||
verdict, cwds = self.judged()
|
||||
self.assertEqual((cwds, verdict.entries_read), (["/code/p"], 3))
|
||||
|
||||
def test_checks_the_evidence_against_the_transcript_the_judge_was_shown_and_not_the_premises(self):
|
||||
self.assertIsInstance(self.judged(evidence="Ship fast.")[0], Rejected)
|
||||
self.assertIsInstance(self.judged(evidence="Worker chose option B.")[0], Unfinished)
|
||||
|
||||
def test_keeps_the_cost_and_the_run_of_a_rejected_reply(self):
|
||||
rejected, _ = self.judged(evidence="Never said.")
|
||||
self.assertEqual((rejected.cost, rejected.run.prompt), (0.25, self.PROMPT))
|
||||
|
||||
def test_keeps_the_cost_of_a_reply_rejected_for_its_shape(self):
|
||||
rejected, _ = self.judged(finished="yes")
|
||||
self.assertEqual((type(rejected), rejected.cost), (Rejected, 0.25))
|
||||
|
||||
def test_checks_neither_the_task_nor_the_rules_as_the_transcript_or_the_premises(self):
|
||||
self.assertIsInstance(self.judged(evidence=JUDGE_RULES)[0], Rejected)
|
||||
self.assertIsInstance(self.judged(answer="Do it.", question="Which?")[0], Interrupt)
|
||||
|
||||
def test_splits_the_prompt_at_the_first_transcript_heading(self):
|
||||
quoting = self.PROMPT.replace("Worker chose", f"Quoted: {TRANSCRIPT_HEADING}Worker chose", 1)
|
||||
self.assertIsInstance(self.judged(quoting, evidence="Quoted:")[0], Unfinished)
|
||||
|
||||
def test_splits_the_prompt_at_the_first_task_heading(self):
|
||||
quoting = self.PROMPT.replace("Do it.", f"Quote:\n\n{TASK_HEADING}Ship slowly.", 1)
|
||||
self.assertIsInstance(self.judged(quoting, answer="Ship fast.", question="Which?")[0], QuotedAnswer)
|
||||
self.assertIsInstance(self.judged(quoting, answer="Quote:", question="Which?")[0], Interrupt)
|
||||
|
||||
def test_fails_before_the_judge_runs_on_a_prompt_without_a_heading(self):
|
||||
for prompt in ("No headings at all.", f"{TASK_HEADING}Do it.", f"\n\n{TASK_HEADING}Do it.\n\nNo transcript heading."):
|
||||
with self.subTest(prompt=prompt):
|
||||
failure, cwds = self.judged(prompt)
|
||||
self.assertEqual((type(failure), cwds), (Failure, []))
|
||||
self.assertIn("lacks the heading", failure.reason)
|
||||
|
||||
def test_checks_a_quoted_answer_against_the_premises(self):
|
||||
self.assertIsInstance(self.judged(answer="Ship fast.", question="Which?")[0], QuotedAnswer)
|
||||
self.assertIsInstance(self.judged(answer="Use Postgres.", question="Which?")[0], Interrupt)
|
||||
|
||||
def test_keeps_an_escalation_that_the_worker_wrote_after_the_start_of_the_transcript(self):
|
||||
self.assertEqual(self.judged(escalation="Worker chose option B.")[0].escalation, "Worker chose option B.")
|
||||
self.assertEqual(self.judged(escalation="Use Postgres.")[0].escalation, "")
|
||||
|
||||
def test_names_the_typed_answer_that_holds_a_reversal(self):
|
||||
self.assertEqual(self.judged(reversed="Use Postgres.")[0].judged.reversed, "Use Postgres.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -11,6 +11,7 @@ from loop_claude import escalation_mark
|
||||
from failure import Failure
|
||||
from loop_claude.verdicts import Finished, Interrupt, Judged, QuotedAnswer, Unfinished, Verdict
|
||||
from claude_code.transcript import human_answers
|
||||
from transcript_archive.judge_run import JudgeRun
|
||||
from loop_claude.quotes import plain, verbatim
|
||||
from loop_claude.cycle.retry import retried
|
||||
from model_routing.ladder import Rung
|
||||
@@ -64,6 +65,10 @@ FINISHED_RULES = (
|
||||
"continue: an open PR, a review not yet run, CI not yet reported, an answer not yet recorded."
|
||||
)
|
||||
JUDGE_RULES = "\n\n".join((INTERRUPT_RULES, QUOTE_RULES, FINISHED_RULES))
|
||||
# Open the task and the transcript in the prompt; the premises and the records come before the task, the rules after the transcript.
|
||||
# judge_replay.archived_prompt splits each archived prompt on them, so rewording one makes every run archived before it replay as failed.
|
||||
TASK_HEADING = "The task the worker was given:\n"
|
||||
TRANSCRIPT_HEADING = "What the worker did and said, what the human typed on [human] lines and what loop-claude told it on [loop] lines, latest last:\n"
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -94,10 +99,15 @@ JUDGE_SCHEMA = {
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Rejected(Failure):
|
||||
"""A judge reply loop-claude rejected: reason says why, reply holds the reply as json, and cost is what the run cost in dollars, 0 where its reply records none."""
|
||||
reply: str
|
||||
"""A judge reply loop-claude rejected: reason says why, run holds the prompt and the reply, and cost is what the run cost in dollars, 0 where its reply records none."""
|
||||
run: JudgeRun
|
||||
cost: float = 0.0
|
||||
|
||||
@property
|
||||
def reply(self) -> str:
|
||||
"""The reply as json."""
|
||||
return json.dumps(self.run.reply, ensure_ascii=False)
|
||||
|
||||
|
||||
class Grounds(typing.NamedTuple):
|
||||
"""The texts quotes_checked checks the judge's quotes against. A quote must appear in its text as written, since a quote reworded or stitched together can say what its source does not; only the evidence may join passages with an ellipsis, each passage in order."""
|
||||
@@ -155,6 +165,12 @@ def judge_once(root: str, brief: Brief, rung: Rung, effects: Effects) -> Verdict
|
||||
return verdict if isinstance(verdict, Failure | Limited) else escalation_mark.stamped(verdict, len(brief.entries))
|
||||
|
||||
|
||||
def judge_archived(root: str, prompt: str, grounds: Grounds, entries_read: int, rung: Rung, effects: Effects) -> Verdict | Failure | Limited:
|
||||
"""One judge run on a prompt the archive kept from an earlier run, as judge-replay replays it: the judge reads what it read then, and its quotes are checked against grounds, which judge_replay.archived_prompt makes from that prompt in place of the records of today."""
|
||||
verdict = run_judge(root, judge_args(rung), prompt, grounds, effects)
|
||||
return verdict if isinstance(verdict, Failure | Limited) else escalation_mark.stamped(verdict, entries_read)
|
||||
|
||||
|
||||
def judge_args(rung: Rung) -> list[str]:
|
||||
# --json-schema has claude check its reply against the schema and return it as the json result's structured_output.
|
||||
# No tools, so the verdict rests on the transcript grounded() checks it against.
|
||||
@@ -172,14 +188,14 @@ def run_judge(root: str, args: list[str], prompt: str, grounds: Grounds, effects
|
||||
return reply
|
||||
output = structured_output(reply)
|
||||
verdict = verdict_from(output)
|
||||
replied = json.dumps(reply, ensure_ascii=False)
|
||||
run = JudgeRun(prompt, reply)
|
||||
cost = cost_of(reply)
|
||||
if isinstance(verdict, Failure):
|
||||
return Rejected(verdict.reason, replied, cost)
|
||||
return Rejected(verdict.reason, run, cost)
|
||||
checked = quotes_checked(verdict, grounds)
|
||||
if isinstance(checked, Failure):
|
||||
return Rejected(checked.reason, replied, cost)
|
||||
return dataclasses.replace(checked, judged=Judged(cost, tokens_of(reply.get("usage")), reversal_of(output, grounds.typed)))
|
||||
return Rejected(checked.reason, run, cost)
|
||||
return dataclasses.replace(checked, judged=Judged(cost, tokens_of(reply.get("usage")), reversal_of(output, grounds.typed), run))
|
||||
|
||||
|
||||
def reversal_of(output: object, typed: tuple[str, ...]) -> str:
|
||||
@@ -255,8 +271,8 @@ def judge_prompt(brief: Brief) -> tuple[str, Grounds]:
|
||||
head = (
|
||||
f"From the goals file, {brief.premises}\n\n"
|
||||
f"The records: the project's working rules and decisions, then the human's rules for every project:\n{brief.records or 'none'}\n\n"
|
||||
f"The task the worker was given:\n{brief.task}\n\n"
|
||||
"What the worker did and said, what the human typed on [human] lines and what loop-claude told it on [loop] lines, latest last:\n"
|
||||
f"{TASK_HEADING}{brief.task}\n\n"
|
||||
f"{TRANSCRIPT_HEADING}"
|
||||
)
|
||||
room = max(0, JUDGE_PROMPT_CHARS - len(head) - len(JUDGE_RULES) - 2)
|
||||
transcript = escalation_mark.transcript(brief.entries, brief.escalated_at, typed_answer)
|
||||
|
||||
@@ -14,6 +14,7 @@ from failure import Failure
|
||||
from loop_claude.verdicts import Finished, Interrupt, Judged, QuotedAnswer, Unfinished, Verdict
|
||||
from loop_claude.cycle.interrupt import interrupt, Talked, WorkerSession
|
||||
from loop_claude.routing import judge_rung, next_route, step_up, StepUp
|
||||
from transcript_archive.judge_run import JudgeRun
|
||||
from model_routing.ladder import Ladder, Rung
|
||||
from transcript_archive.read import last_spent
|
||||
from transcript_archive import outcomes
|
||||
@@ -98,9 +99,9 @@ def archived_stop(project_archive: pathlib.Path, session: Session, task: str, ro
|
||||
return Stop(1, f"{failure.reason}\n{stop.reason}".rstrip()) if failure and failure.reason != stop.reason else stop
|
||||
|
||||
|
||||
def archived(project_archive: pathlib.Path, transcript_path: str, task: str, outcome: str, effects: Effects) -> Failure | None:
|
||||
def archived(project_archive: pathlib.Path, transcript_path: str, task: str, outcome: str, effects: Effects, judge_run: JudgeRun | None = None) -> Failure | None:
|
||||
"""transcript_archive.write.archive, stamped now and saying its notes through effects. A cycle and close.py, which has no Cycle, both archive through it."""
|
||||
return archive(project_archive, transcript_path, task, outcome, effects.now(), lambda text: say(text, effects))
|
||||
return archive(project_archive, transcript_path, task, outcome, effects.now(), lambda text: say(text, effects), judge_run)
|
||||
|
||||
|
||||
def judge_of(cycle: Cycle) -> Rung:
|
||||
@@ -167,7 +168,7 @@ def rule(cycle: Cycle, transcript_path: str, quoted_answers: int) -> Ruled | Sto
|
||||
if isinstance(annotation, Failure):
|
||||
return Stop(1, annotation.reason)
|
||||
stepped = ruling_step_up(cycle, verdict)
|
||||
if failure := archive_stepped(cycle, transcript_path, ruling_outcome(cycle, verdict, quoted_answers, annotation), stepped):
|
||||
if failure := archive_stepped(cycle, transcript_path, ruling_outcome(cycle, verdict, quoted_answers, annotation), stepped, verdict.judged.run):
|
||||
return Stop(1, failure.reason)
|
||||
return Ruled(verdict, stepped.after if stepped else None)
|
||||
|
||||
@@ -187,13 +188,13 @@ def annotation_of(cycle: Cycle, transcript_path: str, entries: list[Entry], judg
|
||||
def ruling_outcome(cycle: Cycle, ruling: Interrupt | QuotedAnswer, quoted_answers: int, annotation: outcomes.Annotation) -> str:
|
||||
"""The archive's outcome for a ruling: interrupted where it goes to the human, with the question put to them, and else answered from the premises or the records."""
|
||||
if isinstance(ruling, QuotedAnswer) and not to_the_human(ruling, quoted_answers):
|
||||
return outcomes.answered(question=ruling.question, sources=prompts.QUOTE_SOURCES, answer=ruling.answer, evidence=ruling.evidence, escalation=ruling.escalation, judge=str(judge_of(cycle)), worker=cycle.route.worker_line(), judge_cost=ruling.judged.cost, annotation=annotation)
|
||||
return outcomes.interrupted(question_for(ruling), ruling.evidence, str(judge_of(cycle)), ruling.escalation, cycle.route.worker_line(), ruling.judged.cost, annotation)
|
||||
return outcomes.answered(question=ruling.question, sources=prompts.QUOTE_SOURCES, answer=ruling.answer, evidence=ruling.evidence, escalation=ruling.escalation, judge=str(judge_of(cycle)), worker=cycle.route.worker_line(), judge_cost=ruling.judged.cost, annotation=annotation, judge_run=ruling.judged.run)
|
||||
return outcomes.interrupted(question_for(ruling), ruling.evidence, str(judge_of(cycle)), ruling.escalation, cycle.route.worker_line(), judge_cost=ruling.judged.cost, annotation=annotation, judge_run=ruling.judged.run)
|
||||
|
||||
|
||||
def keep_rejection(cycle: Cycle, transcript_path: str, rejected: Rejected) -> Stop | None:
|
||||
"""Archives a judge reply loop-claude rejected, so the archive shows what the judge returned before a retry or an exit 1."""
|
||||
if unkept := archived(cycle.project_archive, transcript_path, cycle.task, outcomes.rejected(rejected.reason, rejected.reply, str(judge_of(cycle)), rejected.cost), cycle.effects):
|
||||
if unkept := archived(cycle.project_archive, transcript_path, cycle.task, outcomes.rejected(rejected.reason, rejected.reply, str(judge_of(cycle)), judge_cost=rejected.cost, judge_run=rejected.run), cycle.effects, rejected.run):
|
||||
return Stop(1, unkept.reason)
|
||||
return None
|
||||
|
||||
@@ -211,9 +212,9 @@ def ask(cycle: Cycle, transcript_path: str, ruling: Interrupt | QuotedAnswer) ->
|
||||
return HumanTyped(TypedAnswer(ruling.question, answer))
|
||||
|
||||
|
||||
def archive_stepped(cycle: Cycle, transcript_path: str, outcome: str, step_up: StepUp | None) -> Failure | None:
|
||||
"""Archives the outcome of a ruling, then the step-up it made, if it made one. The first failure is returned."""
|
||||
if failure := archived(cycle.project_archive, transcript_path, cycle.task, outcome, cycle.effects):
|
||||
def archive_stepped(cycle: Cycle, transcript_path: str, outcome: str, step_up: StepUp | None, judge_run: JudgeRun | None) -> Failure | None:
|
||||
"""Archives the outcome of a ruling and the judge run that gave it, then the step-up it made, if it made one. The first failure is returned."""
|
||||
if failure := archived(cycle.project_archive, transcript_path, cycle.task, outcome, cycle.effects, judge_run):
|
||||
return failure
|
||||
return archived_step(cycle.project_archive, transcript_path, cycle.task, step_up, cycle.effects)
|
||||
|
||||
@@ -236,8 +237,8 @@ def close(cycle: Cycle, transcript_path: str, verdict: Finished | Unfinished) ->
|
||||
annotation = entries if isinstance(entries, Failure) else annotation_of(cycle, transcript_path, entries, verdict.judged)
|
||||
if isinstance(annotation, Failure):
|
||||
return Stop(1, annotation.reason)
|
||||
outcome = outcomes.finished(cycle.route.worker_line(), verdict.evidence, str(judge_of(cycle)), verdict.judged.cost, annotation) if isinstance(verdict, Finished) else outcomes.unfinished(cycle.route.worker_line(), verdict.evidence, str(judge_of(cycle)), verdict.escalation, verdict.judged.cost, annotation)
|
||||
if failure := archived(cycle.project_archive, transcript_path, cycle.task, outcome, cycle.effects):
|
||||
outcome = outcomes.finished(cycle.route.worker_line(), verdict.evidence, str(judge_of(cycle)), judge_cost=verdict.judged.cost, annotation=annotation, judge_run=verdict.judged.run) if isinstance(verdict, Finished) else outcomes.unfinished(cycle.route.worker_line(), verdict.evidence, str(judge_of(cycle)), verdict.escalation, judge_cost=verdict.judged.cost, annotation=annotation, judge_run=verdict.judged.run)
|
||||
if failure := archived(cycle.project_archive, transcript_path, cycle.task, outcome, cycle.effects, verdict.judged.run):
|
||||
return Stop(1, failure.reason)
|
||||
if isinstance(verdict, Finished) and (failure := delete_session(cycle.session)):
|
||||
return Stop(1, failure.reason)
|
||||
|
||||
@@ -342,7 +342,9 @@ class Checkout:
|
||||
for archived in (copy, kept):
|
||||
self.test.assertEqual(stat.S_IMODE(archived.stat().st_mode), 0o600)
|
||||
stamped = re.sub(r"^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ ", "<UTC> ", kept.read_text(), flags=re.MULTILINE)
|
||||
outcomes[copy.read_text()] = re.sub(r"^Config: .*$", "Config: <config>", stamped, flags=re.MULTILINE)
|
||||
configured = re.sub(r"^Config: .*$", "Config: <config>", stamped, flags=re.MULTILINE)
|
||||
# The judge's run is named by the hash of its prompt and reply, which test_loop_claude.py checks.
|
||||
outcomes[copy.read_text()] = re.sub(r"^Judge run: judge/[0-9a-f]{64}\.json\n", "", configured, flags=re.MULTILINE)
|
||||
return outcomes
|
||||
|
||||
|
||||
|
||||
@@ -174,6 +174,9 @@ def ran(stdout="", code=0, stderr=""):
|
||||
return subprocess.CompletedProcess([], code, stdout, stderr)
|
||||
|
||||
|
||||
JUDGE_RUN_LINE = re.compile(r"^Judge run: judge/[0-9a-f]{64}\.json\n", re.MULTILINE)
|
||||
|
||||
|
||||
def judged(verdict, evidence="working", answer="", escalation="", release_ready=False, cost=None, reversed=""):
|
||||
reply = {"answer": answer, "escalation": escalation, "evidence": evidence, "finished": verdict == "finished", "interrupt": verdict == "interrupt", "question": "Which one?" if verdict in ("interrupt", "quoted") else "", "answered_before": "", "left_open": "", "release_ready": release_ready, "reversed": reversed}
|
||||
return ran(json.dumps({"structured_output": reply, **({} if cost is None else {"total_cost_usd": cost})}))
|
||||
@@ -289,7 +292,8 @@ class Cycles(unittest.TestCase):
|
||||
return run.Cycle(config=CONFIG, window=WINDOW, effects=fake.effects(), facts=facts, judge_model="judge-model", ladder=LADDER, premises="## Goals", project=None, project_archive=self.archive, records=records, root=str(self.home), route=Route(Rung("worker-model")), session=self.session, task="Do it.", plan=plan_of(""), unfinished_evidence=unfinished_evidence)
|
||||
|
||||
def archived_outcomes(self):
|
||||
return sorted(path.read_text() for path in self.archive.glob("*/*.outcomes.md"))
|
||||
"""Each outcomes file's text without its `Judge run:` lines, which name a file by the hash of a prompt and its reply: the tests of the judge's runs check them."""
|
||||
return sorted(JUDGE_RUN_LINE.sub("", path.read_text()) for path in self.archive.glob("*/*.outcomes.md"))
|
||||
|
||||
def test_a_finished_chunk_is_archived_and_its_session_deleted(self):
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("finished")])
|
||||
@@ -383,6 +387,32 @@ Worker: {OFF_LADDER}
|
||||
f"{stamp} Judge reply rejected: the judge quoted 'never said', which is not in the transcript\n\nReply:\n> {misquoted.stdout}\n\nJudge: judge-model\n\n"
|
||||
f"{stamp} Verdict: finished\n\nEvidence:\n> working\n\nConfig: {CONFIG}\nJudge: judge-model\nWorker: {OFF_LADDER}\n"])
|
||||
|
||||
def test_archives_each_judge_runs_prompt_and_reply_beside_the_rejection_interrupt_and_verdict_it_gave(self):
|
||||
rejected, interrupted, finished = ran(json.dumps({"structured_output": {"interrupt": False}})), judged("interrupt", cost=0.25), judged("finished")
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)] * 2, claude=[rejected, interrupted, finished], inputs=["Use B."])
|
||||
with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)).verdict, Finished("working"))
|
||||
prompts = [stdin for args, stdin in fake.made("claude")[:3]]
|
||||
[chunk] = self.archive.iterdir()
|
||||
kept = "\n".join(path.read_text() for path in chunk.glob("*.outcomes.md"))
|
||||
named = re.findall(r"^Judge run: (judge/[0-9a-f]{64}\.json)$", kept, re.MULTILINE)
|
||||
self.assertEqual(len(named), 3, kept)
|
||||
runs = [json.loads((chunk / name).read_text()) for name in named]
|
||||
replies = [json.loads(reply.stdout) for reply in (rejected, interrupted, finished)]
|
||||
self.assertEqual(sorted((found["prompt"], json.dumps(found["reply"])) for found in runs), sorted((prompt, json.dumps(reply)) for prompt, reply in zip(prompts, replies)))
|
||||
self.assertEqual(sorted(path.name for path in (chunk / "judge").iterdir()), sorted(name.removeprefix("judge/") for name in named))
|
||||
|
||||
def test_stops_when_a_judge_run_cannot_be_archived(self):
|
||||
self.archive.mkdir(parents=True)
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("finished")])
|
||||
def replace_file(path, data):
|
||||
return Failure("disk full") if "/judge/" in str(path) else real_replace_file(path, data)
|
||||
real_replace_file = write.replace_file
|
||||
with unittest.mock.patch("transcript_archive.write.replace_file", replace_file):
|
||||
stop = run.run_cycle(self.cycle(fake))
|
||||
self.assertEqual(stop.code, 1)
|
||||
self.assertRegex(stop.reason, r"^could not archive the transcript: .*/judge/[0-9a-f]{64}\.json: disk full$")
|
||||
|
||||
def test_archives_what_each_judge_run_cost_on_an_interrupt_a_quoted_answer_and_a_rejected_misquote(self):
|
||||
for first, then, inputs in (
|
||||
(judged("interrupt", cost=0.25), judged("finished"), ["Use B."]),
|
||||
@@ -396,6 +426,7 @@ Worker: {OFF_LADDER}
|
||||
run.run_cycle(dataclasses.replace(self.cycle(fake, records="From AGENTS.md:\nUse A."), project_archive=pathlib.Path(archive)))
|
||||
kept = "".join(path.read_text() for path in pathlib.Path(archive).glob("*/*.outcomes.md"))
|
||||
self.assertEqual(kept.count("Judge cost: 0.2500\n"), 1, kept)
|
||||
self.assertIn("Judge cost: 0.2500\nJudge run: judge/", kept)
|
||||
|
||||
def test_stops_with_exit_1_where_a_subagents_transcript_cannot_be_read_for_the_spend(self):
|
||||
for verdict in ("interrupt", "finished"):
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
"""The verdicts cycle.judge.verdict_from builds from the judge's reply; cycle.run.rule then stamps every verdict but Finished with the items the chunk takes. They sit outside cycle/, since routing reads them and cycle imports routing."""
|
||||
import dataclasses
|
||||
|
||||
from transcript_archive.judge_run import JudgeRun
|
||||
from claude_code.tokens import Tokens
|
||||
|
||||
|
||||
@@ -22,6 +23,8 @@ class Judged:
|
||||
tokens: Tokens = Tokens()
|
||||
# The human's whole typed answer that holds the judge's quote of a reversal of a choice the worker took on its own recommendation; empty for none, and where no typed answer holds the quote.
|
||||
reversed: str = ""
|
||||
# The prompt it read and its reply, which the archive keeps beside the verdict. Left out of comparisons and reprs, so that equal verdicts stay equal whatever prompt they came from.
|
||||
run: JudgeRun | None = dataclasses.field(default=None, compare=False, repr=False)
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
<archive root>/<project key>/<chunk>/<sha256>.jsonl
|
||||
<archive root>/<project key>/<chunk>/<sha256>.outcomes.md
|
||||
<archive root>/<project key>/<chunk>/session/...
|
||||
<archive root>/<project key>/<chunk>/judge/<sha256>.json
|
||||
|
||||
- The archive root is $LOOP_CLAUDE_TRANSCRIPTS, or ~/.claude/loop-claude-transcripts (location.archive_root).
|
||||
- The project key is the checkout's path relative to ~/code, or to / outside it (project_key.project_key).
|
||||
@@ -18,6 +19,8 @@
|
||||
Code keeps in a folder named after the session, beside the transcript. A file there is replaced only by a version that
|
||||
grew from it. A file Claude Code rewrote whole lands beside its copy as <name>.<hash of its content>
|
||||
(write.copy_for).
|
||||
- judge/ holds each judge run an outcome names on its `Judge run:` line: the prompt the judge read and claude's
|
||||
json reply, named by the hash of that content (judge_run.JudgeRun). judge-replay reads the prompt back.
|
||||
|
||||
write.archive writes the layout, at every interrupt and as every cycle ends. read.py and the tools that
|
||||
measure finished chunks read it.
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
import dataclasses
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
|
||||
from failure import Failure
|
||||
|
||||
# The folder of a chunk's judge runs, beside its transcript copies.
|
||||
FOLDER = "judge"
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class JudgeRun:
|
||||
"""The prompt a judge run read and claude's json reply to it. The judge has no tools and one turn, so these two are its whole transcript."""
|
||||
prompt: str
|
||||
reply: dict
|
||||
|
||||
def content(self) -> bytes:
|
||||
return json.dumps({"prompt": self.prompt, "reply": self.reply}, ensure_ascii=False, indent=1).encode()
|
||||
|
||||
@property
|
||||
def file(self) -> str:
|
||||
"""Where the archive keeps the run, relative to its chunk's folder. The name is the hash of the content, so a run archived twice is kept once."""
|
||||
return f"{FOLDER}/{hashlib.sha256(self.content()).hexdigest()}.json"
|
||||
|
||||
@staticmethod
|
||||
def read(path: pathlib.Path) -> "JudgeRun | Failure":
|
||||
try:
|
||||
kept = json.loads(path.read_text(encoding="utf-8"))
|
||||
prompt, reply = kept["prompt"], kept["reply"]
|
||||
except (OSError, ValueError, KeyError, TypeError) as failure:
|
||||
return Failure(f"could not read the archived judge run {path}: {failure}")
|
||||
if not (isinstance(prompt, str) and isinstance(reply, dict)):
|
||||
return Failure(f"could not read the archived judge run {path}: its prompt is no string or its reply no object")
|
||||
return JudgeRun(prompt, reply)
|
||||
@@ -1,6 +1,7 @@
|
||||
import re
|
||||
import typing
|
||||
|
||||
from transcript_archive.judge_run import FOLDER, JudgeRun
|
||||
from model_routing.prices import PRICES
|
||||
from text_files.markdown_text import quoted
|
||||
from text_files.utc_stamp import STAMPED
|
||||
@@ -29,6 +30,9 @@ JUDGE = "Judge: "
|
||||
# What the judge's run cost, in dollars, on the line after JUDGE; absent where the verdict recorded none.
|
||||
JUDGE_COST = "Judge cost: "
|
||||
JUDGE_COST_AT = re.compile(rf"^{re.escape(JUDGE)}(.+)\n{re.escape(JUDGE_COST)}([0-9.]+)$", re.MULTILINE)
|
||||
# The judge run the verdict came from, as a path in the chunk's folder (judge_run.JudgeRun.file), on the line after the judge's rung and cost; absent where the archive kept none.
|
||||
JUDGE_RUN = "Judge run: "
|
||||
JUDGE_RUN_AT = re.compile(rf"^{re.escape(JUDGE_RUN)}({FOLDER}/[0-9a-f]{{64}}\.json)$", re.MULTILINE)
|
||||
|
||||
|
||||
# The claude-config release and commit that ran the cycle, on the line before JUDGE, such as `Config: v0.8.0 (<sha>)`.
|
||||
@@ -84,9 +88,18 @@ def spent_stamps(outcomes: str) -> list[str]:
|
||||
return [stamp for stamp, outcome in each_outcome(outcomes) if f"\n{SPENT}" in outcome]
|
||||
|
||||
|
||||
def judge_lines(judge: str, judge_cost: float) -> str:
|
||||
"""The judge's rung, and what its run cost where that is known."""
|
||||
return f"{JUDGE}{judge}" + (f"\n{JUDGE_COST}{judge_cost:.4f}" if judge_cost else "")
|
||||
def judge_lines(judge: str, judge_cost: float, judge_run: JudgeRun | None) -> str:
|
||||
"""The judge's rung, what its run cost where that is known, and the archived run where there is one."""
|
||||
return f"{JUDGE}{judge}" + (f"\n{JUDGE_COST}{judge_cost:.4f}" if judge_cost else "") + (f"\n{JUDGE_RUN}{judge_run.file}" if judge_run else "")
|
||||
|
||||
|
||||
def judge_run_at(outcomes: str, stamp: str) -> str | None:
|
||||
"""The path of the judge run that the outcome stamped stamp names; None where it names none."""
|
||||
for found, outcome in each_outcome(outcomes):
|
||||
# The last such line: an interrupt's question, which comes first, may hold one.
|
||||
if found == stamp and (named := JUDGE_RUN_AT.findall(outcome)):
|
||||
return named[-1]
|
||||
return None
|
||||
|
||||
|
||||
def judge_costs(outcomes: str) -> list[tuple[str, float]]:
|
||||
@@ -98,31 +111,31 @@ def escalation_block(escalation: str) -> str:
|
||||
return f"Escalation:\n{quoted(escalation)}\n\n" if escalation else ""
|
||||
|
||||
|
||||
def finished(worker: WorkerLine, evidence: str, judge: str, judge_cost: float = 0.0, annotation: Annotation = Annotation()) -> str:
|
||||
return f"{FINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{annotation_lines(annotation)}{judge_lines(judge, judge_cost)}\n{WORKER}{worker}"
|
||||
def finished(worker: WorkerLine, evidence: str, judge: str, *, judge_cost: float = 0.0, annotation: Annotation = Annotation(), judge_run: JudgeRun | None = None) -> str:
|
||||
return f"{FINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{annotation_lines(annotation)}{judge_lines(judge, judge_cost, judge_run)}\n{WORKER}{worker}"
|
||||
|
||||
|
||||
def unfinished(worker: WorkerLine, evidence: str, judge: str, escalation: str, judge_cost: float = 0.0, annotation: Annotation = Annotation()) -> str:
|
||||
def unfinished(worker: WorkerLine, evidence: str, judge: str, escalation: str, *, judge_cost: float = 0.0, annotation: Annotation = Annotation(), judge_run: JudgeRun | None = None) -> str:
|
||||
"""escalation is the judge's quote showing that the worker's cited escalation trigger fired; empty for none."""
|
||||
escalated = escalation_block(escalation)
|
||||
return f"{UNFINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{annotation_lines(annotation)}{judge_lines(judge, judge_cost)}\n{WORKER}{worker}"
|
||||
return f"{UNFINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{annotation_lines(annotation)}{judge_lines(judge, judge_cost, judge_run)}\n{WORKER}{worker}"
|
||||
|
||||
|
||||
def interrupted(question: str, evidence: str, judge: str, escalation: str, worker: WorkerLine, judge_cost: float = 0.0, annotation: Annotation = Annotation()) -> str:
|
||||
def interrupted(question: str, evidence: str, judge: str, escalation: str, worker: WorkerLine, *, judge_cost: float = 0.0, annotation: Annotation = Annotation(), judge_run: JudgeRun | None = None) -> str:
|
||||
"""escalation is the judge's quote showing that the worker's cited escalation trigger fired; empty for none. worker is the line naming the route the worker ran on until the interrupt."""
|
||||
escalated = escalation_block(escalation)
|
||||
return f"{INTERRUPTED}\n\n{question}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{annotation_lines(annotation)}{judge_lines(judge, judge_cost)}\n{WORKER}{worker}"
|
||||
return f"{INTERRUPTED}\n\n{question}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{annotation_lines(annotation)}{judge_lines(judge, judge_cost, judge_run)}\n{WORKER}{worker}"
|
||||
|
||||
|
||||
def answered(*, question: str, sources: str, answer: str, evidence: str, escalation: str, judge: str, worker: WorkerLine, judge_cost: float = 0.0, annotation: Annotation = Annotation()) -> str:
|
||||
def answered(*, question: str, sources: str, answer: str, evidence: str, escalation: str, judge: str, worker: WorkerLine, judge_cost: float = 0.0, annotation: Annotation = Annotation(), judge_run: JudgeRun | None = None) -> str:
|
||||
"""escalation and worker are as on interrupted. Every parameter is keyword-only, since a swap of two str ones would still type-check."""
|
||||
escalated = escalation_block(escalation)
|
||||
return f"{ANSWERED}\n\n{question}\n\n{sources} says:\n{quoted(answer)}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{annotation_lines(annotation)}{judge_lines(judge, judge_cost)}\n{WORKER}{worker}"
|
||||
return f"{ANSWERED}\n\n{question}\n\n{sources} says:\n{quoted(answer)}\n\nEvidence:\n{quoted(evidence)}\n\n{escalated}{annotation_lines(annotation)}{judge_lines(judge, judge_cost, judge_run)}\n{WORKER}{worker}"
|
||||
|
||||
|
||||
def rejected(why: str, reply: str, judge: str, judge_cost: float = 0.0) -> str:
|
||||
def rejected(why: str, reply: str, judge: str, *, judge_cost: float = 0.0, judge_run: JudgeRun | None = None) -> str:
|
||||
"""A judge reply loop-claude rejected, with the reason and the reply as json."""
|
||||
return f"{REJECTED}: {why}\n\nReply:\n{quoted(reply)}\n\n{judge_lines(judge, judge_cost)}"
|
||||
return f"{REJECTED}: {why}\n\nReply:\n{quoted(reply)}\n\n{judge_lines(judge, judge_cost, judge_run)}"
|
||||
|
||||
|
||||
def stepped_up(worker: WorkerLine) -> str:
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import contextlib
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
@@ -10,6 +11,7 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from transcript_archive.project_key import checkout_of, project_key
|
||||
from failure import Failure
|
||||
from transcript_archive.judge_run import JudgeRun
|
||||
from transcript_archive import outcomes, read, write
|
||||
from transcript_archive.location import project_archive
|
||||
from text_files.utc_stamp import utc
|
||||
@@ -31,6 +33,81 @@ class Archive(unittest.TestCase):
|
||||
self.assertEqual(kept_outcomes.read_text(), "Task: Do it.\n\n2026-09-21T14:13:20Z Stopped: Ctrl-C\n\nWorker: m\n")
|
||||
|
||||
|
||||
class JudgeRuns(unittest.TestCase):
|
||||
judged = JudgeRun("The prompt.", {"structured_output": {"finished": True}, "total_cost_usd": 0.5})
|
||||
|
||||
def test_a_run_is_kept_once_in_the_chunk_folder_under_the_hash_of_its_content(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
kept = pathlib.Path(root, "archive")
|
||||
transcript = pathlib.Path(root, "t.jsonl")
|
||||
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n')
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
for offset in (0, 1):
|
||||
outcome = outcomes.finished("w", "merged", "j", judge_run=self.judged)
|
||||
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcome, NOW + offset, judge_run=self.judged))
|
||||
(folder,) = kept.glob("*/")
|
||||
(file,) = folder.glob("judge/*")
|
||||
self.assertEqual(file, folder / self.judged.file)
|
||||
self.assertEqual(file.name, hashlib.sha256(file.read_bytes()).hexdigest() + ".json")
|
||||
self.assertEqual(json.loads(file.read_text()), {"prompt": "The prompt.", "reply": {"structured_output": {"finished": True}, "total_cost_usd": 0.5}})
|
||||
self.assertEqual(JudgeRun.read(file), self.judged)
|
||||
|
||||
def test_names_the_run_on_the_line_after_the_judge_lines_and_reads_the_name_back(self):
|
||||
outcome = outcomes.finished("w", "e", "j", judge_cost=0.5, judge_run=self.judged)
|
||||
self.assertEqual(outcome, f"Verdict: finished\n\nEvidence:\n> e\n\nJudge: j\nJudge cost: 0.5000\nJudge run: {self.judged.file}\nWorker: w")
|
||||
self.assertEqual(outcomes.rejected("why", "{}", "j", judge_run=self.judged), f"Judge reply rejected: why\n\nReply:\n> {{}}\n\nJudge: j\nJudge run: {self.judged.file}")
|
||||
self.assertEqual(outcomes.workers(f"{utc(NOW)} {outcome}\n"), [(utc(NOW), "w")])
|
||||
self.assertEqual(outcomes.judge_costs(f"{utc(NOW)} {outcome}\n"), [("j", 0.5)])
|
||||
kept = f"Task: Do it.\n\n{utc(NOW)} {outcomes.stopped('w', 'Ctrl-C')}\n\n{utc(NOW + 1)} {outcome}\n\n{utc(NOW + 2)} {outcomes.finished('w', 'e', 'j')}\n"
|
||||
self.assertEqual(outcomes.judge_run_at(kept, utc(NOW + 1)), self.judged.file)
|
||||
self.assertIsNone(outcomes.judge_run_at(kept, utc(NOW + 2)))
|
||||
|
||||
def test_keeps_a_run_as_readable_utf_8_json_indented_by_one_space(self):
|
||||
run = JudgeRun("Prompt é.", {"a": 1})
|
||||
self.assertEqual(run.content(), b'{\n "prompt": "Prompt \xc3\xa9.",\n "reply": {\n "a": 1\n }\n}')
|
||||
|
||||
def test_reads_a_file_that_is_no_run_as_a_failure(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
for text in ("not json", '{"prompt": "P"}', '{"prompt": 1, "reply": {}}', '{"prompt": "P", "reply": []}'):
|
||||
with self.subTest(text=text):
|
||||
path = pathlib.Path(root, "run.json")
|
||||
path.write_text(text)
|
||||
self.assertRegex(JudgeRun.read(path).reason, f"^could not read the archived judge run {path}: ")
|
||||
|
||||
def test_reads_no_path_that_leaves_the_judge_folder_from_an_outcome(self):
|
||||
outcome = f"{utc(NOW)} Verdict: finished\n\nJudge run: judge/../../secret.json\n"
|
||||
self.assertIsNone(outcomes.judge_run_at(outcome, utc(NOW)))
|
||||
|
||||
def test_names_the_run_on_each_kind_of_outcome_that_has_a_judge(self):
|
||||
named = f"Judge: j\nJudge run: {self.judged.file}\n"
|
||||
kept = [
|
||||
outcomes.unfinished("w", "e", "j", "", judge_run=self.judged),
|
||||
outcomes.interrupted("Q?", "e", "j", "", "w", judge_run=self.judged),
|
||||
outcomes.answered(question="Q?", sources="S", answer="a", evidence="e", escalation="", judge="j", worker="w", judge_run=self.judged),
|
||||
]
|
||||
self.assertEqual([named in outcome for outcome in kept], [True, True, True])
|
||||
|
||||
def test_names_the_folder_it_cannot_make_in_the_failure_to_keep_a_run(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
(pathlib.Path(root) / "judge").write_text("a file where the folder should be")
|
||||
failure = write.keep_judge_run(pathlib.Path(root), self.judged)
|
||||
self.assertRegex(failure.reason, f"^could not archive the transcript: {root}/judge: ")
|
||||
self.assertNotIn("None", failure.reason)
|
||||
|
||||
def test_writes_no_outcome_where_the_run_cannot_be_kept(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
kept = pathlib.Path(root, "archive")
|
||||
transcript = pathlib.Path(root, "t.jsonl")
|
||||
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n')
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.stopped("w", "x"), NOW))
|
||||
(folder,) = kept.glob("*/")
|
||||
(folder / "judge").write_text("a file where the folder should be")
|
||||
failure = write.archive(kept, str(transcript), "Do it.", outcomes.finished("w", "e", "j", judge_run=self.judged), NOW + 1, judge_run=self.judged)
|
||||
self.assertIsInstance(failure, Failure)
|
||||
self.assertNotIn("Verdict: finished", next(folder.glob("*.outcomes.md")).read_text())
|
||||
|
||||
|
||||
class CopyFor(unittest.TestCase):
|
||||
def test_skips_a_version_already_kept_beside_its_copy(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
|
||||
@@ -5,14 +5,15 @@ import typing
|
||||
|
||||
from transcript_archive.location import chunk_folder, outcomes_of
|
||||
from failure import Failure
|
||||
from transcript_archive.judge_run import JudgeRun
|
||||
from text_files.file_writes import make_private, replace_file
|
||||
from claude_code.session import session_files
|
||||
from transcript_archive.outcomes import TASK
|
||||
from text_files.utc_stamp import utc
|
||||
|
||||
|
||||
def archive(project_archive: pathlib.Path, path: str, task: str, outcome: str, at: float, note: typing.Callable[[str], None] = lambda text: None) -> Failure | None:
|
||||
"""Keeps the transcript at path, its outcomes file and its session's files in project_archive, in the layout this package's docstring gives; the failure that stopped it, or None. The outcome is stamped and appended to the outcomes file, which opens with the task only when the copy is new. Where a new copy lands, note is told where."""
|
||||
def archive(project_archive: pathlib.Path, path: str, task: str, outcome: str, at: float, note: typing.Callable[[str], None] = lambda text: None, judge_run: JudgeRun | None = None) -> Failure | None:
|
||||
"""Keeps the transcript at path, its outcomes file and its session's files in project_archive, in the layout this package's docstring gives; the failure that stopped it, or None. The outcome is stamped and appended to the outcomes file, which opens with the task only when the copy is new. Where a new copy lands, note is told where. A judge_run the outcome names is kept first, so the outcome never names a run the archive lacks."""
|
||||
try:
|
||||
with open(path, "rb") as handle:
|
||||
raw = handle.read()
|
||||
@@ -23,6 +24,8 @@ def archive(project_archive: pathlib.Path, path: str, task: str, outcome: str, a
|
||||
for sibling in session_files(path):
|
||||
if sibling != path and (unkept := mirror(pathlib.Path(sibling), folder / "session" / os.path.basename(sibling))):
|
||||
return unkept
|
||||
if judge_run and (unkept := keep_judge_run(folder, judge_run)):
|
||||
return unkept
|
||||
copy = folder / f"{hashlib.sha256(raw).hexdigest()}.jsonl"
|
||||
outcomes = outcomes_of(copy)
|
||||
stamped = f"{utc(at)} {outcome}"
|
||||
@@ -36,6 +39,19 @@ def archive(project_archive: pathlib.Path, path: str, task: str, outcome: str, a
|
||||
return None
|
||||
|
||||
|
||||
def keep_judge_run(folder: pathlib.Path, run: JudgeRun) -> Failure | None:
|
||||
kept = folder / run.file
|
||||
if kept.exists():
|
||||
return None
|
||||
try:
|
||||
make_private(kept.parent)
|
||||
except OSError as failure:
|
||||
return unarchived(failure.filename, failure.strerror)
|
||||
if unwritten := replace_file(kept, run.content()):
|
||||
return unarchived(kept, unwritten.reason)
|
||||
return None
|
||||
|
||||
|
||||
def unarchived(where: object, why: str | None) -> Failure:
|
||||
return Failure(f"could not archive the transcript: {where}: {why}")
|
||||
|
||||
|
||||
+3
-3
@@ -182,16 +182,16 @@ Valid while `~/.claude` is a checkout of this repo.
|
||||
|
||||
Valid while claude takes `--session-id` and resumes by it, and the human runs loop-claude where tmux is installed.
|
||||
|
||||
## loop-claude: the archive keeps the raw transcript, and a failed archive stops the loop
|
||||
## loop-claude: the archive keeps the raw transcript and each judge run, and a failed archive stops the loop
|
||||
|
||||
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. Valid while transcripts stay small enough to copy whole.
|
||||
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. The judge's runs are archived too, as `judge/<sha256>.json` in the chunk's folder: the prompt it read and its json reply. Together they are its whole transcript, since it runs with no tools and one turn. A run is kept whole and deduplicated by hash, and the outcome of the verdict, interrupt or rejected reply it produced names it on a `Judge run:` line. Keeping it whole costs little: a prompt is at most `JUDGE_PROMPT_CHARS`, 400,000 characters. Measured 2026-11-17, the archive's 911 judge lines come to at most 0.36 GB of prompts, against 3.5 GB in its 1,211 transcript copies, the largest 23.4 MB. The judge is still priced from its `Judge cost:` line, which is claude's own total for the run, and the Spent block counts the live reply's tokens; the archived reply keeps its `usage` for a later reader, and nothing reads it back. `judge-replay` replays a verdict on its archived prompt where there is one. Valid while transcripts, and the judge's prompts, stay small enough to copy whole.
|
||||
|
||||
## loop-claude: each verdict records the claude-config release and commit that ran it, the raw tokens each thread spent by model and kind since the previous outcome that recorded a spend, and the human's typed answer the judge quotes as reversing a choice the worker took on its own recommendation
|
||||
|
||||
2026-11-12, lilleman, todo item 305. Serves goals 1, 1.2 and 1.4. A change to the setup can be judged only by comparing chunks run before it with chunks run after, so each verdict, interrupt and quoted answer in loop-claude's archive records three things beside its judge lines.
|
||||
|
||||
- `Config: <release> (<commit>)`: `git describe` and the commit of `~/.claude`, so outcomes group by the setup that produced them. Where git can't tell, it records `unknown` and why, on one line.
|
||||
- A `Spent since` block: the raw tokens each thread spent, by model and by kind (input, cache write 5m, cache write 1h, cache read, output), then the list prices in force, from `bin/model_routing/prices.py`. The threads are the worker's main thread, each subagent by its agent type and description, and the judge run that ruled. Each message counts once, by its id. The stretch runs from the previous outcome that recorded a spend, so what a `/talk` or a stopped run spent between rulings is counted once. It keeps tokens, since prices change. The prices recorded beside them give what the stretch cost when it ran. Claude Code doesn't report thinking apart from output, so it is left in output. The judge's tokens come from its json reply's `usage`, until item 303 archives the judge's own transcript.
|
||||
- A `Spent since` block: the raw tokens each thread spent, by model and by kind (input, cache write 5m, cache write 1h, cache read, output), then the list prices in force, from `bin/model_routing/prices.py`. The threads are the worker's main thread, each subagent by its agent type and description, and the judge run that ruled. Each message counts once, by its id. The stretch runs from the previous outcome that recorded a spend, so what a `/talk` or a stopped run spent between rulings is counted once. It keeps tokens, since prices change. The prices recorded beside them give what the stretch cost when it ran. Claude Code doesn't report thinking apart from output, so it is left in output. The judge's tokens come from its json reply's `usage`, which the archived judge run holds too.
|
||||
- A `Reversed:` quote: the human's typed answer that overrides a choice the worker took on its own recommendation and recorded. The judge quotes the latest such `[human]` line; loop-claude keeps the whole typed answer that holds the quote verbatim, and drops a quote no typed answer holds. `score-outcomes` counts the distinct answers per finished chunk. That count is goal 1.2's counter-signal, and the 2026-11-08 entry of `loop-claude: a judge decides each interrupt…` is valid only while it stays low. It counts only chunks that record a Config line, since earlier judges didn't look for reversals.
|
||||
|
||||
Valid while score-outcomes reads the reversals from the archive.
|
||||
|
||||
@@ -12,9 +12,8 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 303 | 0.12.0 | | defect | **Archive each judge run's prompt and json reply beside the verdict it gave, so judge cost, replays and backups cover the judge as they cover the worker.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3, 3 by 2 | 4.2 |
|
||||
| 306 | 0.12.0 | | | **Set a strategy that keeps loop-claude's archive and larv-review's review folders backed up and small over the years, compressing older entries without losing any.** | 3 | 2 | 4 | 7 | 5 | 3 by 5 | 3.9 |
|
||||
| 304 | 0.12.0 | 303 | defect | **Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 2, 3 by 2 | 3.8 |
|
||||
| 304 | 0.12.0 | | defect | **Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 2, 3 by 2 | 3.8 |
|
||||
| 387 | 0.12.0 | | | **Bind a project's rung to its ladder in one value, so the routing functions and `Cycle` take one argument for both.** | 3 | 1 | 3 | 5 | 5 | code health by 3 | 2.9 |
|
||||
| 338 | 0.12.0 | | owner | **Script each step that 0.9.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.3 |
|
||||
| 288 | 0.12.0 | | | **Point the two reading-cost decisions at code health and goal 1.4's counter-signal, in place of the goal 1.1 they cite.** | 1 | 2 | 1 | 2 | 1 | code health by 1, 1.4 by 1 | 2.2 |
|
||||
@@ -76,23 +75,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 303. Archive each judge run's prompt and json reply beside the verdict it gave, so judge cost, replays and backups cover the judge as they cover the worker.
|
||||
|
||||
Today no judge transcript reaches loop-claude's archive. `bin/claude_code/reply.py` runs every headless reply, the judge's included, with `--no-session-persistence`, so its runs stay out of the session picker; Claude Code then writes no session file for the archive to copy. No decision chose to leave the judge out: `loop-claude: the archive keeps the raw transcript` was written for the worker's `.jsonl`. Goal 3 asks for the transcripts of unattended runs to be backed up, and the judge is one. The gap also leaves the judge's spend measurable only from the `Judge cost:` lines, which older verdicts lack ($136 recorded over 2026-09-30 to 2026-11-11), and leaves `judge-replay` rebuilding each prompt from today's records: of the 39 verdicts it replayed on 2026-10-15, 8 may rest on records that had changed since.
|
||||
|
||||
The judge runs with no tools and one turn (`judge_args` in `bin/loop_claude/cycle/judge.py`), so its whole transcript is the prompt it read and the json reply. Recommended: keep `--no-session-persistence`, and have the archive write the prompt and the reply beside the outcomes entry of the verdict, interrupt or rejected reply they produced, deduplicated by content hash as the worker's copies are. A failed write stops the loop, as that decision says. Persisting the session and copying its file would also work, but fills the picker and needs a cleanup. The prompt repeats the worker's transcript, which the archive already holds: measure the size per cycle first, and if it outgrows "small enough to copy whole", keep the prompt without the transcript part and name the worker copy and entry range it held. Then let `judge-replay` read the archived prompt where one exists, and price the judge from the archived reply's usage. Update that decision entry to say the judge's runs are archived too.
|
||||
|
||||
Touches:
|
||||
|
||||
- `bin/loop_claude/cycle/judge.py`: `run_judge`, `judge_once`
|
||||
- `bin/loop_claude/cycle/run.py`: `(module)` (small)
|
||||
- `bin/transcript_archive/write.py`: `archive`, `copy_for`
|
||||
- `bin/transcript_archive/location.py`
|
||||
- `bin/transcript_archive/test_transcript_archive.py`
|
||||
- `bin/judge_replay/main.py`: `archived_cases`, `rule`
|
||||
- `bin/judge_replay/replay.py`: `case_from`
|
||||
- `bin/judge_replay/test_judge_replay.py`
|
||||
|
||||
### 306. Set a strategy that keeps loop-claude's archive and larv-review's review folders backed up and small over the years, compressing older entries without losing any.
|
||||
|
||||
Measured on 2026-11-11: loop-claude's archive is 6.3 GB (12,721 files from October alone, 0.84 GB older than 30 days), and `~/.local/state/larv-review/` is 997 MB, 984 MB of it under `code/`, with no pruning: `skills/larv-review/SKILL.md` → "Reports" keeps every folder after the merge, because `review-replay` replays rounds from them. The largest archived worker transcript, 23.4 MB, gzips to 9.1 MB at level 6. The disk holds 65 GB free today.
|
||||
|
||||
Reference in New Issue
Block a user