Answer prose round 1: 0 🔴, 5 🟡 (0 misread); the decision entry and the help text reworded with the measurement redone, the title names the judge runs, and the hash comments say prompt and reply
This commit is contained in:
@@ -1,8 +1,8 @@
|
||||
from claude_code.entries import Entry
|
||||
from claude_code.transcript import human_answers
|
||||
from failure import Failure
|
||||
from loop_claude import escalation_mark
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Grounds, JUDGE_RULES, TASK_HEADING, TRANSCRIPT_HEADING
|
||||
from claude_code.transcript import human_answers
|
||||
from loop_claude.cycle.prompts import typed_answer
|
||||
|
||||
|
||||
|
||||
@@ -4,10 +4,10 @@ usage: judge-replay <model>
|
||||
|
||||
<model> is a project model on the routing ladder this setup ships, such as claude-opus-5-5. The tool replays the
|
||||
verdicts archived for the projects that ~/.claude/loop-claude-models.json runs on it: the latest
|
||||
15 of each kind, finished, unfinished, interrupt and quoted answer. Each transcript copy in the
|
||||
archive is what the judge read for the verdict beside it. Where the archive kept the judge's own prompt
|
||||
(docs/decisions.md → `loop-claude: the archive keeps the raw transcript`), the replay hands the judge that
|
||||
prompt; for a verdict from before the archive kept it, the replay hands the judge the copy, with the task,
|
||||
15 of each kind, finished, unfinished, interrupt and quoted answer. Where the archive kept the judge's own
|
||||
prompt (docs/decisions.md → `loop-claude: the archive keeps the raw transcript and each judge run`), the
|
||||
replay hands the judge that prompt, which holds what the judge read then. For a verdict from before the
|
||||
archive kept it, the replay builds the prompt from the transcript copy beside the verdict, with the task,
|
||||
premises and records each project holds now.
|
||||
|
||||
It first replays every verdict on the reference judge: the model at xhigh effort, the rung the
|
||||
@@ -27,7 +27,7 @@ keeps the judge to its own model's rungs; a rung above them that misses no inter
|
||||
evidence to let the judge climb onto it.
|
||||
|
||||
Like the loop, it tries a judge three times running before it counts the reply as failed. A
|
||||
verdict whose project or transcript cannot be read replays as failed too. Exit 1, saying why, on
|
||||
verdict whose project, transcript or archived judge run cannot be read replays as failed too. Exit 1, saying why, on
|
||||
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
|
||||
or a usage limit.
|
||||
"""
|
||||
|
||||
@@ -185,7 +185,7 @@ class Main(unittest.TestCase):
|
||||
self.assertIn("The task the worker was given:\nDo it.", self.prompts[1])
|
||||
self.assertIn("1. Ship.", self.prompts[1])
|
||||
|
||||
def test_replays_a_verdict_whose_archived_judge_run_cannot_be_read_as_failed(self):
|
||||
def test_a_verdict_whose_archived_judge_run_cannot_be_read_replays_as_failed(self):
|
||||
kept = self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1"
|
||||
(kept / "t0.outcomes.md").write_text("Task: Do it.\n\n2026-10-14T10:00:00Z Verdict: finished\n\nJudge: j\nJudge run: judge/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.json\nWorker: x\n")
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
|
||||
@@ -343,7 +343,7 @@ class Checkout:
|
||||
self.test.assertEqual(stat.S_IMODE(archived.stat().st_mode), 0o600)
|
||||
stamped = re.sub(r"^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ ", "<UTC> ", kept.read_text(), flags=re.MULTILINE)
|
||||
configured = re.sub(r"^Config: .*$", "Config: <config>", stamped, flags=re.MULTILINE)
|
||||
# The judge's run is named by the hash of its prompt, which test_loop_claude.py checks.
|
||||
# The judge's run is named by the hash of its prompt and reply, which test_loop_claude.py checks.
|
||||
outcomes[copy.read_text()] = re.sub(r"^Judge run: judge/[0-9a-f]{64}\.json\n", "", configured, flags=re.MULTILINE)
|
||||
return outcomes
|
||||
|
||||
|
||||
@@ -292,7 +292,7 @@ class Cycles(unittest.TestCase):
|
||||
return run.Cycle(config=CONFIG, window=WINDOW, effects=fake.effects(), facts=facts, judge_model="judge-model", ladder=LADDER, premises="## Goals", project=None, project_archive=self.archive, records=records, root=str(self.home), route=Route(Rung("worker-model")), session=self.session, task="Do it.", plan=plan_of(""), unfinished_evidence=unfinished_evidence)
|
||||
|
||||
def archived_outcomes(self):
|
||||
"""Each outcomes file's text without its `Judge run:` lines, which name a file by the hash of a prompt: the tests of the judge's runs check them."""
|
||||
"""Each outcomes file's text without its `Judge run:` lines, which name a file by the hash of a prompt and its reply: the tests of the judge's runs check them."""
|
||||
return sorted(JUDGE_RUN_LINE.sub("", path.read_text()) for path in self.archive.glob("*/*.outcomes.md"))
|
||||
|
||||
def test_a_finished_chunk_is_archived_and_its_session_deleted(self):
|
||||
|
||||
@@ -19,7 +19,6 @@
|
||||
Code keeps in a folder named after the session, beside the transcript. A file there is replaced only by a version that
|
||||
grew from it. A file Claude Code rewrote whole lands beside its copy as <name>.<hash of its content>
|
||||
(write.copy_for).
|
||||
|
||||
- judge/ holds each judge run an outcome names on its `Judge run:` line: the prompt the judge read and claude's
|
||||
json reply, named by the hash of that content (judge_run.JudgeRun). judge-replay reads the prompt back.
|
||||
|
||||
|
||||
+2
-2
@@ -182,9 +182,9 @@ Valid while `~/.claude` is a checkout of this repo.
|
||||
|
||||
Valid while claude takes `--session-id` and resumes by it, and the human runs loop-claude where tmux is installed.
|
||||
|
||||
## loop-claude: the archive keeps the raw transcript, and a failed archive stops the loop
|
||||
## loop-claude: the archive keeps the raw transcript and each judge run, and a failed archive stops the loop
|
||||
|
||||
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. The judge's runs are archived too, as `judge/<sha256>.json` in the chunk's folder: the prompt it read and its json reply, which are its whole transcript, since it runs with no tools and one turn. The run is kept whole, deduplicated by hash, and the outcome of the verdict, interrupt or rejected reply it produced names it on a `Judge run:` line. It is kept whole because a prompt is at most `JUDGE_PROMPT_CHARS`, 400,000 characters, against copies of up to 14.8 MB: measured 2026-11-16 over the archive's 140 copies, 603 MB, 136 recorded verdicts times that ceiling is under a tenth of it. `judge-replay` replays a verdict on its archived prompt where there is one. Valid while transcripts, and the judge's prompts, stay small enough to copy whole.
|
||||
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. The judge's runs are archived too, as `judge/<sha256>.json` in the chunk's folder: the prompt it read and its json reply. Together they are its whole transcript, since it runs with no tools and one turn. A run is kept whole and deduplicated by hash, and the outcome of the verdict, interrupt or rejected reply it produced names it on a `Judge run:` line. Keeping it whole costs little: a prompt is at most `JUDGE_PROMPT_CHARS`, 400,000 characters. Measured 2026-11-17, the archive's 911 judge lines come to at most 0.36 GB of prompts, against 3.5 GB in its 1,211 transcript copies, the largest 23.4 MB. `judge-replay` replays a verdict on its archived prompt where there is one. Valid while transcripts, and the judge's prompts, stay small enough to copy whole.
|
||||
|
||||
## loop-claude: each verdict records the claude-config release and commit that ran it, the raw tokens each thread spent by model and kind since the previous outcome that recorded a spend, and the human's typed answer the judge quotes as reversing a choice the worker took on its own recommendation
|
||||
|
||||
|
||||
Reference in New Issue
Block a user