Answer product-owner round 1: 0 🔴, 2 🟡 (0 misread); the report names the prompt each missed verdict was replayed on, the help says where records change the kind, and the decision drops pricing from usage
This commit is contained in:
@@ -12,7 +12,8 @@ premises and records each project holds now.
|
||||
|
||||
It first replays every verdict on the reference judge: the model at xhigh effort, the rung the
|
||||
judge ran on before loop-claude routed it. A verdict the reference judge returns another kind on
|
||||
may rest on records or judge rules that have changed since; either way no rung is held to it. It
|
||||
may rest, where its prompt was rebuilt, on records or judge rules that have changed since; where the archived prompt was
|
||||
replayed, the judge's own variance is the cause. Either way no rung is held to it. It
|
||||
then replays the rest on each rung above the model's cheapest that a chunk's E + I raises a
|
||||
review to, except the reference judge's rung. Then it replays them on each candidate rung,
|
||||
highest first: from the model's cheapest rung down to the lowest rung a worker routes to. It stops
|
||||
|
||||
@@ -91,4 +91,4 @@ def report(model: str, reference: Rung, cases: list[Case], replay: Replay[Case,
|
||||
|
||||
|
||||
def missed_lines(missed: list[tuple[Case, Kind]]) -> list[str]:
|
||||
return [f" {found.root} {found.at}: archived {found.archived_kind}, replayed {kind}" for found, kind in missed]
|
||||
return [f" {found.root} {found.at}: archived {found.archived_kind}, replayed {kind} on {'its archived prompt' if found.judge_run else 'a prompt rebuilt from today\'s records'}" for found, kind in missed]
|
||||
|
||||
@@ -76,16 +76,21 @@ class Reproduced(unittest.TestCase):
|
||||
|
||||
|
||||
class Report(unittest.TestCase):
|
||||
def test_names_whether_a_missed_verdict_was_replayed_on_its_archived_prompt(self):
|
||||
kept = Case("/code/p", "Do it.", TRANSCRIPT, "2026-10-14T10:00:00Z", FINISHED, pathlib.Path("/archive/p/c/judge/run.json"))
|
||||
lines = report("claude-opus-5-5", REFERENCE, [kept], Replay([UNFINISHED], [], [])).split("\n")
|
||||
self.assertIn(" /code/p 2026-10-14T10:00:00Z: archived Verdict: finished, replayed Verdict: unfinished on its archived prompt", lines)
|
||||
|
||||
def test_counts_the_verdicts_and_names_each_miss_and_the_lowest_rung(self):
|
||||
cases = [case("2026-10-14T10:00:00Z", FINISHED), case("2026-10-14T10:00:01Z", INTERRUPTED)]
|
||||
replay = Replay([UNFINISHED, INTERRUPTED], [Tried(MIDDLE, []), Tried(CHEAP, [Miss(cases[1], INTERRUPTED, FINISHED)])], MIDDLE)
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, replay).splitlines(), [
|
||||
"Replayed 2 verdicts of claude-opus-5-5 projects: 1 Verdict: finished, 1 Verdict: interrupt.",
|
||||
"The reference judge, claude-opus-5-5 at xhigh effort, reproduced 1. It returned another kind on:",
|
||||
" /code/p 2026-10-14T10:00:00Z: archived Verdict: finished, replayed Verdict: unfinished",
|
||||
" /code/p 2026-10-14T10:00:00Z: archived Verdict: finished, replayed Verdict: unfinished on a prompt rebuilt from today's records",
|
||||
"claude-sonnet-5-5 at high effort missed 0.",
|
||||
"claude-haiku-5-5 at xhigh effort missed 1:",
|
||||
" /code/p 2026-10-14T10:00:01Z: archived Verdict: interrupt, replayed Verdict: finished",
|
||||
" /code/p 2026-10-14T10:00:01Z: archived Verdict: interrupt, replayed Verdict: finished on a prompt rebuilt from today's records",
|
||||
"Lowest candidate that, like each candidate above it, returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort",
|
||||
])
|
||||
missed_all = report("claude-opus-5-5", REFERENCE, cases, Replay([FINISHED, INTERRUPTED], [Tried(MIDDLE, [Miss(cases[1], INTERRUPTED, FINISHED)])], None)).splitlines()
|
||||
@@ -98,7 +103,7 @@ class Report(unittest.TestCase):
|
||||
climbs = [Tried(Rung("claude-fable-5-2", "low"), [Miss(cases[1], INTERRUPTED, FINISHED)]), Tried(Rung("claude-fable-5-2", "high"), [])]
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, Replay([FINISHED, INTERRUPTED], [Tried(MIDDLE, [])], MIDDLE, climbs)).splitlines()[2:5], [
|
||||
"claude-fable-5-2 at low effort, a rung a review climbs to, missed 1:",
|
||||
" /code/p 2026-10-14T10:00:01Z: archived Verdict: interrupt, replayed Verdict: finished",
|
||||
" /code/p 2026-10-14T10:00:01Z: archived Verdict: interrupt, replayed Verdict: finished on a prompt rebuilt from today's records",
|
||||
"claude-fable-5-2 at high effort, a rung a review climbs to, missed 0.",
|
||||
])
|
||||
|
||||
@@ -312,6 +317,7 @@ class JudgeArchivedRuns(unittest.TestCase):
|
||||
def test_splits_the_prompt_at_the_first_task_heading(self):
|
||||
quoting = self.PROMPT.replace("Do it.", f"Quote:\n\n{TASK_HEADING}Ship slowly.", 1)
|
||||
self.assertIsInstance(self.judged(quoting, answer="Ship fast.", question="Which?")[0], QuotedAnswer)
|
||||
self.assertIsInstance(self.judged(quoting, answer="Quote:", question="Which?")[0], Interrupt)
|
||||
|
||||
def test_fails_before_the_judge_runs_on_a_prompt_without_a_heading(self):
|
||||
for prompt in ("No headings at all.", f"{TASK_HEADING}Do it.", f"\n\n{TASK_HEADING}Do it.\n\nNo transcript heading."):
|
||||
|
||||
@@ -66,6 +66,7 @@ FINISHED_RULES = (
|
||||
)
|
||||
JUDGE_RULES = "\n\n".join((INTERRUPT_RULES, QUOTE_RULES, FINISHED_RULES))
|
||||
# Open the task and the transcript in the prompt; the premises and the records come before the task, the rules after the transcript.
|
||||
# judge_replay.archived_prompt splits each archived prompt on them, so rewording one makes every run archived before it replay as failed.
|
||||
TASK_HEADING = "The task the worker was given:\n"
|
||||
TRANSCRIPT_HEADING = "What the worker did and said, what the human typed on [human] lines and what loop-claude told it on [loop] lines, latest last:\n"
|
||||
|
||||
|
||||
+1
-1
@@ -184,7 +184,7 @@ Valid while claude takes `--session-id` and resumes by it, and the human runs lo
|
||||
|
||||
## loop-claude: the archive keeps the raw transcript and each judge run, and a failed archive stops the loop
|
||||
|
||||
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. The judge's runs are archived too, as `judge/<sha256>.json` in the chunk's folder: the prompt it read and its json reply. Together they are its whole transcript, since it runs with no tools and one turn. A run is kept whole and deduplicated by hash, and the outcome of the verdict, interrupt or rejected reply it produced names it on a `Judge run:` line. Keeping it whole costs little: a prompt is at most `JUDGE_PROMPT_CHARS`, 400,000 characters. Measured 2026-11-17, the archive's 911 judge lines come to at most 0.36 GB of prompts, against 3.5 GB in its 1,211 transcript copies, the largest 23.4 MB. `judge-replay` replays a verdict on its archived prompt where there is one. Valid while transcripts, and the judge's prompts, stay small enough to copy whole.
|
||||
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. The judge's runs are archived too, as `judge/<sha256>.json` in the chunk's folder: the prompt it read and its json reply. Together they are its whole transcript, since it runs with no tools and one turn. A run is kept whole and deduplicated by hash, and the outcome of the verdict, interrupt or rejected reply it produced names it on a `Judge run:` line. Keeping it whole costs little: a prompt is at most `JUDGE_PROMPT_CHARS`, 400,000 characters. Measured 2026-11-17, the archive's 911 judge lines come to at most 0.36 GB of prompts, against 3.5 GB in its 1,211 transcript copies, the largest 23.4 MB. The judge is still priced from its `Judge cost:` line, which is claude's own total for the run, and the Spent block counts the live reply's tokens; the archived reply keeps its `usage` for a later reader, and nothing reads it back. `judge-replay` replays a verdict on its archived prompt where there is one. Valid while transcripts, and the judge's prompts, stay small enough to copy whole.
|
||||
|
||||
## loop-claude: each verdict records the claude-config release and commit that ran it, the raw tokens each thread spent by model and kind since the previous outcome that recorded a spend, and the human's typed answer the judge quotes as reversing a choice the worker took on its own recommendation
|
||||
|
||||
|
||||
Reference in New Issue
Block a user