Answer prose round 1: 0 🔴, 16 🟡 (0 misread); name the offline deciders and runs, and fix the help, the decision and the docstrings

This commit is contained in:
2026-11-17 10:22:54 +01:00
parent 155a65117c
commit 5f098aa315
6 changed files with 82 additions and 56 deletions
+1 -1
View File
@@ -181,7 +181,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `command-patterns [<version>] [--calls <file>]` prints a table of the shell command patterns a release's archived chunks ran, such as `tea pr create`, for an audit of the release, and with `--calls <file>` saves each call as a JSON line; `command-patterns --help` states the columns and the line's keys.
- `compress-archives` gzips each transcript in loop-claude's archive and each file in the review folders 30 days after it last changed, keeping all of them; Install's crontab line runs it daily.
- `deploy-release [--cut] [<checkout>]` moves `~/.claude`, or the checkout named, to the newest release, as Update says; with `--cut`, run in the development clone, it first cuts the next release, as Release says.
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints what each rung a review climbs to misses, and the lowest rung below them that, like each candidate above it, returns them. With `--offline` it calls no judge and counts what a "?" rule, a decision table and a classifier trained on the archive decide of each verdict's kind, and how many interrupts they miss. `judge-replay --help` states the replay.
- `judge-replay <model> [--offline]` replays loop-claude's judge on archived verdicts of the projects run on the model. It prints what each rung a review climbs to misses, and the lowest rung below those that returns every verdict, as each candidate above it does. With `--offline` it doesn't call a judge: it counts what a "?" rule, a decision table and a classifier trained on the archive decide of each verdict's kind from the worker's final message, and how many interrupts each misses. `judge-replay --help` states both replays.
- `larv-review <command>` holds larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's round count and the opening of its round commit's title, the scoring run's trigger and setup; the comprehension history's format, its latest places and its count of how often seats agree; and each reviewer's prompt header. `larv-review --help` states them. `settings.json` → `hooks` runs `larv-review reviewer-context` as each subagent starts. A reviewer under `agents/` whose frontmatter has `omitClaudeMd: true` starts without `CLAUDE.md`, and the command hands it that file's "Shared goal", "Prose" and "Vocabulary" sections.
- `loop-claude <task>` runs a project's next chunks, each in its own tmux window, cycle after cycle; `loop-claude --help` states its contract.
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
+17 -9
View File
@@ -32,15 +32,22 @@ replays/judge-replay-<model>-<UTC stamp>/ under ~/.claude/loop-claude-transcript
A run's .jsonl file starts with a line holding claude's arguments, working directory, prompt, exit code and
stderr, then holds claude's stream-json output; result.md holds the report.
With --offline it calls no judge. It reads every archived verdict of the model's projects, not a sample, and
replays what code could decide of its kind from the worker's final message: a "?" rule, a decision table and
a classifier trained on the archive (judge_replay/offline.py). It prints, for each, how many verdicts it
decided and how many interrupts it missed, on held-out folds and on the latest fifth.
Like the loop, it tries a judge three times running before it counts the reply as failed. A
verdict whose project, transcript or archived judge run cannot be read replays as failed too. Exit 1, saying why, on
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
or a usage limit.
With --offline it doesn't call a judge, so the runs, retries and usage limit above don't apply. It reads every
archived verdict of the model's projects, not a sample. It counts how many of them each of three code
deciders decides from the worker's final message, and how many archived interrupts it decides as another
kind, which are its missed interrupts: a "?" rule, a decision table, and a classifier trained on the archive
(judge_replay/offline.py). The rule and the table run on the whole archive; the table's rows were read from
it, so its run isn't a held-out test. The classifier runs on held-out folds, each tested by a model trained on
the other folds, and on the latest fifth of the verdicts, tested by a model trained on the earlier four
fifths. The report is kept in replays/judge-replay-<model>-offline-<UTC stamp>/result.md. A verdict whose
transcript cannot be read is skipped, and the report counts the skipped. Exit 1, saying why, on bad
arguments, an unreadable archive or models file, no archived verdict of the model's projects, or none with a
transcript that can be read.
"""
import pathlib
import sys
@@ -51,7 +58,7 @@ from transcript_archive.read import archived_outcomes
from rung_replay.archive import archiving, print_report, replay_folder
from rung_replay.command import arguments_or_exit, down_from, failed, limited_exit, projects_on
from loop_claude.cycle.judge import Brief, judge_archived, judge_once
from judge_replay.offline import final_message, METHODS, offline_report, Sample
from judge_replay.offline import final_message, LEARNERS, offline_report, RULES, Sample
from judge_replay.replay import Case, case_from, Kind, report, reproduced, sample, verdict_kind
from transcript_archive.project_key import checkout_of
from loop_effects.effects import Effects
@@ -68,7 +75,7 @@ from claude_code.transcript import read_transcript
from loop_claude.verdicts import Verdict
TOOL = "judge-replay"
USAGE = f"{TOOL} <model> [--offline], a model on the routing ladder"
USAGE = f"{TOOL} <model> [--offline]; <model> is a model on the routing ladder"
OFFLINE = "--offline"
PER_KIND = 15
# The effort settings.json gave every judge before loop-claude routed it; the archived verdicts came from it.
@@ -104,16 +111,17 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
def offline(model: str, argv: list[str], cases: list[Case], root: pathlib.Path, effects: Effects) -> int:
"""Replays the verdicts of cases offline, oldest first, and keeps the report in the archive as a judge replay does."""
samples = []
samples, skipped = [], 0
for found in sorted(cases, key=lambda found: (found.at, found.root, str(found.transcript))):
entries = read_transcript(str(found.transcript))
if isinstance(entries, Failure):
skipped += 1
print(f"{TOOL}: skipped a verdict: {entries.reason}", file=sys.stderr)
else:
samples.append(Sample(found.archived_kind, final_message(entries)))
if not samples:
return failed(TOOL, "no archived verdict has a transcript that can be read")
print_report(replay_folder(root, TOOL, argv, effects.now()), offline_report(model, samples, METHODS))
print_report(replay_folder(root, TOOL, argv, effects.now()), offline_report(model, samples, RULES, LEARNERS, skipped=skipped))
return 0
+26 -16
View File
@@ -86,9 +86,9 @@ def logistic(samples: typing.Sequence[Sample]) -> typing.Callable[[str], float]:
def classifier(samples: typing.Sequence[Sample]) -> Decider:
"""Finished below the lowest chance any interrupt among the samples got when held out, interrupt above the highest chance any other kind got, and the judge's model between.
"""Finished where the chance of an interrupt is below one half and below that of every interrupt among the samples; interrupt where it is at least one half and above that of every other kind; otherwise the judge's model decides.
Each sample's chance is the one a model trained on the other folds of the samples gives it, so the bounds don't rest on how well the model fits what it trained on. A bound is where the samples held no miss; that doesn't promise the next verdict holds none."""
A sample's chance is the one a model trained on the other folds of the samples gives it, so the bounds don't rest on how well the model fits what it trained on. A bound is where the samples held no miss; that doesn't promise the next verdict holds none."""
interrupts = [found for found in samples if found.kind == INTERRUPTED]
if not interrupts or len(interrupts) == len(samples):
return lambda text: None
@@ -105,11 +105,13 @@ def classifier(samples: typing.Sequence[Sample]) -> Decider:
return decide
METHODS: dict[str, Method] = {"a question mark": question_mark, "decision table": decision_table, "classifier": classifier}
# Rules are fixed, and run on the whole archive; a learner trains on samples, and runs on samples it didn't train on.
RULES: dict[str, Method] = {'"?" rule': question_mark, "decision table": decision_table}
LEARNERS: dict[str, Method] = {"classifier": classifier}
def folds_of(samples: typing.Sequence[Sample], count: int) -> list[tuple[list[Sample], list[Sample]]]:
"""count folds of the samples, dealt in turn; each is what to learn from and what to test on, the samples the fold's test leaves out."""
"""count folds of the samples, dealt in turn. Each fold is the samples to learn from, which are all those its test leaves out, and the samples to test on."""
return [([found for place, found in enumerate(samples) if place % count != fold], [found for place, found in enumerate(samples) if place % count == fold]) for fold in range(count)]
@@ -123,18 +125,26 @@ def plain(kind: Kind) -> str:
return kind.removeprefix("Verdict: ")
def offline_report(model: str, samples: typing.Sequence[Sample], methods: typing.Mapping[str, Method], folds: int = FOLDS) -> str:
def offline_report(model: str, samples: typing.Sequence[Sample], rules: typing.Mapping[str, Method], learners: typing.Mapping[str, Method], folds: int = FOLDS, skipped: int = 0) -> str:
"""samples are in the order the verdicts were judged, oldest first, so the latest fifth is the newest verdicts."""
kinds = collections.Counter(found.kind for found in samples)
lines = [f"Replayed {len(samples)} archived verdicts of {model} projects offline, nothing sent to an outside service: {', '.join(f'{count} {plain(kind)}' for kind, count in sorted(kinds.items(), key=lambda each: plain(each[0])))}."]
lines = [f"Replayed {len(samples)} archived verdicts of {model} projects offline, without sending anything to an outside service. The archive gives them as {', '.join(f'{count} {plain(kind)}' for kind, count in sorted(kinds.items(), key=lambda each: plain(each[0])))}."]
if skipped:
lines.append(f"Skipped {skipped} verdicts whose transcript could not be read.")
for name, rule in rules.items():
lines += decided_lines(f"{name}, on the whole archive", evaluate(rule, [], samples), kinds)
split = len(samples) - max(1, len(samples) // LATEST)
for name, method in methods.items():
held_out = sum((evaluate(method, trained, tested) for trained, tested in folds_of(samples, folds)), Counts())
lines += [result_line(f"{name}, on held-out folds", held_out), *decides_lines(name, held_out, kinds)]
lines.append(result_line(f"{name}, on the latest fifth after learning from the rest", evaluate(method, samples[:split], samples[split:])))
for name, learner in learners.items():
held_out = sum((evaluate(learner, trained, tested) for trained, tested in folds_of(samples, folds)), Counts())
lines += decided_lines(f"{name}, on held-out folds", held_out, kinds)
lines.append(result_line(f"{name}, on the latest fifth after learning from the rest", evaluate(learner, samples[:split], samples[split:])))
return "\n".join(lines)
def decided_lines(how: str, found: Counts, kinds: collections.Counter[Kind]) -> list[str]:
return [result_line(how, found), *decides_lines(how, found, kinds)]
def result_line(how: str, found: Counts) -> str:
decided = collections.Counter({kind: sum(count for (_, said), count in found.items() if said == kind) for kind in {said for _, said in found if said}})
missed = sum(count for (archived, said), count in found.items() if archived == INTERRUPTED and said not in (None, INTERRUPTED))
@@ -143,18 +153,18 @@ def result_line(how: str, found: Counts) -> str:
def wrong_kinds(decided: Kind, found: Counts) -> str:
"""The archived kinds a decider returned decided for, where the archive gave another kind."""
"""The kinds the archive gives the verdicts that were decided as decided, where the archive gives another kind than decided."""
wrong = sorted(((plain(archived), count) for (archived, said), count in found.items() if said == decided and archived != decided), key=lambda each: each[0])
return f" (archived {', '.join(f'{kind} {count}' for kind, count in wrong)})" if wrong else ""
return f" (of these, archived {', '.join(f'{kind} {count}' for kind, count in wrong)})" if wrong else ""
def decides_lines(name: str, found: Counts, kinds: collections.Counter[Kind]) -> list[str]:
"""For finished and for unfinished: whether the decider decided it where the archive holds no interrupt, and how much of that kind it got right."""
def decides_lines(how: str, found: Counts, kinds: collections.Counter[Kind]) -> list[str]:
"""For finished and for unfinished that the decider decided at all: whether it decided no archived interrupt as that kind, and how many of the kind it got right."""
lines = []
for kind in (FINISHED, UNFINISHED):
decided = sum(count for (_, said), count in found.items() if said == kind)
if decided and not found[(INTERRUPTED, kind)]:
lines.append(f"{name} decides {plain(kind)} with no missed interrupt: {found[(kind, kind)]} of {kinds[kind]} {plain(kind)}.")
lines.append(f"{how}, decides {plain(kind)} with no missed interrupt: {found[(kind, kind)]} of {kinds[kind]} {plain(kind)}.")
elif decided:
lines.append(f"{name} does not decide {plain(kind)} with no missed interrupt.")
lines.append(f"{how}, does not decide {plain(kind)} with no missed interrupt: it decided {found[(INTERRUPTED, kind)]} archived interrupt(s) as {plain(kind)}.")
return lines
+2 -2
View File
@@ -179,8 +179,8 @@ class Main(unittest.TestCase):
code, out, err = self.run_main(["claude-opus-5-5", "--offline"], unused)
self.assertEqual(code, 0, err)
self.assertEqual(self.calls, [])
self.assertIn("Replayed 2 archived verdicts of claude-opus-5-5 projects offline, nothing sent to an outside service: 1 finished, 1 interrupt.", out)
self.assertIn("a question mark, on held-out folds: decided 2 of 2: finished 2 (archived interrupt 1). Missed interrupts: 1.", out)
self.assertIn("Replayed 2 archived verdicts of claude-opus-5-5 projects offline, without sending anything to an outside service. The archive gives them as 1 finished, 1 interrupt.", out)
self.assertIn("\"?\" rule, on the whole archive: decided 2 of 2: finished 2 (of these, archived interrupt 1). Missed interrupts: 1.", out)
folder = self.home / ".claude" / "loop-claude-transcripts" / "replays" / "judge-replay-claude-opus-5-5-offline-20260921T141320Z"
self.assertEqual((folder / "result.md").read_text(encoding="utf-8"), out)
+22 -14
View File
@@ -7,7 +7,7 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINISHED
from claude_code.entries import AssistantTurn, HumanTurn, Text, ToolCall
from judge_replay.offline import classifier, decision_table, evaluate, final_message, folds_of, METHODS, offline_report, question_mark, Sample
from judge_replay.offline import classifier, decision_table, evaluate, final_message, folds_of, LEARNERS, offline_report, question_mark, RULES, Sample
def turn(*parts):
@@ -56,12 +56,12 @@ class Classifier(unittest.TestCase):
done = [sample(FINISHED, f"merged pull request {n} and removed the worktree, no findings left") for n in range(12)]
return classifier([*asking, *done, *extra])
def test_decides_a_message_like_each_kind_it_learned_and_nothing_else(self):
def test_decides_a_message_like_each_kind_it_learned(self):
decide = self.trained()
self.assertEqual(decide("I recommend option 3; tell me which you want"), INTERRUPTED)
self.assertEqual(decide("merged pull request 99 and removed the worktree, no findings left"), FINISHED)
def test_abstains_on_every_message_where_it_learned_no_interrupt_or_no_other_kind(self):
def test_abstains_where_it_learned_no_interrupt_or_no_other_kind(self):
self.assertIsNone(classifier([sample(FINISHED, "merged")])("merged"))
self.assertIsNone(classifier([sample(INTERRUPTED, "which?")])("which?"))
self.assertIsNone(classifier([])("anything"))
@@ -91,25 +91,33 @@ class Report(unittest.TestCase):
def samples(self):
return [sample(FINISHED, "Merged."), sample(FINISHED, "Merged. Release 1.0.0 is ready to cut."), sample(INTERRUPTED, "I recommend merging."), sample(INTERRUPTED, "Merged.")]
def test_names_each_method_and_counts_what_it_decided_and_missed_by_kind(self):
text = offline_report("claude-opus-5-5", self.samples(), {"decision table": decision_table})
self.assertIn("Replayed 4 archived verdicts of claude-opus-5-5 projects offline, nothing sent to an outside service: 2 finished, 2 interrupt.", text)
self.assertIn("decision table, on held-out folds: decided 3 of 4: finished 2 (archived interrupt 1), interrupt 1. Missed interrupts: 1.", text)
self.assertIn("does not decide finished with no missed interrupt", text)
def test_counts_what_a_rule_decided_on_the_whole_archive_and_missed_by_kind(self):
text = offline_report("claude-opus-5-5", self.samples(), {"decision table": decision_table}, {})
self.assertIn("Replayed 4 archived verdicts of claude-opus-5-5 projects offline, without sending anything to an outside service. The archive gives them as 2 finished, 2 interrupt.", text)
self.assertIn("decision table, on the whole archive: decided 3 of 4: finished 2 (of these, archived interrupt 1), interrupt 1. Missed interrupts: 1.", text)
self.assertIn("decision table, on the whole archive, does not decide finished with no missed interrupt: it decided 1 archived interrupt(s) as finished.", text)
def test_says_a_method_that_decided_a_kind_and_missed_no_interrupt_decides_that_kind(self):
text = offline_report("claude-opus-5-5", [sample(FINISHED, "Merged."), sample(INTERRUPTED, "I recommend merging.")], {"decision table": decision_table})
text = offline_report("claude-opus-5-5", [sample(FINISHED, "Merged."), sample(INTERRUPTED, "I recommend merging.")], {"decision table": decision_table}, {})
self.assertIn("Missed interrupts: 0.", text)
self.assertIn("decides finished with no missed interrupt: 1 of 1 finished", text)
self.assertIn("decision table, on the whole archive, decides finished with no missed interrupt: 1 of 1 finished.", text)
def test_runs_a_method_that_learns_once_for_each_fold_and_once_on_the_latest_fifth(self):
def test_runs_a_learner_once_for_each_fold_and_once_on_the_latest_fifth_and_a_rule_only_on_the_whole_archive(self):
samples = [sample(INTERRUPTED if n % 2 else FINISHED, "I recommend it" if n % 2 else "merged it") for n in range(20)]
text = offline_report("m", samples, {"classifier": classifier}, folds=4)
text = offline_report("m", samples, {"rule": question_mark}, {"classifier": classifier}, folds=4)
self.assertIn("classifier, on held-out folds: decided", text)
self.assertIn("classifier, on the latest fifth after learning from the rest: decided", text)
self.assertIn("rule, on the whole archive: decided 20 of 20", text)
self.assertNotIn("rule, on held-out", text)
self.assertNotIn("rule, on the latest fifth", text)
def test_the_shipped_methods_are_the_one_rule_the_table_and_the_classifier(self):
self.assertEqual(list(METHODS), ["a question mark", "decision table", "classifier"])
def test_counts_the_verdicts_it_skipped_only_where_it_skipped_some(self):
self.assertNotIn("Skipped", offline_report("m", self.samples(), {}, {}))
self.assertIn("Skipped 2 verdicts whose transcript could not be read.", offline_report("m", self.samples(), {}, {}, skipped=2))
def test_the_shipped_methods_are_the_question_mark_rule_the_table_and_the_classifier(self):
self.assertEqual(list(RULES), ['"?" rule', "decision table"])
self.assertEqual(list(LEARNERS), ["classifier"])
if __name__ == "__main__":
+14 -14
View File
@@ -624,20 +624,6 @@ On 2026-11-12, `bin/judge-replay claude-opus-5-5` replayed 60 verdicts, 15 of ea
Valid while the archived verdicts show what a judge should return, and until the rungs the replays tested change: the project's rung, those below it and each rung above it that a review climbs to. Replay then.
## routing: the judge's model keeps the verdict kind, since no code rule and no classifier trained on the archive decided a kind of the worker's final message with no missed interrupt
2026-11-17, claude. Serves goals 1.2 and 1.4, and applies the principle that a cost cut waits for a measurement that shows quality holds.
On 2026-11-17, `bin/judge-replay claude-opus-5-5 --offline` replayed all 1,104 archived verdicts of the Opus 5.5 projects (660 finished, 333 interrupt, 33 quoted answer, 78 unfinished) through three deciders that read only the worker's final message and sent nothing to an outside service. Each decides a kind or leaves the verdict to the judge's model. A missed interrupt is an archived interrupt decided as another kind, which lets the worker settle a decision that is lilleman's.
- **A "?" rule** (interrupt where the message holds one, finished where it holds none) decided every verdict and missed 197 interrupts.
- **A decision table** (the rows `DECISION_TABLE` in `bin/judge_replay/offline.py` holds, read from the archive's interrupts, so its run is partly in-sample) decided 974 and missed 55. Most of its misses are a worker reporting a merged chunk while the release is ready to cut, or asking for an answer in words no row names. Both depend on state the message doesn't hold, such as the plan's items.
- **A logistic-regression classifier over the message's words**, trained on the archive, bounded where the training folds held no miss, decided 93 on held-out folds and missed 2 interrupts. On the latest fifth, learning from the rest, it decided 25 and missed 1.
No method meets the bar of no missed interrupt, so no item is filed for a verdict kind and the judge's model keeps the verdict kind, the escalation it quotes and whether a release is ready to cut.
Valid while the archived verdicts hold interrupts that read like a finished chunk. Replay with `judge-replay <model> --offline` when the archive has doubled, or when a decider reads more than the final message.
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index, shipped as `bin/model_routing/routing-ladder.json`, which a project may replace with a file of that name in its config's folder; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one shipped ladder and one set of steps for every project, read by loop-claude and by the worker through `bin/subagent-rung`; a project runs on other rungs only through a ladder file of its own (2026-11-15 note). The judge's rung is set by `routing: on a model judge-replay covered`. `bin/model_routing/routing-ladder.json` holds the rungs and `ladder.py` beside it the steps.
@@ -664,6 +650,20 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
Valid while the index ranks these models, and while no newer model joins the catalog.
## routing: the judge's model keeps the verdict kind, since no code rule and no classifier trained on the archive decided finished from the worker's final message with no missed interrupt
2026-11-17, claude. Serves goals 1.2 and 1.4, and applies project principle 1: a cost cut waits for a measurement that shows quality holds.
On 2026-11-17, `bin/judge-replay claude-opus-5-5 --offline` replayed all 1,104 archived verdicts of the Opus 5.5 projects (660 finished, 333 interrupt, 33 quoted answer, 78 unfinished) through three deciders. Each reads only the worker's final message, doesn't send anything to an outside service, and decides a kind or leaves the verdict to the judge's model. A missed interrupt is an archived interrupt decided as another kind, which lets the worker settle a decision that is lilleman's. Deciding interrupt can't miss one: a wrong interrupt only asks lilleman more.
- **A "?" rule** (interrupt where the message holds a "?", finished where it holds none) decided every verdict and missed 197 interrupts.
- **A decision table**, the rows of `DECISION_TABLE` in `bin/judge_replay/offline.py`. Its rows were read from this archive's interrupts, so its run isn't a held-out test. It decided 974 and missed 55. Most of its misses are a worker reporting a merged chunk while a release is ready to cut, which depends on the plan; the rest ask for an answer in words no row names.
- **A classifier**, a logistic regression over the message's words, trained on the archive. It decides finished only where its chance of an interrupt is below that of every interrupt among its training folds when held out, and interrupt only where it is above that of every other kind. On held-out folds it decided 93 and missed 2 interrupts. On the latest fifth, learning from the rest, it decided 25 and missed 1.
No method meets the bar of no missed interrupt, so no item is filed for a verdict kind, and the judge's model keeps the verdict kind. It also keeps the escalation it quotes and whether a release is ready to cut, which this replay didn't test.
Valid while the archived verdicts hold interrupts that read like a finished chunk. Replay with `judge-replay <model> --offline` when the archive has doubled, or when a decider reads more than the final message.
## routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds
2026-10-14, lilleman. Serves goal 1.4.