Replay the archived judge verdicts offline through code rules and a classifier trained on the archive
CI / tests (pull_request) Failing after 5m8s
CI / tests (pull_request) Failing after 5m8s
This commit is contained in:
@@ -181,7 +181,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `command-patterns [<version>] [--calls <file>]` prints a table of the shell command patterns a release's archived chunks ran, such as `tea pr create`, for an audit of the release, and with `--calls <file>` saves each call as a JSON line; `command-patterns --help` states the columns and the line's keys.
|
||||
- `compress-archives` gzips each transcript in loop-claude's archive and each file in the review folders 30 days after it last changed, keeping all of them; Install's crontab line runs it daily.
|
||||
- `deploy-release [--cut] [<checkout>]` moves `~/.claude`, or the checkout named, to the newest release, as Update says; with `--cut`, run in the development clone, it first cuts the next release, as Release says.
|
||||
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints what each rung a review climbs to misses, and the lowest rung below them that, like each candidate above it, returns them; `judge-replay --help` states the replay.
|
||||
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints what each rung a review climbs to misses, and the lowest rung below them that, like each candidate above it, returns them. With `--offline` it calls no judge and counts what a "?" rule, a decision table and a classifier trained on the archive decide of each verdict's kind, and how many interrupts they miss. `judge-replay --help` states the replay.
|
||||
- `larv-review <command>` holds larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's round count and the opening of its round commit's title, the scoring run's trigger and setup; the comprehension history's format, its latest places and its count of how often seats agree; and each reviewer's prompt header. `larv-review --help` states them. `settings.json` → `hooks` runs `larv-review reviewer-context` as each subagent starts. A reviewer under `agents/` whose frontmatter has `omitClaudeMd: true` starts without `CLAUDE.md`, and the command hands it that file's "Shared goal", "Prose" and "Vocabulary" sections.
|
||||
- `loop-claude <task>` runs a project's next chunks, each in its own tmux window, cycle after cycle; `loop-claude --help` states its contract.
|
||||
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""Replay loop-claude's judge on archived verdicts, to find the lowest rung that returns them: docs/decisions.md → `routing: on a model judge-replay covered`.
|
||||
|
||||
usage: judge-replay <model>
|
||||
usage: judge-replay <model> [--offline]
|
||||
|
||||
<model> is a project model on the routing ladder this setup ships, such as claude-opus-5-5. The tool replays the
|
||||
verdicts archived for the projects that ~/.claude/loop-claude-models.json runs on it: the latest
|
||||
@@ -32,12 +32,18 @@ replays/judge-replay-<model>-<UTC stamp>/ under ~/.claude/loop-claude-transcript
|
||||
A run's .jsonl file starts with a line holding claude's arguments, working directory, prompt, exit code and
|
||||
stderr, then holds claude's stream-json output; result.md holds the report.
|
||||
|
||||
With --offline it calls no judge. It reads every archived verdict of the model's projects, not a sample, and
|
||||
replays what code could decide of its kind from the worker's final message: a "?" rule, a decision table and
|
||||
a classifier trained on the archive (judge_replay/offline.py). It prints, for each, how many verdicts it
|
||||
decided and how many interrupts it missed, on held-out folds and on the latest fifth.
|
||||
|
||||
Like the loop, it tries a judge three times running before it counts the reply as failed. A
|
||||
verdict whose project, transcript or archived judge run cannot be read replays as failed too. Exit 1, saying why, on
|
||||
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
|
||||
or a usage limit.
|
||||
"""
|
||||
import pathlib
|
||||
import sys
|
||||
import typing
|
||||
|
||||
from transcript_archive.location import archive_root, transcript_of
|
||||
@@ -45,6 +51,7 @@ from transcript_archive.read import archived_outcomes
|
||||
from rung_replay.archive import archiving, print_report, replay_folder
|
||||
from rung_replay.command import arguments_or_exit, down_from, failed, limited_exit, projects_on
|
||||
from loop_claude.cycle.judge import Brief, judge_archived, judge_once
|
||||
from judge_replay.offline import final_message, METHODS, offline_report, Sample
|
||||
from judge_replay.replay import Case, case_from, Kind, report, reproduced, sample, verdict_kind
|
||||
from transcript_archive.project_key import checkout_of
|
||||
from loop_effects.effects import Effects
|
||||
@@ -61,7 +68,8 @@ from claude_code.transcript import read_transcript
|
||||
from loop_claude.verdicts import Verdict
|
||||
|
||||
TOOL = "judge-replay"
|
||||
USAGE = f"{TOOL} <model>, a model on the routing ladder"
|
||||
USAGE = f"{TOOL} <model> [--offline], a model on the routing ladder"
|
||||
OFFLINE = "--offline"
|
||||
PER_KIND = 15
|
||||
# The effort settings.json gave every judge before loop-claude routed it; the archived verdicts came from it.
|
||||
REFERENCE_EFFORT = "xhigh"
|
||||
@@ -72,7 +80,7 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
ladder = read_ladder(SHIPPED)
|
||||
if isinstance(ladder, Failure):
|
||||
return failed(TOOL, ladder.reason)
|
||||
project = arguments_or_exit(TOOL, argv, __doc__, USAGE, lambda argv: model_argument(ladder, argv))
|
||||
project = arguments_or_exit(TOOL, argv, __doc__, USAGE, lambda argv: model_argument(ladder, [arg for arg in argv if arg != OFFLINE]))
|
||||
if isinstance(project, int):
|
||||
return project
|
||||
cases = archived_cases(project.rung.model, archive_root(home, env), home)
|
||||
@@ -80,6 +88,8 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
return failed(TOOL, cases.reason)
|
||||
if not cases:
|
||||
return failed(TOOL, f"no archived verdicts of projects on {project.rung.model} in {archive_root(home, env)}; run loop-claude on a project on that model first")
|
||||
if OFFLINE in argv:
|
||||
return offline(project.rung.model, argv, cases, archive_root(home, env), effects)
|
||||
cases = sample(cases, PER_KIND)
|
||||
folder = replay_folder(archive_root(home, env), TOOL, argv, effects.now())
|
||||
effects = archiving(effects, folder)
|
||||
@@ -92,6 +102,21 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
return 0
|
||||
|
||||
|
||||
def offline(model: str, argv: list[str], cases: list[Case], root: pathlib.Path, effects: Effects) -> int:
|
||||
"""Replays the verdicts of cases offline, oldest first, and keeps the report in the archive as a judge replay does."""
|
||||
samples = []
|
||||
for found in sorted(cases, key=lambda found: (found.at, found.root, str(found.transcript))):
|
||||
entries = read_transcript(str(found.transcript))
|
||||
if isinstance(entries, Failure):
|
||||
print(f"{TOOL}: skipped a verdict: {entries.reason}", file=sys.stderr)
|
||||
else:
|
||||
samples.append(Sample(found.archived_kind, final_message(entries)))
|
||||
if not samples:
|
||||
return failed(TOOL, "no archived verdict has a transcript that can be read")
|
||||
print_report(replay_folder(root, TOOL, argv, effects.now()), offline_report(model, samples, METHODS))
|
||||
return 0
|
||||
|
||||
|
||||
def model_argument(ladder: Ladder, argv: list[str]) -> ProjectRung | None:
|
||||
"""The project's rung on ladder of the one model argv names; None for any other argv."""
|
||||
return ladder.project_rung(argv[0]) if len(argv) == 1 else None
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
"""Deciders of a verdict's kind that run offline on the worker's final message, replayed on the archive: docs/decisions.md → `routing: the judge's model keeps the verdict kind, …`."""
|
||||
import collections
|
||||
import math
|
||||
import random
|
||||
import re
|
||||
import typing
|
||||
|
||||
from transcript_archive.outcomes import FINISHED, INTERRUPTED, UNFINISHED
|
||||
from claude_code.entries import AssistantTurn, Entry, Text
|
||||
from judge_replay.replay import Kind
|
||||
|
||||
|
||||
class Sample(typing.NamedTuple):
|
||||
"""What a decider reads of one archived verdict: the kind the archive gives it and the worker's final message."""
|
||||
kind: Kind
|
||||
text: str
|
||||
|
||||
|
||||
# A decider names a kind, or None to leave the verdict to the judge's model. A method builds one from the samples it may learn from.
|
||||
Decider = typing.Callable[[str], Kind | None]
|
||||
Method = typing.Callable[[typing.Sequence[Sample]], Decider]
|
||||
Counts = collections.Counter[tuple[Kind, Kind | None]]
|
||||
|
||||
FOLDS = 10
|
||||
INNER_FOLDS = 5
|
||||
# A word in fewer messages than this is left out of the classifier.
|
||||
MIN_MESSAGES = 3
|
||||
EPOCHS = 8
|
||||
RATE = 0.05
|
||||
DECAY = 1e-3
|
||||
# The latest 1/LATEST of the archive is the second test: the verdicts a decider would meet next.
|
||||
LATEST = 5
|
||||
|
||||
# The first row whose pattern matches the final message decides it; a message no row matches is left to the judge. The rows come from reading the archive's interrupts, so a run of the table on the whole archive is not a held-out test.
|
||||
DECISION_TABLE = (
|
||||
(INTERRUPTED, re.compile(r"(?i)\?|recommend|your call|needs? you|tell me|you answer|please|unmerged|not merged|do you|should i|i need|need your|you say|decisions? for|stays until")),
|
||||
(FINISHED, re.compile(r"(?i)\bmerged\b(?!.*ready to cut)")),
|
||||
)
|
||||
|
||||
|
||||
def final_message(entries: list[Entry]) -> str:
|
||||
"""The text of the worker's last turn that has any, "" where it wrote none."""
|
||||
for entry in reversed(entries):
|
||||
if isinstance(entry, AssistantTurn):
|
||||
text = "\n".join(part.text for part in entry.parts if isinstance(part, Text) and part.text.strip())
|
||||
if text:
|
||||
return text
|
||||
return ""
|
||||
|
||||
|
||||
def question_mark(samples: typing.Sequence[Sample]) -> Decider:
|
||||
"""The one rule: an interrupt where the message holds a "?", finished where it holds none."""
|
||||
return lambda text: INTERRUPTED if "?" in text else FINISHED
|
||||
|
||||
|
||||
def decision_table(samples: typing.Sequence[Sample]) -> Decider:
|
||||
return lambda text: next((kind for kind, pattern in DECISION_TABLE if pattern.search(text)), None)
|
||||
|
||||
|
||||
def words_of(text: str) -> frozenset[str]:
|
||||
"""Each word and each pair of neighbouring words of the message, and its question marks."""
|
||||
words = re.findall(r"[a-z0-9']+|\?", text.lower())
|
||||
return frozenset(words) | {f"{first} {second}" for first, second in zip(words, words[1:])}
|
||||
|
||||
|
||||
def logistic(samples: typing.Sequence[Sample]) -> typing.Callable[[str], float]:
|
||||
"""The chance that a message is an interrupt, from a logistic regression over its words that the samples train."""
|
||||
seen = collections.Counter(word for found in samples for word in words_of(found.text))
|
||||
vocabulary = {word for word, count in seen.items() if count >= MIN_MESSAGES}
|
||||
data = [(words_of(found.text) & vocabulary, 1.0 if found.kind == INTERRUPTED else 0.0) for found in samples]
|
||||
weights: dict[str, float] = collections.defaultdict(float)
|
||||
bias = 0.0
|
||||
shuffler = random.Random(0)
|
||||
|
||||
def chance(words: typing.Iterable[str]) -> float:
|
||||
return 1 / (1 + math.exp(-max(-30.0, min(30.0, bias + sum(weights[word] for word in words)))))
|
||||
|
||||
for _ in range(EPOCHS):
|
||||
shuffler.shuffle(data)
|
||||
for words, interrupt in data:
|
||||
error = chance(words) - interrupt
|
||||
bias -= RATE * error
|
||||
for word in words:
|
||||
weights[word] -= RATE * (error + DECAY * weights[word])
|
||||
return lambda text: chance(words_of(text) & vocabulary)
|
||||
|
||||
|
||||
def classifier(samples: typing.Sequence[Sample]) -> Decider:
|
||||
"""Finished below the lowest chance any interrupt among the samples got when held out, interrupt above the highest chance any other kind got, and the judge's model between.
|
||||
|
||||
Each sample's chance is the one a model trained on the other folds of the samples gives it, so the bounds don't rest on how well the model fits what it trained on. A bound is where the samples held no miss; that doesn't promise the next verdict holds none."""
|
||||
interrupts = [found for found in samples if found.kind == INTERRUPTED]
|
||||
if not interrupts or len(interrupts) == len(samples):
|
||||
return lambda text: None
|
||||
held_out = [(tested.kind, chance) for trained, tests in folds_of(samples, INNER_FOLDS) for chance, tested in zip(map(logistic(trained), (found.text for found in tests)), tests)]
|
||||
below = min(chance for kind, chance in held_out if kind == INTERRUPTED)
|
||||
above = max((chance for kind, chance in held_out if kind != INTERRUPTED), default=0.0)
|
||||
chance = logistic(samples)
|
||||
|
||||
def decide(text: str) -> Kind | None:
|
||||
found = chance(text)
|
||||
if found < below and found < 0.5:
|
||||
return FINISHED
|
||||
return INTERRUPTED if found > above and found >= 0.5 else None
|
||||
return decide
|
||||
|
||||
|
||||
METHODS: dict[str, Method] = {"a question mark": question_mark, "decision table": decision_table, "classifier": classifier}
|
||||
|
||||
|
||||
def folds_of(samples: typing.Sequence[Sample], count: int) -> list[tuple[list[Sample], list[Sample]]]:
|
||||
"""count folds of the samples, dealt in turn; each is what to learn from and what to test on, the samples the fold's test leaves out."""
|
||||
return [([found for place, found in enumerate(samples) if place % count != fold], [found for place, found in enumerate(samples) if place % count == fold]) for fold in range(count)]
|
||||
|
||||
|
||||
def evaluate(method: Method, trained: typing.Sequence[Sample], tested: typing.Sequence[Sample]) -> Counts:
|
||||
"""How often the method, learning from trained, decided each kind for each archived kind of tested; None where it left the verdict to the judge."""
|
||||
decide = method(trained)
|
||||
return collections.Counter((found.kind, decide(found.text)) for found in tested)
|
||||
|
||||
|
||||
def plain(kind: Kind) -> str:
|
||||
return kind.removeprefix("Verdict: ")
|
||||
|
||||
|
||||
def offline_report(model: str, samples: typing.Sequence[Sample], methods: typing.Mapping[str, Method], folds: int = FOLDS) -> str:
|
||||
"""samples are in the order the verdicts were judged, oldest first, so the latest fifth is the newest verdicts."""
|
||||
kinds = collections.Counter(found.kind for found in samples)
|
||||
lines = [f"Replayed {len(samples)} archived verdicts of {model} projects offline, nothing sent to an outside service: {', '.join(f'{count} {plain(kind)}' for kind, count in sorted(kinds.items(), key=lambda each: plain(each[0])))}."]
|
||||
split = len(samples) - max(1, len(samples) // LATEST)
|
||||
for name, method in methods.items():
|
||||
held_out = sum((evaluate(method, trained, tested) for trained, tested in folds_of(samples, folds)), Counts())
|
||||
lines += [result_line(f"{name}, on held-out folds", held_out), *decides_lines(name, held_out, kinds)]
|
||||
lines.append(result_line(f"{name}, on the latest fifth after learning from the rest", evaluate(method, samples[:split], samples[split:])))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def result_line(how: str, found: Counts) -> str:
|
||||
decided = collections.Counter({kind: sum(count for (_, said), count in found.items() if said == kind) for kind in {said for _, said in found if said}})
|
||||
missed = sum(count for (archived, said), count in found.items() if archived == INTERRUPTED and said not in (None, INTERRUPTED))
|
||||
parts = [f"{plain(kind)} {count}{wrong_kinds(kind, found)}" for kind, count in sorted(decided.items(), key=lambda each: plain(each[0]))]
|
||||
return f"{how}: decided {sum(decided.values())} of {sum(found.values())}{': ' + ', '.join(parts) if parts else ''}. Missed interrupts: {missed}."
|
||||
|
||||
|
||||
def wrong_kinds(decided: Kind, found: Counts) -> str:
|
||||
"""The archived kinds a decider returned decided for, where the archive gave another kind."""
|
||||
wrong = sorted(((plain(archived), count) for (archived, said), count in found.items() if said == decided and archived != decided), key=lambda each: each[0])
|
||||
return f" (archived {', '.join(f'{kind} {count}' for kind, count in wrong)})" if wrong else ""
|
||||
|
||||
|
||||
def decides_lines(name: str, found: Counts, kinds: collections.Counter[Kind]) -> list[str]:
|
||||
"""For finished and for unfinished: whether the decider decided it where the archive holds no interrupt, and how much of that kind it got right."""
|
||||
lines = []
|
||||
for kind in (FINISHED, UNFINISHED):
|
||||
decided = sum(count for (_, said), count in found.items() if said == kind)
|
||||
if decided and not found[(INTERRUPTED, kind)]:
|
||||
lines.append(f"{name} decides {plain(kind)} with no missed interrupt: {found[(kind, kind)]} of {kinds[kind]} {plain(kind)}.")
|
||||
elif decided:
|
||||
lines.append(f"{name} does not decide {plain(kind)} with no missed interrupt.")
|
||||
return lines
|
||||
@@ -175,6 +175,22 @@ class Main(unittest.TestCase):
|
||||
# The replay shows the judge the whole transcript, with no step-up marked in it.
|
||||
self.assertNotIn("[step]", self.prompts[0])
|
||||
|
||||
def test_offline_replays_every_verdict_through_the_code_deciders_without_calling_the_judge(self):
|
||||
code, out, err = self.run_main(["claude-opus-5-5", "--offline"], unused)
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual(self.calls, [])
|
||||
self.assertIn("Replayed 2 archived verdicts of claude-opus-5-5 projects offline, nothing sent to an outside service: 1 finished, 1 interrupt.", out)
|
||||
self.assertIn("a question mark, on held-out folds: decided 2 of 2: finished 2 (archived interrupt 1). Missed interrupts: 1.", out)
|
||||
folder = self.home / ".claude" / "loop-claude-transcripts" / "replays" / "judge-replay-claude-opus-5-5-offline-20260921T141320Z"
|
||||
self.assertEqual((folder / "result.md").read_text(encoding="utf-8"), out)
|
||||
|
||||
def test_offline_exits_1_where_no_transcript_can_be_read(self):
|
||||
for path in (self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1").glob("*.jsonl"):
|
||||
path.unlink()
|
||||
code, out, err = self.run_main(["claude-opus-5-5", "--offline"], unused)
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("no archived verdict has a transcript that can be read", err)
|
||||
|
||||
def test_keeps_each_run_and_the_printed_report_in_one_folder_of_the_archive_beside_the_projects(self):
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
self.assertEqual(code, 0, err)
|
||||
@@ -272,7 +288,7 @@ class Main(unittest.TestCase):
|
||||
|
||||
def test_prints_the_help(self):
|
||||
code, out, _ = self.run_main(["-h"], unused)
|
||||
self.assertEqual((code, out.splitlines()[2]), (0, "usage: judge-replay <model>"))
|
||||
self.assertEqual((code, out.splitlines()[2]), (0, "usage: judge-replay <model> [--offline]"))
|
||||
|
||||
def test_the_launcher_prints_the_help(self):
|
||||
ran = subprocess.run([sys.executable, str(TOOL), "--help"], capture_output=True, text=True)
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
import collections
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINISHED
|
||||
from claude_code.entries import AssistantTurn, HumanTurn, Text, ToolCall
|
||||
from judge_replay.offline import classifier, decision_table, evaluate, final_message, folds_of, METHODS, offline_report, question_mark, Sample
|
||||
|
||||
|
||||
def turn(*parts):
|
||||
return AssistantTurn(parts=parts)
|
||||
|
||||
|
||||
def sample(kind, text):
|
||||
return Sample(kind, text)
|
||||
|
||||
|
||||
class FinalMessage(unittest.TestCase):
|
||||
def test_is_the_text_of_the_workers_last_turn_that_has_any(self):
|
||||
entries = [turn(Text("first")), HumanTurn(text="ok"), turn(Text("Done. Merged."), ToolCall("1", "Bash", {})), turn(ToolCall("2", "Bash", {}))]
|
||||
self.assertEqual(final_message(entries), "Done. Merged.")
|
||||
|
||||
def test_is_empty_where_the_worker_wrote_no_text(self):
|
||||
self.assertEqual(final_message([HumanTurn(text="hi"), turn(ToolCall("1", "Bash", {}))]), "")
|
||||
|
||||
|
||||
class QuestionMark(unittest.TestCase):
|
||||
def test_decides_interrupt_on_a_question_mark_and_finished_on_none(self):
|
||||
decide = question_mark([])
|
||||
self.assertEqual(decide("Which one?"), INTERRUPTED)
|
||||
self.assertEqual(decide("Merged."), FINISHED)
|
||||
|
||||
|
||||
class DecisionTable(unittest.TestCase):
|
||||
def test_reads_the_workers_asking_for_the_humans_answer_as_an_interrupt(self):
|
||||
decide = decision_table([])
|
||||
for text in ("My recommendation is option 1.", "Your call: merge now or wait.", "The PR stays unmerged.", "Tell me which."):
|
||||
with self.subTest(text=text):
|
||||
self.assertEqual(decide(text), INTERRUPTED)
|
||||
|
||||
def test_reads_a_merge_with_no_ask_and_no_release_to_cut_as_finished(self):
|
||||
decide = decision_table([])
|
||||
self.assertEqual(decide("PR 12 merged and the worktree is removed."), FINISHED)
|
||||
self.assertIsNone(decide("PR 12 merged. Release 0.3.0 is ready to cut."))
|
||||
|
||||
def test_leaves_any_other_message_to_the_judge(self):
|
||||
self.assertIsNone(decision_table([])("I stopped at the gate."))
|
||||
|
||||
|
||||
class Classifier(unittest.TestCase):
|
||||
def trained(self, extra=()):
|
||||
asking = [sample(INTERRUPTED, f"I recommend option {n}; tell me which of the options you want") for n in range(12)]
|
||||
done = [sample(FINISHED, f"merged pull request {n} and removed the worktree, no findings left") for n in range(12)]
|
||||
return classifier([*asking, *done, *extra])
|
||||
|
||||
def test_decides_a_message_like_each_kind_it_learned_and_nothing_else(self):
|
||||
decide = self.trained()
|
||||
self.assertEqual(decide("I recommend option 3; tell me which you want"), INTERRUPTED)
|
||||
self.assertEqual(decide("merged pull request 99 and removed the worktree, no findings left"), FINISHED)
|
||||
|
||||
def test_abstains_on_every_message_where_it_learned_no_interrupt_or_no_other_kind(self):
|
||||
self.assertIsNone(classifier([sample(FINISHED, "merged")])("merged"))
|
||||
self.assertIsNone(classifier([sample(INTERRUPTED, "which?")])("which?"))
|
||||
self.assertIsNone(classifier([])("anything"))
|
||||
|
||||
def test_decides_no_finished_where_an_interrupt_it_learned_looks_like_one(self):
|
||||
lookalike = "merged pull request 7 and removed the worktree, no findings left"
|
||||
decide = self.trained([sample(INTERRUPTED, lookalike)])
|
||||
self.assertNotEqual(decide(lookalike), FINISHED)
|
||||
|
||||
|
||||
class Folds(unittest.TestCase):
|
||||
def test_deals_samples_to_folds_in_turn_so_each_is_tested_once(self):
|
||||
samples = [sample(FINISHED, str(n)) for n in range(7)]
|
||||
dealt = folds_of(samples, 3)
|
||||
self.assertEqual([[found.text for found in tested] for _, tested in dealt], [["0", "3", "6"], ["1", "4"], ["2", "5"]])
|
||||
self.assertEqual([found.text for found in dealt[1][0]], ["0", "2", "3", "5", "6"])
|
||||
|
||||
|
||||
class Evaluate(unittest.TestCase):
|
||||
def test_counts_each_archived_kind_by_what_the_method_decided_or_abstained(self):
|
||||
tested = [sample(FINISHED, "merged"), sample(INTERRUPTED, "merged"), sample(UNFINISHED, "stopped"), sample(ANSWERED, "Which?")]
|
||||
found = evaluate(decision_table, [], tested)
|
||||
self.assertEqual(found, collections.Counter({(FINISHED, FINISHED): 1, (INTERRUPTED, FINISHED): 1, (UNFINISHED, None): 1, (ANSWERED, INTERRUPTED): 1}))
|
||||
|
||||
|
||||
class Report(unittest.TestCase):
|
||||
def samples(self):
|
||||
return [sample(FINISHED, "Merged."), sample(FINISHED, "Merged. Release 1.0.0 is ready to cut."), sample(INTERRUPTED, "I recommend merging."), sample(INTERRUPTED, "Merged.")]
|
||||
|
||||
def test_names_each_method_and_counts_what_it_decided_and_missed_by_kind(self):
|
||||
text = offline_report("claude-opus-5-5", self.samples(), {"decision table": decision_table})
|
||||
self.assertIn("Replayed 4 archived verdicts of claude-opus-5-5 projects offline, nothing sent to an outside service: 2 finished, 2 interrupt.", text)
|
||||
self.assertIn("decision table, on held-out folds: decided 3 of 4: finished 2 (archived interrupt 1), interrupt 1. Missed interrupts: 1.", text)
|
||||
self.assertIn("does not decide finished with no missed interrupt", text)
|
||||
|
||||
def test_says_a_method_that_decided_a_kind_and_missed_no_interrupt_decides_that_kind(self):
|
||||
text = offline_report("claude-opus-5-5", [sample(FINISHED, "Merged."), sample(INTERRUPTED, "I recommend merging.")], {"decision table": decision_table})
|
||||
self.assertIn("Missed interrupts: 0.", text)
|
||||
self.assertIn("decides finished with no missed interrupt: 1 of 1 finished", text)
|
||||
|
||||
def test_runs_a_method_that_learns_once_for_each_fold_and_once_on_the_latest_fifth(self):
|
||||
samples = [sample(INTERRUPTED if n % 2 else FINISHED, "I recommend it" if n % 2 else "merged it") for n in range(20)]
|
||||
text = offline_report("m", samples, {"classifier": classifier}, folds=4)
|
||||
self.assertIn("classifier, on held-out folds: decided", text)
|
||||
self.assertIn("classifier, on the latest fifth after learning from the rest: decided", text)
|
||||
|
||||
def test_the_shipped_methods_are_the_one_rule_the_table_and_the_classifier(self):
|
||||
self.assertEqual(list(METHODS), ["a question mark", "decision table", "classifier"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -624,6 +624,20 @@ On 2026-11-12, `bin/judge-replay claude-opus-5-5` replayed 60 verdicts, 15 of ea
|
||||
|
||||
Valid while the archived verdicts show what a judge should return, and until the rungs the replays tested change: the project's rung, those below it and each rung above it that a review climbs to. Replay then.
|
||||
|
||||
## routing: the judge's model keeps the verdict kind, since no code rule and no classifier trained on the archive decided a kind of the worker's final message with no missed interrupt
|
||||
|
||||
2026-11-17, claude. Serves goals 1.2 and 1.4, and applies the principle that a cost cut waits for a measurement that shows quality holds.
|
||||
|
||||
On 2026-11-17, `bin/judge-replay claude-opus-5-5 --offline` replayed all 1,104 archived verdicts of the Opus 5.5 projects (660 finished, 333 interrupt, 33 quoted answer, 78 unfinished) through three deciders that read only the worker's final message and sent nothing to an outside service. Each decides a kind or leaves the verdict to the judge's model. A missed interrupt is an archived interrupt decided as another kind, which lets the worker settle a decision that is lilleman's.
|
||||
|
||||
- **A "?" rule** (interrupt where the message holds one, finished where it holds none) decided every verdict and missed 197 interrupts.
|
||||
- **A decision table** (the rows `DECISION_TABLE` in `bin/judge_replay/offline.py` holds, read from the archive's interrupts, so its run is partly in-sample) decided 974 and missed 55. Most of its misses are a worker reporting a merged chunk while the release is ready to cut, or asking for an answer in words no row names. Both depend on state the message doesn't hold, such as the plan's items.
|
||||
- **A logistic-regression classifier over the message's words**, trained on the archive, bounded where the training folds held no miss, decided 93 on held-out folds and missed 2 interrupts. On the latest fifth, learning from the rest, it decided 25 and missed 1.
|
||||
|
||||
No method meets the bar of no missed interrupt, so no item is filed for a verdict kind and the judge's model keeps the verdict kind, the escalation it quotes and whether a release is ready to cut.
|
||||
|
||||
Valid while the archived verdicts hold interrupts that read like a finished chunk. Replay with `judge-replay <model> --offline` when the archive has doubled, or when a decider reads more than the final message.
|
||||
|
||||
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index, shipped as `bin/model_routing/routing-ladder.json`, which a project may replace with a file of that name in its config's folder; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
|
||||
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one shipped ladder and one set of steps for every project, read by loop-claude and by the worker through `bin/subagent-rung`; a project runs on other rungs only through a ladder file of its own (2026-11-15 note). The judge's rung is set by `routing: on a model judge-replay covered`. `bin/model_routing/routing-ladder.json` holds the rungs and `ladder.py` beside it the steps.
|
||||
|
||||
@@ -2525,3 +2525,19 @@ Read most (opens and searches):
|
||||
- `bin/loop_claude/cycle/run.py` 4
|
||||
- `bin/loop_claude/loop/close.py` 3
|
||||
- `bin/model_routing/test_review_choice.py` 3
|
||||
|
||||
## Reading cost, chunk cfb3b6b8c710, finished 2026-11-17T08:13:19Z
|
||||
|
||||
182657 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `docs/decisions.md` 13
|
||||
- `README.md` 5
|
||||
- `skills/larv-review/SKILL.md` 4
|
||||
- `CLAUDE.md` 3
|
||||
- `CHANGELOG.md` 1
|
||||
- `bin/loop_claude/cycle/prompts.py` 1
|
||||
- `bin/todo-item` 1
|
||||
- `rules/decisions.md` 1
|
||||
- `todo.md` 1
|
||||
|
||||
Reference in New Issue
Block a user