Replay the archived judge verdicts offline through code rules and a classifier trained on the archive #585
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- judge-replay <model> --offline replays every archived verdict through a "?" rule, a decision table and a classifier trained on the archive, with no judge call, and counts the interrupts each misses.
|
||||
- do-epic finds an epic repo's checkout and project config by the repo's name, and clones a missing checkout from the host and owner of the epic's other repos, asking the human where they differ or the clone fails.
|
||||
- do-epic reads its machine-wide settings from ~/.claude-projects/do-epic/config.toml and its repo rule from repo-rule.md beside it, and refuses to start while either is missing or misspelt.
|
||||
- `project-record save` now writes a goals file kept as a Jira epic's description, except a change to `## Goals` or `## Audience`, which stays an edit in Jira.
|
||||
|
||||
@@ -181,7 +181,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `command-patterns [<version>] [--calls <file>]` prints a table of the shell command patterns a release's archived chunks ran, such as `tea pr create`, for an audit of the release, and with `--calls <file>` saves each call as a JSON line; `command-patterns --help` states the columns and the line's keys.
|
||||
- `compress-archives` gzips each transcript in loop-claude's archive and each file in the review folders 30 days after it last changed, keeping all of them; Install's crontab line runs it daily.
|
||||
- `deploy-release [--cut] [<checkout>]` moves `~/.claude`, or the checkout named, to the newest release, as Update says; with `--cut`, run in the development clone, it first cuts the next release, as Release says.
|
||||
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints what each rung a review climbs to misses, and the lowest rung below them that, like each candidate above it, returns them; `judge-replay --help` states the replay.
|
||||
- `judge-replay <model> [--offline]` replays loop-claude's judge on archived verdicts of the projects run on the model. It prints what each rung a review climbs to misses, and the lowest rung below those that returns every verdict, as each candidate above it does. With `--offline` it doesn't call a judge: it counts what a "?" rule, a decision table and a classifier trained on the archive decide of each verdict's kind from the worker's final message, and how many interrupts each misses. `judge-replay --help` states both replays.
|
||||
- `larv-review <command>` holds larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's round count and the opening of its round commit's title, the scoring run's trigger and setup; the comprehension history's format, its latest places and its count of how often seats agree; and each reviewer's prompt header. `larv-review --help` states them. `settings.json` → `hooks` runs `larv-review reviewer-context` as each subagent starts. A reviewer under `agents/` whose frontmatter has `omitClaudeMd: true` starts without `CLAUDE.md`, and the command hands it that file's "Shared goal", "Prose" and "Vocabulary" sections.
|
||||
- `loop-claude <task>` runs a project's next chunks, each in its own tmux window, cycle after cycle; `loop-claude --help` states its contract.
|
||||
- `markdown-section <file> <fragment>` prints the section of a markdown file under the first heading that holds the fragment, to the next heading of its level or a higher one, instead of a `grep -n` and a guessed `sed -n` range; `markdown-section --help` says what it does where several headings match.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""Replay loop-claude's judge on archived verdicts, to find the lowest rung that returns them: docs/decisions.md → `routing: on a model judge-replay covered`.
|
||||
|
||||
usage: judge-replay <model>
|
||||
usage: judge-replay <model> [--offline]
|
||||
|
||||
<model> is a project model on the routing ladder this setup ships, such as claude-opus-5-5. The tool replays the
|
||||
verdicts archived for the projects that ~/.claude/loop-claude-models.json runs on it: the latest
|
||||
@@ -36,8 +36,21 @@ Like the loop, it tries a judge three times running before it counts the reply a
|
||||
verdict whose project, transcript or archived judge run cannot be read replays as failed too. Exit 1, saying why, on
|
||||
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
|
||||
or a usage limit.
|
||||
|
||||
With --offline it doesn't call a judge, so the runs, retries and usage limit above don't apply. It reads every
|
||||
archived verdict of the model's projects, not a sample. It counts how many of them each of three code
|
||||
deciders decides from the worker's final message, and how many archived interrupts it decides as another
|
||||
kind, which are its missed interrupts: a "?" rule, a decision table, and a classifier trained on the archive
|
||||
(judge_replay/offline.py). The rule and the table run on the whole archive; the table's rows were read from
|
||||
it, so its run isn't a held-out test. The classifier runs on held-out folds, each tested by a model trained on
|
||||
the other folds, and on the latest fifth of the verdicts, tested by a model trained on the earlier four
|
||||
fifths. The classifier trains 330 regressions in Python, about 12 minutes on 1,100 verdicts. The report is
|
||||
kept in replays/judge-replay-<model>-offline-<UTC stamp>/result.md. A verdict whose transcript cannot be read is skipped, and the report counts the skipped. Exit 1, saying why, on bad
|
||||
arguments, an unreadable archive or models file, no archived verdict of the model's projects, or none with a
|
||||
transcript that can be read.
|
||||
"""
|
||||
import pathlib
|
||||
import sys
|
||||
import typing
|
||||
|
||||
from transcript_archive.location import archive_root, transcript_of
|
||||
@@ -49,6 +62,7 @@ from judge_replay.replay import Case, case_from, Kind, report, reproduced, sampl
|
||||
from transcript_archive.project_key import checkout_of
|
||||
from loop_effects.effects import Effects
|
||||
from failure import Failure
|
||||
from judge_replay.offline import final_message, LEARNERS, offline_report, RULES, Sample
|
||||
from rung_replay.search import first_reply, replayed
|
||||
from judge_replay.archived_prompt import grounds_of
|
||||
from transcript_archive.judge_run import JudgeRun
|
||||
@@ -61,7 +75,8 @@ from claude_code.transcript import read_transcript
|
||||
from loop_claude.verdicts import Verdict
|
||||
|
||||
TOOL = "judge-replay"
|
||||
USAGE = f"{TOOL} <model>, a model on the routing ladder"
|
||||
USAGE = f"{TOOL} <model> [--offline]; <model> is a model on the routing ladder"
|
||||
OFFLINE = "--offline"
|
||||
PER_KIND = 15
|
||||
# The effort settings.json gave every judge before loop-claude routed it; the archived verdicts came from it.
|
||||
REFERENCE_EFFORT = "xhigh"
|
||||
@@ -72,7 +87,7 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
ladder = read_ladder(SHIPPED)
|
||||
if isinstance(ladder, Failure):
|
||||
return failed(TOOL, ladder.reason)
|
||||
project = arguments_or_exit(TOOL, argv, __doc__, USAGE, lambda argv: model_argument(ladder, argv))
|
||||
project = arguments_or_exit(TOOL, argv, __doc__, USAGE, lambda argv: model_argument(ladder, [arg for arg in argv if arg != OFFLINE]))
|
||||
if isinstance(project, int):
|
||||
return project
|
||||
cases = archived_cases(project.rung.model, archive_root(home, env), home)
|
||||
@@ -80,6 +95,8 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
return failed(TOOL, cases.reason)
|
||||
if not cases:
|
||||
return failed(TOOL, f"no archived verdicts of projects on {project.rung.model} in {archive_root(home, env)}; run loop-claude on a project on that model first")
|
||||
if OFFLINE in argv:
|
||||
return offline(project.rung.model, cases, archive_root(home, env), effects)
|
||||
cases = sample(cases, PER_KIND)
|
||||
folder = replay_folder(archive_root(home, env), TOOL, argv, effects.now())
|
||||
effects = archiving(effects, folder)
|
||||
@@ -92,6 +109,22 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
return 0
|
||||
|
||||
|
||||
def offline(model: str, cases: list[Case], root: pathlib.Path, effects: Effects) -> int:
|
||||
"""Replays the verdicts of cases offline, oldest first, and keeps the report in the archive as a judge replay does."""
|
||||
samples, skipped = [], 0
|
||||
for found in sorted(cases, key=lambda found: (found.at, found.root, str(found.transcript))):
|
||||
entries = read_transcript(str(found.transcript))
|
||||
if isinstance(entries, Failure):
|
||||
skipped += 1
|
||||
print(f"{TOOL}: skipped a verdict: {entries.reason}", file=sys.stderr)
|
||||
else:
|
||||
samples.append(Sample(found.archived_kind, final_message(entries)))
|
||||
if not samples:
|
||||
return failed(TOOL, "no archived verdict has a transcript that can be read")
|
||||
print_report(replay_folder(root, TOOL, [model, OFFLINE], effects.now()), offline_report(model, samples, RULES, LEARNERS, skipped=skipped))
|
||||
return 0
|
||||
|
||||
|
||||
def model_argument(ladder: Ladder, argv: list[str]) -> ProjectRung | None:
|
||||
"""The project's rung on ladder of the one model argv names; None for any other argv."""
|
||||
return ladder.project_rung(argv[0]) if len(argv) == 1 else None
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
"""Deciders of a verdict's kind that run offline on the worker's final message, replayed on the archive: docs/decisions.md → `routing: the judge's model keeps the verdict kind, …`."""
|
||||
import collections
|
||||
import math
|
||||
import random
|
||||
import re
|
||||
import typing
|
||||
|
||||
from claude_code.entries import AssistantTurn, Entry, Text
|
||||
from transcript_archive.outcomes import FINISHED, INTERRUPTED, UNFINISHED
|
||||
from judge_replay.replay import Kind
|
||||
|
||||
|
||||
class Sample(typing.NamedTuple):
|
||||
"""What a decider reads of one archived verdict: the kind the archive gives it and the worker's final message."""
|
||||
kind: Kind
|
||||
text: str
|
||||
|
||||
|
||||
# A decider names a kind, or None to leave the verdict to the judge's model. A method builds one from the samples it may learn from.
|
||||
Decider = typing.Callable[[str], Kind | None]
|
||||
Method = typing.Callable[[typing.Sequence[Sample]], Decider]
|
||||
Counts = collections.Counter[tuple[Kind, Kind | None]]
|
||||
|
||||
FOLDS = 10
|
||||
INNER_FOLDS = 5
|
||||
# A word in fewer messages than this is left out of the classifier.
|
||||
MIN_MESSAGES = 3
|
||||
EPOCHS = 8
|
||||
# The classifier averages this many regressions, each shuffling the samples in its own order: one shuffle alone moved the missed interrupts of a run between 0 and 2.
|
||||
SEEDS = 5
|
||||
RATE = 0.05
|
||||
DECAY = 1e-3
|
||||
# The latest 1/LATEST of the archive is the second test: the verdicts a decider would meet next.
|
||||
LATEST = 5
|
||||
|
||||
# The first row whose pattern matches the final message decides it; a message no row matches is left to the judge. The rows come from reading the archive's interrupts, so a run of the table on the whole archive is not a held-out test.
|
||||
DECISION_TABLE = (
|
||||
(INTERRUPTED, re.compile(r"(?i)\?|recommend|your call|needs? you|tell me|you answer|please|unmerged|not merged|do you|should i|i need|need your|you say|decisions? for|stays until")),
|
||||
(FINISHED, re.compile(r"(?i)\bmerged\b(?!.*ready to cut)")),
|
||||
)
|
||||
|
||||
|
||||
def final_message(entries: list[Entry]) -> str:
|
||||
"""The text of the worker's last turn that has any, "" where it wrote none."""
|
||||
for entry in reversed(entries):
|
||||
if isinstance(entry, AssistantTurn):
|
||||
text = "\n".join(part.text for part in entry.parts if isinstance(part, Text) and part.text.strip())
|
||||
if text:
|
||||
return text
|
||||
return ""
|
||||
|
||||
|
||||
def question_mark(samples: typing.Sequence[Sample]) -> Decider:
|
||||
"""The one rule: an interrupt where the message holds a "?", finished where it holds none."""
|
||||
return lambda text: INTERRUPTED if "?" in text else FINISHED
|
||||
|
||||
|
||||
def decision_table(samples: typing.Sequence[Sample]) -> Decider:
|
||||
return lambda text: next((kind for kind, pattern in DECISION_TABLE if pattern.search(text)), None)
|
||||
|
||||
|
||||
def words_of(text: str) -> frozenset[str]:
|
||||
"""Each word and each pair of neighbouring words of the message, and its question marks."""
|
||||
words = re.findall(r"[a-z0-9']+|\?", text.lower())
|
||||
return frozenset(words) | {f"{first} {second}" for first, second in zip(words, words[1:])}
|
||||
|
||||
|
||||
def logistic(samples: typing.Sequence[Sample], seed: int) -> typing.Callable[[str], float]:
|
||||
"""The chance that a message is an interrupt, from a logistic regression over its words that the samples train, shuffled in the order seed gives."""
|
||||
seen = collections.Counter(word for found in samples for word in words_of(found.text))
|
||||
vocabulary = {word for word, count in seen.items() if count >= MIN_MESSAGES}
|
||||
data = [(words_of(found.text) & vocabulary, 1.0 if found.kind == INTERRUPTED else 0.0) for found in samples]
|
||||
weights: dict[str, float] = collections.defaultdict(float)
|
||||
bias = 0.0
|
||||
shuffler = random.Random(seed)
|
||||
|
||||
def chance(words: typing.Iterable[str]) -> float:
|
||||
return 1 / (1 + math.exp(-max(-30.0, min(30.0, bias + sum(weights[word] for word in words)))))
|
||||
|
||||
for _ in range(EPOCHS):
|
||||
shuffler.shuffle(data)
|
||||
for words, interrupt in data:
|
||||
error = chance(words) - interrupt
|
||||
bias -= RATE * error
|
||||
for word in words:
|
||||
weights[word] -= RATE * (error + DECAY * weights[word])
|
||||
return lambda text: chance(words_of(text) & vocabulary)
|
||||
|
||||
|
||||
def averaged(samples: typing.Sequence[Sample]) -> typing.Callable[[str], float]:
|
||||
"""The mean chance of SEEDS regressions that differ only in the order they shuffle the samples."""
|
||||
regressions = [logistic(samples, seed) for seed in range(SEEDS)]
|
||||
return lambda text: sum(regression(text) for regression in regressions) / SEEDS
|
||||
|
||||
|
||||
def classifier(samples: typing.Sequence[Sample]) -> Decider:
|
||||
"""Finished where the chance of an interrupt is below one half and below that of every interrupt among the samples; interrupt where it is at least one half and above that of every other kind; otherwise the judge's model decides.
|
||||
|
||||
A sample's chance is the one a model trained on the other folds of the samples gives it, so the bounds don't rest on how well the model fits what it trained on. A bound is where the samples held no miss; that doesn't promise the next verdict holds none."""
|
||||
interrupts = [found for found in samples if found.kind == INTERRUPTED]
|
||||
if not interrupts or len(interrupts) == len(samples):
|
||||
return lambda text: None
|
||||
held_out = [(tested.kind, chance) for trained, tests in folds_of(samples, INNER_FOLDS) for chance, tested in zip(map(averaged(trained), (found.text for found in tests)), tests)]
|
||||
below = min(chance for kind, chance in held_out if kind == INTERRUPTED)
|
||||
above = max(chance for kind, chance in held_out if kind != INTERRUPTED)
|
||||
chance = averaged(samples)
|
||||
|
||||
def decide(text: str) -> Kind | None:
|
||||
found = chance(text)
|
||||
if found < below and found < 0.5:
|
||||
return FINISHED
|
||||
return INTERRUPTED if found > above and found >= 0.5 else None
|
||||
return decide
|
||||
|
||||
|
||||
# Rules are fixed, and run on the whole archive; a learner trains on samples, and runs on samples it didn't train on.
|
||||
RULES: dict[str, Method] = {'"?" rule': question_mark, "decision table": decision_table}
|
||||
LEARNERS: dict[str, Method] = {"classifier": classifier}
|
||||
|
||||
|
||||
def folds_of(samples: typing.Sequence[Sample], count: int) -> list[tuple[list[Sample], list[Sample]]]:
|
||||
"""count folds of the samples, dealt in turn. Each fold is the samples to learn from, which are all those its test leaves out, and the samples to test on."""
|
||||
return [([found for place, found in enumerate(samples) if place % count != fold], [found for place, found in enumerate(samples) if place % count == fold]) for fold in range(count)]
|
||||
|
||||
|
||||
def evaluate(method: Method, trained: typing.Sequence[Sample], tested: typing.Sequence[Sample]) -> Counts:
|
||||
"""How often the method, learning from trained, decided each kind for each archived kind of tested; None where it left the verdict to the judge."""
|
||||
decide = method(trained)
|
||||
return collections.Counter((found.kind, decide(found.text)) for found in tested)
|
||||
|
||||
|
||||
def plain(kind: Kind) -> str:
|
||||
return kind.removeprefix("Verdict: ")
|
||||
|
||||
|
||||
def offline_report(model: str, samples: typing.Sequence[Sample], rules: typing.Mapping[str, Method], learners: typing.Mapping[str, Method], folds: int = FOLDS, skipped: int = 0) -> str:
|
||||
"""samples are in the order the verdicts were judged, oldest first, so the latest fifth is the newest verdicts."""
|
||||
kinds = collections.Counter(found.kind for found in samples)
|
||||
lines = [f"Replayed {len(samples)} archived verdicts of {model} projects offline, without sending anything to an outside service. The archive gives them as {', '.join(f'{count} {plain(kind)}' for kind, count in sorted(kinds.items(), key=lambda each: plain(each[0])))}."]
|
||||
if skipped:
|
||||
lines.append(f"Skipped {skipped} verdicts whose transcript could not be read.")
|
||||
for name, rule in rules.items():
|
||||
found = evaluate(rule, [], samples)
|
||||
lines += [result_line(f"{name}, on the whole archive", found), *decides_lines(f"{name}, on the whole archive", found)]
|
||||
split = len(samples) - max(1, len(samples) // LATEST)
|
||||
for name, learner in learners.items():
|
||||
held_out = sum((evaluate(learner, trained, tested) for trained, tested in folds_of(samples, folds)), Counts())
|
||||
latest = evaluate(learner, samples[:split], samples[split:])
|
||||
lines += [result_line(f"{name}, on held-out folds", held_out), result_line(f"{name}, on the latest fifth after learning from the rest", latest), *decides_lines(f"{name}, on both tests", held_out + latest)]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def result_line(how: str, found: Counts) -> str:
|
||||
decided = collections.Counter({kind: sum(count for (_, said), count in found.items() if said == kind) for kind in {said for _, said in found if said}})
|
||||
missed = sum(count for (archived, said), count in found.items() if archived == INTERRUPTED and said not in (None, INTERRUPTED))
|
||||
parts = [f"{plain(kind)} {count}{wrong_kinds(kind, found)}" for kind, count in sorted(decided.items(), key=lambda each: plain(each[0]))]
|
||||
return f"{how}: decided {sum(decided.values())} of {sum(found.values())}{': ' + ', '.join(parts) if parts else ''}. Missed interrupts: {missed}."
|
||||
|
||||
|
||||
def wrong_kinds(decided: Kind, found: Counts) -> str:
|
||||
"""The kinds the archive gives the verdicts that were decided as decided, where the archive gives another kind than decided."""
|
||||
wrong = sorted(((plain(archived), count) for (archived, said), count in found.items() if said == decided and archived != decided), key=lambda each: each[0])
|
||||
return f" (of these, archived {', '.join(f'{kind} {count}' for kind, count in wrong)})" if wrong else ""
|
||||
|
||||
|
||||
def decides_lines(how: str, found: Counts) -> list[str]:
|
||||
"""For finished and for unfinished that the decider decided at all: whether it decided no archived interrupt as that kind."""
|
||||
lines = []
|
||||
for kind in (FINISHED, UNFINISHED):
|
||||
decided = sum(count for (_, said), count in found.items() if said == kind)
|
||||
if decided and not found[(INTERRUPTED, kind)]:
|
||||
lines.append(f"{how}, decides {plain(kind)} with no missed interrupt.")
|
||||
elif decided:
|
||||
lines.append(f"{how}, does not decide {plain(kind)} with no missed interrupt: it decided {found[(INTERRUPTED, kind)]} archived interrupt(s) as {plain(kind)}.")
|
||||
return lines
|
||||
@@ -175,6 +175,44 @@ class Main(unittest.TestCase):
|
||||
# The replay shows the judge the whole transcript, with no step-up marked in it.
|
||||
self.assertNotIn("[step]", self.prompts[0])
|
||||
|
||||
def test_offline_replays_every_verdict_through_the_code_deciders_without_calling_the_judge(self):
|
||||
code, out, err = self.run_main(["claude-opus-5-5", "--offline"], unused)
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual(self.calls, [])
|
||||
self.assertIn("Replayed 2 archived verdicts of claude-opus-5-5 projects offline, without sending anything to an outside service. The archive gives them as 1 finished, 1 interrupt.", out)
|
||||
self.assertIn("\"?\" rule, on the whole archive: decided 2 of 2: finished 2 (of these, archived interrupt 1). Missed interrupts: 1.", out)
|
||||
folder = self.home / ".claude" / "loop-claude-transcripts" / "replays" / "judge-replay-claude-opus-5-5-offline-20260921T141320Z"
|
||||
self.assertEqual((folder / "result.md").read_text(encoding="utf-8"), out)
|
||||
|
||||
def test_offline_exits_1_where_no_transcript_can_be_read(self):
|
||||
for path in (self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1").glob("*.jsonl"):
|
||||
path.unlink()
|
||||
code, out, err = self.run_main(["claude-opus-5-5", "--offline"], unused)
|
||||
self.assertEqual((code, out), (1, ""))
|
||||
self.assertEqual(err.splitlines()[-1], "judge-replay: no archived verdict has a transcript that can be read")
|
||||
|
||||
def test_offline_skips_each_verdict_whose_transcript_cannot_be_read_and_says_how_many_on_stderr_and_in_the_report(self):
|
||||
kept = self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1"
|
||||
(kept / "t2.outcomes.md").write_text(f"Task: Do it.\n\n2026-10-14T10:00:02Z {FINISHED}\n\nEvidence:\n> e\n")
|
||||
(kept / "t0.jsonl").unlink()
|
||||
code, out, err = self.run_main(["claude-opus-5-5", "--offline"], unused)
|
||||
self.assertEqual(code, 0)
|
||||
self.assertEqual(len(err.splitlines()), 2)
|
||||
self.assertTrue(all(line.startswith("judge-replay: skipped a verdict: could not read the transcript ") for line in err.splitlines()))
|
||||
self.assertIn("Replayed 1 archived verdicts of claude-opus-5-5 projects offline", out)
|
||||
self.assertIn("Skipped 2 verdicts whose transcript could not be read.", out)
|
||||
|
||||
def test_offline_keeps_its_report_in_the_same_folder_wherever_the_flag_stands(self):
|
||||
code, out, err = self.run_main(["--offline", "claude-opus-5-5"], unused)
|
||||
self.assertEqual(code, 0, err)
|
||||
folder = self.home / ".claude" / "loop-claude-transcripts" / "replays" / "judge-replay-claude-opus-5-5-offline-20260921T141320Z"
|
||||
self.assertEqual((folder / "result.md").read_text(encoding="utf-8"), out)
|
||||
|
||||
def test_offline_says_the_usage_on_arguments_that_name_no_model(self):
|
||||
code, out, err = self.run_main(["--offline"], unused)
|
||||
self.assertEqual((code, out), (1, ""))
|
||||
self.assertEqual(err, "judge-replay: cannot read the arguments --offline; usage: judge-replay <model> [--offline]; <model> is a model on the routing ladder\n")
|
||||
|
||||
def test_keeps_each_run_and_the_printed_report_in_one_folder_of_the_archive_beside_the_projects(self):
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
self.assertEqual(code, 0, err)
|
||||
@@ -272,7 +310,7 @@ class Main(unittest.TestCase):
|
||||
|
||||
def test_prints_the_help(self):
|
||||
code, out, _ = self.run_main(["-h"], unused)
|
||||
self.assertEqual((code, out.splitlines()[2]), (0, "usage: judge-replay <model>"))
|
||||
self.assertEqual((code, out.splitlines()[2]), (0, "usage: judge-replay <model> [--offline]"))
|
||||
|
||||
def test_the_launcher_prints_the_help(self):
|
||||
ran = subprocess.run([sys.executable, str(TOOL), "--help"], capture_output=True, text=True)
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
import collections
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINISHED
|
||||
from claude_code.entries import AssistantTurn, HumanTurn, Text, ToolCall
|
||||
from judge_replay.offline import averaged, classifier, decision_table, evaluate, final_message, folds_of, LEARNERS, logistic, offline_report, question_mark, RULES, Sample, words_of
|
||||
|
||||
|
||||
def turn(*parts):
|
||||
return AssistantTurn(parts=parts)
|
||||
|
||||
|
||||
def sample(kind, text):
|
||||
return Sample(kind, text)
|
||||
|
||||
|
||||
class FinalMessage(unittest.TestCase):
|
||||
def test_is_the_text_of_the_workers_last_turn_that_has_any(self):
|
||||
entries = [turn(Text("first")), HumanTurn(text="ok"), turn(Text("Done. Merged."), ToolCall("1", "Bash", {})), turn(ToolCall("2", "Bash", {}))]
|
||||
self.assertEqual(final_message(entries), "Done. Merged.")
|
||||
|
||||
def test_joins_the_texts_of_one_turn_on_separate_lines_and_leaves_out_blank_ones(self):
|
||||
self.assertEqual(final_message([turn(Text("Merged."), Text(" "), Text("Which one?"))]), "Merged.\nWhich one?")
|
||||
|
||||
def test_is_empty_where_the_worker_wrote_no_text(self):
|
||||
self.assertEqual(final_message([HumanTurn(text="hi"), turn(ToolCall("1", "Bash", {}))]), "")
|
||||
|
||||
|
||||
class WordsOf(unittest.TestCase):
|
||||
def test_is_each_lowercase_word_each_pair_of_neighbours_and_each_question_mark(self):
|
||||
self.assertEqual(words_of("Which one? Yes"), {"which", "one", "?", "yes", "which one", "one ?", "? yes"})
|
||||
|
||||
|
||||
class QuestionMark(unittest.TestCase):
|
||||
def test_decides_interrupt_on_a_question_mark_and_finished_on_none(self):
|
||||
decide = question_mark([])
|
||||
self.assertEqual(decide("Which one?"), INTERRUPTED)
|
||||
self.assertEqual(decide("Merged."), FINISHED)
|
||||
|
||||
|
||||
class DecisionTable(unittest.TestCase):
|
||||
def test_reads_the_workers_asking_for_the_humans_answer_as_an_interrupt(self):
|
||||
decide = decision_table([])
|
||||
for text in ("My recommendation is option 1.", "Your call: merge now or wait.", "The PR stays unmerged.", "Tell me which."):
|
||||
with self.subTest(text=text):
|
||||
self.assertEqual(decide(text), INTERRUPTED)
|
||||
|
||||
def test_reads_a_merge_with_no_ask_and_no_release_to_cut_as_finished(self):
|
||||
decide = decision_table([])
|
||||
self.assertEqual(decide("PR 12 merged and the worktree is removed."), FINISHED)
|
||||
self.assertIsNone(decide("PR 12 merged. Release 0.3.0 is ready to cut."))
|
||||
|
||||
def test_leaves_any_other_message_to_the_judge(self):
|
||||
self.assertIsNone(decision_table([])("I stopped at the gate."))
|
||||
|
||||
|
||||
class Logistic(unittest.TestCase):
|
||||
def test_gives_a_message_with_no_learned_word_about_the_share_of_interrupts(self):
|
||||
chance = logistic([sample(INTERRUPTED, "")] * 10 + [sample(FINISHED, "")] * 30, 0)
|
||||
self.assertAlmostEqual(chance(""), 0.25, delta=0.05)
|
||||
|
||||
def test_learns_a_word_found_in_as_few_messages_as_the_minimum(self):
|
||||
chance = logistic([sample(INTERRUPTED, "ask")] * 3 + [sample(FINISHED, "")] * 3, 0)
|
||||
self.assertGreater(chance("ask"), chance(""))
|
||||
|
||||
def test_averaged_is_the_mean_of_the_regressions_that_shuffle_in_each_seed_order(self):
|
||||
samples = [sample(INTERRUPTED if n % 3 else FINISHED, f"word{n % 4} ask" if n % 3 else f"word{n % 4}") for n in range(30)]
|
||||
mean = sum(logistic(samples, seed)("ask") for seed in range(5)) / 5
|
||||
self.assertAlmostEqual(averaged(samples)("ask"), mean)
|
||||
|
||||
|
||||
class Classifier(unittest.TestCase):
|
||||
def trained(self, extra=()):
|
||||
asking = [sample(INTERRUPTED, f"I recommend option {n}; tell me which of the options you want") for n in range(12)]
|
||||
done = [sample(FINISHED, f"merged pull request {n} and removed the worktree, no findings left") for n in range(12)]
|
||||
return classifier([*asking, *done, *extra])
|
||||
|
||||
def test_decides_a_message_like_each_kind_it_learned(self):
|
||||
decide = self.trained()
|
||||
self.assertEqual(decide("I recommend option 3; tell me which you want"), INTERRUPTED)
|
||||
self.assertEqual(decide("merged pull request 99 and removed the worktree, no findings left"), FINISHED)
|
||||
|
||||
def test_decides_finished_on_a_message_of_words_only_finished_verdicts_held(self):
|
||||
self.assertEqual(self.trained()("merged pull request"), FINISHED)
|
||||
|
||||
def test_abstains_where_it_learned_no_interrupt_or_no_other_kind(self):
|
||||
self.assertIsNone(classifier([sample(FINISHED, "merged")])("merged"))
|
||||
self.assertIsNone(classifier([sample(INTERRUPTED, "which?")])("which?"))
|
||||
self.assertIsNone(classifier([])("anything"))
|
||||
|
||||
def test_decides_no_finished_where_an_interrupt_it_learned_looks_like_one(self):
|
||||
lookalike = "merged pull request 7 and removed the worktree, no findings left"
|
||||
decide = self.trained([sample(INTERRUPTED, lookalike)])
|
||||
self.assertNotEqual(decide(lookalike), FINISHED)
|
||||
|
||||
|
||||
def test_abstains_where_a_lookalike_of_each_kind_sits_in_the_samples_of_the_other(self):
|
||||
decide = self.trained([sample(INTERRUPTED, "merged pull request 7 and removed the worktree, no findings left"), sample(FINISHED, "I recommend option 7; tell me which of the options you want")])
|
||||
for text in ("merged pull request", "I recommend option", "zzz qqq"):
|
||||
with self.subTest(text=text):
|
||||
self.assertIsNone(decide(text))
|
||||
|
||||
|
||||
class Folds(unittest.TestCase):
|
||||
def test_deals_samples_to_folds_in_turn_so_each_is_tested_once(self):
|
||||
samples = [sample(FINISHED, str(n)) for n in range(7)]
|
||||
dealt = folds_of(samples, 3)
|
||||
self.assertEqual([[found.text for found in tested] for _, tested in dealt], [["0", "3", "6"], ["1", "4"], ["2", "5"]])
|
||||
self.assertEqual([found.text for found in dealt[1][0]], ["0", "2", "3", "5", "6"])
|
||||
|
||||
|
||||
class Evaluate(unittest.TestCase):
|
||||
def test_counts_each_archived_kind_by_what_the_method_decided_or_abstained(self):
|
||||
tested = [sample(FINISHED, "merged"), sample(INTERRUPTED, "merged"), sample(UNFINISHED, "stopped"), sample(ANSWERED, "Which?")]
|
||||
found = evaluate(decision_table, [], tested)
|
||||
self.assertEqual(found, collections.Counter({(FINISHED, FINISHED): 1, (INTERRUPTED, FINISHED): 1, (UNFINISHED, None): 1, (ANSWERED, INTERRUPTED): 1}))
|
||||
|
||||
|
||||
class Report(unittest.TestCase):
|
||||
def samples(self):
|
||||
return [sample(FINISHED, "Merged."), sample(FINISHED, "Merged. Release 1.0.0 is ready to cut."), sample(INTERRUPTED, "I recommend merging."), sample(INTERRUPTED, "Merged.")]
|
||||
|
||||
def test_counts_what_a_rule_decided_on_the_whole_archive_and_missed_by_kind(self):
|
||||
text = offline_report("claude-opus-5-5", self.samples(), {"decision table": decision_table}, {})
|
||||
self.assertIn("Replayed 4 archived verdicts of claude-opus-5-5 projects offline, without sending anything to an outside service. The archive gives them as 2 finished, 2 interrupt.", text)
|
||||
self.assertIn("decision table, on the whole archive: decided 3 of 4: finished 2 (of these, archived interrupt 1), interrupt 1. Missed interrupts: 1.", text)
|
||||
self.assertIn("decision table, on the whole archive, does not decide finished with no missed interrupt: it decided 1 archived interrupt(s) as finished.", text)
|
||||
|
||||
def test_says_a_method_that_decided_a_kind_and_missed_no_interrupt_decides_that_kind(self):
|
||||
text = offline_report("claude-opus-5-5", [sample(FINISHED, "Merged."), sample(INTERRUPTED, "I recommend merging.")], {"decision table": decision_table}, {})
|
||||
self.assertIn("Missed interrupts: 0.", text)
|
||||
self.assertIn("decision table, on the whole archive, decides finished with no missed interrupt.", text)
|
||||
|
||||
def test_runs_a_learner_once_for_each_fold_and_once_on_the_latest_fifth_and_a_rule_only_on_the_whole_archive(self):
|
||||
samples = [sample(INTERRUPTED if n % 2 else FINISHED, "I recommend it" if n % 2 else "merged it") for n in range(20)]
|
||||
text = offline_report("m", samples, {"rule": question_mark}, {"classifier": classifier}, folds=4)
|
||||
self.assertIn("classifier, on held-out folds: decided", text)
|
||||
self.assertIn("classifier, on the latest fifth after learning from the rest: decided", text)
|
||||
self.assertIn("rule, on the whole archive: decided 20 of 20", text)
|
||||
self.assertNotIn("rule, on held-out", text)
|
||||
self.assertNotIn("rule, on the latest fifth", text)
|
||||
|
||||
def test_words_the_whole_report_line_by_line_with_the_wrong_kinds_by_name(self):
|
||||
answers = {"f": FINISHED, "i1": FINISHED, "i2": FINISHED, "u": FINISHED, "q": None}
|
||||
samples = [sample(FINISHED, "f"), sample(INTERRUPTED, "i1"), sample(INTERRUPTED, "i2"), sample(UNFINISHED, "u"), sample(ANSWERED, "q")]
|
||||
text = offline_report("m", samples, {"rule": lambda trained: lambda said: answers[said]}, {})
|
||||
self.assertEqual(text.splitlines(), [
|
||||
"Replayed 5 archived verdicts of m projects offline, without sending anything to an outside service. The archive gives them as 1 finished, 2 interrupt, 1 quoted answer, 1 unfinished.",
|
||||
"rule, on the whole archive: decided 4 of 5: finished 4 (of these, archived interrupt 2, unfinished 1). Missed interrupts: 2.",
|
||||
"rule, on the whole archive, does not decide finished with no missed interrupt: it decided 2 archived interrupt(s) as finished.",
|
||||
])
|
||||
|
||||
def test_says_a_method_that_decided_nothing_decided_0_and_names_no_kind(self):
|
||||
text = offline_report("m", self.samples(), {"rule": lambda trained: lambda said: None}, {})
|
||||
self.assertEqual(text.splitlines()[1:], ["rule, on the whole archive: decided 0 of 4. Missed interrupts: 0."])
|
||||
|
||||
def test_tests_the_latest_fifth_of_the_samples_after_learning_from_the_earlier_ones(self):
|
||||
trained = []
|
||||
|
||||
def learner(samples):
|
||||
trained.append(len(samples))
|
||||
return lambda said: FINISHED
|
||||
|
||||
text = offline_report("m", [sample(FINISHED, str(n)) for n in range(7)], {}, {"learner": learner}, folds=2)
|
||||
self.assertEqual(trained, [3, 4, 6])
|
||||
self.assertEqual(text.splitlines()[-2:], ["learner, on the latest fifth after learning from the rest: decided 1 of 1: finished 1. Missed interrupts: 0.", "learner, on both tests, decides finished with no missed interrupt."])
|
||||
|
||||
def test_says_a_learner_that_missed_an_interrupt_on_either_test_does_not_decide_the_kind(self):
|
||||
def learner(samples):
|
||||
return lambda said: FINISHED if said == "miss" else None
|
||||
|
||||
samples = [sample(FINISHED, "ok")] * 4 + [sample(INTERRUPTED, "miss")]
|
||||
text = offline_report("m", samples, {}, {"learner": learner}, folds=2)
|
||||
self.assertEqual(text.splitlines()[-1], "learner, on both tests, does not decide finished with no missed interrupt: it decided 2 archived interrupt(s) as finished.")
|
||||
|
||||
def test_counts_the_verdicts_it_skipped_only_where_it_skipped_some(self):
|
||||
self.assertNotIn("Skipped", offline_report("m", self.samples(), {}, {}))
|
||||
self.assertIn("Skipped 2 verdicts whose transcript could not be read.", offline_report("m", self.samples(), {}, {}, skipped=2))
|
||||
|
||||
def test_the_shipped_methods_are_the_question_mark_rule_the_table_and_the_classifier(self):
|
||||
self.assertEqual(list(RULES), ['"?" rule', "decision table"])
|
||||
self.assertEqual(list(LEARNERS), ["classifier"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -674,6 +674,22 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
|
||||
|
||||
Valid while the index ranks these models, and while no newer model joins the catalog.
|
||||
|
||||
## routing: the judge's model keeps the verdict kind until lilleman rules on the offline classifier, which decided finished with no missed interrupt on 1,116 archived verdicts but missed one on the first 1,104
|
||||
|
||||
2026-11-18, claude. Serves goals 1.2 and 1.4, and applies project principle 1: a cost cut waits for a measurement that shows quality holds.
|
||||
|
||||
`bin/judge-replay claude-opus-5-5 --offline` replays all archived verdicts of the Opus 5.5 projects through three deciders. Each reads only the worker's final message, doesn't send anything to an outside service, and decides a kind or leaves the verdict to the judge's model. A missed interrupt is an archived interrupt decided as another kind, which lets the worker settle a decision that is lilleman's. Deciding interrupt can't miss one: a wrong interrupt only asks lilleman more. On 2026-11-17 the archive held 1,104 verdicts (660 finished, 333 interrupt, 33 quoted answer, 78 unfinished); on 2026-11-18 it held 1,116 (668, 336, 34, 78).
|
||||
|
||||
- **A "?" rule** (interrupt where the message holds a "?", finished where it holds none) decided every one of the 1,104 and missed 197 interrupts.
|
||||
- **A decision table**, the rows of `DECISION_TABLE` in `bin/judge_replay/offline.py`. Its rows were read from this archive's interrupts, so its run isn't a held-out test. It decided 974 of the 1,104 and missed 55. Most of its misses are a worker reporting a merged chunk while a release is ready to cut, which depends on the plan; the rest ask for an answer in words no row names.
|
||||
- **A classifier**, a logistic regression over the message's words, trained on the archive. It decides finished only where its chance of an interrupt is below that of every interrupt among its training folds when held out, and interrupt only where it is above that of every other kind. It averages five regressions that differ only in the order they shuffle the samples, since a single shuffle moved its missed interrupts between 0 and 2 on the same archive. Its bar is no missed interrupt on both tests: held-out folds, each tested by a model trained on the other folds, and the latest fifth of the verdicts, learning from the rest.
|
||||
- On the 1,104 verdicts, held-out folds: 103 decided, none missed. Latest fifth: 29 decided, all as finished, and 1 interrupt missed. That fails the bar, and shuffles 5 to 9 and 10 to 14 failed it alike.
|
||||
- On the 1,116 verdicts, held-out folds: 95 decided (84 finished, 11 interrupt), none missed. Latest fifth: 20 decided, all as finished, none missed. That meets the bar for finished.
|
||||
|
||||
The classifier decides about 1 in 12 verdicts, and about 1 in 100 of what it decides were an interrupt on the 1,104. A latest fifth that starts a few verdicts later no longer holds the one it missed, so the bar flips with the archive's size and doesn't separate a classifier that never misses from one that rarely does. The judge's model keeps the verdict kind, the escalation it quotes and whether a release is ready to cut until lilleman rules whether a miss rate of about 1 in 100 decided is close enough to none. The plan holds that question as an item marked `question`.
|
||||
|
||||
Valid while the archived verdicts hold interrupts that read like a finished chunk. Replay with `judge-replay <model> --offline` when the archive has doubled, or when a decider reads more than the final message.
|
||||
|
||||
## routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds
|
||||
|
||||
2026-10-14, lilleman. Serves goal 1.4.
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 423`
|
||||
`Next ID = 424`
|
||||
|
||||
## Items
|
||||
|
||||
@@ -28,11 +28,11 @@
|
||||
| 414 | 0.12.0 | 410 | | **DM the human through the notifier when a human-task issue blocks the epic's next item, and poll until it closes.** | 2 | 2 | 2 | 4 | 3 | 1 by 5 | 1.9 |
|
||||
| 417 | 0.12.0 | 410 | | **Run do-epic end to end against a sandbox epic on a real Jira site with a command of its own, outside `run-tests`.** | 2 | 1 | 1 | 2 | 3 | 1 by 5 | 1.9 |
|
||||
| 421 | 0.12.0 | | | **Refuse in rule-hook a Bash or Monitor wait that has no end: a loop without a time or count cap, and a `pgrep -f` or `pkill -f` whose pattern the same command line contains.** | 2 | 2 | 3 | 5 | 3 | 1.5 by 4 | 1.3 |
|
||||
| 299 | 0.12.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
|
||||
| 311 | 0.12.0 | | | **Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.0 |
|
||||
| 316 | 0.12.0 | | | **Launch comprehension seats as an agent type of their own under `agents/`, without `CLAUDE.md` and the skill and agent listings a general-purpose subagent carries.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
|
||||
| 310 | 0.12.0 | | | **Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 0.9 |
|
||||
| 422 | 0.12.0 | 401, 420 | | **Check every 5 minutes whether a tmux window of the loop looks stuck by other signs than a lingering worker, investigate it with a headless claude, and kill only the process the investigation names.** | 3 | 2 | 3 | 6 | 5 | 1.5 by 4 | 0.8 |
|
||||
| 423 | 0.13.0 | | question | **Skip the judge's model call on a worker's final message that the offline classifier decides is finished, once lilleman rules that its miss rate of about 1 in 100 decided is close enough to no missed interrupt.** | 3 | 3 | 3 | 7 | 5 | 1.4 by 2 | 0.4 |
|
||||
| 365 | 0.13.0 | | | **Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
|
||||
| 368 | 0.13.0 | | | **Call a release review due, call a release ready to cut, and tell whether a chunk owes a scoring run, for a plan outside the checkout, from the project repo's history or from Jira.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
|
||||
| 309 | 0.13.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.2 |
|
||||
@@ -278,15 +278,6 @@ Touches:
|
||||
- `bin/rule_hook/test_rule_hook.py`: the new rule's tests (small)
|
||||
- `settings.json`: the PreToolUse matcher (small)
|
||||
|
||||
### 299. Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.
|
||||
|
||||
The judge's LLM still decides three things: the verdict kind, the escalation it quotes, and whether a release is ready to cut (`docs/decisions.md` → `routing: loop-claude reads the items a chunk takes …`). This item replays the verdict kind. No single rule found so far separates interrupts: a "?" in the worker's final message appears in 48 interrupts and misses 66. A smaller LLM already failed the replay (`docs/decisions.md` → `routing: on a model judge-replay covered, …`: Sonnet returned another kind on 3 interrupts, Haiku missed 7). Untested are combined code rules, a decision table, and a small classifier trained and run on this machine on the archived verdicts (372 finished, 234 interrupt, 42 unfinished, 20 quoted). Run each offline, nothing sent to an outside service (goal 4), and count disagreements by kind; the bar is zero missed interrupts. A hosted decision model is out of scope, since it would send transcripts to an outside service.
|
||||
|
||||
Touches:
|
||||
- `bin/judge_replay/main.py`: `main`, `archived_cases` (small)
|
||||
- `bin/judge_replay/replay.py`: `Case`, `verdict_kind`, `report`
|
||||
- `bin/judge_replay/test_judge_replay.py`
|
||||
|
||||
### 311. Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.
|
||||
|
||||
Workers took 7,870 Bash results over 3,000 characters, $305 of carried context, against the reading rule in their prompt; stability reviewers another $145. `BASH_MAX_OUTPUT_LENGTH` (or `bashOutputMaxChars`) makes Claude Code save a longer result to a file and hand the model its path and the first 2,000 characters. loop-claude can hand it as it hands `CLAUDE_CODE_AUTO_COMPACT_WINDOW`. A ceiling of 6,000 characters would have removed $228 of carried output, 10,000 characters $122. Reads of old session transcripts cost $46; a rule-hook rule can refuse them, but not a Read of the files Claude Code saves a long Bash result to: they sit under `~/.claude/projects/<project>/<session>/tool-results/`, the archive holds 85 of them (item 312's round, 2026-11-12), and the saved-to-file ceiling hands the model their path to read.
|
||||
@@ -349,6 +340,21 @@ Touches:
|
||||
- `bin/loop_claude/tmux.py`: `Listed`, `listed`
|
||||
- `README.md`: loop-claude's section (small)
|
||||
|
||||
### 423. Skip the judge's model call on a worker's final message that the offline classifier decides is finished, once lilleman rules that its miss rate of about 1 in 100 decided is close enough to no missed interrupt.
|
||||
|
||||
Question for lilleman: is a miss rate of about 1 in 100 decided close enough to none? A missed interrupt lets a worker settle a decision that is yours, unasked. Until you rule, the judge's model keeps the verdict kind (`docs/decisions.md` → `routing: the judge's model keeps the verdict kind until lilleman rules on the offline classifier …`).
|
||||
|
||||
Taking the item builds this: before the judge call, loop-claude runs the classifier trained on its archive over the worker's final message. Where the classifier decides finished, loop-claude skips the call. The judge's model keeps every verdict the classifier leaves, and the escalation it quotes and whether a release is ready to cut. The classifier is re-trained as the archive grows. The judge call sits in `bin/loop_claude/cycle/run.py` → `rule`, a place the comprehension seats found hard, since the order of stamping, archiving and stepping matters across three functions: make it easier to read.
|
||||
|
||||
Evidence, from `judge-replay claude-opus-5-5 --offline`: the classifier reads only the final message, averages five seeded logistic regressions, and decides finished only where its chance of an interrupt is below that of every interrupt among its held-out training folds.
|
||||
- On the 1,116 archived verdicts it decided finished with no missed interrupt on both tests: held-out folds, 95 decided and none missed; latest fifth, 20 decided and none missed.
|
||||
- On the first 1,104 it missed one interrupt on the latest fifth, while the held-out folds decided 103 and missed none.
|
||||
- It decides about 1 in 12 verdicts. About 1 in 100 of those it decided were an interrupt, on the 1,104.
|
||||
|
||||
Savings: one judge call per about 12 verdicts. That is small, and nobody has measured it.
|
||||
|
||||
Fallbacks: decline, and close this item. Or first tighten the bar in `judge-replay`, so it separates a classifier that never misses from one that rarely does, and replay again.
|
||||
|
||||
### 365. Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.
|
||||
|
||||
lilleman's call, 2026-11-12: 0.10.0 and 0.11.0 switch the history tools off for a plan outside the checkout, and 0.13.0 holds only their ports. In Jira a dropped item, which `Drop item` commit titles mark today, is an issue resolved by a resolution the config names.
|
||||
|
||||
Reference in New Issue
Block a user