Merge remote-tracking branch 'origin/main' into ci-cancel-superseded
CI / tests (pull_request) Successful in 4m2s

This commit was merged in pull request #601.
This commit is contained in:
2026-11-18 14:24:24 +01:00
15 changed files with 868 additions and 34 deletions
+2
View File
@@ -2,6 +2,8 @@
## Unreleased
- judge-replay <model> --offline replays every archived verdict through a "?" rule, a decision table and a classifier trained on the archive, with no judge call, and counts the interrupts each misses.
- do-epic finds an epic repo's checkout and project config by the repo's name, and clones a missing checkout from the host and owner of the epic's other repos, asking the human where they differ or the clone fails.
- do-epic reads its machine-wide settings from ~/.claude-projects/do-epic/config.toml and its repo rule from repo-rule.md beside it, and refuses to start while either is missing or misspelt.
- `project-record save` now writes a goals file kept as a Jira epic's description, except a change to `## Goals` or `## Audience`, which stays an edit in Jira.
- Add do-epic's epic backend: one interface that reads a Jira epic's child issues and their sub-tasks, with labels, assignee, status and parent, and files new ones, met by a Jira backend and an in-memory fake that pass the same contract tests.
+3 -3
View File
@@ -181,7 +181,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `command-patterns [<version>] [--calls <file>]` prints a table of the shell command patterns a release's archived chunks ran, such as `tea pr create`, for an audit of the release, and with `--calls <file>` saves each call as a JSON line; `command-patterns --help` states the columns and the line's keys.
- `compress-archives` gzips each transcript in loop-claude's archive and each file in the review folders 30 days after it last changed, keeping all of them; Install's crontab line runs it daily.
- `deploy-release [--cut] [<checkout>]` moves `~/.claude`, or the checkout named, to the newest release, as Update says; with `--cut`, run in the development clone, it first cuts the next release, as Release says.
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints what each rung a review climbs to misses, and the lowest rung below them that, like each candidate above it, returns them; `judge-replay --help` states the replay.
- `judge-replay <model> [--offline]` replays loop-claude's judge on archived verdicts of the projects run on the model. It prints what each rung a review climbs to misses, and the lowest rung below those that returns every verdict, as each candidate above it does. With `--offline` it doesn't call a judge: it counts what a "?" rule, a decision table and a classifier trained on the archive decide of each verdict's kind from the worker's final message, and how many interrupts each misses. `judge-replay --help` states both replays.
- `larv-review <command>` holds larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's round count and the opening of its round commit's title, the scoring run's trigger and setup; the comprehension history's format, its latest places and its count of how often seats agree; and each reviewer's prompt header. `larv-review --help` states them. `settings.json` → `hooks` runs `larv-review reviewer-context` as each subagent starts. A reviewer under `agents/` whose frontmatter has `omitClaudeMd: true` starts without `CLAUDE.md`, and the command hands it that file's "Shared goal", "Prose" and "Vocabulary" sections.
- `loop-claude <task>` runs a project's next chunks, each in its own tmux window, cycle after cycle; `loop-claude --help` states its contract.
- `markdown-section <file> <fragment>` prints the section of a markdown file under the first heading that holds the fragment, to the next heading of its level or a higher one, instead of a `grep -n` and a guessed `sed -n` range; `markdown-section --help` says what it does where several headings match.
@@ -213,8 +213,8 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `branch_chunks/` cuts a default branch into chunks and finds, for each, the items it shipped and the later chunks that fixed its code, for `score_outcomes` and `routing_figures`. It imports `reading_cost`'s record path.
- `command_lines/` reads a shell command: its simple commands, and a docker call's subcommand and memory limits, for `command_patterns`, `reading_cost`, `routing_figures`, `rule_hook` and `run_heavy`.
- `deploy_release` imports `release_plan`'s `release_standing`.
- `do_epic/` will hold the tool that drives one Jira epic. So far `config.py` reads its machine-wide settings, importing `project_config`'s settings and `model_routing`'s ladder file, and `epic_record.py` reads and writes an epic's description by section and language line, through `record_store`'s `EpicDescription`.
- `forge/` tells a GitHub origin from a Gitea one and runs that forge's CLI (`commands.py`), and finds a pull request and waits for its checks (`pulls.py`), for `chunk_preflight`, `chunk_pr`, `deploy_release` and `larv_review`.
- `do_epic/` will hold the tool that drives one Jira epic. So far `config.py` reads its machine-wide settings, importing `project_config`'s settings and `model_routing`'s ladder file, and `epic_record.py` reads and writes an epic's description by section and language line, through `record_store`'s `EpicDescription`. `repos.py` finds a repo's checkout and project config by its name, and clones a missing checkout from the host and owner of the epic's other repos, through `forge`'s `url_beside`.
- `forge/` tells a GitHub origin from a Gitea one, names a repo beside an origin's and runs that forge's CLI (`commands.py`), and finds a pull request and waits for its checks (`pulls.py`), for `chunk_preflight`, `chunk_pr`, `deploy_release`, `do_epic` and `larv_review`.
- `git_cli/` runs git for every tool (`git_repo.py`), and reads the git calls in a shell command (`git_command.py`).
- `judge_replay` imports `loop_claude`'s judge, verdicts, premises and records.
- `loop_claude` runs `chunk-preflight` as a process of its own.
+96
View File
@@ -0,0 +1,96 @@
"""Where an epic's repo is on this machine: its checkout and its project config, found by the repo's name.
A missing checkout is cloned from the host and owner of the epic's other repos (docs/decisions.md → `do-epic: drives one Jira epic …`). Where that is unclear, the answer is an Unsure."""
import dataclasses
import pathlib
import re
import shutil
from project_config.settings import by_name, CONFIG, ProjectConfig, PROJECTS
from do_epic.config import DoEpicConfig
from failure import Failure
from git_cli.git_repo import Git
from forge.commands import url_beside
# One folder name, which can't be a path or a git option.
REPO_NAME = r"\w[\w.-]*"
# The addresses a clone may use: https, ssh, or scp-style `user@host:path`. Not ext:: or file:, which run a program or read this machine's files, and an origin can spell either.
CLONE_ADDRESS = rf"https://[\w.-]+(:\d+)?/\S+|ssh://[\w.@-]+(:\d+)?/\S+|\w[\w.-]*@[\w.-]+:[\w./~-]+"
# A clone that waits longer is stopped.
CLONE_SECONDS = 600
# git's options that make a password or a host key fail at once instead of prompting: an unattended run never waits on a prompt (docs/decisions.md → principle 6).
NO_PROMPTS = ("-c", "core.askPass=true", "-c", "core.sshCommand=ssh -o BatchMode=yes")
@dataclasses.dataclass(frozen=True)
class Located:
"""checkout is the repo's git checkout, and config the project config that names it."""
checkout: pathlib.Path
config: ProjectConfig
@dataclasses.dataclass(frozen=True)
class Unsure(Failure):
"""reason is a question for the human."""
def locate(name: str, others: list[pathlib.Path], config: DoEpicConfig, home: pathlib.Path, git: Git) -> Located | Failure:
"""The checkout and project config of the repo called name; others are the checkouts of the epic's other repos.
A project config names the repo's checkout, and it is cloned where missing. Where no config names the repo, it is cloned into the clone folder, if it isn't there, and the human is asked to write the config, since the repo's plan, goals and records are theirs to place.
A name that is no repo's name, or a config that can't be read, is a plain Failure. Any other fault is an Unsure."""
if not re.fullmatch(REPO_NAME, name):
return Failure(f"{name!r} is not a repo's name: one folder name that starts with a letter, digit or underscore")
found = by_name(name, home)
if isinstance(found, Failure):
return found
checkout = found.top if found else config.clone_folder / name
cloned = ensure(name, checkout, others, git)
if isinstance(cloned, Failure):
return cloned
if found is None:
return Unsure(f"{checkout} {'was cloned' if cloned else 'holds the repo'}, but no project config names it. Write {home / PROJECTS}/<key>/{CONFIG} with `checkout = \"{checkout}\"`, as `project-config --help` shows, and run do-epic again.")
return Located(checkout, found)
def ensure(name: str, checkout: pathlib.Path, others: list[pathlib.Path], git: Git) -> bool | Failure:
"""Whether checkout was cloned just now: False where it is a git checkout already. Else it is cloned there from the host and owner of the others, or an Unsure says why not."""
if (checkout / ".git").exists():
return False
if checkout.exists() and (not checkout.is_dir() or any(checkout.iterdir())):
return Unsure(f"{checkout} exists and is not a git checkout, so {name} cannot be cloned there. Where should it be cloned?")
url = clone_url(name, others, git)
if isinstance(url, Failure):
return url
cloned = git(*NO_PROMPTS, "clone", "--", url, str(checkout), timeout=CLONE_SECONDS)
if isinstance(cloned, Failure):
# What the clone began, and nothing else: checkout was absent or empty. A kept half-clone would pass for the repo on the next run.
shutil.rmtree(checkout, ignore_errors=True)
return Unsure(f"cannot clone {url} to {checkout}: {cloned.reason}. Where does {name} live?")
return True
def without_credentials(url: str) -> str:
"""url without a `user:token@` after its scheme, so that a clone and a question for the team never carry one. The clone then uses git's own credentials."""
return re.sub(r"^(https?://)[^/]*@", r"\1", url)
def clone_url(name: str, others: list[pathlib.Path], git: Git) -> str | Failure:
"""The address of the repo called name, under the host and owner all the others share. An Unsure where they share none or an address can't be built."""
if not others:
return Unsure(f"{name} is missing and no other repo of the epic shows what host to clone it from. Where does it live?")
urls = set()
for other in others:
origin = git("remote", "get-url", "origin", cwd=str(other))
if isinstance(origin, Failure):
return Unsure(f"cannot read the origin of {other}: {origin.reason}. Where does {name} live?")
beside = url_beside(without_credentials(origin.strip()), name)
if beside is None:
return Unsure(f"{other} has an origin, {without_credentials(origin.strip())}, with no host and owner to put another repo under. Where does {name} live?")
urls.add(beside)
if len(urls) > 1:
return Unsure(f"The epic's other repos sit under more than one host or owner, so {name} could be at {' or '.join(sorted(urls))}. Which holds it?")
url = next(iter(urls))
if not re.fullmatch(CLONE_ADDRESS, url):
return Unsure(f"{url} is not an https, ssh or scp-style address. Where does {name} live?")
return url
+184
View File
@@ -0,0 +1,184 @@
import pathlib
import sys
import tempfile
import unittest
import unittest.mock
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from do_epic.config import DoEpicConfig
from failure import Failure
from do_epic.repos import locate, Located, Unsure
from project_config.settings import ProjectConfig
from project_config.test_project_config import write_config
class FakeGit:
"""git as locate calls it: origins maps a checkout to its origin URL, cloned holds each `clone` that worked, and refused names a URL whose clone fails after it made the checkout's `.git`, as a stopped git does."""
def __init__(self, origins: dict[str, str | Failure], refused: str | None = None) -> None:
self.origins, self.refused = origins, refused
self.cloned: list[tuple[str, str]] = []
self.clone_timeouts: list[float | None] = []
self.clone_options: list[tuple[str, ...]] = []
def __call__(self, *args: str, cwd: str | None = None, timeout: float | None = None, guarded: bool = False, errors: str = "strict") -> str | Failure:
if args[:3] == ("remote", "get-url", "origin"):
return self.origins[cwd or ""]
clone = args.index("clone")
self.clone_options.append(args[:clone])
assert args[clone + 1] == "--", args
url, destination = args[clone + 2:clone + 4]
pathlib.Path(destination, ".git").mkdir(parents=True)
if url == self.refused:
return Failure("git clone: repository not found")
self.cloned.append((url, destination))
self.clone_timeouts.append(timeout)
return ""
class LocateTest(unittest.TestCase):
def setUp(self) -> None:
made = tempfile.TemporaryDirectory()
self.addCleanup(made.cleanup)
self.home = pathlib.Path(made.name).resolve()
self.projects = self.home / ".claude-projects"
self.config = DoEpicConfig(self.projects / "do-epic", "https://acme.atlassian.net", self.home / "jira.env", self.home / "clones", "English", "claude-opus-5-5", "rule")
self.billing, self.payments = self.home / "elsewhere" / "billing", self.home / "elsewhere" / "payments"
self.origins: dict[str, str | Failure] = {str(self.billing): "git@github.com:acme/billing.git", str(self.payments): "git@github.com:acme/payments"}
def configure(self) -> None:
write_config(self.home, self.home / "code" / "ledger")
def locate(self, git: FakeGit, name: str = "ledger") -> Located | Failure:
return locate(name, [self.billing, self.payments], self.config, self.home, git)
def unsure(self, git: FakeGit, name: str = "ledger") -> str:
found = self.locate(git, name)
assert isinstance(found, Unsure), found
return found.reason
def test_finds_a_checkout_that_exists_with_its_config_and_clones_nothing(self) -> None:
self.configure()
(self.home / "code" / "ledger" / ".git").mkdir(parents=True)
git = FakeGit(self.origins)
found = self.locate(git)
assert isinstance(found, Located), found
self.assertEqual((found.checkout, found.config.folder, git.cloned), (self.home / "code" / "ledger", self.projects / "ledger", []))
def test_clones_a_checkout_the_config_names_where_it_is_missing_from_the_place_of_the_other_repos(self) -> None:
self.configure()
git = FakeGit(self.origins)
found = self.locate(git)
assert isinstance(found, Located), found
self.assertEqual((found.checkout, git.cloned), (self.home / "code" / "ledger", [("git@github.com:acme/ledger", str(self.home / "code" / "ledger"))]))
assert isinstance(found.config, ProjectConfig)
def test_clones_into_the_clone_folder_and_asks_for_a_config_where_none_names_the_repo(self) -> None:
git = FakeGit(self.origins)
asked = self.unsure(git)
clone = self.home / "clones" / "ledger"
self.assertEqual(git.cloned, [("git@github.com:acme/ledger", str(clone))])
self.assertIn(f"{clone} was cloned, but no project config names it", asked)
self.assertIn(f"{self.projects}/<key>/config.toml with `checkout = \"{clone}\"`", asked)
self.assertEqual(self.unsure(git), asked.replace("was cloned", "holds the repo"), "the second ask finds the clone and clones nothing")
self.assertEqual(len(git.cloned), 1)
def test_asks_where_the_other_repos_sit_in_more_than_one_place(self) -> None:
self.configure()
self.origins[str(self.payments)] = "git@gitea.larvit.se:acme/payments"
git = FakeGit(self.origins)
asked = self.unsure(git)
self.assertIn("git@gitea.larvit.se:acme/ledger or git@github.com:acme/ledger", asked)
self.assertEqual(git.cloned, [])
def test_asks_where_the_other_repos_differ_only_by_owner(self) -> None:
self.configure()
self.origins[str(self.payments)] = "git@github.com:other/payments"
self.assertIn("more than one host or owner", self.unsure(FakeGit(self.origins)))
def test_asks_where_no_other_repo_shows_where_to_clone_from(self) -> None:
self.configure()
found = locate("ledger", [], self.config, self.home, FakeGit({}))
assert isinstance(found, Unsure), found
self.assertIn("no other repo of the epic", found.reason)
def test_asks_where_an_others_origin_cannot_be_read_or_has_no_place(self) -> None:
self.configure()
self.origins[str(self.billing)] = Failure("git remote: no such remote 'origin'")
self.assertIn(f"cannot read the origin of {self.billing}: git remote: no such remote 'origin'", self.unsure(FakeGit(self.origins)))
self.origins[str(self.billing)] = "billing"
self.assertIn(f"{self.billing} has an origin, billing, with no host and owner to put another repo under", self.unsure(FakeGit(self.origins)))
def test_asks_all_the_same_where_the_leftovers_of_a_failed_clone_cannot_be_removed(self) -> None:
self.configure()
with unittest.mock.patch("do_epic.repos.shutil.rmtree", side_effect=lambda path, ignore_errors: None if ignore_errors is True else self.fail("rmtree must ignore errors")):
self.assertIn("cannot clone", self.unsure(FakeGit(self.origins, refused="git@github.com:acme/ledger")))
def test_asks_where_the_clone_fails_naming_the_url_and_the_reason(self) -> None:
self.configure()
asked = self.unsure(FakeGit(self.origins, refused="git@github.com:acme/ledger"))
self.assertIn(f"cannot clone git@github.com:acme/ledger to {self.home / 'code' / 'ledger'}: git clone: repository not found", asked)
def test_asks_where_the_place_to_clone_into_holds_something_that_is_not_a_checkout(self) -> None:
self.configure()
(self.home / "code" / "ledger").mkdir(parents=True)
(self.home / "code" / "ledger" / "notes.txt").write_text("x", encoding="utf-8")
git = FakeGit(self.origins)
self.assertIn(f"{self.home / 'code' / 'ledger'} exists and is not a git checkout", self.unsure(git))
self.assertEqual(git.cloned, [])
def test_refuses_a_name_that_is_not_a_repo_name_before_it_forms_a_path_or_a_command(self) -> None:
for name in ("../escape", "-flag", "a/b", ""):
with self.subTest(name=name):
found = self.locate(FakeGit(self.origins), name)
assert isinstance(found, Failure) and not isinstance(found, Unsure), found
self.assertIn("is not a repo's name", found.reason)
def test_refuses_an_origin_that_is_not_https_ssh_or_an_scp_style_address(self) -> None:
self.configure()
for origin in ("ext::sh -c touch% x/billing", "file:///srv/acme/billing", "/srv/acme/billing", "-oProxy=x:a/b"):
with self.subTest(origin=origin):
self.origins[str(self.billing)] = origin
self.origins[str(self.payments)] = origin
git = FakeGit(self.origins)
self.assertIn("is not an https, ssh or scp-style address", self.unsure(git))
self.assertEqual(git.cloned, [])
def test_clones_and_asks_without_the_credentials_an_origin_holds(self) -> None:
self.configure()
self.origins[str(self.billing)] = "https://bot:s3cret@gitea.larvit.se/acme/billing"
self.origins[str(self.payments)] = "https://other:t0@ken@gitea.larvit.se/acme/payments.git"
git = FakeGit(self.origins)
self.assertIsInstance(self.locate(git), Located)
self.assertEqual(git.cloned, [("https://gitea.larvit.se/acme/ledger", str(self.home / "code" / "ledger"))])
self.origins[str(self.payments)] = "https://other:t0@ken@gitea.larvit.se/acme/payments"
self.origins[str(self.billing)] = "https://bot:s3cret@gitea.larvit.se"
self.assertNotIn("s3cret", self.unsure(FakeGit(self.origins), "stock"))
def test_stops_a_clone_that_waits_and_makes_a_prompt_fail(self) -> None:
self.configure()
git = FakeGit(self.origins)
self.locate(git)
self.assertEqual(git.clone_timeouts, [600])
self.assertEqual(git.clone_options, [("-c", "core.askPass=true", "-c", "core.sshCommand=ssh -o BatchMode=yes")])
def test_removes_what_a_failed_clone_began_so_that_the_next_run_clones_again(self) -> None:
self.configure()
failing = FakeGit(self.origins, refused="git@github.com:acme/ledger")
self.unsure(failing)
self.assertFalse((self.home / "code" / "ledger").exists())
found = self.locate(FakeGit(self.origins))
self.assertIsInstance(found, Located)
def test_asks_where_the_place_to_clone_into_is_a_file(self) -> None:
self.configure()
(self.home / "code").mkdir()
(self.home / "code" / "ledger").write_text("x", encoding="utf-8")
self.assertIn("exists and is not a git checkout", self.unsure(FakeGit(self.origins)))
def test_fails_with_the_config_failure_where_a_config_cannot_be_read(self) -> None:
(self.projects / "broken").mkdir(parents=True)
(self.projects / "broken" / "config.toml").write_text("this is not toml", encoding="utf-8")
found = self.locate(FakeGit(self.origins))
self.assertIsInstance(found, Failure)
self.assertNotIsInstance(found, Unsure)
+9 -1
View File
@@ -1,5 +1,6 @@
"""Which forge a checkout's origin is on, and running that forge's CLI in the checkout."""
"""Which forge a checkout's origin is on, a repo's URL beside that origin's, and running the forge's CLI in the checkout."""
import json
import re
import subprocess
import typing
@@ -16,6 +17,13 @@ def forge_of(origin_url: str) -> Forge:
return "gh" if "github.com" in origin_url else "tea"
def url_beside(origin_url: str, name: str) -> str | None:
"""The origin URL of the repo called name under the same host and owner as origin_url's repo; None where origin_url has no place, such as a bare name. The URL has no `.git`, which git clones without."""
place = re.fullmatch(r"(.+[/:])[^/:]+", origin_url.rstrip("/"))
# An origin of only a scheme and a host, such as `https://host`, has no owner folder to put a name in.
return place.group(1) + name if place and not place.group(1).endswith("//") else None
def runner(cwd: str) -> Run:
def run(command: list[str]) -> str | Failure:
try:
+14 -1
View File
@@ -7,7 +7,7 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from forge.pulls import checks_passed, open_pull, POLL_SECONDS, required_checks, WAIT_SECONDS
from forge.commands import forge_of, gitea
from forge.commands import forge_of, gitea, url_beside
class ForgeOf(unittest.TestCase):
@@ -16,6 +16,19 @@ class ForgeOf(unittest.TestCase):
self.assertEqual(forge_of("https://gitea.larvit.se/o/r.git"), "tea")
class UrlBeside(unittest.TestCase):
def test_names_a_repo_in_the_same_place_of_the_same_host_in_each_spelling_of_an_origin(self):
self.assertEqual(url_beside("git@github.com:acme/billing.git", "ledger"), "git@github.com:acme/ledger")
self.assertEqual(url_beside("https://gitea.larvit.se/acme/billing", "ledger"), "https://gitea.larvit.se/acme/ledger")
self.assertEqual(url_beside("https://gitea.larvit.se/acme/billing.git/", "ledger"), "https://gitea.larvit.se/acme/ledger")
self.assertEqual(url_beside("gitea:billing", "ledger"), "gitea:ledger", "a host with no owner")
def test_names_none_where_the_origin_has_no_place_to_put_a_repo_in(self):
self.assertIsNone(url_beside("billing", "ledger"))
self.assertIsNone(url_beside("", "ledger"))
self.assertIsNone(url_beside("https://gitea.larvit.se", "ledger"))
# An answer no call can read, longer than the 200 characters of it that a failure quotes.
LONG = "[" + ", ".join(["1"] * 100) + "]"
+36 -3
View File
@@ -1,6 +1,6 @@
"""Replay loop-claude's judge on archived verdicts, to find the lowest rung that returns them: docs/decisions.md → `routing: on a model judge-replay covered`.
usage: judge-replay <model>
usage: judge-replay <model> [--offline]
<model> is a project model on the routing ladder this setup ships, such as claude-opus-5-5. The tool replays the
verdicts archived for the projects that ~/.claude/loop-claude-models.json runs on it: the latest
@@ -36,8 +36,21 @@ Like the loop, it tries a judge three times running before it counts the reply a
verdict whose project, transcript or archived judge run cannot be read replays as failed too. Exit 1, saying why, on
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
or a usage limit.
With --offline it doesn't call a judge, so the runs, retries and usage limit above don't apply. It reads every
archived verdict of the model's projects, not a sample. It counts how many of them each of three code
deciders decides from the worker's final message, and how many archived interrupts it decides as another
kind, which are its missed interrupts: a "?" rule, a decision table, and a classifier trained on the archive
(judge_replay/offline.py). The rule and the table run on the whole archive; the table's rows were read from
it, so its run isn't a held-out test. The classifier runs on held-out folds, each tested by a model trained on
the other folds, and on the latest fifth of the verdicts, tested by a model trained on the earlier four
fifths. The classifier trains 330 regressions in Python, about 12 minutes on 1,100 verdicts. The report is
kept in replays/judge-replay-<model>-offline-<UTC stamp>/result.md. A verdict whose transcript cannot be read is skipped, and the report counts the skipped. Exit 1, saying why, on bad
arguments, an unreadable archive or models file, no archived verdict of the model's projects, or none with a
transcript that can be read.
"""
import pathlib
import sys
import typing
from transcript_archive.location import archive_root, transcript_of
@@ -49,6 +62,7 @@ from judge_replay.replay import Case, case_from, Kind, report, reproduced, sampl
from transcript_archive.project_key import checkout_of
from loop_effects.effects import Effects
from failure import Failure
from judge_replay.offline import final_message, LEARNERS, offline_report, RULES, Sample
from rung_replay.search import first_reply, replayed
from judge_replay.archived_prompt import grounds_of
from transcript_archive.judge_run import JudgeRun
@@ -61,7 +75,8 @@ from claude_code.transcript import read_transcript
from loop_claude.verdicts import Verdict
TOOL = "judge-replay"
USAGE = f"{TOOL} <model>, a model on the routing ladder"
USAGE = f"{TOOL} <model> [--offline]; <model> is a model on the routing ladder"
OFFLINE = "--offline"
PER_KIND = 15
# The effort settings.json gave every judge before loop-claude routed it; the archived verdicts came from it.
REFERENCE_EFFORT = "xhigh"
@@ -72,7 +87,7 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
ladder = read_ladder(SHIPPED)
if isinstance(ladder, Failure):
return failed(TOOL, ladder.reason)
project = arguments_or_exit(TOOL, argv, __doc__, USAGE, lambda argv: model_argument(ladder, argv))
project = arguments_or_exit(TOOL, argv, __doc__, USAGE, lambda argv: model_argument(ladder, [arg for arg in argv if arg != OFFLINE]))
if isinstance(project, int):
return project
cases = archived_cases(project.rung.model, archive_root(home, env), home)
@@ -80,6 +95,8 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
return failed(TOOL, cases.reason)
if not cases:
return failed(TOOL, f"no archived verdicts of projects on {project.rung.model} in {archive_root(home, env)}; run loop-claude on a project on that model first")
if OFFLINE in argv:
return offline(project.rung.model, cases, archive_root(home, env), effects)
cases = sample(cases, PER_KIND)
folder = replay_folder(archive_root(home, env), TOOL, argv, effects.now())
effects = archiving(effects, folder)
@@ -92,6 +109,22 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
return 0
def offline(model: str, cases: list[Case], root: pathlib.Path, effects: Effects) -> int:
"""Replays the verdicts of cases offline, oldest first, and keeps the report in the archive as a judge replay does."""
samples, skipped = [], 0
for found in sorted(cases, key=lambda found: (found.at, found.root, str(found.transcript))):
entries = read_transcript(str(found.transcript))
if isinstance(entries, Failure):
skipped += 1
print(f"{TOOL}: skipped a verdict: {entries.reason}", file=sys.stderr)
else:
samples.append(Sample(found.archived_kind, final_message(entries)))
if not samples:
return failed(TOOL, "no archived verdict has a transcript that can be read")
print_report(replay_folder(root, TOOL, [model, OFFLINE], effects.now()), offline_report(model, samples, RULES, LEARNERS, skipped=skipped))
return 0
def model_argument(ladder: Ladder, argv: list[str]) -> ProjectRung | None:
"""The project's rung on ladder of the one model argv names; None for any other argv."""
return ladder.project_rung(argv[0]) if len(argv) == 1 else None
+175
View File
@@ -0,0 +1,175 @@
"""Deciders of a verdict's kind that run offline on the worker's final message, replayed on the archive: docs/decisions.md → `routing: the judge's model keeps the verdict kind, …`."""
import collections
import math
import random
import re
import typing
from claude_code.entries import AssistantTurn, Entry, Text
from transcript_archive.outcomes import FINISHED, INTERRUPTED, UNFINISHED
from judge_replay.replay import Kind
class Sample(typing.NamedTuple):
"""What a decider reads of one archived verdict: the kind the archive gives it and the worker's final message."""
kind: Kind
text: str
# A decider names a kind, or None to leave the verdict to the judge's model. A method builds one from the samples it may learn from.
Decider = typing.Callable[[str], Kind | None]
Method = typing.Callable[[typing.Sequence[Sample]], Decider]
Counts = collections.Counter[tuple[Kind, Kind | None]]
FOLDS = 10
INNER_FOLDS = 5
# A word in fewer messages than this is left out of the classifier.
MIN_MESSAGES = 3
EPOCHS = 8
# The classifier averages this many regressions, each shuffling the samples in its own order: one shuffle alone moved the missed interrupts of a run between 0 and 2.
SEEDS = 5
RATE = 0.05
DECAY = 1e-3
# The latest 1/LATEST of the archive is the second test: the verdicts a decider would meet next.
LATEST = 5
# The first row whose pattern matches the final message decides it; a message no row matches is left to the judge. The rows come from reading the archive's interrupts, so a run of the table on the whole archive is not a held-out test.
DECISION_TABLE = (
(INTERRUPTED, re.compile(r"(?i)\?|recommend|your call|needs? you|tell me|you answer|please|unmerged|not merged|do you|should i|i need|need your|you say|decisions? for|stays until")),
(FINISHED, re.compile(r"(?i)\bmerged\b(?!.*ready to cut)")),
)
def final_message(entries: list[Entry]) -> str:
"""The text of the worker's last turn that has any, "" where it wrote none."""
for entry in reversed(entries):
if isinstance(entry, AssistantTurn):
text = "\n".join(part.text for part in entry.parts if isinstance(part, Text) and part.text.strip())
if text:
return text
return ""
def question_mark(samples: typing.Sequence[Sample]) -> Decider:
"""The one rule: an interrupt where the message holds a "?", finished where it holds none."""
return lambda text: INTERRUPTED if "?" in text else FINISHED
def decision_table(samples: typing.Sequence[Sample]) -> Decider:
return lambda text: next((kind for kind, pattern in DECISION_TABLE if pattern.search(text)), None)
def words_of(text: str) -> frozenset[str]:
"""Each word and each pair of neighbouring words of the message, and its question marks."""
words = re.findall(r"[a-z0-9']+|\?", text.lower())
return frozenset(words) | {f"{first} {second}" for first, second in zip(words, words[1:])}
def logistic(samples: typing.Sequence[Sample], seed: int) -> typing.Callable[[str], float]:
"""The chance that a message is an interrupt, from a logistic regression over its words that the samples train, shuffled in the order seed gives."""
seen = collections.Counter(word for found in samples for word in words_of(found.text))
vocabulary = {word for word, count in seen.items() if count >= MIN_MESSAGES}
data = [(words_of(found.text) & vocabulary, 1.0 if found.kind == INTERRUPTED else 0.0) for found in samples]
weights: dict[str, float] = collections.defaultdict(float)
bias = 0.0
shuffler = random.Random(seed)
def chance(words: typing.Iterable[str]) -> float:
return 1 / (1 + math.exp(-max(-30.0, min(30.0, bias + sum(weights[word] for word in words)))))
for _ in range(EPOCHS):
shuffler.shuffle(data)
for words, interrupt in data:
error = chance(words) - interrupt
bias -= RATE * error
for word in words:
weights[word] -= RATE * (error + DECAY * weights[word])
return lambda text: chance(words_of(text) & vocabulary)
def averaged(samples: typing.Sequence[Sample]) -> typing.Callable[[str], float]:
"""The mean chance of SEEDS regressions that differ only in the order they shuffle the samples."""
regressions = [logistic(samples, seed) for seed in range(SEEDS)]
return lambda text: sum(regression(text) for regression in regressions) / SEEDS
def classifier(samples: typing.Sequence[Sample]) -> Decider:
"""Finished where the chance of an interrupt is below one half and below that of every interrupt among the samples; interrupt where it is at least one half and above that of every other kind; otherwise the judge's model decides.
A sample's chance is the one a model trained on the other folds of the samples gives it, so the bounds don't rest on how well the model fits what it trained on. A bound is where the samples held no miss; that doesn't promise the next verdict holds none."""
interrupts = [found for found in samples if found.kind == INTERRUPTED]
if not interrupts or len(interrupts) == len(samples):
return lambda text: None
held_out = [(tested.kind, chance) for trained, tests in folds_of(samples, INNER_FOLDS) for chance, tested in zip(map(averaged(trained), (found.text for found in tests)), tests)]
below = min(chance for kind, chance in held_out if kind == INTERRUPTED)
above = max(chance for kind, chance in held_out if kind != INTERRUPTED)
chance = averaged(samples)
def decide(text: str) -> Kind | None:
found = chance(text)
if found < below and found < 0.5:
return FINISHED
return INTERRUPTED if found > above and found >= 0.5 else None
return decide
# Rules are fixed, and run on the whole archive; a learner trains on samples, and runs on samples it didn't train on.
RULES: dict[str, Method] = {'"?" rule': question_mark, "decision table": decision_table}
LEARNERS: dict[str, Method] = {"classifier": classifier}
def folds_of(samples: typing.Sequence[Sample], count: int) -> list[tuple[list[Sample], list[Sample]]]:
"""count folds of the samples, dealt in turn. Each fold is the samples to learn from, which are all those its test leaves out, and the samples to test on."""
return [([found for place, found in enumerate(samples) if place % count != fold], [found for place, found in enumerate(samples) if place % count == fold]) for fold in range(count)]
def evaluate(method: Method, trained: typing.Sequence[Sample], tested: typing.Sequence[Sample]) -> Counts:
"""How often the method, learning from trained, decided each kind for each archived kind of tested; None where it left the verdict to the judge."""
decide = method(trained)
return collections.Counter((found.kind, decide(found.text)) for found in tested)
def plain(kind: Kind) -> str:
return kind.removeprefix("Verdict: ")
def offline_report(model: str, samples: typing.Sequence[Sample], rules: typing.Mapping[str, Method], learners: typing.Mapping[str, Method], folds: int = FOLDS, skipped: int = 0) -> str:
"""samples are in the order the verdicts were judged, oldest first, so the latest fifth is the newest verdicts."""
kinds = collections.Counter(found.kind for found in samples)
lines = [f"Replayed {len(samples)} archived verdicts of {model} projects offline, without sending anything to an outside service. The archive gives them as {', '.join(f'{count} {plain(kind)}' for kind, count in sorted(kinds.items(), key=lambda each: plain(each[0])))}."]
if skipped:
lines.append(f"Skipped {skipped} verdicts whose transcript could not be read.")
for name, rule in rules.items():
found = evaluate(rule, [], samples)
lines += [result_line(f"{name}, on the whole archive", found), *decides_lines(f"{name}, on the whole archive", found)]
split = len(samples) - max(1, len(samples) // LATEST)
for name, learner in learners.items():
held_out = sum((evaluate(learner, trained, tested) for trained, tested in folds_of(samples, folds)), Counts())
latest = evaluate(learner, samples[:split], samples[split:])
lines += [result_line(f"{name}, on held-out folds", held_out), result_line(f"{name}, on the latest fifth after learning from the rest", latest), *decides_lines(f"{name}, on both tests", held_out + latest)]
return "\n".join(lines)
def result_line(how: str, found: Counts) -> str:
decided = collections.Counter({kind: sum(count for (_, said), count in found.items() if said == kind) for kind in {said for _, said in found if said}})
missed = sum(count for (archived, said), count in found.items() if archived == INTERRUPTED and said not in (None, INTERRUPTED))
parts = [f"{plain(kind)} {count}{wrong_kinds(kind, found)}" for kind, count in sorted(decided.items(), key=lambda each: plain(each[0]))]
return f"{how}: decided {sum(decided.values())} of {sum(found.values())}{': ' + ', '.join(parts) if parts else ''}. Missed interrupts: {missed}."
def wrong_kinds(decided: Kind, found: Counts) -> str:
"""The kinds the archive gives the verdicts that were decided as decided, where the archive gives another kind than decided."""
wrong = sorted(((plain(archived), count) for (archived, said), count in found.items() if said == decided and archived != decided), key=lambda each: each[0])
return f" (of these, archived {', '.join(f'{kind} {count}' for kind, count in wrong)})" if wrong else ""
def decides_lines(how: str, found: Counts) -> list[str]:
"""For finished and for unfinished that the decider decided at all: whether it decided no archived interrupt as that kind."""
lines = []
for kind in (FINISHED, UNFINISHED):
decided = sum(count for (_, said), count in found.items() if said == kind)
if decided and not found[(INTERRUPTED, kind)]:
lines.append(f"{how}, decides {plain(kind)} with no missed interrupt.")
elif decided:
lines.append(f"{how}, does not decide {plain(kind)} with no missed interrupt: it decided {found[(INTERRUPTED, kind)]} archived interrupt(s) as {plain(kind)}.")
return lines
+39 -1
View File
@@ -175,6 +175,44 @@ class Main(unittest.TestCase):
# The replay shows the judge the whole transcript, with no step-up marked in it.
self.assertNotIn("[step]", self.prompts[0])
def test_offline_replays_every_verdict_through_the_code_deciders_without_calling_the_judge(self):
code, out, err = self.run_main(["claude-opus-5-5", "--offline"], unused)
self.assertEqual(code, 0, err)
self.assertEqual(self.calls, [])
self.assertIn("Replayed 2 archived verdicts of claude-opus-5-5 projects offline, without sending anything to an outside service. The archive gives them as 1 finished, 1 interrupt.", out)
self.assertIn("\"?\" rule, on the whole archive: decided 2 of 2: finished 2 (of these, archived interrupt 1). Missed interrupts: 1.", out)
folder = self.home / ".claude" / "loop-claude-transcripts" / "replays" / "judge-replay-claude-opus-5-5-offline-20260921T141320Z"
self.assertEqual((folder / "result.md").read_text(encoding="utf-8"), out)
def test_offline_exits_1_where_no_transcript_can_be_read(self):
for path in (self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1").glob("*.jsonl"):
path.unlink()
code, out, err = self.run_main(["claude-opus-5-5", "--offline"], unused)
self.assertEqual((code, out), (1, ""))
self.assertEqual(err.splitlines()[-1], "judge-replay: no archived verdict has a transcript that can be read")
def test_offline_skips_each_verdict_whose_transcript_cannot_be_read_and_says_how_many_on_stderr_and_in_the_report(self):
kept = self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1"
(kept / "t2.outcomes.md").write_text(f"Task: Do it.\n\n2026-10-14T10:00:02Z {FINISHED}\n\nEvidence:\n> e\n")
(kept / "t0.jsonl").unlink()
code, out, err = self.run_main(["claude-opus-5-5", "--offline"], unused)
self.assertEqual(code, 0)
self.assertEqual(len(err.splitlines()), 2)
self.assertTrue(all(line.startswith("judge-replay: skipped a verdict: could not read the transcript ") for line in err.splitlines()))
self.assertIn("Replayed 1 archived verdicts of claude-opus-5-5 projects offline", out)
self.assertIn("Skipped 2 verdicts whose transcript could not be read.", out)
def test_offline_keeps_its_report_in_the_same_folder_wherever_the_flag_stands(self):
code, out, err = self.run_main(["--offline", "claude-opus-5-5"], unused)
self.assertEqual(code, 0, err)
folder = self.home / ".claude" / "loop-claude-transcripts" / "replays" / "judge-replay-claude-opus-5-5-offline-20260921T141320Z"
self.assertEqual((folder / "result.md").read_text(encoding="utf-8"), out)
def test_offline_says_the_usage_on_arguments_that_name_no_model(self):
code, out, err = self.run_main(["--offline"], unused)
self.assertEqual((code, out), (1, ""))
self.assertEqual(err, "judge-replay: cannot read the arguments --offline; usage: judge-replay <model> [--offline]; <model> is a model on the routing ladder\n")
def test_keeps_each_run_and_the_printed_report_in_one_folder_of_the_archive_beside_the_projects(self):
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
self.assertEqual(code, 0, err)
@@ -272,7 +310,7 @@ class Main(unittest.TestCase):
def test_prints_the_help(self):
code, out, _ = self.run_main(["-h"], unused)
self.assertEqual((code, out.splitlines()[2]), (0, "usage: judge-replay <model>"))
self.assertEqual((code, out.splitlines()[2]), (0, "usage: judge-replay <model> [--offline]"))
def test_the_launcher_prints_the_help(self):
ran = subprocess.run([sys.executable, str(TOOL), "--help"], capture_output=True, text=True)
+190
View File
@@ -0,0 +1,190 @@
import collections
import pathlib
import sys
import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINISHED
from claude_code.entries import AssistantTurn, HumanTurn, Text, ToolCall
from judge_replay.offline import averaged, classifier, decision_table, evaluate, final_message, folds_of, LEARNERS, logistic, offline_report, question_mark, RULES, Sample, words_of
def turn(*parts):
return AssistantTurn(parts=parts)
def sample(kind, text):
return Sample(kind, text)
class FinalMessage(unittest.TestCase):
def test_is_the_text_of_the_workers_last_turn_that_has_any(self):
entries = [turn(Text("first")), HumanTurn(text="ok"), turn(Text("Done. Merged."), ToolCall("1", "Bash", {})), turn(ToolCall("2", "Bash", {}))]
self.assertEqual(final_message(entries), "Done. Merged.")
def test_joins_the_texts_of_one_turn_on_separate_lines_and_leaves_out_blank_ones(self):
self.assertEqual(final_message([turn(Text("Merged."), Text(" "), Text("Which one?"))]), "Merged.\nWhich one?")
def test_is_empty_where_the_worker_wrote_no_text(self):
self.assertEqual(final_message([HumanTurn(text="hi"), turn(ToolCall("1", "Bash", {}))]), "")
class WordsOf(unittest.TestCase):
def test_is_each_lowercase_word_each_pair_of_neighbours_and_each_question_mark(self):
self.assertEqual(words_of("Which one? Yes"), {"which", "one", "?", "yes", "which one", "one ?", "? yes"})
class QuestionMark(unittest.TestCase):
def test_decides_interrupt_on_a_question_mark_and_finished_on_none(self):
decide = question_mark([])
self.assertEqual(decide("Which one?"), INTERRUPTED)
self.assertEqual(decide("Merged."), FINISHED)
class DecisionTable(unittest.TestCase):
def test_reads_the_workers_asking_for_the_humans_answer_as_an_interrupt(self):
decide = decision_table([])
for text in ("My recommendation is option 1.", "Your call: merge now or wait.", "The PR stays unmerged.", "Tell me which."):
with self.subTest(text=text):
self.assertEqual(decide(text), INTERRUPTED)
def test_reads_a_merge_with_no_ask_and_no_release_to_cut_as_finished(self):
decide = decision_table([])
self.assertEqual(decide("PR 12 merged and the worktree is removed."), FINISHED)
self.assertIsNone(decide("PR 12 merged. Release 0.3.0 is ready to cut."))
def test_leaves_any_other_message_to_the_judge(self):
self.assertIsNone(decision_table([])("I stopped at the gate."))
class Logistic(unittest.TestCase):
def test_gives_a_message_with_no_learned_word_about_the_share_of_interrupts(self):
chance = logistic([sample(INTERRUPTED, "")] * 10 + [sample(FINISHED, "")] * 30, 0)
self.assertAlmostEqual(chance(""), 0.25, delta=0.05)
def test_learns_a_word_found_in_as_few_messages_as_the_minimum(self):
chance = logistic([sample(INTERRUPTED, "ask")] * 3 + [sample(FINISHED, "")] * 3, 0)
self.assertGreater(chance("ask"), chance(""))
def test_averaged_is_the_mean_of_the_regressions_that_shuffle_in_each_seed_order(self):
samples = [sample(INTERRUPTED if n % 3 else FINISHED, f"word{n % 4} ask" if n % 3 else f"word{n % 4}") for n in range(30)]
mean = sum(logistic(samples, seed)("ask") for seed in range(5)) / 5
self.assertAlmostEqual(averaged(samples)("ask"), mean)
class Classifier(unittest.TestCase):
def trained(self, extra=()):
asking = [sample(INTERRUPTED, f"I recommend option {n}; tell me which of the options you want") for n in range(12)]
done = [sample(FINISHED, f"merged pull request {n} and removed the worktree, no findings left") for n in range(12)]
return classifier([*asking, *done, *extra])
def test_decides_a_message_like_each_kind_it_learned(self):
decide = self.trained()
self.assertEqual(decide("I recommend option 3; tell me which you want"), INTERRUPTED)
self.assertEqual(decide("merged pull request 99 and removed the worktree, no findings left"), FINISHED)
def test_decides_finished_on_a_message_of_words_only_finished_verdicts_held(self):
self.assertEqual(self.trained()("merged pull request"), FINISHED)
def test_abstains_where_it_learned_no_interrupt_or_no_other_kind(self):
self.assertIsNone(classifier([sample(FINISHED, "merged")])("merged"))
self.assertIsNone(classifier([sample(INTERRUPTED, "which?")])("which?"))
self.assertIsNone(classifier([])("anything"))
def test_decides_no_finished_where_an_interrupt_it_learned_looks_like_one(self):
lookalike = "merged pull request 7 and removed the worktree, no findings left"
decide = self.trained([sample(INTERRUPTED, lookalike)])
self.assertNotEqual(decide(lookalike), FINISHED)
def test_abstains_where_a_lookalike_of_each_kind_sits_in_the_samples_of_the_other(self):
decide = self.trained([sample(INTERRUPTED, "merged pull request 7 and removed the worktree, no findings left"), sample(FINISHED, "I recommend option 7; tell me which of the options you want")])
for text in ("merged pull request", "I recommend option", "zzz qqq"):
with self.subTest(text=text):
self.assertIsNone(decide(text))
class Folds(unittest.TestCase):
def test_deals_samples_to_folds_in_turn_so_each_is_tested_once(self):
samples = [sample(FINISHED, str(n)) for n in range(7)]
dealt = folds_of(samples, 3)
self.assertEqual([[found.text for found in tested] for _, tested in dealt], [["0", "3", "6"], ["1", "4"], ["2", "5"]])
self.assertEqual([found.text for found in dealt[1][0]], ["0", "2", "3", "5", "6"])
class Evaluate(unittest.TestCase):
def test_counts_each_archived_kind_by_what_the_method_decided_or_abstained(self):
tested = [sample(FINISHED, "merged"), sample(INTERRUPTED, "merged"), sample(UNFINISHED, "stopped"), sample(ANSWERED, "Which?")]
found = evaluate(decision_table, [], tested)
self.assertEqual(found, collections.Counter({(FINISHED, FINISHED): 1, (INTERRUPTED, FINISHED): 1, (UNFINISHED, None): 1, (ANSWERED, INTERRUPTED): 1}))
class Report(unittest.TestCase):
def samples(self):
return [sample(FINISHED, "Merged."), sample(FINISHED, "Merged. Release 1.0.0 is ready to cut."), sample(INTERRUPTED, "I recommend merging."), sample(INTERRUPTED, "Merged.")]
def test_counts_what_a_rule_decided_on_the_whole_archive_and_missed_by_kind(self):
text = offline_report("claude-opus-5-5", self.samples(), {"decision table": decision_table}, {})
self.assertIn("Replayed 4 archived verdicts of claude-opus-5-5 projects offline, without sending anything to an outside service. The archive gives them as 2 finished, 2 interrupt.", text)
self.assertIn("decision table, on the whole archive: decided 3 of 4: finished 2 (of these, archived interrupt 1), interrupt 1. Missed interrupts: 1.", text)
self.assertIn("decision table, on the whole archive, does not decide finished with no missed interrupt: it decided 1 archived interrupt(s) as finished.", text)
def test_says_a_method_that_decided_a_kind_and_missed_no_interrupt_decides_that_kind(self):
text = offline_report("claude-opus-5-5", [sample(FINISHED, "Merged."), sample(INTERRUPTED, "I recommend merging.")], {"decision table": decision_table}, {})
self.assertIn("Missed interrupts: 0.", text)
self.assertIn("decision table, on the whole archive, decides finished with no missed interrupt.", text)
def test_runs_a_learner_once_for_each_fold_and_once_on_the_latest_fifth_and_a_rule_only_on_the_whole_archive(self):
samples = [sample(INTERRUPTED if n % 2 else FINISHED, "I recommend it" if n % 2 else "merged it") for n in range(20)]
text = offline_report("m", samples, {"rule": question_mark}, {"classifier": classifier}, folds=4)
self.assertIn("classifier, on held-out folds: decided", text)
self.assertIn("classifier, on the latest fifth after learning from the rest: decided", text)
self.assertIn("rule, on the whole archive: decided 20 of 20", text)
self.assertNotIn("rule, on held-out", text)
self.assertNotIn("rule, on the latest fifth", text)
def test_words_the_whole_report_line_by_line_with_the_wrong_kinds_by_name(self):
answers = {"f": FINISHED, "i1": FINISHED, "i2": FINISHED, "u": FINISHED, "q": None}
samples = [sample(FINISHED, "f"), sample(INTERRUPTED, "i1"), sample(INTERRUPTED, "i2"), sample(UNFINISHED, "u"), sample(ANSWERED, "q")]
text = offline_report("m", samples, {"rule": lambda trained: lambda said: answers[said]}, {})
self.assertEqual(text.splitlines(), [
"Replayed 5 archived verdicts of m projects offline, without sending anything to an outside service. The archive gives them as 1 finished, 2 interrupt, 1 quoted answer, 1 unfinished.",
"rule, on the whole archive: decided 4 of 5: finished 4 (of these, archived interrupt 2, unfinished 1). Missed interrupts: 2.",
"rule, on the whole archive, does not decide finished with no missed interrupt: it decided 2 archived interrupt(s) as finished.",
])
def test_says_a_method_that_decided_nothing_decided_0_and_names_no_kind(self):
text = offline_report("m", self.samples(), {"rule": lambda trained: lambda said: None}, {})
self.assertEqual(text.splitlines()[1:], ["rule, on the whole archive: decided 0 of 4. Missed interrupts: 0."])
def test_tests_the_latest_fifth_of_the_samples_after_learning_from_the_earlier_ones(self):
trained = []
def learner(samples):
trained.append(len(samples))
return lambda said: FINISHED
text = offline_report("m", [sample(FINISHED, str(n)) for n in range(7)], {}, {"learner": learner}, folds=2)
self.assertEqual(trained, [3, 4, 6])
self.assertEqual(text.splitlines()[-2:], ["learner, on the latest fifth after learning from the rest: decided 1 of 1: finished 1. Missed interrupts: 0.", "learner, on both tests, decides finished with no missed interrupt."])
def test_says_a_learner_that_missed_an_interrupt_on_either_test_does_not_decide_the_kind(self):
def learner(samples):
return lambda said: FINISHED if said == "miss" else None
samples = [sample(FINISHED, "ok")] * 4 + [sample(INTERRUPTED, "miss")]
text = offline_report("m", samples, {}, {"learner": learner}, folds=2)
self.assertEqual(text.splitlines()[-1], "learner, on both tests, does not decide finished with no missed interrupt: it decided 2 archived interrupt(s) as finished.")
def test_counts_the_verdicts_it_skipped_only_where_it_skipped_some(self):
self.assertNotIn("Skipped", offline_report("m", self.samples(), {}, {}))
self.assertIn("Skipped 2 verdicts whose transcript could not be read.", offline_report("m", self.samples(), {}, {}, skipped=2))
def test_the_shipped_methods_are_the_question_mark_rule_the_table_and_the_classifier(self):
self.assertEqual(list(RULES), ['"?" rule', "decision table"])
self.assertEqual(list(LEARNERS), ["classifier"])
if __name__ == "__main__":
unittest.main()
+14
View File
@@ -350,6 +350,20 @@ def held(folder: pathlib.Path, home: pathlib.Path) -> ProjectConfig | Failure |
return config if isinstance(config, Failure) else load(config[0], config[0], home)
def by_name(name: str, home: pathlib.Path) -> ProjectConfig | Failure | None:
"""The config of the project whose main checkout is a folder called name, though that folder may not exist yet, loaded as `held` does; None where no config names one. A config that cannot be read fails every name, since it may be the one asked for."""
files = []
for file in configs(home):
config = read(file, home)
if isinstance(config, Failure):
return config
if config[0].name == name:
files.append(file)
if len(files) > 1:
return Failure(f"{' and '.join(str(file) for file in files)} each name a checkout called {name}; only one config may")
return held(files[0].parent, home) if files else None
def configs(home: pathlib.Path) -> list[pathlib.Path]:
"""Every project's config file under home, in key order. do-epic's machine-wide config is not a project's."""
return sorted(file for file in (home / PROJECTS).glob(f"*/{CONFIG}") if file.parent.name != DO_EPIC)
+27 -1
View File
@@ -9,7 +9,7 @@ import unittest.mock
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from project_config.settings import env_files, Epic, Finish, held, JiraScoring, load, NOTIFIERS, OUTSIDE, Page, ProjectConfig, SETTINGS, Unnamed
from project_config.settings import by_name, env_files, Epic, Finish, held, JiraScoring, load, NOTIFIERS, OUTSIDE, Page, ProjectConfig, SETTINGS, Unnamed
from failure import Failure
from test_env import isolated_env
from project_config.main import located, title_templates
@@ -598,6 +598,32 @@ class HeldTest(ConfigCase):
self.assertIsInstance(held(target, self.home), ProjectConfig, "git names the folder by its resolved path")
class ByNameTest(ConfigCase):
def test_finds_the_config_of_the_project_whose_main_checkout_has_the_name_though_the_checkout_is_missing(self) -> None:
folder = self.write("acme", CLIENT)
found = by_name("acme-billing", self.home)
assert isinstance(found, ProjectConfig), found
self.assertEqual((found.folder, found.top), (folder, self.checkout))
def test_finds_none_where_no_config_names_a_checkout_of_that_name(self) -> None:
self.write("acme", CLIENT)
self.assertIsNone(by_name("billing", self.home), "a part of a name is not the name")
self.assertIsNone(by_name("acme", self.home), "a config's key is not a repo's name")
self.assertIsNone(by_name("do-epic", self.home))
def test_fails_where_two_configs_name_a_checkout_of_that_name(self) -> None:
self.write("acme", CLIENT)
self.write("acme-two", CLIENT.replace("~/code/acme-billing", "~/other/acme-billing"))
found = by_name("acme-billing", self.home)
assert isinstance(found, Failure), found
self.assertIn("acme/config.toml and ", found.reason)
self.assertIn("acme-two/config.toml each name a checkout called acme-billing", found.reason)
def test_fails_where_a_config_cannot_be_read_since_it_may_be_the_one_asked_for(self) -> None:
self.write("acme", "this is not toml")
self.assertIsInstance(by_name("acme-billing", self.home), Failure)
class EnvFilesTest(ConfigCase):
def test_lists_each_readable_configs_env_file_even_one_load_refuses(self) -> None:
self.write("acme", CLIENT + 'env-file = "~/acme.env"\n')
+17 -1
View File
@@ -151,7 +151,7 @@ Valid while `~/.claude` is a checkout of this repo.
- **One item, several repos.** do-epic splits such an item when it plans it, into one Jira sub-task per repo, each labelled for its repo. The parent tracks the whole change. A person's issue is split too, with a comment for the team.
- **Ranking.** Score, then Blocked by, in one weight chain: the epic's achieve goals, then code health, then the repo's goals. Where the epic and a repo disagree, the epic wins. do-epic re-ranks Jira's backlog to match, and people's issues keep their rank.
- **Releases.** An epic item has no release. It ships into its repo's unreleased work: a bullet under `CHANGELOG.md` → `## Unreleased`, or nothing where the repo's plan is in Jira.
- **Repos.** A missing checkout is cloned from the host of the epic's other repos; where unsure, do-epic asks. Each chunk uses its repo's model, routing, and language for commits and PRs. Jira text is in the epic's language, or else the machine's default.
- **Repos.** A repo's name is its main checkout's folder name, and a project config names that checkout. A missing checkout is cloned from the host of the epic's other repos; where unsure, do-epic asks. Unsure means the other repos don't all sit under one host and owner (the same host spelt as https and as ssh counts as two), an origin can't be read or isn't an https, ssh or scp-style address, the folder to clone into holds something else, or the clone fails. A repo no config names is cloned into the clone folder, and the human is asked to write its config. Each chunk uses its repo's model, routing, and language for commits and PRs. Jira text is in the epic's language, or else the machine's default.
- **Running.** The epic caps the chunks running at once, and each repo's `parallel-chunks` still applies. do-epic and loop-claude never run chunks in the same repo at once.
- **Long gates.** A chunk whose PR waits on a gate, such as a human approval, parks. It keeps its worktree, PR and session, and its state is in a file that survives a reboot. A parked chunk takes no running slot, and the items it blocks wait. do-epic polls the forge and resumes the chunk's session when the gate passes or a reviewer comments. It fixes reasonable feedback, replies to the rest, and changes the setup so review would have caught it. `do-epic <key> --status` and the tmux window names show what each chunk waits on.
- **Human tasks.** do-epic DMs the human when one blocks the next item, and polls until its issue closes.
@@ -674,6 +674,22 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
Valid while the index ranks these models, and while no newer model joins the catalog.
## routing: the judge's model keeps the verdict kind until lilleman rules on the offline classifier, which decided finished with no missed interrupt on 1,116 archived verdicts but missed one on the first 1,104
2026-11-18, claude. Serves goals 1.2 and 1.4, and applies project principle 1: a cost cut waits for a measurement that shows quality holds.
`bin/judge-replay claude-opus-5-5 --offline` replays all archived verdicts of the Opus 5.5 projects through three deciders. Each reads only the worker's final message, doesn't send anything to an outside service, and decides a kind or leaves the verdict to the judge's model. A missed interrupt is an archived interrupt decided as another kind, which lets the worker settle a decision that is lilleman's. Deciding interrupt can't miss one: a wrong interrupt only asks lilleman more. On 2026-11-17 the archive held 1,104 verdicts (660 finished, 333 interrupt, 33 quoted answer, 78 unfinished); on 2026-11-18 it held 1,116 (668, 336, 34, 78).
- **A "?" rule** (interrupt where the message holds a "?", finished where it holds none) decided every one of the 1,104 and missed 197 interrupts.
- **A decision table**, the rows of `DECISION_TABLE` in `bin/judge_replay/offline.py`. Its rows were read from this archive's interrupts, so its run isn't a held-out test. It decided 974 of the 1,104 and missed 55. Most of its misses are a worker reporting a merged chunk while a release is ready to cut, which depends on the plan; the rest ask for an answer in words no row names.
- **A classifier**, a logistic regression over the message's words, trained on the archive. It decides finished only where its chance of an interrupt is below that of every interrupt among its training folds when held out, and interrupt only where it is above that of every other kind. It averages five regressions that differ only in the order they shuffle the samples, since a single shuffle moved its missed interrupts between 0 and 2 on the same archive. Its bar is no missed interrupt on both tests: held-out folds, each tested by a model trained on the other folds, and the latest fifth of the verdicts, learning from the rest.
- On the 1,104 verdicts, held-out folds: 103 decided, none missed. Latest fifth: 29 decided, all as finished, and 1 interrupt missed. That fails the bar, and shuffles 5 to 9 and 10 to 14 failed it alike.
- On the 1,116 verdicts, held-out folds: 95 decided (84 finished, 11 interrupt), none missed. Latest fifth: 20 decided, all as finished, none missed. That meets the bar for finished.
The classifier decides about 1 in 12 verdicts, and about 1 in 100 of what it decides were an interrupt on the 1,104. A latest fifth that starts a few verdicts later no longer holds the one it missed, so the bar flips with the archive's size and doesn't separate a classifier that never misses from one that rarely does. The judge's model keeps the verdict kind, the escalation it quotes and whether a release is ready to cut until lilleman rules whether a miss rate of about 1 in 100 decided is close enough to none. The plan holds that question as an item marked `question`.
Valid while the archived verdicts hold interrupts that read like a finished chunk. Replay with `judge-replay <model> --offline` when the archive has doubled, or when a decider reads more than the final message.
## routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds
2026-10-14, lilleman. Serves goal 1.4.
+44
View File
@@ -2596,3 +2596,47 @@ Read most (opens and searches):
- `docs/decisions.md` 6
- `skills/larv-review/SKILL.md` 5
## Reading cost, chunk 3cc43a838262, finished 2026-11-18T09:22:31Z
57869 new input tokens before the first commit.
Read most (opens and searches):
- `CLAUDE.md` 5
- `skills/larv-review/SKILL.md` 4
- `docs/decisions.md` 1
## Reading cost, chunk e880525b42ae, finished 2026-11-18T09:56:42Z
89443 new input tokens before the first commit.
Read most (opens and searches):
- `README.md` 6
- `bin/plan_store/jira_epic.py` 6
- `bin/plan_store/test_jira_epic.py` 5
- `.gitignore` 4
- `bin/atlassian/client.py` 3
- `bin/plan_store/contract.py` 2
- `docs/decisions.md` 2
- `bin/adf_codec/convert.py` 1
- `bin/do_epic/backend/contract.py` 1
- `bin/do_epic/backend/protocol.py` 1
## Reading cost, chunk 3312afb8e236, finished 2026-11-18T11:56:17Z
502050 new input tokens before the first commit.
Read most (opens and searches):
- `README.md` 14
- `bin/project_config/settings.py` 13
- `docs/decisions.md` 10
- `bin/project_config/main.py` 9
- `.gitignore` 7
- `skills/larv-review/SKILL.md` 7
- `bin/project_config/test_project_config.py` 6
- `bin/model_routing/ladder.py` 3
- `ci.sh` 3
- `docs/reading-cost.md` 3
+18 -23
View File
@@ -6,17 +6,16 @@
`Bar = 0.1`
`Next ID = 423`
`Next ID = 424`
## Items
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 407 | 0.12.0 | | | **Find an epic repo's checkout and project config by the repo's name, clone a missing one from the host of the epic's other repos, and ask the human where unsure.** | 2 | 2 | 2 | 4 | 3 | 1 by 5 | 1.9 |
| 401 | 0.12.0 | | | **Move loop-claude's cycle, judge, interrupt, tmux and routing into a package a second entry point drives, with loop-claude's behaviour unchanged.** | 3 | 2 | 3 | 6 | 6 | 1 by 3, code health by 2 | 1.8 |
| 406 | 0.12.0 | | | **Pick each epic item's repo, or for an item without one whether it is Claude's work or a human task, by `repo-rule.md`, record the pick on the issue, and ask the human where the rule leaves it unclear.** | 2 | 2 | 2 | 4 | 4 | 1 by 5 | 1.6 |
| 408 | 0.12.0 | | | **Rank an epic's items by one weight chain, the epic's achieve goals, then code health, then the item's repo goals, then by Blocked by, and re-rank Jira's backlog to match, with people's issues keeping their rank.** | 3 | 2 | 2 | 5 | 4 | 1 by 5 | 1.6 |
| 410 | 0.12.0 | 401, 406, 407, 408 | | **Run `do-epic <EPIC-KEY>`: one tmux session per epic, a window per chunk, up to the epic's cap of running chunks and each repo's `parallel-chunks`, and never in a repo a loop-claude is running chunks in.** | 3 | 2 | 3 | 6 | 6 | 1 by 5 | 0.8 |
| 410 | 0.12.0 | 401, 406, 408 | | **Run `do-epic <EPIC-KEY>`: one tmux session per epic, a window per chunk, up to the epic's cap of running chunks and each repo's `parallel-chunks`, and never in a repo a loop-claude is running chunks in.** | 3 | 2 | 3 | 6 | 6 | 1 by 5 | 0.8 |
| 416 | 0.12.0 | 410 | | **Finish an epic's repo item with a bullet under its repo's `CHANGELOG.md` → `## Unreleased`, or with none where the repo's plan is in Jira.** | 1 | 1 | 1 | 1 | 2 | 1 by 5 | 2.1 |
| 413 | 0.12.0 | 410 | | **File a finding about an epic goal in the epic and one about a repo goal by that repo's rules, ask the human where unsure which, and give the judge the epic's records beside the repo's.** | 3 | 2 | 3 | 6 | 4 | 1 by 5 | 1.6 |
| 415 | 0.12.0 | 410, 413 | | **Finish an epic: run release-review passes on each repo it changed, work what they file, have the human confirm each goal, post a summary comment, and close the epic in Jira last.** | 3 | 2 | 3 | 6 | 5 | 1 by 5 | 1.2 |
@@ -29,11 +28,11 @@
| 414 | 0.12.0 | 410 | | **DM the human through the notifier when a human-task issue blocks the epic's next item, and poll until it closes.** | 2 | 2 | 2 | 4 | 3 | 1 by 5 | 1.9 |
| 417 | 0.12.0 | 410 | | **Run do-epic end to end against a sandbox epic on a real Jira site with a command of its own, outside `run-tests`.** | 2 | 1 | 1 | 2 | 3 | 1 by 5 | 1.9 |
| 421 | 0.12.0 | | | **Refuse in rule-hook a Bash or Monitor wait that has no end: a loop without a time or count cap, and a `pgrep -f` or `pkill -f` whose pattern the same command line contains.** | 2 | 2 | 3 | 5 | 3 | 1.5 by 4 | 1.3 |
| 299 | 0.12.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
| 311 | 0.12.0 | | | **Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.0 |
| 316 | 0.12.0 | | | **Launch comprehension seats as an agent type of their own under `agents/`, without `CLAUDE.md` and the skill and agent listings a general-purpose subagent carries.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
| 310 | 0.12.0 | | | **Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 0.9 |
| 422 | 0.12.0 | 401, 420 | | **Check every 5 minutes whether a tmux window of the loop looks stuck by other signs than a lingering worker, investigate it with a headless claude, and kill only the process the investigation names.** | 3 | 2 | 3 | 6 | 5 | 1.5 by 4 | 0.8 |
| 423 | 0.13.0 | | question | **Skip the judge's model call on a worker's final message that the offline classifier decides is finished, once lilleman rules that its miss rate of about 1 in 100 decided is close enough to no missed interrupt.** | 3 | 3 | 3 | 7 | 5 | 1.4 by 2 | 0.4 |
| 365 | 0.13.0 | | | **Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
| 368 | 0.13.0 | | | **Call a release review due, call a release ready to cut, and tell whether a chunk owes a scoring run, for a plan outside the checkout, from the project repo's history or from Jira.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
| 309 | 0.13.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.2 |
@@ -84,16 +83,6 @@
## Details
### 407. Find an epic repo's checkout and project config by the repo's name, clone a missing one from the host of the epic's other repos, and ask the human where unsure.
Unsure means the epic's other repos sit on more than one host, or the clone fails (`docs/decisions.md` → `do-epic: drives one Jira epic …`). A by-name lookup goes beside `held` and `load`.
Touches:
- `bin/project_config/settings.py`: `held` (small)
- `bin/forge/commands.py`: `forge_of` (small)
- `bin/do_epic/repos.py`
### 401. Move loop-claude's cycle, judge, interrupt, tmux and routing into a package a second entry point drives, with loop-claude's behaviour unchanged.
do-epic runs on these, not on a copy (`docs/decisions.md` → `do-epic: drives one Jira epic …`). Move the reusable modules into `bin/agent_cycle/`: `tmux`, `judge`, `interrupt`, `verdicts`, `retry`, `spend`, `worker`, `states`, `stop`, and the parts of `routing` that take a `ProjectRung` and a `Route`. Keep `routed_item`, `loop/`, `project/`, `escalation_mark` and `chunk_items` in `loop_claude`. Decouple `cycle/run.py`'s archive and `Chunk items:` handling, and `cycle/prompts.py`'s goals-file and `todo.md` names, behind parameters. The places the comprehension seats found hard here, `routing.py` `unfinished_route` and `cycle/judge.py` `quotes_checked`, must read no harder after the move.
@@ -289,15 +278,6 @@ Touches:
- `bin/rule_hook/test_rule_hook.py`: the new rule's tests (small)
- `settings.json`: the PreToolUse matcher (small)
### 299. Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.
The judge's LLM still decides three things: the verdict kind, the escalation it quotes, and whether a release is ready to cut (`docs/decisions.md` → `routing: loop-claude reads the items a chunk takes …`). This item replays the verdict kind. No single rule found so far separates interrupts: a "?" in the worker's final message appears in 48 interrupts and misses 66. A smaller LLM already failed the replay (`docs/decisions.md` → `routing: on a model judge-replay covered, …`: Sonnet returned another kind on 3 interrupts, Haiku missed 7). Untested are combined code rules, a decision table, and a small classifier trained and run on this machine on the archived verdicts (372 finished, 234 interrupt, 42 unfinished, 20 quoted). Run each offline, nothing sent to an outside service (goal 4), and count disagreements by kind; the bar is zero missed interrupts. A hosted decision model is out of scope, since it would send transcripts to an outside service.
Touches:
- `bin/judge_replay/main.py`: `main`, `archived_cases` (small)
- `bin/judge_replay/replay.py`: `Case`, `verdict_kind`, `report`
- `bin/judge_replay/test_judge_replay.py`
### 311. Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.
Workers took 7,870 Bash results over 3,000 characters, $305 of carried context, against the reading rule in their prompt; stability reviewers another $145. `BASH_MAX_OUTPUT_LENGTH` (or `bashOutputMaxChars`) makes Claude Code save a longer result to a file and hand the model its path and the first 2,000 characters. loop-claude can hand it as it hands `CLAUDE_CODE_AUTO_COMPACT_WINDOW`. A ceiling of 6,000 characters would have removed $228 of carried output, 10,000 characters $122. Reads of old session transcripts cost $46; a rule-hook rule can refuse them, but not a Read of the files Claude Code saves a long Bash result to: they sit under `~/.claude/projects/<project>/<session>/tool-results/`, the archive holds 85 of them (item 312's round, 2026-11-12), and the saved-to-file ceiling hands the model their path to read.
@@ -360,6 +340,21 @@ Touches:
- `bin/loop_claude/tmux.py`: `Listed`, `listed`
- `README.md`: loop-claude's section (small)
### 423. Skip the judge's model call on a worker's final message that the offline classifier decides is finished, once lilleman rules that its miss rate of about 1 in 100 decided is close enough to no missed interrupt.
Question for lilleman: is a miss rate of about 1 in 100 decided close enough to none? A missed interrupt lets a worker settle a decision that is yours, unasked. Until you rule, the judge's model keeps the verdict kind (`docs/decisions.md` → `routing: the judge's model keeps the verdict kind until lilleman rules on the offline classifier …`).
Taking the item builds this: before the judge call, loop-claude runs the classifier trained on its archive over the worker's final message. Where the classifier decides finished, loop-claude skips the call. The judge's model keeps every verdict the classifier leaves, and the escalation it quotes and whether a release is ready to cut. The classifier is re-trained as the archive grows. The judge call sits in `bin/loop_claude/cycle/run.py` → `rule`, a place the comprehension seats found hard, since the order of stamping, archiving and stepping matters across three functions: make it easier to read.
Evidence, from `judge-replay claude-opus-5-5 --offline`: the classifier reads only the final message, averages five seeded logistic regressions, and decides finished only where its chance of an interrupt is below that of every interrupt among its held-out training folds.
- On the 1,116 archived verdicts it decided finished with no missed interrupt on both tests: held-out folds, 95 decided and none missed; latest fifth, 20 decided and none missed.
- On the first 1,104 it missed one interrupt on the latest fifth, while the held-out folds decided 103 and missed none.
- It decides about 1 in 12 verdicts. About 1 in 100 of those it decided were an interrupt, on the 1,104.
Savings: one judge call per about 12 verdicts. That is small, and nobody has measured it.
Fallbacks: decline, and close this item. Or first tighten the bar in `judge-replay`, so it separates a classifier that never misses from one that rarely does, and replay again.
### 365. Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.
lilleman's call, 2026-11-12: 0.10.0 and 0.11.0 switch the history tools off for a plan outside the checkout, and 0.13.0 holds only their ports. In Jira a dropped item, which `Drop item` commit titles mark today, is an issue resolved by a resolution the config names.