Merge remote-tracking branch 'origin/main' into item-406
# Conflicts: # README.md # docs/reading-cost.md
This commit is contained in:
@@ -1,6 +1,12 @@
|
||||
name: CI
|
||||
on: pull_request
|
||||
|
||||
# A push to a pull request cancels that pull request's older runs; the merge gate reads only the head commit's status.
|
||||
# The group is the ref: refs/pull/<number>/head for a pull request, the branch for a push, so no two refs cancel each other's runs.
|
||||
concurrency:
|
||||
group: ci-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
tests:
|
||||
runs-on: docker-host
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- judge-replay <model> --offline replays every archived verdict through a "?" rule, a decision table and a classifier trained on the archive, with no judge call, and counts the interrupts each misses.
|
||||
- do-epic finds an epic repo's checkout and project config by the repo's name, and clones a missing checkout from the host and owner of the epic's other repos, asking the human where they differ or the clone fails.
|
||||
- do-epic reads its machine-wide settings from ~/.claude-projects/do-epic/config.toml and its repo rule from repo-rule.md beside it, and refuses to start while either is missing or misspelt.
|
||||
- `project-record save` now writes a goals file kept as a Jira epic's description, except a change to `## Goals` or `## Audience`, which stays an edit in Jira.
|
||||
- Add do-epic's epic backend: one interface that reads a Jira epic's child issues and their sub-tasks, with labels, assignee, status and parent, and files new ones, met by a Jira backend and an in-memory fake that pass the same contract tests.
|
||||
|
||||
@@ -181,7 +181,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `command-patterns [<version>] [--calls <file>]` prints a table of the shell command patterns a release's archived chunks ran, such as `tea pr create`, for an audit of the release, and with `--calls <file>` saves each call as a JSON line; `command-patterns --help` states the columns and the line's keys.
|
||||
- `compress-archives` gzips each transcript in loop-claude's archive and each file in the review folders 30 days after it last changed, keeping all of them; Install's crontab line runs it daily.
|
||||
- `deploy-release [--cut] [<checkout>]` moves `~/.claude`, or the checkout named, to the newest release, as Update says; with `--cut`, run in the development clone, it first cuts the next release, as Release says.
|
||||
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints what each rung a review climbs to misses, and the lowest rung below them that, like each candidate above it, returns them; `judge-replay --help` states the replay.
|
||||
- `judge-replay <model> [--offline]` replays loop-claude's judge on archived verdicts of the projects run on the model. It prints what each rung a review climbs to misses, and the lowest rung below those that returns every verdict, as each candidate above it does. With `--offline` it doesn't call a judge: it counts what a "?" rule, a decision table and a classifier trained on the archive decide of each verdict's kind from the worker's final message, and how many interrupts each misses. `judge-replay --help` states both replays.
|
||||
- `larv-review <command>` holds larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's round count and the opening of its round commit's title, the scoring run's trigger and setup; the comprehension history's format, its latest places and its count of how often seats agree; and each reviewer's prompt header. `larv-review --help` states them. `settings.json` → `hooks` runs `larv-review reviewer-context` as each subagent starts. A reviewer under `agents/` whose frontmatter has `omitClaudeMd: true` starts without `CLAUDE.md`, and the command hands it that file's "Shared goal", "Prose" and "Vocabulary" sections.
|
||||
- `loop-claude <task>` runs a project's next chunks, each in its own tmux window, cycle after cycle; `loop-claude --help` states its contract.
|
||||
- `markdown-section <file> <fragment>` prints the section of a markdown file under the first heading that holds the fragment, to the next heading of its level or a higher one, instead of a `grep -n` and a guessed `sed -n` range; `markdown-section --help` says what it does where several headings match.
|
||||
@@ -213,8 +213,8 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `branch_chunks/` cuts a default branch into chunks and finds, for each, the items it shipped and the later chunks that fixed its code, for `score_outcomes` and `routing_figures`. It imports `reading_cost`'s record path.
|
||||
- `command_lines/` reads a shell command: its simple commands, and a docker call's subcommand and memory limits, for `command_patterns`, `reading_cost`, `routing_figures`, `rule_hook` and `run_heavy`.
|
||||
- `deploy_release` imports `release_plan`'s `release_standing`.
|
||||
- `do_epic/` will hold the tool that drives one Jira epic. So far `config.py` reads its machine-wide settings, importing `project_config`'s settings and `model_routing`'s ladder file, and `epic_record.py` reads and writes an epic's description by section and language line, through `record_store`'s `EpicDescription`. `repo_pick.py` asks claude, by `repo-rule.md`, where an epic item is worked, and records the answer as a label on the issue. `backend/` reads and files the epic's issues and adds those labels.
|
||||
- `forge/` tells a GitHub origin from a Gitea one and runs that forge's CLI (`commands.py`), and finds a pull request and waits for its checks (`pulls.py`), for `chunk_preflight`, `chunk_pr`, `deploy_release` and `larv_review`.
|
||||
- `do_epic/` will hold the tool that drives one Jira epic. So far `config.py` reads its machine-wide settings, importing `project_config`'s settings and `model_routing`'s ladder file, and `epic_record.py` reads and writes an epic's description by section and language line, through `record_store`'s `EpicDescription`. `repos.py` finds a repo's checkout and project config by its name, and clones a missing checkout from the host and owner of the epic's other repos, through `forge`'s `url_beside`. `repo_pick.py` asks claude, by `repo-rule.md`, where an epic item is worked, and records the answer as a label on the issue. `backend/` reads and files the epic's issues and adds those labels.
|
||||
- `forge/` tells a GitHub origin from a Gitea one, names a repo beside an origin's and runs that forge's CLI (`commands.py`), and finds a pull request and waits for its checks (`pulls.py`), for `chunk_preflight`, `chunk_pr`, `deploy_release`, `do_epic` and `larv_review`.
|
||||
- `git_cli/` runs git for every tool (`git_repo.py`), and reads the git calls in a shell command (`git_command.py`).
|
||||
- `judge_replay` imports `loop_claude`'s judge, verdicts, premises and records.
|
||||
- `loop_claude` runs `chunk-preflight` as a process of its own.
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
"""Where an epic's repo is on this machine: its checkout and its project config, found by the repo's name.
|
||||
|
||||
A missing checkout is cloned from the host and owner of the epic's other repos (docs/decisions.md → `do-epic: drives one Jira epic …`). Where that is unclear, the answer is an Unsure."""
|
||||
import dataclasses
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
|
||||
from project_config.settings import by_name, CONFIG, ProjectConfig, PROJECTS
|
||||
from do_epic.config import DoEpicConfig
|
||||
from failure import Failure
|
||||
from git_cli.git_repo import Git
|
||||
from forge.commands import url_beside
|
||||
|
||||
# One folder name, which can't be a path or a git option.
|
||||
REPO_NAME = r"\w[\w.-]*"
|
||||
# The addresses a clone may use: https, ssh, or scp-style `user@host:path`. Not ext:: or file:, which run a program or read this machine's files, and an origin can spell either.
|
||||
CLONE_ADDRESS = rf"https://[\w.-]+(:\d+)?/\S+|ssh://[\w.@-]+(:\d+)?/\S+|\w[\w.-]*@[\w.-]+:[\w./~-]+"
|
||||
# A clone that waits longer is stopped.
|
||||
CLONE_SECONDS = 600
|
||||
# git's options that make a password or a host key fail at once instead of prompting: an unattended run never waits on a prompt (docs/decisions.md → principle 6).
|
||||
NO_PROMPTS = ("-c", "core.askPass=true", "-c", "core.sshCommand=ssh -o BatchMode=yes")
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Located:
|
||||
"""checkout is the repo's git checkout, and config the project config that names it."""
|
||||
checkout: pathlib.Path
|
||||
config: ProjectConfig
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Unsure(Failure):
|
||||
"""reason is a question for the human."""
|
||||
|
||||
|
||||
def locate(name: str, others: list[pathlib.Path], config: DoEpicConfig, home: pathlib.Path, git: Git) -> Located | Failure:
|
||||
"""The checkout and project config of the repo called name; others are the checkouts of the epic's other repos.
|
||||
|
||||
A project config names the repo's checkout, and it is cloned where missing. Where no config names the repo, it is cloned into the clone folder, if it isn't there, and the human is asked to write the config, since the repo's plan, goals and records are theirs to place.
|
||||
A name that is no repo's name, or a config that can't be read, is a plain Failure. Any other fault is an Unsure."""
|
||||
if not re.fullmatch(REPO_NAME, name):
|
||||
return Failure(f"{name!r} is not a repo's name: one folder name that starts with a letter, digit or underscore")
|
||||
found = by_name(name, home)
|
||||
if isinstance(found, Failure):
|
||||
return found
|
||||
checkout = found.top if found else config.clone_folder / name
|
||||
cloned = ensure(name, checkout, others, git)
|
||||
if isinstance(cloned, Failure):
|
||||
return cloned
|
||||
if found is None:
|
||||
return Unsure(f"{checkout} {'was cloned' if cloned else 'holds the repo'}, but no project config names it. Write {home / PROJECTS}/<key>/{CONFIG} with `checkout = \"{checkout}\"`, as `project-config --help` shows, and run do-epic again.")
|
||||
return Located(checkout, found)
|
||||
|
||||
|
||||
def ensure(name: str, checkout: pathlib.Path, others: list[pathlib.Path], git: Git) -> bool | Failure:
|
||||
"""Whether checkout was cloned just now: False where it is a git checkout already. Else it is cloned there from the host and owner of the others, or an Unsure says why not."""
|
||||
if (checkout / ".git").exists():
|
||||
return False
|
||||
if checkout.exists() and (not checkout.is_dir() or any(checkout.iterdir())):
|
||||
return Unsure(f"{checkout} exists and is not a git checkout, so {name} cannot be cloned there. Where should it be cloned?")
|
||||
url = clone_url(name, others, git)
|
||||
if isinstance(url, Failure):
|
||||
return url
|
||||
cloned = git(*NO_PROMPTS, "clone", "--", url, str(checkout), timeout=CLONE_SECONDS)
|
||||
if isinstance(cloned, Failure):
|
||||
# What the clone began, and nothing else: checkout was absent or empty. A kept half-clone would pass for the repo on the next run.
|
||||
shutil.rmtree(checkout, ignore_errors=True)
|
||||
return Unsure(f"cannot clone {url} to {checkout}: {cloned.reason}. Where does {name} live?")
|
||||
return True
|
||||
|
||||
|
||||
def without_credentials(url: str) -> str:
|
||||
"""url without a `user:token@` after its scheme, so that a clone and a question for the team never carry one. The clone then uses git's own credentials."""
|
||||
return re.sub(r"^(https?://)[^/]*@", r"\1", url)
|
||||
|
||||
|
||||
def clone_url(name: str, others: list[pathlib.Path], git: Git) -> str | Failure:
|
||||
"""The address of the repo called name, under the host and owner all the others share. An Unsure where they share none or an address can't be built."""
|
||||
if not others:
|
||||
return Unsure(f"{name} is missing and no other repo of the epic shows what host to clone it from. Where does it live?")
|
||||
urls = set()
|
||||
for other in others:
|
||||
origin = git("remote", "get-url", "origin", cwd=str(other))
|
||||
if isinstance(origin, Failure):
|
||||
return Unsure(f"cannot read the origin of {other}: {origin.reason}. Where does {name} live?")
|
||||
beside = url_beside(without_credentials(origin.strip()), name)
|
||||
if beside is None:
|
||||
return Unsure(f"{other} has an origin, {without_credentials(origin.strip())}, with no host and owner to put another repo under. Where does {name} live?")
|
||||
urls.add(beside)
|
||||
if len(urls) > 1:
|
||||
return Unsure(f"The epic's other repos sit under more than one host or owner, so {name} could be at {' or '.join(sorted(urls))}. Which holds it?")
|
||||
url = next(iter(urls))
|
||||
if not re.fullmatch(CLONE_ADDRESS, url):
|
||||
return Unsure(f"{url} is not an https, ssh or scp-style address. Where does {name} live?")
|
||||
return url
|
||||
@@ -0,0 +1,184 @@
|
||||
import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import unittest.mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from do_epic.config import DoEpicConfig
|
||||
from failure import Failure
|
||||
from do_epic.repos import locate, Located, Unsure
|
||||
from project_config.settings import ProjectConfig
|
||||
from project_config.test_project_config import write_config
|
||||
|
||||
|
||||
class FakeGit:
|
||||
"""git as locate calls it: origins maps a checkout to its origin URL, cloned holds each `clone` that worked, and refused names a URL whose clone fails after it made the checkout's `.git`, as a stopped git does."""
|
||||
def __init__(self, origins: dict[str, str | Failure], refused: str | None = None) -> None:
|
||||
self.origins, self.refused = origins, refused
|
||||
self.cloned: list[tuple[str, str]] = []
|
||||
self.clone_timeouts: list[float | None] = []
|
||||
self.clone_options: list[tuple[str, ...]] = []
|
||||
|
||||
def __call__(self, *args: str, cwd: str | None = None, timeout: float | None = None, guarded: bool = False, errors: str = "strict") -> str | Failure:
|
||||
if args[:3] == ("remote", "get-url", "origin"):
|
||||
return self.origins[cwd or ""]
|
||||
clone = args.index("clone")
|
||||
self.clone_options.append(args[:clone])
|
||||
assert args[clone + 1] == "--", args
|
||||
url, destination = args[clone + 2:clone + 4]
|
||||
pathlib.Path(destination, ".git").mkdir(parents=True)
|
||||
if url == self.refused:
|
||||
return Failure("git clone: repository not found")
|
||||
self.cloned.append((url, destination))
|
||||
self.clone_timeouts.append(timeout)
|
||||
return ""
|
||||
|
||||
|
||||
class LocateTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
made = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(made.cleanup)
|
||||
self.home = pathlib.Path(made.name).resolve()
|
||||
self.projects = self.home / ".claude-projects"
|
||||
self.config = DoEpicConfig(self.projects / "do-epic", "https://acme.atlassian.net", self.home / "jira.env", self.home / "clones", "English", "claude-opus-5-5", "rule")
|
||||
self.billing, self.payments = self.home / "elsewhere" / "billing", self.home / "elsewhere" / "payments"
|
||||
self.origins: dict[str, str | Failure] = {str(self.billing): "git@github.com:acme/billing.git", str(self.payments): "git@github.com:acme/payments"}
|
||||
|
||||
def configure(self) -> None:
|
||||
write_config(self.home, self.home / "code" / "ledger")
|
||||
|
||||
def locate(self, git: FakeGit, name: str = "ledger") -> Located | Failure:
|
||||
return locate(name, [self.billing, self.payments], self.config, self.home, git)
|
||||
|
||||
def unsure(self, git: FakeGit, name: str = "ledger") -> str:
|
||||
found = self.locate(git, name)
|
||||
assert isinstance(found, Unsure), found
|
||||
return found.reason
|
||||
|
||||
def test_finds_a_checkout_that_exists_with_its_config_and_clones_nothing(self) -> None:
|
||||
self.configure()
|
||||
(self.home / "code" / "ledger" / ".git").mkdir(parents=True)
|
||||
git = FakeGit(self.origins)
|
||||
found = self.locate(git)
|
||||
assert isinstance(found, Located), found
|
||||
self.assertEqual((found.checkout, found.config.folder, git.cloned), (self.home / "code" / "ledger", self.projects / "ledger", []))
|
||||
|
||||
def test_clones_a_checkout_the_config_names_where_it_is_missing_from_the_place_of_the_other_repos(self) -> None:
|
||||
self.configure()
|
||||
git = FakeGit(self.origins)
|
||||
found = self.locate(git)
|
||||
assert isinstance(found, Located), found
|
||||
self.assertEqual((found.checkout, git.cloned), (self.home / "code" / "ledger", [("git@github.com:acme/ledger", str(self.home / "code" / "ledger"))]))
|
||||
assert isinstance(found.config, ProjectConfig)
|
||||
|
||||
def test_clones_into_the_clone_folder_and_asks_for_a_config_where_none_names_the_repo(self) -> None:
|
||||
git = FakeGit(self.origins)
|
||||
asked = self.unsure(git)
|
||||
clone = self.home / "clones" / "ledger"
|
||||
self.assertEqual(git.cloned, [("git@github.com:acme/ledger", str(clone))])
|
||||
self.assertIn(f"{clone} was cloned, but no project config names it", asked)
|
||||
self.assertIn(f"{self.projects}/<key>/config.toml with `checkout = \"{clone}\"`", asked)
|
||||
self.assertEqual(self.unsure(git), asked.replace("was cloned", "holds the repo"), "the second ask finds the clone and clones nothing")
|
||||
self.assertEqual(len(git.cloned), 1)
|
||||
|
||||
def test_asks_where_the_other_repos_sit_in_more_than_one_place(self) -> None:
|
||||
self.configure()
|
||||
self.origins[str(self.payments)] = "git@gitea.larvit.se:acme/payments"
|
||||
git = FakeGit(self.origins)
|
||||
asked = self.unsure(git)
|
||||
self.assertIn("git@gitea.larvit.se:acme/ledger or git@github.com:acme/ledger", asked)
|
||||
self.assertEqual(git.cloned, [])
|
||||
|
||||
def test_asks_where_the_other_repos_differ_only_by_owner(self) -> None:
|
||||
self.configure()
|
||||
self.origins[str(self.payments)] = "git@github.com:other/payments"
|
||||
self.assertIn("more than one host or owner", self.unsure(FakeGit(self.origins)))
|
||||
|
||||
def test_asks_where_no_other_repo_shows_where_to_clone_from(self) -> None:
|
||||
self.configure()
|
||||
found = locate("ledger", [], self.config, self.home, FakeGit({}))
|
||||
assert isinstance(found, Unsure), found
|
||||
self.assertIn("no other repo of the epic", found.reason)
|
||||
|
||||
def test_asks_where_an_others_origin_cannot_be_read_or_has_no_place(self) -> None:
|
||||
self.configure()
|
||||
self.origins[str(self.billing)] = Failure("git remote: no such remote 'origin'")
|
||||
self.assertIn(f"cannot read the origin of {self.billing}: git remote: no such remote 'origin'", self.unsure(FakeGit(self.origins)))
|
||||
self.origins[str(self.billing)] = "billing"
|
||||
self.assertIn(f"{self.billing} has an origin, billing, with no host and owner to put another repo under", self.unsure(FakeGit(self.origins)))
|
||||
|
||||
def test_asks_all_the_same_where_the_leftovers_of_a_failed_clone_cannot_be_removed(self) -> None:
|
||||
self.configure()
|
||||
with unittest.mock.patch("do_epic.repos.shutil.rmtree", side_effect=lambda path, ignore_errors: None if ignore_errors is True else self.fail("rmtree must ignore errors")):
|
||||
self.assertIn("cannot clone", self.unsure(FakeGit(self.origins, refused="git@github.com:acme/ledger")))
|
||||
|
||||
def test_asks_where_the_clone_fails_naming_the_url_and_the_reason(self) -> None:
|
||||
self.configure()
|
||||
asked = self.unsure(FakeGit(self.origins, refused="git@github.com:acme/ledger"))
|
||||
self.assertIn(f"cannot clone git@github.com:acme/ledger to {self.home / 'code' / 'ledger'}: git clone: repository not found", asked)
|
||||
|
||||
def test_asks_where_the_place_to_clone_into_holds_something_that_is_not_a_checkout(self) -> None:
|
||||
self.configure()
|
||||
(self.home / "code" / "ledger").mkdir(parents=True)
|
||||
(self.home / "code" / "ledger" / "notes.txt").write_text("x", encoding="utf-8")
|
||||
git = FakeGit(self.origins)
|
||||
self.assertIn(f"{self.home / 'code' / 'ledger'} exists and is not a git checkout", self.unsure(git))
|
||||
self.assertEqual(git.cloned, [])
|
||||
|
||||
def test_refuses_a_name_that_is_not_a_repo_name_before_it_forms_a_path_or_a_command(self) -> None:
|
||||
for name in ("../escape", "-flag", "a/b", ""):
|
||||
with self.subTest(name=name):
|
||||
found = self.locate(FakeGit(self.origins), name)
|
||||
assert isinstance(found, Failure) and not isinstance(found, Unsure), found
|
||||
self.assertIn("is not a repo's name", found.reason)
|
||||
|
||||
def test_refuses_an_origin_that_is_not_https_ssh_or_an_scp_style_address(self) -> None:
|
||||
self.configure()
|
||||
for origin in ("ext::sh -c touch% x/billing", "file:///srv/acme/billing", "/srv/acme/billing", "-oProxy=x:a/b"):
|
||||
with self.subTest(origin=origin):
|
||||
self.origins[str(self.billing)] = origin
|
||||
self.origins[str(self.payments)] = origin
|
||||
git = FakeGit(self.origins)
|
||||
self.assertIn("is not an https, ssh or scp-style address", self.unsure(git))
|
||||
self.assertEqual(git.cloned, [])
|
||||
|
||||
def test_clones_and_asks_without_the_credentials_an_origin_holds(self) -> None:
|
||||
self.configure()
|
||||
self.origins[str(self.billing)] = "https://bot:s3cret@gitea.larvit.se/acme/billing"
|
||||
self.origins[str(self.payments)] = "https://other:t0@ken@gitea.larvit.se/acme/payments.git"
|
||||
git = FakeGit(self.origins)
|
||||
self.assertIsInstance(self.locate(git), Located)
|
||||
self.assertEqual(git.cloned, [("https://gitea.larvit.se/acme/ledger", str(self.home / "code" / "ledger"))])
|
||||
self.origins[str(self.payments)] = "https://other:t0@ken@gitea.larvit.se/acme/payments"
|
||||
self.origins[str(self.billing)] = "https://bot:s3cret@gitea.larvit.se"
|
||||
self.assertNotIn("s3cret", self.unsure(FakeGit(self.origins), "stock"))
|
||||
|
||||
def test_stops_a_clone_that_waits_and_makes_a_prompt_fail(self) -> None:
|
||||
self.configure()
|
||||
git = FakeGit(self.origins)
|
||||
self.locate(git)
|
||||
self.assertEqual(git.clone_timeouts, [600])
|
||||
self.assertEqual(git.clone_options, [("-c", "core.askPass=true", "-c", "core.sshCommand=ssh -o BatchMode=yes")])
|
||||
|
||||
def test_removes_what_a_failed_clone_began_so_that_the_next_run_clones_again(self) -> None:
|
||||
self.configure()
|
||||
failing = FakeGit(self.origins, refused="git@github.com:acme/ledger")
|
||||
self.unsure(failing)
|
||||
self.assertFalse((self.home / "code" / "ledger").exists())
|
||||
found = self.locate(FakeGit(self.origins))
|
||||
self.assertIsInstance(found, Located)
|
||||
|
||||
def test_asks_where_the_place_to_clone_into_is_a_file(self) -> None:
|
||||
self.configure()
|
||||
(self.home / "code").mkdir()
|
||||
(self.home / "code" / "ledger").write_text("x", encoding="utf-8")
|
||||
self.assertIn("exists and is not a git checkout", self.unsure(FakeGit(self.origins)))
|
||||
|
||||
def test_fails_with_the_config_failure_where_a_config_cannot_be_read(self) -> None:
|
||||
(self.projects / "broken").mkdir(parents=True)
|
||||
(self.projects / "broken" / "config.toml").write_text("this is not toml", encoding="utf-8")
|
||||
found = self.locate(FakeGit(self.origins))
|
||||
self.assertIsInstance(found, Failure)
|
||||
self.assertNotIsInstance(found, Unsure)
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Which forge a checkout's origin is on, and running that forge's CLI in the checkout."""
|
||||
"""Which forge a checkout's origin is on, a repo's URL beside that origin's, and running the forge's CLI in the checkout."""
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import typing
|
||||
|
||||
@@ -16,6 +17,13 @@ def forge_of(origin_url: str) -> Forge:
|
||||
return "gh" if "github.com" in origin_url else "tea"
|
||||
|
||||
|
||||
def url_beside(origin_url: str, name: str) -> str | None:
|
||||
"""The origin URL of the repo called name under the same host and owner as origin_url's repo; None where origin_url has no place, such as a bare name. The URL has no `.git`, which git clones without."""
|
||||
place = re.fullmatch(r"(.+[/:])[^/:]+", origin_url.rstrip("/"))
|
||||
# An origin of only a scheme and a host, such as `https://host`, has no owner folder to put a name in.
|
||||
return place.group(1) + name if place and not place.group(1).endswith("//") else None
|
||||
|
||||
|
||||
def runner(cwd: str) -> Run:
|
||||
def run(command: list[str]) -> str | Failure:
|
||||
try:
|
||||
|
||||
+14
-1
@@ -7,7 +7,7 @@ import unittest
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from forge.pulls import checks_passed, open_pull, POLL_SECONDS, required_checks, WAIT_SECONDS
|
||||
from forge.commands import forge_of, gitea
|
||||
from forge.commands import forge_of, gitea, url_beside
|
||||
|
||||
|
||||
class ForgeOf(unittest.TestCase):
|
||||
@@ -16,6 +16,19 @@ class ForgeOf(unittest.TestCase):
|
||||
self.assertEqual(forge_of("https://gitea.larvit.se/o/r.git"), "tea")
|
||||
|
||||
|
||||
class UrlBeside(unittest.TestCase):
|
||||
def test_names_a_repo_in_the_same_place_of_the_same_host_in_each_spelling_of_an_origin(self):
|
||||
self.assertEqual(url_beside("git@github.com:acme/billing.git", "ledger"), "git@github.com:acme/ledger")
|
||||
self.assertEqual(url_beside("https://gitea.larvit.se/acme/billing", "ledger"), "https://gitea.larvit.se/acme/ledger")
|
||||
self.assertEqual(url_beside("https://gitea.larvit.se/acme/billing.git/", "ledger"), "https://gitea.larvit.se/acme/ledger")
|
||||
self.assertEqual(url_beside("gitea:billing", "ledger"), "gitea:ledger", "a host with no owner")
|
||||
|
||||
def test_names_none_where_the_origin_has_no_place_to_put_a_repo_in(self):
|
||||
self.assertIsNone(url_beside("billing", "ledger"))
|
||||
self.assertIsNone(url_beside("", "ledger"))
|
||||
self.assertIsNone(url_beside("https://gitea.larvit.se", "ledger"))
|
||||
|
||||
|
||||
# An answer no call can read, longer than the 200 characters of it that a failure quotes.
|
||||
LONG = "[" + ", ".join(["1"] * 100) + "]"
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""Replay loop-claude's judge on archived verdicts, to find the lowest rung that returns them: docs/decisions.md → `routing: on a model judge-replay covered`.
|
||||
|
||||
usage: judge-replay <model>
|
||||
usage: judge-replay <model> [--offline]
|
||||
|
||||
<model> is a project model on the routing ladder this setup ships, such as claude-opus-5-5. The tool replays the
|
||||
verdicts archived for the projects that ~/.claude/loop-claude-models.json runs on it: the latest
|
||||
@@ -36,8 +36,21 @@ Like the loop, it tries a judge three times running before it counts the reply a
|
||||
verdict whose project, transcript or archived judge run cannot be read replays as failed too. Exit 1, saying why, on
|
||||
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
|
||||
or a usage limit.
|
||||
|
||||
With --offline it doesn't call a judge, so the runs, retries and usage limit above don't apply. It reads every
|
||||
archived verdict of the model's projects, not a sample. It counts how many of them each of three code
|
||||
deciders decides from the worker's final message, and how many archived interrupts it decides as another
|
||||
kind, which are its missed interrupts: a "?" rule, a decision table, and a classifier trained on the archive
|
||||
(judge_replay/offline.py). The rule and the table run on the whole archive; the table's rows were read from
|
||||
it, so its run isn't a held-out test. The classifier runs on held-out folds, each tested by a model trained on
|
||||
the other folds, and on the latest fifth of the verdicts, tested by a model trained on the earlier four
|
||||
fifths. The classifier trains 330 regressions in Python, about 12 minutes on 1,100 verdicts. The report is
|
||||
kept in replays/judge-replay-<model>-offline-<UTC stamp>/result.md. A verdict whose transcript cannot be read is skipped, and the report counts the skipped. Exit 1, saying why, on bad
|
||||
arguments, an unreadable archive or models file, no archived verdict of the model's projects, or none with a
|
||||
transcript that can be read.
|
||||
"""
|
||||
import pathlib
|
||||
import sys
|
||||
import typing
|
||||
|
||||
from transcript_archive.location import archive_root, transcript_of
|
||||
@@ -49,6 +62,7 @@ from judge_replay.replay import Case, case_from, Kind, report, reproduced, sampl
|
||||
from transcript_archive.project_key import checkout_of
|
||||
from loop_effects.effects import Effects
|
||||
from failure import Failure
|
||||
from judge_replay.offline import final_message, LEARNERS, offline_report, RULES, Sample
|
||||
from rung_replay.search import first_reply, replayed
|
||||
from judge_replay.archived_prompt import grounds_of
|
||||
from transcript_archive.judge_run import JudgeRun
|
||||
@@ -61,7 +75,8 @@ from claude_code.transcript import read_transcript
|
||||
from loop_claude.verdicts import Verdict
|
||||
|
||||
TOOL = "judge-replay"
|
||||
USAGE = f"{TOOL} <model>, a model on the routing ladder"
|
||||
USAGE = f"{TOOL} <model> [--offline]; <model> is a model on the routing ladder"
|
||||
OFFLINE = "--offline"
|
||||
PER_KIND = 15
|
||||
# The effort settings.json gave every judge before loop-claude routed it; the archived verdicts came from it.
|
||||
REFERENCE_EFFORT = "xhigh"
|
||||
@@ -72,7 +87,7 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
ladder = read_ladder(SHIPPED)
|
||||
if isinstance(ladder, Failure):
|
||||
return failed(TOOL, ladder.reason)
|
||||
project = arguments_or_exit(TOOL, argv, __doc__, USAGE, lambda argv: model_argument(ladder, argv))
|
||||
project = arguments_or_exit(TOOL, argv, __doc__, USAGE, lambda argv: model_argument(ladder, [arg for arg in argv if arg != OFFLINE]))
|
||||
if isinstance(project, int):
|
||||
return project
|
||||
cases = archived_cases(project.rung.model, archive_root(home, env), home)
|
||||
@@ -80,6 +95,8 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
return failed(TOOL, cases.reason)
|
||||
if not cases:
|
||||
return failed(TOOL, f"no archived verdicts of projects on {project.rung.model} in {archive_root(home, env)}; run loop-claude on a project on that model first")
|
||||
if OFFLINE in argv:
|
||||
return offline(project.rung.model, cases, archive_root(home, env), effects)
|
||||
cases = sample(cases, PER_KIND)
|
||||
folder = replay_folder(archive_root(home, env), TOOL, argv, effects.now())
|
||||
effects = archiving(effects, folder)
|
||||
@@ -92,6 +109,22 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
return 0
|
||||
|
||||
|
||||
def offline(model: str, cases: list[Case], root: pathlib.Path, effects: Effects) -> int:
|
||||
"""Replays the verdicts of cases offline, oldest first, and keeps the report in the archive as a judge replay does."""
|
||||
samples, skipped = [], 0
|
||||
for found in sorted(cases, key=lambda found: (found.at, found.root, str(found.transcript))):
|
||||
entries = read_transcript(str(found.transcript))
|
||||
if isinstance(entries, Failure):
|
||||
skipped += 1
|
||||
print(f"{TOOL}: skipped a verdict: {entries.reason}", file=sys.stderr)
|
||||
else:
|
||||
samples.append(Sample(found.archived_kind, final_message(entries)))
|
||||
if not samples:
|
||||
return failed(TOOL, "no archived verdict has a transcript that can be read")
|
||||
print_report(replay_folder(root, TOOL, [model, OFFLINE], effects.now()), offline_report(model, samples, RULES, LEARNERS, skipped=skipped))
|
||||
return 0
|
||||
|
||||
|
||||
def model_argument(ladder: Ladder, argv: list[str]) -> ProjectRung | None:
|
||||
"""The project's rung on ladder of the one model argv names; None for any other argv."""
|
||||
return ladder.project_rung(argv[0]) if len(argv) == 1 else None
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
"""Deciders of a verdict's kind that run offline on the worker's final message, replayed on the archive: docs/decisions.md → `routing: the offline classifier may decide finished, …`."""
|
||||
import collections
|
||||
import math
|
||||
import random
|
||||
import re
|
||||
import typing
|
||||
|
||||
from claude_code.entries import AssistantTurn, Entry, Text
|
||||
from transcript_archive.outcomes import FINISHED, INTERRUPTED, UNFINISHED
|
||||
from judge_replay.replay import Kind
|
||||
|
||||
|
||||
class Sample(typing.NamedTuple):
|
||||
"""What a decider reads of one archived verdict: the kind the archive gives it and the worker's final message."""
|
||||
kind: Kind
|
||||
text: str
|
||||
|
||||
|
||||
# A decider names a kind, or None to leave the verdict to the judge's model. A method builds one from the samples it may learn from.
|
||||
Decider = typing.Callable[[str], Kind | None]
|
||||
Method = typing.Callable[[typing.Sequence[Sample]], Decider]
|
||||
Counts = collections.Counter[tuple[Kind, Kind | None]]
|
||||
|
||||
FOLDS = 10
|
||||
INNER_FOLDS = 5
|
||||
# A word in fewer messages than this is left out of the classifier.
|
||||
MIN_MESSAGES = 3
|
||||
EPOCHS = 8
|
||||
# The classifier averages this many regressions, each shuffling the samples in its own order: one shuffle alone moved the missed interrupts of a run between 0 and 2.
|
||||
SEEDS = 5
|
||||
RATE = 0.05
|
||||
DECAY = 1e-3
|
||||
# The latest 1/LATEST of the archive is the second test: the verdicts a decider would meet next.
|
||||
LATEST = 5
|
||||
|
||||
# The first row whose pattern matches the final message decides it; a message no row matches is left to the judge. The rows come from reading the archive's interrupts, so a run of the table on the whole archive is not a held-out test.
|
||||
DECISION_TABLE = (
|
||||
(INTERRUPTED, re.compile(r"(?i)\?|recommend|your call|needs? you|tell me|you answer|please|unmerged|not merged|do you|should i|i need|need your|you say|decisions? for|stays until")),
|
||||
(FINISHED, re.compile(r"(?i)\bmerged\b(?!.*ready to cut)")),
|
||||
)
|
||||
|
||||
|
||||
def final_message(entries: list[Entry]) -> str:
|
||||
"""The text of the worker's last turn that has any, "" where it wrote none."""
|
||||
for entry in reversed(entries):
|
||||
if isinstance(entry, AssistantTurn):
|
||||
text = "\n".join(part.text for part in entry.parts if isinstance(part, Text) and part.text.strip())
|
||||
if text:
|
||||
return text
|
||||
return ""
|
||||
|
||||
|
||||
def question_mark(samples: typing.Sequence[Sample]) -> Decider:
|
||||
"""The one rule: an interrupt where the message holds a "?", finished where it holds none."""
|
||||
return lambda text: INTERRUPTED if "?" in text else FINISHED
|
||||
|
||||
|
||||
def decision_table(samples: typing.Sequence[Sample]) -> Decider:
|
||||
return lambda text: next((kind for kind, pattern in DECISION_TABLE if pattern.search(text)), None)
|
||||
|
||||
|
||||
def words_of(text: str) -> frozenset[str]:
|
||||
"""Each word and each pair of neighbouring words of the message, and its question marks."""
|
||||
words = re.findall(r"[a-z0-9']+|\?", text.lower())
|
||||
return frozenset(words) | {f"{first} {second}" for first, second in zip(words, words[1:])}
|
||||
|
||||
|
||||
def logistic(samples: typing.Sequence[Sample], seed: int) -> typing.Callable[[str], float]:
|
||||
"""The chance that a message is an interrupt, from a logistic regression over its words that the samples train, shuffled in the order seed gives."""
|
||||
seen = collections.Counter(word for found in samples for word in words_of(found.text))
|
||||
vocabulary = {word for word, count in seen.items() if count >= MIN_MESSAGES}
|
||||
data = [(words_of(found.text) & vocabulary, 1.0 if found.kind == INTERRUPTED else 0.0) for found in samples]
|
||||
weights: dict[str, float] = collections.defaultdict(float)
|
||||
bias = 0.0
|
||||
shuffler = random.Random(seed)
|
||||
|
||||
def chance(words: typing.Iterable[str]) -> float:
|
||||
return 1 / (1 + math.exp(-max(-30.0, min(30.0, bias + sum(weights[word] for word in words)))))
|
||||
|
||||
for _ in range(EPOCHS):
|
||||
shuffler.shuffle(data)
|
||||
for words, interrupt in data:
|
||||
error = chance(words) - interrupt
|
||||
bias -= RATE * error
|
||||
for word in words:
|
||||
weights[word] -= RATE * (error + DECAY * weights[word])
|
||||
return lambda text: chance(words_of(text) & vocabulary)
|
||||
|
||||
|
||||
def averaged(samples: typing.Sequence[Sample]) -> typing.Callable[[str], float]:
|
||||
"""The mean chance of SEEDS regressions that differ only in the order they shuffle the samples."""
|
||||
regressions = [logistic(samples, seed) for seed in range(SEEDS)]
|
||||
return lambda text: sum(regression(text) for regression in regressions) / SEEDS
|
||||
|
||||
|
||||
def classifier(samples: typing.Sequence[Sample]) -> Decider:
|
||||
"""Finished where the chance of an interrupt is below one half and below that of every interrupt among the samples; interrupt where it is at least one half and above that of every other kind; otherwise the judge's model decides.
|
||||
|
||||
A sample's chance is the one a model trained on the other folds of the samples gives it, so the bounds don't rest on how well the model fits what it trained on. A bound is where the samples held no miss; that doesn't promise the next verdict holds none."""
|
||||
interrupts = [found for found in samples if found.kind == INTERRUPTED]
|
||||
if not interrupts or len(interrupts) == len(samples):
|
||||
return lambda text: None
|
||||
held_out = [(tested.kind, chance) for trained, tests in folds_of(samples, INNER_FOLDS) for chance, tested in zip(map(averaged(trained), (found.text for found in tests)), tests)]
|
||||
below = min(chance for kind, chance in held_out if kind == INTERRUPTED)
|
||||
above = max(chance for kind, chance in held_out if kind != INTERRUPTED)
|
||||
chance = averaged(samples)
|
||||
|
||||
def decide(text: str) -> Kind | None:
|
||||
found = chance(text)
|
||||
if found < below and found < 0.5:
|
||||
return FINISHED
|
||||
return INTERRUPTED if found > above and found >= 0.5 else None
|
||||
return decide
|
||||
|
||||
|
||||
# Rules are fixed, and run on the whole archive; a learner trains on samples, and runs on samples it didn't train on.
|
||||
RULES: dict[str, Method] = {'"?" rule': question_mark, "decision table": decision_table}
|
||||
LEARNERS: dict[str, Method] = {"classifier": classifier}
|
||||
|
||||
|
||||
def folds_of(samples: typing.Sequence[Sample], count: int) -> list[tuple[list[Sample], list[Sample]]]:
|
||||
"""count folds of the samples, dealt in turn. Each fold is the samples to learn from, which are all those its test leaves out, and the samples to test on."""
|
||||
return [([found for place, found in enumerate(samples) if place % count != fold], [found for place, found in enumerate(samples) if place % count == fold]) for fold in range(count)]
|
||||
|
||||
|
||||
def evaluate(method: Method, trained: typing.Sequence[Sample], tested: typing.Sequence[Sample]) -> Counts:
|
||||
"""How often the method, learning from trained, decided each kind for each archived kind of tested; None where it left the verdict to the judge."""
|
||||
decide = method(trained)
|
||||
return collections.Counter((found.kind, decide(found.text)) for found in tested)
|
||||
|
||||
|
||||
def plain(kind: Kind) -> str:
|
||||
return kind.removeprefix("Verdict: ")
|
||||
|
||||
|
||||
def offline_report(model: str, samples: typing.Sequence[Sample], rules: typing.Mapping[str, Method], learners: typing.Mapping[str, Method], folds: int = FOLDS, skipped: int = 0) -> str:
|
||||
"""samples are in the order the verdicts were judged, oldest first, so the latest fifth is the newest verdicts."""
|
||||
kinds = collections.Counter(found.kind for found in samples)
|
||||
lines = [f"Replayed {len(samples)} archived verdicts of {model} projects offline, without sending anything to an outside service. The archive gives them as {', '.join(f'{count} {plain(kind)}' for kind, count in sorted(kinds.items(), key=lambda each: plain(each[0])))}."]
|
||||
if skipped:
|
||||
lines.append(f"Skipped {skipped} verdicts whose transcript could not be read.")
|
||||
for name, rule in rules.items():
|
||||
found = evaluate(rule, [], samples)
|
||||
lines += [result_line(f"{name}, on the whole archive", found), *decides_lines(f"{name}, on the whole archive", found)]
|
||||
split = len(samples) - max(1, len(samples) // LATEST)
|
||||
for name, learner in learners.items():
|
||||
held_out = sum((evaluate(learner, trained, tested) for trained, tested in folds_of(samples, folds)), Counts())
|
||||
latest = evaluate(learner, samples[:split], samples[split:])
|
||||
lines += [result_line(f"{name}, on held-out folds", held_out), result_line(f"{name}, on the latest fifth after learning from the rest", latest), *decides_lines(f"{name}, on both tests", held_out + latest)]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def result_line(how: str, found: Counts) -> str:
|
||||
decided = collections.Counter({kind: sum(count for (_, said), count in found.items() if said == kind) for kind in {said for _, said in found if said}})
|
||||
missed = sum(count for (archived, said), count in found.items() if archived == INTERRUPTED and said not in (None, INTERRUPTED))
|
||||
parts = [f"{plain(kind)} {count}{wrong_kinds(kind, found)}" for kind, count in sorted(decided.items(), key=lambda each: plain(each[0]))]
|
||||
return f"{how}: decided {sum(decided.values())} of {sum(found.values())}{': ' + ', '.join(parts) if parts else ''}. Missed interrupts: {missed}."
|
||||
|
||||
|
||||
def wrong_kinds(decided: Kind, found: Counts) -> str:
|
||||
"""The kinds the archive gives the verdicts that were decided as decided, where the archive gives another kind than decided."""
|
||||
wrong = sorted(((plain(archived), count) for (archived, said), count in found.items() if said == decided and archived != decided), key=lambda each: each[0])
|
||||
return f" (of these, archived {', '.join(f'{kind} {count}' for kind, count in wrong)})" if wrong else ""
|
||||
|
||||
|
||||
def decides_lines(how: str, found: Counts) -> list[str]:
|
||||
"""For finished and for unfinished that the decider decided at all: whether it decided no archived interrupt as that kind."""
|
||||
lines = []
|
||||
for kind in (FINISHED, UNFINISHED):
|
||||
decided = sum(count for (_, said), count in found.items() if said == kind)
|
||||
if decided and not found[(INTERRUPTED, kind)]:
|
||||
lines.append(f"{how}, decides {plain(kind)} with no missed interrupt.")
|
||||
elif decided:
|
||||
lines.append(f"{how}, does not decide {plain(kind)} with no missed interrupt: it decided {found[(INTERRUPTED, kind)]} archived interrupt(s) as {plain(kind)}.")
|
||||
return lines
|
||||
@@ -175,6 +175,44 @@ class Main(unittest.TestCase):
|
||||
# The replay shows the judge the whole transcript, with no step-up marked in it.
|
||||
self.assertNotIn("[step]", self.prompts[0])
|
||||
|
||||
def test_offline_replays_every_verdict_through_the_code_deciders_without_calling_the_judge(self):
|
||||
code, out, err = self.run_main(["claude-opus-5-5", "--offline"], unused)
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual(self.calls, [])
|
||||
self.assertIn("Replayed 2 archived verdicts of claude-opus-5-5 projects offline, without sending anything to an outside service. The archive gives them as 1 finished, 1 interrupt.", out)
|
||||
self.assertIn("\"?\" rule, on the whole archive: decided 2 of 2: finished 2 (of these, archived interrupt 1). Missed interrupts: 1.", out)
|
||||
folder = self.home / ".claude" / "loop-claude-transcripts" / "replays" / "judge-replay-claude-opus-5-5-offline-20260921T141320Z"
|
||||
self.assertEqual((folder / "result.md").read_text(encoding="utf-8"), out)
|
||||
|
||||
def test_offline_exits_1_where_no_transcript_can_be_read(self):
|
||||
for path in (self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1").glob("*.jsonl"):
|
||||
path.unlink()
|
||||
code, out, err = self.run_main(["claude-opus-5-5", "--offline"], unused)
|
||||
self.assertEqual((code, out), (1, ""))
|
||||
self.assertEqual(err.splitlines()[-1], "judge-replay: no archived verdict has a transcript that can be read")
|
||||
|
||||
def test_offline_skips_each_verdict_whose_transcript_cannot_be_read_and_says_how_many_on_stderr_and_in_the_report(self):
|
||||
kept = self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1"
|
||||
(kept / "t2.outcomes.md").write_text(f"Task: Do it.\n\n2026-10-14T10:00:02Z {FINISHED}\n\nEvidence:\n> e\n")
|
||||
(kept / "t0.jsonl").unlink()
|
||||
code, out, err = self.run_main(["claude-opus-5-5", "--offline"], unused)
|
||||
self.assertEqual(code, 0)
|
||||
self.assertEqual(len(err.splitlines()), 2)
|
||||
self.assertTrue(all(line.startswith("judge-replay: skipped a verdict: could not read the transcript ") for line in err.splitlines()))
|
||||
self.assertIn("Replayed 1 archived verdicts of claude-opus-5-5 projects offline", out)
|
||||
self.assertIn("Skipped 2 verdicts whose transcript could not be read.", out)
|
||||
|
||||
def test_offline_keeps_its_report_in_the_same_folder_wherever_the_flag_stands(self):
|
||||
code, out, err = self.run_main(["--offline", "claude-opus-5-5"], unused)
|
||||
self.assertEqual(code, 0, err)
|
||||
folder = self.home / ".claude" / "loop-claude-transcripts" / "replays" / "judge-replay-claude-opus-5-5-offline-20260921T141320Z"
|
||||
self.assertEqual((folder / "result.md").read_text(encoding="utf-8"), out)
|
||||
|
||||
def test_offline_says_the_usage_on_arguments_that_name_no_model(self):
|
||||
code, out, err = self.run_main(["--offline"], unused)
|
||||
self.assertEqual((code, out), (1, ""))
|
||||
self.assertEqual(err, "judge-replay: cannot read the arguments --offline; usage: judge-replay <model> [--offline]; <model> is a model on the routing ladder\n")
|
||||
|
||||
def test_keeps_each_run_and_the_printed_report_in_one_folder_of_the_archive_beside_the_projects(self):
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
self.assertEqual(code, 0, err)
|
||||
@@ -272,7 +310,7 @@ class Main(unittest.TestCase):
|
||||
|
||||
def test_prints_the_help(self):
|
||||
code, out, _ = self.run_main(["-h"], unused)
|
||||
self.assertEqual((code, out.splitlines()[2]), (0, "usage: judge-replay <model>"))
|
||||
self.assertEqual((code, out.splitlines()[2]), (0, "usage: judge-replay <model> [--offline]"))
|
||||
|
||||
def test_the_launcher_prints_the_help(self):
|
||||
ran = subprocess.run([sys.executable, str(TOOL), "--help"], capture_output=True, text=True)
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
import collections
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINISHED
|
||||
from claude_code.entries import AssistantTurn, HumanTurn, Text, ToolCall
|
||||
from judge_replay.offline import averaged, classifier, decision_table, evaluate, final_message, folds_of, LEARNERS, logistic, offline_report, question_mark, RULES, Sample, words_of
|
||||
|
||||
|
||||
def turn(*parts):
|
||||
return AssistantTurn(parts=parts)
|
||||
|
||||
|
||||
def sample(kind, text):
|
||||
return Sample(kind, text)
|
||||
|
||||
|
||||
class FinalMessage(unittest.TestCase):
|
||||
def test_is_the_text_of_the_workers_last_turn_that_has_any(self):
|
||||
entries = [turn(Text("first")), HumanTurn(text="ok"), turn(Text("Done. Merged."), ToolCall("1", "Bash", {})), turn(ToolCall("2", "Bash", {}))]
|
||||
self.assertEqual(final_message(entries), "Done. Merged.")
|
||||
|
||||
def test_joins_the_texts_of_one_turn_on_separate_lines_and_leaves_out_blank_ones(self):
|
||||
self.assertEqual(final_message([turn(Text("Merged."), Text(" "), Text("Which one?"))]), "Merged.\nWhich one?")
|
||||
|
||||
def test_is_empty_where_the_worker_wrote_no_text(self):
|
||||
self.assertEqual(final_message([HumanTurn(text="hi"), turn(ToolCall("1", "Bash", {}))]), "")
|
||||
|
||||
|
||||
class WordsOf(unittest.TestCase):
|
||||
def test_is_each_lowercase_word_each_pair_of_neighbours_and_each_question_mark(self):
|
||||
self.assertEqual(words_of("Which one? Yes"), {"which", "one", "?", "yes", "which one", "one ?", "? yes"})
|
||||
|
||||
|
||||
class QuestionMark(unittest.TestCase):
|
||||
def test_decides_interrupt_on_a_question_mark_and_finished_on_none(self):
|
||||
decide = question_mark([])
|
||||
self.assertEqual(decide("Which one?"), INTERRUPTED)
|
||||
self.assertEqual(decide("Merged."), FINISHED)
|
||||
|
||||
|
||||
class DecisionTable(unittest.TestCase):
|
||||
def test_reads_the_workers_asking_for_the_humans_answer_as_an_interrupt(self):
|
||||
decide = decision_table([])
|
||||
for text in ("My recommendation is option 1.", "Your call: merge now or wait.", "The PR stays unmerged.", "Tell me which."):
|
||||
with self.subTest(text=text):
|
||||
self.assertEqual(decide(text), INTERRUPTED)
|
||||
|
||||
def test_reads_a_merge_with_no_ask_and_no_release_to_cut_as_finished(self):
|
||||
decide = decision_table([])
|
||||
self.assertEqual(decide("PR 12 merged and the worktree is removed."), FINISHED)
|
||||
self.assertIsNone(decide("PR 12 merged. Release 0.3.0 is ready to cut."))
|
||||
|
||||
def test_leaves_any_other_message_to_the_judge(self):
|
||||
self.assertIsNone(decision_table([])("I stopped at the gate."))
|
||||
|
||||
|
||||
class Logistic(unittest.TestCase):
|
||||
def test_gives_a_message_with_no_learned_word_about_the_share_of_interrupts(self):
|
||||
chance = logistic([sample(INTERRUPTED, "")] * 10 + [sample(FINISHED, "")] * 30, 0)
|
||||
self.assertAlmostEqual(chance(""), 0.25, delta=0.05)
|
||||
|
||||
def test_learns_a_word_found_in_as_few_messages_as_the_minimum(self):
|
||||
chance = logistic([sample(INTERRUPTED, "ask")] * 3 + [sample(FINISHED, "")] * 3, 0)
|
||||
self.assertGreater(chance("ask"), chance(""))
|
||||
|
||||
def test_averaged_is_the_mean_of_the_regressions_that_shuffle_in_each_seed_order(self):
|
||||
samples = [sample(INTERRUPTED if n % 3 else FINISHED, f"word{n % 4} ask" if n % 3 else f"word{n % 4}") for n in range(30)]
|
||||
mean = sum(logistic(samples, seed)("ask") for seed in range(5)) / 5
|
||||
self.assertAlmostEqual(averaged(samples)("ask"), mean)
|
||||
|
||||
|
||||
class Classifier(unittest.TestCase):
|
||||
def trained(self, extra=()):
|
||||
asking = [sample(INTERRUPTED, f"I recommend option {n}; tell me which of the options you want") for n in range(12)]
|
||||
done = [sample(FINISHED, f"merged pull request {n} and removed the worktree, no findings left") for n in range(12)]
|
||||
return classifier([*asking, *done, *extra])
|
||||
|
||||
def test_decides_a_message_like_each_kind_it_learned(self):
|
||||
decide = self.trained()
|
||||
self.assertEqual(decide("I recommend option 3; tell me which you want"), INTERRUPTED)
|
||||
self.assertEqual(decide("merged pull request 99 and removed the worktree, no findings left"), FINISHED)
|
||||
|
||||
def test_decides_finished_on_a_message_of_words_only_finished_verdicts_held(self):
|
||||
self.assertEqual(self.trained()("merged pull request"), FINISHED)
|
||||
|
||||
def test_abstains_where_it_learned_no_interrupt_or_no_other_kind(self):
|
||||
self.assertIsNone(classifier([sample(FINISHED, "merged")])("merged"))
|
||||
self.assertIsNone(classifier([sample(INTERRUPTED, "which?")])("which?"))
|
||||
self.assertIsNone(classifier([])("anything"))
|
||||
|
||||
def test_decides_no_finished_where_an_interrupt_it_learned_looks_like_one(self):
|
||||
lookalike = "merged pull request 7 and removed the worktree, no findings left"
|
||||
decide = self.trained([sample(INTERRUPTED, lookalike)])
|
||||
self.assertNotEqual(decide(lookalike), FINISHED)
|
||||
|
||||
|
||||
def test_abstains_where_a_lookalike_of_each_kind_sits_in_the_samples_of_the_other(self):
|
||||
decide = self.trained([sample(INTERRUPTED, "merged pull request 7 and removed the worktree, no findings left"), sample(FINISHED, "I recommend option 7; tell me which of the options you want")])
|
||||
for text in ("merged pull request", "I recommend option", "zzz qqq"):
|
||||
with self.subTest(text=text):
|
||||
self.assertIsNone(decide(text))
|
||||
|
||||
|
||||
class Folds(unittest.TestCase):
|
||||
def test_deals_samples_to_folds_in_turn_so_each_is_tested_once(self):
|
||||
samples = [sample(FINISHED, str(n)) for n in range(7)]
|
||||
dealt = folds_of(samples, 3)
|
||||
self.assertEqual([[found.text for found in tested] for _, tested in dealt], [["0", "3", "6"], ["1", "4"], ["2", "5"]])
|
||||
self.assertEqual([found.text for found in dealt[1][0]], ["0", "2", "3", "5", "6"])
|
||||
|
||||
|
||||
class Evaluate(unittest.TestCase):
|
||||
def test_counts_each_archived_kind_by_what_the_method_decided_or_abstained(self):
|
||||
tested = [sample(FINISHED, "merged"), sample(INTERRUPTED, "merged"), sample(UNFINISHED, "stopped"), sample(ANSWERED, "Which?")]
|
||||
found = evaluate(decision_table, [], tested)
|
||||
self.assertEqual(found, collections.Counter({(FINISHED, FINISHED): 1, (INTERRUPTED, FINISHED): 1, (UNFINISHED, None): 1, (ANSWERED, INTERRUPTED): 1}))
|
||||
|
||||
|
||||
class Report(unittest.TestCase):
|
||||
def samples(self):
|
||||
return [sample(FINISHED, "Merged."), sample(FINISHED, "Merged. Release 1.0.0 is ready to cut."), sample(INTERRUPTED, "I recommend merging."), sample(INTERRUPTED, "Merged.")]
|
||||
|
||||
def test_counts_what_a_rule_decided_on_the_whole_archive_and_missed_by_kind(self):
|
||||
text = offline_report("claude-opus-5-5", self.samples(), {"decision table": decision_table}, {})
|
||||
self.assertIn("Replayed 4 archived verdicts of claude-opus-5-5 projects offline, without sending anything to an outside service. The archive gives them as 2 finished, 2 interrupt.", text)
|
||||
self.assertIn("decision table, on the whole archive: decided 3 of 4: finished 2 (of these, archived interrupt 1), interrupt 1. Missed interrupts: 1.", text)
|
||||
self.assertIn("decision table, on the whole archive, does not decide finished with no missed interrupt: it decided 1 archived interrupt(s) as finished.", text)
|
||||
|
||||
def test_says_a_method_that_decided_a_kind_and_missed_no_interrupt_decides_that_kind(self):
|
||||
text = offline_report("claude-opus-5-5", [sample(FINISHED, "Merged."), sample(INTERRUPTED, "I recommend merging.")], {"decision table": decision_table}, {})
|
||||
self.assertIn("Missed interrupts: 0.", text)
|
||||
self.assertIn("decision table, on the whole archive, decides finished with no missed interrupt.", text)
|
||||
|
||||
def test_runs_a_learner_once_for_each_fold_and_once_on_the_latest_fifth_and_a_rule_only_on_the_whole_archive(self):
|
||||
samples = [sample(INTERRUPTED if n % 2 else FINISHED, "I recommend it" if n % 2 else "merged it") for n in range(20)]
|
||||
text = offline_report("m", samples, {"rule": question_mark}, {"classifier": classifier}, folds=4)
|
||||
self.assertIn("classifier, on held-out folds: decided", text)
|
||||
self.assertIn("classifier, on the latest fifth after learning from the rest: decided", text)
|
||||
self.assertIn("rule, on the whole archive: decided 20 of 20", text)
|
||||
self.assertNotIn("rule, on held-out", text)
|
||||
self.assertNotIn("rule, on the latest fifth", text)
|
||||
|
||||
def test_words_the_whole_report_line_by_line_with_the_wrong_kinds_by_name(self):
|
||||
answers = {"f": FINISHED, "i1": FINISHED, "i2": FINISHED, "u": FINISHED, "q": None}
|
||||
samples = [sample(FINISHED, "f"), sample(INTERRUPTED, "i1"), sample(INTERRUPTED, "i2"), sample(UNFINISHED, "u"), sample(ANSWERED, "q")]
|
||||
text = offline_report("m", samples, {"rule": lambda trained: lambda said: answers[said]}, {})
|
||||
self.assertEqual(text.splitlines(), [
|
||||
"Replayed 5 archived verdicts of m projects offline, without sending anything to an outside service. The archive gives them as 1 finished, 2 interrupt, 1 quoted answer, 1 unfinished.",
|
||||
"rule, on the whole archive: decided 4 of 5: finished 4 (of these, archived interrupt 2, unfinished 1). Missed interrupts: 2.",
|
||||
"rule, on the whole archive, does not decide finished with no missed interrupt: it decided 2 archived interrupt(s) as finished.",
|
||||
])
|
||||
|
||||
def test_says_a_method_that_decided_nothing_decided_0_and_names_no_kind(self):
|
||||
text = offline_report("m", self.samples(), {"rule": lambda trained: lambda said: None}, {})
|
||||
self.assertEqual(text.splitlines()[1:], ["rule, on the whole archive: decided 0 of 4. Missed interrupts: 0."])
|
||||
|
||||
def test_tests_the_latest_fifth_of_the_samples_after_learning_from_the_earlier_ones(self):
|
||||
trained = []
|
||||
|
||||
def learner(samples):
|
||||
trained.append(len(samples))
|
||||
return lambda said: FINISHED
|
||||
|
||||
text = offline_report("m", [sample(FINISHED, str(n)) for n in range(7)], {}, {"learner": learner}, folds=2)
|
||||
self.assertEqual(trained, [3, 4, 6])
|
||||
self.assertEqual(text.splitlines()[-2:], ["learner, on the latest fifth after learning from the rest: decided 1 of 1: finished 1. Missed interrupts: 0.", "learner, on both tests, decides finished with no missed interrupt."])
|
||||
|
||||
def test_says_a_learner_that_missed_an_interrupt_on_either_test_does_not_decide_the_kind(self):
|
||||
def learner(samples):
|
||||
return lambda said: FINISHED if said == "miss" else None
|
||||
|
||||
samples = [sample(FINISHED, "ok")] * 4 + [sample(INTERRUPTED, "miss")]
|
||||
text = offline_report("m", samples, {}, {"learner": learner}, folds=2)
|
||||
self.assertEqual(text.splitlines()[-1], "learner, on both tests, does not decide finished with no missed interrupt: it decided 2 archived interrupt(s) as finished.")
|
||||
|
||||
def test_counts_the_verdicts_it_skipped_only_where_it_skipped_some(self):
|
||||
self.assertNotIn("Skipped", offline_report("m", self.samples(), {}, {}))
|
||||
self.assertIn("Skipped 2 verdicts whose transcript could not be read.", offline_report("m", self.samples(), {}, {}, skipped=2))
|
||||
|
||||
def test_the_shipped_methods_are_the_question_mark_rule_the_table_and_the_classifier(self):
|
||||
self.assertEqual(list(RULES), ['"?" rule', "decision table"])
|
||||
self.assertEqual(list(LEARNERS), ["classifier"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -350,6 +350,20 @@ def held(folder: pathlib.Path, home: pathlib.Path) -> ProjectConfig | Failure |
|
||||
return config if isinstance(config, Failure) else load(config[0], config[0], home)
|
||||
|
||||
|
||||
def by_name(name: str, home: pathlib.Path) -> ProjectConfig | Failure | None:
|
||||
"""The config of the project whose main checkout is a folder called name, though that folder may not exist yet, loaded as `held` does; None where no config names one. A config that cannot be read fails every name, since it may be the one asked for."""
|
||||
files = []
|
||||
for file in configs(home):
|
||||
config = read(file, home)
|
||||
if isinstance(config, Failure):
|
||||
return config
|
||||
if config[0].name == name:
|
||||
files.append(file)
|
||||
if len(files) > 1:
|
||||
return Failure(f"{' and '.join(str(file) for file in files)} each name a checkout called {name}; only one config may")
|
||||
return held(files[0].parent, home) if files else None
|
||||
|
||||
|
||||
def configs(home: pathlib.Path) -> list[pathlib.Path]:
|
||||
"""Every project's config file under home, in key order. do-epic's machine-wide config is not a project's."""
|
||||
return sorted(file for file in (home / PROJECTS).glob(f"*/{CONFIG}") if file.parent.name != DO_EPIC)
|
||||
|
||||
@@ -9,7 +9,7 @@ import unittest.mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from project_config.settings import env_files, Epic, Finish, held, JiraScoring, load, NOTIFIERS, OUTSIDE, Page, ProjectConfig, SETTINGS, Unnamed
|
||||
from project_config.settings import by_name, env_files, Epic, Finish, held, JiraScoring, load, NOTIFIERS, OUTSIDE, Page, ProjectConfig, SETTINGS, Unnamed
|
||||
from failure import Failure
|
||||
from test_env import isolated_env
|
||||
from project_config.main import located, title_templates
|
||||
@@ -598,6 +598,32 @@ class HeldTest(ConfigCase):
|
||||
self.assertIsInstance(held(target, self.home), ProjectConfig, "git names the folder by its resolved path")
|
||||
|
||||
|
||||
class ByNameTest(ConfigCase):
|
||||
def test_finds_the_config_of_the_project_whose_main_checkout_has_the_name_though_the_checkout_is_missing(self) -> None:
|
||||
folder = self.write("acme", CLIENT)
|
||||
found = by_name("acme-billing", self.home)
|
||||
assert isinstance(found, ProjectConfig), found
|
||||
self.assertEqual((found.folder, found.top), (folder, self.checkout))
|
||||
|
||||
def test_finds_none_where_no_config_names_a_checkout_of_that_name(self) -> None:
|
||||
self.write("acme", CLIENT)
|
||||
self.assertIsNone(by_name("billing", self.home), "a part of a name is not the name")
|
||||
self.assertIsNone(by_name("acme", self.home), "a config's key is not a repo's name")
|
||||
self.assertIsNone(by_name("do-epic", self.home))
|
||||
|
||||
def test_fails_where_two_configs_name_a_checkout_of_that_name(self) -> None:
|
||||
self.write("acme", CLIENT)
|
||||
self.write("acme-two", CLIENT.replace("~/code/acme-billing", "~/other/acme-billing"))
|
||||
found = by_name("acme-billing", self.home)
|
||||
assert isinstance(found, Failure), found
|
||||
self.assertIn("acme/config.toml and ", found.reason)
|
||||
self.assertIn("acme-two/config.toml each name a checkout called acme-billing", found.reason)
|
||||
|
||||
def test_fails_where_a_config_cannot_be_read_since_it_may_be_the_one_asked_for(self) -> None:
|
||||
self.write("acme", "this is not toml")
|
||||
self.assertIsInstance(by_name("acme-billing", self.home), Failure)
|
||||
|
||||
|
||||
class EnvFilesTest(ConfigCase):
|
||||
def test_lists_each_readable_configs_env_file_even_one_load_refuses(self) -> None:
|
||||
self.write("acme", CLIENT + 'env-file = "~/acme.env"\n')
|
||||
|
||||
+17
-1
@@ -151,7 +151,7 @@ Valid while `~/.claude` is a checkout of this repo.
|
||||
- **One item, several repos.** do-epic splits such an item when it plans it, into one Jira sub-task per repo, each labelled for its repo. The parent tracks the whole change. A person's issue is split too, with a comment for the team.
|
||||
- **Ranking.** Score, then Blocked by, in one weight chain: the epic's achieve goals, then code health, then the repo's goals. Where the epic and a repo disagree, the epic wins. do-epic re-ranks Jira's backlog to match, and people's issues keep their rank.
|
||||
- **Releases.** An epic item has no release. It ships into its repo's unreleased work: a bullet under `CHANGELOG.md` → `## Unreleased`, or nothing where the repo's plan is in Jira.
|
||||
- **Repos.** A missing checkout is cloned from the host of the epic's other repos; where unsure, do-epic asks. Each chunk uses its repo's model, routing, and language for commits and PRs. Jira text is in the epic's language, or else the machine's default.
|
||||
- **Repos.** A repo's name is its main checkout's folder name, and a project config names that checkout. A missing checkout is cloned from the host of the epic's other repos; where unsure, do-epic asks. Unsure means the other repos don't all sit under one host and owner (the same host spelt as https and as ssh counts as two), an origin can't be read or isn't an https, ssh or scp-style address, the folder to clone into holds something else, or the clone fails. A repo no config names is cloned into the clone folder, and the human is asked to write its config. Each chunk uses its repo's model, routing, and language for commits and PRs. Jira text is in the epic's language, or else the machine's default.
|
||||
- **Running.** The epic caps the chunks running at once, and each repo's `parallel-chunks` still applies. do-epic and loop-claude never run chunks in the same repo at once.
|
||||
- **Long gates.** A chunk whose PR waits on a gate, such as a human approval, parks. It keeps its worktree, PR and session, and its state is in a file that survives a reboot. A parked chunk takes no running slot, and the items it blocks wait. do-epic polls the forge and resumes the chunk's session when the gate passes or a reviewer comments. It fixes reasonable feedback, replies to the rest, and changes the setup so review would have caught it. `do-epic <key> --status` and the tmux window names show what each chunk waits on.
|
||||
- **Human tasks.** do-epic DMs the human when one blocks the next item, and polls until its issue closes.
|
||||
@@ -674,6 +674,22 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
|
||||
|
||||
Valid while the index ranks these models, and while no newer model joins the catalog.
|
||||
|
||||
## routing: the offline classifier may decide finished, since lilleman ruled a miss rate of about 1 in 100 decided close enough to none; the judge's model keeps every other verdict kind
|
||||
|
||||
2026-11-18, claude. Serves goals 1.2 and 1.4, and applies project principle 1: a cost cut waits for a measurement that shows quality holds.
|
||||
|
||||
`bin/judge-replay claude-opus-5-5 --offline` replays all archived verdicts of the Opus 5.5 projects through three deciders. Each reads only the worker's final message, doesn't send anything to an outside service, and decides a kind or leaves the verdict to the judge's model. A missed interrupt is an archived interrupt decided as another kind, which lets the worker settle a decision that is lilleman's. Deciding interrupt can't miss one: a wrong interrupt only asks lilleman more. On 2026-11-17 the archive held 1,104 verdicts (660 finished, 333 interrupt, 33 quoted answer, 78 unfinished); on 2026-11-18 it held 1,116 (668, 336, 34, 78).
|
||||
|
||||
- **A "?" rule** (interrupt where the message holds a "?", finished where it holds none) decided every one of the 1,104 and missed 197 interrupts.
|
||||
- **A decision table**, the rows of `DECISION_TABLE` in `bin/judge_replay/offline.py`. Its rows were read from this archive's interrupts, so its run isn't a held-out test. It decided 974 of the 1,104 and missed 55. Most of its misses are a worker reporting a merged chunk while a release is ready to cut, which depends on the plan; the rest ask for an answer in words no row names.
|
||||
- **A classifier**, a logistic regression over the message's words, trained on the archive. It decides finished only where its chance of an interrupt is below that of every interrupt among its training folds when held out, and interrupt only where it is above that of every other kind. It averages five regressions that differ only in the order they shuffle the samples, since a single shuffle moved its missed interrupts between 0 and 2 on the same archive. Its bar is no missed interrupt on both tests: held-out folds, each tested by a model trained on the other folds, and the latest fifth of the verdicts, learning from the rest.
|
||||
- On the 1,104 verdicts, held-out folds: 103 decided, none missed. Latest fifth: 29 decided, all as finished, and 1 interrupt missed. That fails the bar, and shuffles 5 to 9 and 10 to 14 failed it alike.
|
||||
- On the 1,116 verdicts, held-out folds: 95 decided (84 finished, 11 interrupt), none missed. Latest fifth: 20 decided, all as finished, none missed. That meets the bar for finished.
|
||||
|
||||
The classifier decides about 1 in 12 verdicts, and about 1 in 100 of what it decides were an interrupt on the 1,104. A latest fifth that starts a few verdicts later no longer holds the one it missed, so the bar flips with the archive's size and doesn't separate a classifier that never misses from one that rarely does. On 2026-11-18 lilleman ruled, at loop-claude's terminal, that a miss rate of about 1 in 100 decided is close enough to none. So the classifier may decide finished and skip the judge's call; the plan holds that change as an item. The judge's model keeps every verdict the classifier leaves, the escalation it quotes and whether a release is ready to cut, since the replay didn't test the classifier on those.
|
||||
|
||||
Valid while the archived verdicts hold interrupts that read like a finished chunk. Replay with `judge-replay <model> --offline` when the archive has doubled, or when a decider reads more than the final message.
|
||||
|
||||
## routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds
|
||||
|
||||
2026-10-14, lilleman. Serves goal 1.4.
|
||||
|
||||
@@ -2640,20 +2640,3 @@ Read most (opens and searches):
|
||||
- `bin/model_routing/ladder.py` 3
|
||||
- `ci.sh` 3
|
||||
- `docs/reading-cost.md` 3
|
||||
|
||||
## Reading cost, chunk 88da1a2883a6, finished 2026-11-18T12:23:47Z
|
||||
|
||||
104466 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `README.md` 8
|
||||
- `bin/record_store/confluence_page.py` 7
|
||||
- `bin/record_store/test_jira_description.py` 5
|
||||
- `docs/decisions.md` 4
|
||||
- `.gitignore` 3
|
||||
- `bin/adf_codec/convert.py` 3
|
||||
- `bin/atlassian/client.py` 3
|
||||
- `bin/record_store/fakes.py` 3
|
||||
- `bin/do_epic/epic_record.py` 2
|
||||
- `bin/project_config/main.py` 2
|
||||
|
||||
@@ -6,31 +6,32 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 420`
|
||||
`Next ID = 424`
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 407 | 0.12.0 | | | **Find an epic repo's checkout and project config by the repo's name, clone a missing one from the host of the epic's other repos, and ask the human where unsure.** | 2 | 2 | 2 | 4 | 3 | 1 by 5 | 1.9 |
|
||||
| 401 | 0.12.0 | | | **Move loop-claude's cycle, judge, interrupt, tmux and routing into a package a second entry point drives, with loop-claude's behaviour unchanged.** | 3 | 2 | 3 | 6 | 6 | 1 by 3, code health by 2 | 1.8 |
|
||||
| 406 | 0.12.0 | | | **Pick each epic item's repo, or for an item without one whether it is Claude's work or a human task, by `repo-rule.md`, record the pick on the issue, and ask the human where the rule leaves it unclear.** | 2 | 2 | 2 | 4 | 4 | 1 by 5 | 1.6 |
|
||||
| 408 | 0.12.0 | | | **Rank an epic's items by one weight chain, the epic's achieve goals, then code health, then the item's repo goals, then by Blocked by, and re-rank Jira's backlog to match, with people's issues keeping their rank.** | 3 | 2 | 2 | 5 | 4 | 1 by 5 | 1.6 |
|
||||
| 410 | 0.12.0 | 401, 406, 407, 408 | | **Run `do-epic <EPIC-KEY>`: one tmux session per epic, a window per chunk, up to the epic's cap of running chunks and each repo's `parallel-chunks`, and never in a repo a loop-claude is running chunks in.** | 3 | 2 | 3 | 6 | 6 | 1 by 5 | 0.8 |
|
||||
| 410 | 0.12.0 | 401, 406, 408 | | **Run `do-epic <EPIC-KEY>`: one tmux session per epic, a window per chunk, up to the epic's cap of running chunks and each repo's `parallel-chunks`, and never in a repo a loop-claude is running chunks in.** | 3 | 2 | 3 | 6 | 6 | 1 by 5 | 0.8 |
|
||||
| 416 | 0.12.0 | 410 | | **Finish an epic's repo item with a bullet under its repo's `CHANGELOG.md` → `## Unreleased`, or with none where the repo's plan is in Jira.** | 1 | 1 | 1 | 1 | 2 | 1 by 5 | 2.1 |
|
||||
| 413 | 0.12.0 | 410 | | **File a finding about an epic goal in the epic and one about a repo goal by that repo's rules, ask the human where unsure which, and give the judge the epic's records beside the repo's.** | 3 | 2 | 3 | 6 | 4 | 1 by 5 | 1.6 |
|
||||
| 415 | 0.12.0 | 410, 413 | | **Finish an epic: run release-review passes on each repo it changed, work what they file, have the human confirm each goal, post a summary comment, and close the epic in Jira last.** | 3 | 2 | 3 | 6 | 5 | 1 by 5 | 1.2 |
|
||||
| 418 | 0.12.0 | 415 | | **Describe do-epic in `README.md`'s tools and setup, with its machine settings and `repo-rule.md`.** | 1 | 2 | 1 | 2 | 2 | 1 by 5 | 2.1 |
|
||||
| 400 | 0.12.0 | | | **Forbid a test that decides at run time whether to check anything in `principles/technical-principles.md`, and point stability-reviewer's Tests checks at it.** | 1 | 2 | 3 | 4 | 2 | 1.1 by 5 | 2.0 |
|
||||
| 420 | 0.12.0 | | | **End a worker that printed its final result and lingers, by killing the background shells it left, after a few minutes, and print each kill with its evidence on loop-claude's terminal.** | 2 | 2 | 2 | 4 | 3 | 1.5 by 6 | 2.0 |
|
||||
| 409 | 0.12.0 | 406 | | **Split an epic item that changes several repos into one Jira sub-task per repo, each labelled for its repo, with the parent tracking the whole change, when do-epic plans the item.** | 2 | 2 | 3 | 5 | 3 | 1 by 5 | 1.9 |
|
||||
| 411 | 0.12.0 | 410 | | **Park a chunk whose PR waits on a long gate, keeping its worktree, PR and session in a state file that survives a reboot, and resume its session when the gate passes or a reviewer comments.** | 3 | 2 | 3 | 6 | 5 | 1 by 5 | 1.2 |
|
||||
| 412 | 0.12.0 | 410, 411 | | **Print each chunk of an epic, running, parked with its gate and since when, or stopped, each human task and each open interrupt with `do-epic <key> --status`, and name a parked chunk's gate in its tmux window's name.** | 2 | 1 | 1 | 2 | 3 | 1 by 5 | 1.9 |
|
||||
| 414 | 0.12.0 | 410 | | **DM the human through the notifier when a human-task issue blocks the epic's next item, and poll until it closes.** | 2 | 2 | 2 | 4 | 3 | 1 by 5 | 1.9 |
|
||||
| 417 | 0.12.0 | 410 | | **Run do-epic end to end against a sandbox epic on a real Jira site with a command of its own, outside `run-tests`.** | 2 | 1 | 1 | 2 | 3 | 1 by 5 | 1.9 |
|
||||
| 299 | 0.12.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
|
||||
| 421 | 0.12.0 | | | **Refuse in rule-hook a Bash or Monitor wait that has no end: a loop without a time or count cap, and a `pgrep -f` or `pkill -f` whose pattern the same command line contains.** | 2 | 2 | 3 | 5 | 3 | 1.5 by 4 | 1.3 |
|
||||
| 311 | 0.12.0 | | | **Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.0 |
|
||||
| 316 | 0.12.0 | | | **Launch comprehension seats as an agent type of their own under `agents/`, without `CLAUDE.md` and the skill and agent listings a general-purpose subagent carries.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
|
||||
| 310 | 0.12.0 | | | **Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 0.9 |
|
||||
| 422 | 0.12.0 | 401, 420 | | **Check every 5 minutes whether a tmux window of the loop looks stuck by other signs than a lingering worker, investigate it with a headless claude, and kill only the process the investigation names.** | 3 | 2 | 3 | 6 | 5 | 1.5 by 4 | 0.8 |
|
||||
| 365 | 0.13.0 | | | **Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
|
||||
| 368 | 0.13.0 | | | **Call a release review due, call a release ready to cut, and tell whether a chunk owes a scoring run, for a plan outside the checkout, from the project repo's history or from Jira.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
|
||||
| 309 | 0.13.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.2 |
|
||||
@@ -41,6 +42,7 @@
|
||||
| 298 | 0.13.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 0.9 |
|
||||
| 395 | 0.13.0 | | decision | **Keep each block a save didn't change, with its localId, and the spacing paragraphs beside it as Jira or Confluence holds them.** | 3 | 2 | 2 | 5 | 5 | 1.6 by 4 | 0.8 |
|
||||
| 367 | 0.13.0 | | | **Check that every write to a Jira plan leaves it ranked, as rule-hook's `todo-rank-check` checks `todo.md` today.** | 2 | 2 | 3 | 5 | 3 | 1.6 by 2 | 0.7 |
|
||||
| 423 | 0.13.0 | | | **Skip the judge's model call on a worker's final message that the offline classifier decides is finished.** | 3 | 3 | 3 | 7 | 5 | 1.4 by 2 | 0.4 |
|
||||
| 77 | 0.14.0 | | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 4 | 2 | 1 | 5 | 6 | 1.1 by 7, 1.4 by 7 | 2.1 |
|
||||
| 196 | 0.14.0 | | decision | **Replay the architecture, principles, overrides and product-owner passes with `review-replay` on Opus 5.5 projects, record each run in the review-replay routing decision, and, where a run qualifies a rung, add back the code that lowers a pass to it.** | 2 | 2 | 3 | 5 | 2 | 1.4 by 3 | 1.2 |
|
||||
| 321 | 0.14.0 | 196 | decision | **Evaluate the review passes that run on a rung `review-replay` qualified, against the chunks before them, and revert any pass whose findings or later fixes worsened.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3, 1.4 by 2 | 2.0 |
|
||||
@@ -81,16 +83,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 407. Find an epic repo's checkout and project config by the repo's name, clone a missing one from the host of the epic's other repos, and ask the human where unsure.
|
||||
|
||||
Unsure means the epic's other repos sit on more than one host, or the clone fails (`docs/decisions.md` → `do-epic: drives one Jira epic …`). A by-name lookup goes beside `held` and `load`.
|
||||
|
||||
Touches:
|
||||
|
||||
- `bin/project_config/settings.py`: `held` (small)
|
||||
- `bin/forge/commands.py`: `forge_of` (small)
|
||||
- `bin/do_epic/repos.py`
|
||||
|
||||
### 401. Move loop-claude's cycle, judge, interrupt, tmux and routing into a package a second entry point drives, with loop-claude's behaviour unchanged.
|
||||
|
||||
do-epic runs on these, not on a copy (`docs/decisions.md` → `do-epic: drives one Jira epic …`). Move the reusable modules into `bin/agent_cycle/`: `tmux`, `judge`, `interrupt`, `verdicts`, `retry`, `spend`, `worker`, `states`, `stop`, and the parts of `routing` that take a `ProjectRung` and a `Route`. Keep `routed_item`, `loop/`, `project/`, `escalation_mark` and `chunk_items` in `loop_claude`. Decouple `cycle/run.py`'s archive and `Chunk items:` handling, and `cycle/prompts.py`'s goals-file and `todo.md` names, behind parameters. The places the comprehension seats found hard here, `routing.py` `unfinished_route` and `cycle/judge.py` `quotes_checked`, must read no harder after the move.
|
||||
@@ -111,11 +103,11 @@ Touches:
|
||||
- `bin/loop_claude/cycle/prompts.py`: `QUOTE_SOURCES`, `ANSWER_DESTINATIONS`
|
||||
- `bin/loop_claude/test_loop_claude.py`
|
||||
- `bin/loop_claude/test_black_box.py`
|
||||
- `bin/judge_replay/main.py` (small)
|
||||
- `bin/judge_replay/replay.py` (small)
|
||||
- `bin/judge_replay/archived_prompt.py` (small)
|
||||
- `bin/rung_replay/command.py` (small)
|
||||
- `bin/rung_replay/search.py` (small)
|
||||
- `bin/judge_replay/main.py`
|
||||
- `bin/judge_replay/replay.py`
|
||||
- `bin/judge_replay/archived_prompt.py`
|
||||
- `bin/rung_replay/command.py`
|
||||
- `bin/rung_replay/search.py`
|
||||
- `README.md`: `` ### Packages under `bin/` `` (small)
|
||||
|
||||
### 406. Pick each epic item's repo, or for an item without one whether it is Claude's work or a human task, by `repo-rule.md`, record the pick on the issue, and ask the human where the rule leaves it unclear.
|
||||
@@ -156,7 +148,7 @@ An epic item has no release (`docs/decisions.md` → `do-epic: drives one Jira e
|
||||
|
||||
Touches:
|
||||
|
||||
- `bin/do_epic/main.py` (small)
|
||||
- `bin/do_epic/main.py`
|
||||
|
||||
### 413. File a finding about an epic goal in the epic and one about a repo goal by that repo's rules, ask the human where unsure which, and give the judge the epic's records beside the repo's.
|
||||
|
||||
@@ -199,9 +191,25 @@ Source: a client project's pull request, where a fix hid a regression behind a c
|
||||
|
||||
Touches:
|
||||
|
||||
- `principles/technical-principles.md`: the tests bullet group, `A valuable test checks behaviour` (small)
|
||||
- `principles/technical-principles.md`
|
||||
- `agents/stability-reviewer.md`: `### 11. Tests` (small)
|
||||
|
||||
### 420. End a worker that printed its final result and lingers, by killing the background shells it left, after a few minutes, and print each kill with its evidence on loop-claude's terminal.
|
||||
|
||||
On 2026-11-18 the worker of the item 402 chunk merged its pull request at about 11:47, printed its summary and then waited 95 minutes: it had started `until [ -s /tmp/merge-402.log ] && ! pgrep -f "chunk-pr merg"; do sleep 15; done` in the background, and `pgrep -f` matched the loop's own command line, so the condition never held. Its `claude -p` stayed alive ("Waiting for background work to finish"), `loop-claude` waited for it to exit, and the window's slot started no new chunk until the human found it and killed the shell.
|
||||
|
||||
Every 5 minutes loop-claude checks each worker it runs: when the worker's last transcript entry is its final result and its `claude -p` is still alive more than 3 minutes later, it kills the background shells that are children of that process, and the worker then exits. Nothing else is killed: no worker that is still working, no process of another session, and no window with a sign short of this one. A worker whose shell is still writing output or holds a `run-heavy` slot is left alone. Each kill is printed on loop-claude's terminal with the process, its age and its command, and appended to the judge's records so the pattern can become a rule-hook ban (item 421). Item 422 handles the vaguer signs of a stuck window.
|
||||
|
||||
The check lives in the main run beside the other per-cycle work; item 401 moves that code into a package, and a conflict with it is cheaper than waiting for it.
|
||||
|
||||
Before building the check, try the platform's own bound: Claude Code documents `BASH_MAX_TIMEOUT_MS` and `BASH_DEFAULT_TIMEOUT_MS` as the time limit of a background command, 10 minutes by default in `claude -p` (docs: code.claude.com/docs/en/headless.md and env-vars). Nothing in this repo sets them, yet the lingering shell ran 95 minutes on Claude Code 2.1.327, so first find out why the limit did not apply (an explicit timeout in the worker's call, or a gap in the docs). Hand the workers a cap through those variables where that bounds the wait, and build the lingering-worker check only for what the cap leaves.
|
||||
|
||||
Touches:
|
||||
|
||||
- `bin/loop_claude/loop/main.py`: `run_loop` (small)
|
||||
- `bin/loop_claude/loop/lingering_workers.py`
|
||||
- `README.md`: `## Tools` (small)
|
||||
|
||||
### 409. Split an epic item that changes several repos into one Jira sub-task per repo, each labelled for its repo, with the parent tracking the whole change, when do-epic plans the item.
|
||||
|
||||
do-epic predicts the repos from the code, as `Touches:` predictions are made. A person's issue is split too, with a comment for the team in the epic's language (`docs/decisions.md` → `do-epic: drives one Jira epic …`). The parent is finished once every sub-task is.
|
||||
@@ -253,14 +261,22 @@ Touches:
|
||||
- `bin/do-epic-e2e`
|
||||
- `bin/do_epic/e2e/`
|
||||
|
||||
### 299. Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.
|
||||
### 421. Refuse in rule-hook a Bash or Monitor wait that has no end: a loop without a time or count cap, and a `pgrep -f` or `pkill -f` whose pattern the same command line contains.
|
||||
|
||||
The judge's LLM still decides three things: the verdict kind, the escalation it quotes, and whether a release is ready to cut (`docs/decisions.md` → `routing: loop-claude reads the items a chunk takes …`). This item replays the verdict kind. No single rule found so far separates interrupts: a "?" in the worker's final message appears in 48 interrupts and misses 66. A smaller LLM already failed the replay (`docs/decisions.md` → `routing: on a model judge-replay covered, …`: Sonnet returned another kind on 3 interrupts, Haiku missed 7). Untested are combined code rules, a decision table, and a small classifier trained and run on this machine on the archived verdicts (372 finished, 234 interrupt, 42 unfinished, 20 quoted). Run each offline, nothing sent to an outside service (goal 4), and count disagreements by kind; the bar is zero missed interrupts. A hosted decision model is out of scope, since it would send transcripts to an outside service.
|
||||
On 2026-11-18 a worker in the item 402 chunk started `until [ -s /tmp/merge-402.log ] && ! pgrep -f "chunk-pr merg"; do sleep 15; done` in the background. The `pgrep -f` pattern is part of the loop's own command line, so it matched itself and the loop never ended; the worker's `claude -p` waited on it for 95 minutes and its slot started no new chunk (item 420 ends such a worker; item 422 handles other stuck windows). A wait that cannot end is cheaper to refuse before it starts.
|
||||
|
||||
rule-hook refuses a Bash command, or a Monitor call, that starts a wait of either kind. A loop (`until`, `while`, `for`) that sleeps and has neither a time limit (`timeout`, a deadline compared with `date`) nor a bounded count (`for i in $(seq …)`); `~/.claude/bin/wait-for` always passes, since it takes `--timeout`. A `pgrep -f` or `pkill -f` whose pattern text also appears in the command line that holds it, quoted or not, so it can match its own shell. The refusal names which kind it found and says to use `wait-for <file> [<pattern>] --timeout <seconds>`, or a `pgrep -x` on the process name.
|
||||
|
||||
`settings.json` hooks only `Agent|Bash|Edit|Grep|MultiEdit|Read|Skill|Write`, so rule-hook never sees a Monitor call: add `Monitor` to that matcher, and have rule-hook read a Monitor call's command as it reads a Bash one. Claude Code 2.1.314 and later send Monitor-started commands to a PreToolUse hook; a hook's allow and updatedInput are ignored for them, so rule-hook can refuse but not rewrite (docs: code.claude.com/docs/en/hooks.md).
|
||||
|
||||
The test covers each kind refused, a bounded loop that passes, `wait-for` passing, and a `pgrep -f` on a pattern absent from the command passing. The rule's reason goes in rule-hook's rule table beside the other rules.
|
||||
|
||||
Touches:
|
||||
- `bin/judge_replay/main.py`: `main`, `archived_cases` (small)
|
||||
- `bin/judge_replay/replay.py`: `Case`, `verdict_kind`, `report`
|
||||
- `bin/judge_replay/test_judge_replay.py`
|
||||
|
||||
- `bin/rule_hook/rules.py`: `RULES` (small)
|
||||
- `bin/rule_hook/endless_wait.py`
|
||||
- `bin/rule_hook/test_rule_hook.py`: `CommandBreaches` (small)
|
||||
- `settings.json`
|
||||
|
||||
### 311. Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.
|
||||
|
||||
@@ -310,6 +326,20 @@ Touches:
|
||||
- `bin/loop_claude/test_loop_claude.py`: `FakeEffects`, `Interrupts`
|
||||
- `bin/loop_effects/test_effects.py`
|
||||
|
||||
### 422. Check every 5 minutes whether a tmux window of the loop looks stuck by other signs than a lingering worker, investigate it with a headless claude, and kill only the process the investigation names.
|
||||
|
||||
Item 420 ends a worker that printed its final result and lingers. A window can also stall earlier: a test run that hangs on a shell ignoring SIGINT (item 402's first full run), a pane with no output while its process tree does no work, or a background shell older than the chunk's usual length.
|
||||
|
||||
Every 5 minutes loop-claude lists its own session's windows and, for one with such a sign, launches a headless claude on the project's rung that reads the pane, the process tree and the chunk's log files and returns `working`, `waiting on a person` or `stuck`, with the process that holds the window and why. Only `stuck` kills, and only the named process, never the worker or the loop. A window waiting on a gate (`run-heavy`, a test run that still writes output) or on the human's answer is not `stuck`. Each check and kill is printed on loop-claude's terminal with the evidence.
|
||||
|
||||
Run it first as a report that kills nothing, and turn the kill on once its verdicts over a few days match what the human would have done; record that in the item's decision. Blocked by 401, which moves loop-claude's cycle into a package; this item lands in the package.
|
||||
|
||||
Touches:
|
||||
|
||||
- `bin/loop_claude/loop/stuck_windows.py`
|
||||
- `bin/loop_claude/tmux.py`: `Listed`, `listed`
|
||||
- `README.md`: `## Tools` (small)
|
||||
|
||||
### 365. Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.
|
||||
|
||||
lilleman's call, 2026-11-12: 0.10.0 and 0.11.0 switch the history tools off for a plan outside the checkout, and 0.13.0 holds only their ports. In Jira a dropped item, which `Drop item` commit titles mark today, is an issue resolved by a resolution the config names.
|
||||
@@ -414,6 +444,21 @@ Today a Jira detail edited in one line, or a Confluence record with one entry ap
|
||||
|
||||
One of 0.13.0's history ports, per lilleman's call of 2026-11-12.
|
||||
|
||||
### 423. Skip the judge's model call on a worker's final message that the offline classifier decides is finished.
|
||||
|
||||
On 2026-11-18 lilleman ruled that the classifier's miss rate of about 1 in 100 decided is close enough to no missed interrupt (`docs/decisions.md` → `routing: the offline classifier may decide finished, since lilleman ruled a miss rate of about 1 in 100 decided close enough to none; the judge's model keeps every other verdict kind`).
|
||||
|
||||
Taking the item builds this: before the judge call, loop-claude runs the classifier trained on its archive over the worker's final message. Where the classifier decides finished, loop-claude skips the call. The classifier is re-trained as the archive grows. The judge call sits in `bin/loop_claude/cycle/run.py` → `rule`, a place the comprehension seats found hard, since the order of stamping, archiving and stepping matters across three functions: make it easier to read.
|
||||
|
||||
The judge's model still decides every verdict the classifier leaves, the escalation a worker quotes, and whether a release is ready to cut, since the replay didn't test the classifier on those.
|
||||
|
||||
Evidence, from `judge-replay claude-opus-5-5 --offline`: the classifier reads only the final message, averages five seeded logistic regressions, and decides finished only where its chance of an interrupt is below that of every interrupt among its held-out training folds.
|
||||
- On the 1,116 archived verdicts it decided finished with no missed interrupt on both tests: held-out folds, 95 decided and none missed; latest fifth, 20 decided and none missed.
|
||||
- On the first 1,104 it missed one interrupt on the latest fifth, while the held-out folds decided 103 and missed none.
|
||||
- It decides about 1 in 12 verdicts. About 1 in 100 of those it decided were an interrupt, on the 1,104.
|
||||
|
||||
Savings: one judge call per about 12 verdicts. That is small, and nobody has measured it.
|
||||
|
||||
### 77. Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.
|
||||
|
||||
2026-10-08, lilleman: no metric enters review until a rerun on more data shows it tracking the panel.
|
||||
|
||||
Reference in New Issue
Block a user