Merge pull request 'Run the judge on the rung its chunk's item picks, never below the project's rung' (#266) from judge-routing into main
This commit was merged in pull request #266.
This commit is contained in:
@@ -24,6 +24,9 @@
|
||||
!bin/file_writes.py
|
||||
!bin/git_command.py
|
||||
!bin/git_repo.py
|
||||
!bin/judge-replay
|
||||
!bin/judge_replay/
|
||||
!bin/judge_replay/*.py
|
||||
!bin/larv-review
|
||||
!bin/larv_review/
|
||||
!bin/larv_review/*.py
|
||||
|
||||
@@ -34,6 +34,7 @@ Decisions live in `docs/decisions.md`:
|
||||
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
|
||||
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
|
||||
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model, and a chunk's worker may run below that rung, launching its reviewers on that rung; the worker routes each subagent it delegates to by the R + C of the subagent's task
|
||||
- routing: on a model judge-replay covered, the judge runs on the rung its chunk's item picks, never below the project's rung
|
||||
|
||||
## Working rules
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- `loop-claude` runs the judge of an Opus 5.5 project on the rung the chunk's `todo.md` item picks, never below Opus 5.5 at medium effort, the project's rung. Before, the judge ran at the effort `settings.json` sets, xhigh. A replay of 39 archived verdicts found every cheaper rung missing verdicts. The judge keeps its rung when the worker steps up or takes in another item, and runs on the project's rung when no item routed the chunk. A project on another model keeps its judge at its settings' effort. Each verdict in the archive now names the judge's rung on a `Judge:` line. The new `bin/judge-replay <model>` replays the judge on archived verdicts and prints the cheapest rung that returns them. Expect the judge to cost less on items of R + C 12 or less, and more on items of R + C 15 or more.
|
||||
- Every 20 cycles across all projects, `loop-claude` measures every finished chunk in its archive, with no model run, and writes the figures to `routing-figures.md` at the archive's root: tokens by project, subagent type, rung and tool, and, for each chunk a `todo.md` item routed, grouped by the rung it started on: its review rounds past each pass's first, its unfinished verdicts and its subagents' rungs. It interrupts you, with `claude` on the project's model on the terminal in `~/.claude`, when chunks routed onto a rung took more review rounds or unfinished verdicts than chunks routed onto the next rung up, each rung with 5 chunks or more, or when 5 or more subagents of one type wrote reports longer than the tool results they read. Each call is raised once, then listed in `routing-calls-raised.md` beside the figures; the next chunk of this repo files a `todo.md` item for it. Off a terminal, the calls are said on stderr and the loop goes on. They are raised at the next measurement on a terminal. After a routing change, a `- <UTC time> Reset` line in that file makes the figures count only chunks finished after it, and lets its calls be raised again. Expect a pause of about ten seconds at each measurement.
|
||||
- A session delegates a task to a subagent only when it needs just the task's conclusion, such as a search across many files, and keeps edits in its own thread. It scores the task's R and C as for a `todo.md` item, and launches the subagent on the model and effort that the new `bin/subagent-rung <model> <R+C>` picks from the project's model, such as `model sonnet, effort high`. `CLAUDE.md` states the rule. Expect searches to run on cheaper models, and less of each session's context to fill with search output.
|
||||
- `loop-claude` routes each chunk's worker on a ladder of model and effort rungs, from Haiku 5.5 at low effort to Fable 5.1 at max, when the project's model is on it. As a chunk starts, the R + C of `todo.md`'s top item in the next release picks the rung: 11–12 picks one rung above the project's rung, so Opus 5.5 at high for an Opus 5.5 project, and so on up to four rungs. An item of R + C 10 or less runs the worker at or below the project's model's cheapest rung, which for Opus 5.5 is medium effort, below the xhigh `settings.json` sets. While the worker runs below the project's model's cheapest rung, it launches every reviewer, comprehension seat and security pass, the full scan included, in a subagent on that rung. While it runs above that rung, it launches each comprehension seat and the security pass on that rung. It steps up one rung after every second unfinished verdict on a chunk, and to a higher rung when it takes in an item of higher R + C. Each cycle's header, and each finished or unfinished verdict, stop and step-up in the archive, name the worker's rung; a rerun resumes an open chunk on the latest rung archived for it. `docs/decisions.md` → `routing:` gives the ladder, its source and the bands. Expect a chunk on an easy item to cost much less than under xhigh, and perhaps to take more review rounds, and one on an item of R + C 15 or more to cost more.
|
||||
|
||||
@@ -67,6 +67,7 @@ Rename `## Unreleased` in `CHANGELOG.md` to the version, commit, tag that commit
|
||||
|
||||
Install puts only `loop-claude` on `PATH`; call the others by their path, such as `~/.claude/bin/metric-bench`, from the project's checkout.
|
||||
|
||||
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints the cheapest rung that returns them; `judge-replay --help` states the replay.
|
||||
- `larv-review <command>` holds larv-review's mechanical rules: the paired ruling's base and vote count, each pass's round count, the scoring run's trigger, and the comprehension history's format and latest places; `larv-review --help` states them.
|
||||
- `loop-claude <task>` runs a project's next chunk cycle after cycle; `loop-claude --help` states its contract.
|
||||
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
|
||||
@@ -92,7 +93,7 @@ pytest_add_cli_args=-k
|
||||
|
||||
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
|
||||
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, and `score_outcomes` imports `reading_cost`'s record path. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, `score_outcomes` imports `reading_cost`'s record path, and `judge_replay` imports `loop_claude`'s judge. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
|
||||
## Tracking a new file
|
||||
|
||||
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env python3
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
|
||||
from judge_replay import main
|
||||
from loop_claude.effects import system_effects
|
||||
|
||||
sys.exit(main.main(sys.argv[1:], os.environ, pathlib.Path.home(), system_effects()))
|
||||
@@ -0,0 +1,132 @@
|
||||
"""Replay loop-claude's judge on archived verdicts, to find the cheapest rung that returns them: docs/decisions.md → `routing: on a model judge-replay covered`.
|
||||
|
||||
usage: judge-replay <model>
|
||||
|
||||
<model> is a project model on the routing ladder, such as claude-opus-5-5. The tool replays the
|
||||
verdicts archived for the projects that ~/.claude/loop-claude-models.json runs on it: the latest
|
||||
15 of each kind, finished, unfinished, interrupt and quoted answer. Each transcript copy in the
|
||||
archive is what the judge read for the verdict beside it, so the replay hands the judge that copy,
|
||||
with the task, premises and records each project holds now.
|
||||
|
||||
It first replays every verdict on the reference judge: the model at xhigh effort, the rung the
|
||||
judge ran on before loop-claude routed it. A verdict the reference judge returns another kind on
|
||||
may rest on records or judge rules that have changed since; either way no rung is held to it. It
|
||||
then replays the rest on each candidate rung, cheapest first: from the rung the lowest R + C picks
|
||||
up to the model's cheapest rung. It stops at the first rung that returns all of them. It prints
|
||||
each verdict the reference judge did not reproduce, each verdict a rung missed, and the cheapest
|
||||
rung that missed none. A rung it names is the evidence to add the model to JUDGE_REPLAYED in
|
||||
bin/loop_claude/routing.py, which routes that model's judge, and to record the run in
|
||||
docs/decisions.md → `routing: on a model judge-replay covered`.
|
||||
|
||||
Like the loop, it tries a judge three times running before it counts the reply as failed. A
|
||||
verdict whose project or transcript cannot be read replays as failed too. Exit 1, saying why, on
|
||||
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
|
||||
or a usage limit.
|
||||
"""
|
||||
import concurrent.futures
|
||||
import pathlib
|
||||
import sys
|
||||
import typing
|
||||
|
||||
from transcript_archive.location import archive_root, transcript_of
|
||||
from transcript_archive.read import archived_outcomes
|
||||
from judge_replay.replay import Case, case_from, Kind, replayed, report, sample, verdict_kind
|
||||
from project_key import checkout_of
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.retry import FAILURES_RUNNING, Limited
|
||||
from loop_claude.cycle.judge import judge_once, Verdict
|
||||
from model_routing.ladder import LADDER, picked, position, project_rung, Rung
|
||||
from loop_claude.project.model_choice import read_models_file
|
||||
from loop_claude.project.premises import read_premises
|
||||
from loop_claude.project.records import read_records
|
||||
from claude_code.transcript import read_transcript, render
|
||||
from loop_claude.stop import Stop
|
||||
|
||||
PER_KIND = 15
|
||||
# The effort settings.json gave every judge before loop-claude routed it; the archived verdicts came from it.
|
||||
REFERENCE_EFFORT = "xhigh"
|
||||
# Judges replayed at once.
|
||||
AT_ONCE = 4
|
||||
LOWEST_R_PLUS_C = 2
|
||||
USAGE = "usage: judge-replay <model>, a model on the routing ladder"
|
||||
|
||||
|
||||
def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, effects: Effects) -> int:
|
||||
if argv[:1] in (["-h"], ["--help"]):
|
||||
print(__doc__.strip())
|
||||
return 0
|
||||
anchor = project_rung(argv[0]) if len(argv) == 1 else None
|
||||
if anchor is None:
|
||||
print(f"judge-replay: cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; {USAGE}", file=sys.stderr)
|
||||
return 1
|
||||
cases = archived_cases(argv[0], archive_root(home, env), home)
|
||||
if isinstance(cases, Failure):
|
||||
print(f"judge-replay: {cases.reason}", file=sys.stderr)
|
||||
return 1
|
||||
if not cases:
|
||||
print(f"judge-replay: no archived verdicts of projects on {argv[0]} in {archive_root(home, env)}; run loop-claude on a project on that model first", file=sys.stderr)
|
||||
return 1
|
||||
cases = sample(cases, PER_KIND)
|
||||
reference = Rung(argv[0], REFERENCE_EFFORT)
|
||||
candidates = list(LADDER[position(picked(anchor, LOWEST_R_PLUS_C)):position(anchor) + 1])
|
||||
replay = replayed(cases, reference, candidates, lambda on, rung: rule_all(on, rung, home / ".claude", effects))
|
||||
if isinstance(replay, Limited):
|
||||
print(f"judge-replay: claude hit a usage limit: {replay.said}; replay again once it resets", file=sys.stderr)
|
||||
return 1
|
||||
print(report(argv[0], reference, cases, replay))
|
||||
return 0
|
||||
|
||||
|
||||
def archived_cases(model: str, root: pathlib.Path, home: pathlib.Path) -> list[Case] | Failure:
|
||||
models = read_models_file(home / ".claude" / "loop-claude-models.json")
|
||||
if isinstance(models, Stop):
|
||||
return Failure(models.reason)
|
||||
listed = archived_outcomes(root)
|
||||
if isinstance(listed, Failure):
|
||||
return listed
|
||||
found = []
|
||||
try:
|
||||
for key, outcomes in listed:
|
||||
case = case_from(checkout_of(key, home), outcomes.read_text(encoding="utf-8"), transcript_of(outcomes)) if models.get(key) == model else None
|
||||
found += [case] if case else []
|
||||
except (OSError, UnicodeDecodeError) as failure:
|
||||
return Failure(f"could not read the archive {root}: {failure}")
|
||||
return found
|
||||
|
||||
|
||||
def rule_all(cases: list[Case], rung: Rung, claude_dir: pathlib.Path, effects: Effects) -> list[Kind] | Limited:
|
||||
print(f"replaying {len(cases)} verdicts on {rung}", file=sys.stderr, flush=True)
|
||||
with concurrent.futures.ThreadPoolExecutor(AT_ONCE) as pool:
|
||||
running = [pool.submit(rule, found, rung, claude_dir, effects) for found in cases]
|
||||
kinds = []
|
||||
for future in running:
|
||||
kind = future.result()
|
||||
if isinstance(kind, Limited):
|
||||
pool.shutdown(cancel_futures=True)
|
||||
return kind
|
||||
kinds.append(kind)
|
||||
return kinds
|
||||
|
||||
|
||||
def rule(case: Case, rung: Rung, claude_dir: pathlib.Path, effects: Effects) -> Kind | Limited:
|
||||
premises, records, entries = read_premises(case.root), read_records(case.root, claude_dir), read_transcript(str(case.transcript))
|
||||
if isinstance(premises, Stop):
|
||||
return verdict_kind(Failure(premises.reason))
|
||||
if isinstance(records, Stop):
|
||||
return verdict_kind(Failure(records.reason))
|
||||
if isinstance(entries, Failure):
|
||||
return verdict_kind(entries)
|
||||
verdict = first_reply(lambda: judge_once(case.root, case.task, premises, records, render(entries), rung, effects))
|
||||
return verdict if isinstance(verdict, Limited) else verdict_kind(verdict)
|
||||
|
||||
|
||||
def first_reply(attempt: typing.Callable[[], Verdict | Failure | Limited]) -> Verdict | Failure | Limited:
|
||||
"""The first reply that is no Failure, or the last of FAILURES_RUNNING failures.
|
||||
|
||||
It counts failures as the loop's retried does, but neither waits between attempts nor waits out a usage limit, which ends the replay.
|
||||
"""
|
||||
for _ in range(FAILURES_RUNNING - 1):
|
||||
if not isinstance(verdict := attempt(), Failure):
|
||||
return verdict
|
||||
return attempt()
|
||||
@@ -0,0 +1,117 @@
|
||||
"""Which archived verdicts to replay, the cheapest rung that returns them, and the report: docs/decisions.md → `routing: on a model judge-replay covered`."""
|
||||
import collections
|
||||
import dataclasses
|
||||
import pathlib
|
||||
import typing
|
||||
|
||||
from transcript_archive.outcomes import ANSWERED, FINISHED, first_verdict, INTERRUPTED, task_of, UNFINISHED
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict
|
||||
from loop_claude.cycle.retry import Limited
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
# claude's reply as the replay compares it: a verdict as the archive words it, such as FINISHED, or a failure with its reason.
|
||||
Kind = str
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Case:
|
||||
"""One archived verdict: on_record, judged at its UTC stamp."""
|
||||
root: str
|
||||
task: str
|
||||
transcript: pathlib.Path
|
||||
at: str
|
||||
on_record: Kind
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Tried:
|
||||
"""A rung, and each case it returned another kind on, with that kind."""
|
||||
rung: Rung
|
||||
missed: list[tuple[Case, Kind]]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Replay:
|
||||
"""unreproduced holds each case the reference judge returned another kind on; tried, each rung tried, cheapest first; cheapest, the first rung that missed none, or None."""
|
||||
unreproduced: list[tuple[Case, Kind]]
|
||||
tried: list[Tried]
|
||||
cheapest: Rung | None
|
||||
|
||||
|
||||
def case_from(root: str, outcomes: str, transcript: pathlib.Path) -> Case | None:
|
||||
"""The first verdict an outcomes file holds: the judge ruled on the transcript copy beside it as the copy stands. None when it holds none.
|
||||
|
||||
A fourth quoted answer in a cycle is archived as an interrupt. The reference judge replays it as a quoted answer, so it lands among the unreproduced verdicts and no rung is held to it.
|
||||
"""
|
||||
task, verdict = task_of(outcomes), first_verdict(outcomes)
|
||||
if task is None or verdict is None:
|
||||
return None
|
||||
return Case(root, task, transcript, *verdict)
|
||||
|
||||
|
||||
def sample(cases: list[Case], per_kind: int) -> list[Case]:
|
||||
"""The latest per_kind cases of each kind of verdict, in the order they were judged."""
|
||||
seen: collections.Counter[Kind] = collections.Counter()
|
||||
kept = []
|
||||
for found in sorted(cases, key=lambda found: (found.at, found.root, str(found.transcript)), reverse=True):
|
||||
seen[found.on_record] += 1
|
||||
if seen[found.on_record] <= per_kind:
|
||||
kept.append(found)
|
||||
return kept[::-1]
|
||||
|
||||
|
||||
def verdict_kind(verdict: Verdict | Failure) -> Kind:
|
||||
match verdict:
|
||||
case Finished():
|
||||
return FINISHED
|
||||
case Unfinished():
|
||||
return UNFINISHED
|
||||
case Interrupt():
|
||||
return INTERRUPTED
|
||||
case QuotedAnswer():
|
||||
return ANSWERED
|
||||
case Failure(reason):
|
||||
return f"failed: {reason}"
|
||||
case _:
|
||||
typing.assert_never(verdict)
|
||||
|
||||
|
||||
def replayed(cases: list[Case], reference: Rung, candidates: list[Rung], rule_all: typing.Callable[[list[Case], Rung], list[Kind] | Limited]) -> Replay | Limited:
|
||||
"""Replays the cases on the reference rung, then only those it reproduces on each candidate, cheapest first, until one returns every verdict."""
|
||||
ruled = rule_all(cases, reference)
|
||||
if isinstance(ruled, Limited):
|
||||
return ruled
|
||||
reproduced = [found for found, kind in zip(cases, ruled) if kind == found.on_record]
|
||||
unreproduced = [(found, kind) for found, kind in zip(cases, ruled) if kind != found.on_record]
|
||||
tried: list[Tried] = []
|
||||
# With no verdict to hold a rung to, every rung would miss none.
|
||||
for rung in candidates if reproduced else []:
|
||||
ruled = rule_all(reproduced, rung)
|
||||
if isinstance(ruled, Limited):
|
||||
return ruled
|
||||
tried.append(Tried(rung, [(found, kind) for found, kind in zip(reproduced, ruled) if kind != found.on_record]))
|
||||
if not tried[-1].missed:
|
||||
return Replay(unreproduced, tried, rung)
|
||||
return Replay(unreproduced, tried, None)
|
||||
|
||||
|
||||
def report(model: str, reference: Rung, cases: list[Case], replay: Replay) -> str:
|
||||
kinds = collections.Counter(found.on_record for found in cases)
|
||||
lines = [
|
||||
f"Replayed {len(cases)} verdicts of {model} projects: {', '.join(f'{count} {kind}' for kind, count in sorted(kinds.items()))}.",
|
||||
f"The reference judge, {reference}, reproduced {len(cases) - len(replay.unreproduced)}{'. It returned another kind on:' if replay.unreproduced else '.'}",
|
||||
*missed_lines(replay.unreproduced),
|
||||
]
|
||||
for tried in replay.tried:
|
||||
lines += [f"{tried.rung} missed {len(tried.missed)}{':' if tried.missed else '.'}", *missed_lines(tried.missed)]
|
||||
if not replay.tried:
|
||||
lines.append("No rung tried: the reference judge reproduced no verdict to hold one to.")
|
||||
else:
|
||||
cheapest = replay.cheapest or "none up to the project's rung"
|
||||
lines.append(f"Cheapest rung that returns every verdict the reference judge reproduces: {cheapest}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def missed_lines(missed: list[tuple[Case, Kind]]) -> list[str]:
|
||||
return [f" {found.root} {found.at}: on record {found.on_record}, replayed {kind}" for found, kind in missed]
|
||||
@@ -0,0 +1,209 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINISHED
|
||||
from judge_replay.replay import Case, case_from, Replay, replayed, report, sample, Tried, verdict_kind
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinished
|
||||
from loop_claude.cycle.retry import Limited
|
||||
from judge_replay import main
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
TOOL = pathlib.Path(__file__).resolve().parents[1] / "judge-replay"
|
||||
TRANSCRIPT = pathlib.Path("/archive/p/c/a.jsonl")
|
||||
CHEAP = Rung("claude-haiku-5-5", "xhigh")
|
||||
MIDDLE = Rung("claude-sonnet-5-5", "high")
|
||||
REFERENCE = Rung("claude-opus-5-5", "xhigh")
|
||||
|
||||
|
||||
def case(at, on_record):
|
||||
return Case("/code/p", "Do the next chunk.", TRANSCRIPT, at, on_record)
|
||||
|
||||
|
||||
class CaseFrom(unittest.TestCase):
|
||||
def test_takes_the_task_and_the_first_verdict_on_the_copy(self):
|
||||
outcomes = "Task: Do the next chunk.\nThen merge it.\n\n2026-10-14T10:00:00Z Stopped: claude failed\n\nWorker: x\n2026-10-14T11:00:00Z Verdict: unfinished\n\nEvidence:\n> on\n\nWorker: x\n2026-10-14T11:00:01Z Stepped up\n\nWorker: y\n"
|
||||
self.assertEqual(case_from("/code/p", outcomes, TRANSCRIPT), Case("/code/p", "Do the next chunk.\nThen merge it.", TRANSCRIPT, "2026-10-14T11:00:00Z", UNFINISHED))
|
||||
|
||||
def test_reads_each_kind_of_verdict(self):
|
||||
for verdict in (FINISHED, UNFINISHED, INTERRUPTED, ANSWERED):
|
||||
found = case_from("/code/p", f"Task: t\n\n2026-10-14T10:00:00Z {verdict}\n\nmore\n", TRANSCRIPT)
|
||||
self.assertEqual(found.on_record, verdict)
|
||||
|
||||
def test_skips_a_copy_with_no_verdict(self):
|
||||
self.assertIsNone(case_from("/code/p", "Task: t\n\n2026-10-14T10:00:00Z Stepped up\n\nWorker: y\n", TRANSCRIPT))
|
||||
self.assertIsNone(case_from("/code/p", "not an outcomes file", TRANSCRIPT))
|
||||
|
||||
|
||||
class Sample(unittest.TestCase):
|
||||
def test_keeps_the_latest_of_each_kind_in_the_order_they_were_judged(self):
|
||||
cases = [case("2026-10-14T10:00:01Z", FINISHED), case("2026-10-14T10:00:03Z", FINISHED), case("2026-10-14T10:00:02Z", FINISHED), case("2026-10-14T10:00:00Z", INTERRUPTED)]
|
||||
self.assertEqual(sample(cases, 2), [cases[3], cases[2], cases[1]])
|
||||
|
||||
|
||||
class VerdictKind(unittest.TestCase):
|
||||
def test_names_each_verdict_as_the_archive_does(self):
|
||||
self.assertEqual(verdict_kind(Finished("e")), FINISHED)
|
||||
self.assertEqual(verdict_kind(Unfinished("e", 3)), UNFINISHED)
|
||||
self.assertEqual(verdict_kind(Interrupt("e", "q?")), INTERRUPTED)
|
||||
self.assertEqual(verdict_kind(QuotedAnswer("e", "q?", "a")), ANSWERED)
|
||||
self.assertEqual(verdict_kind(Failure("the judge quoted x")), "failed: the judge quoted x")
|
||||
|
||||
|
||||
class Replayed(unittest.TestCase):
|
||||
def test_tries_rungs_cheapest_first_on_the_verdicts_the_reference_reproduces_and_stops_at_the_first_that_returns_all(self):
|
||||
cases = [case("2026-10-14T10:00:00Z", FINISHED), case("2026-10-14T10:00:01Z", INTERRUPTED), case("2026-10-14T10:00:02Z", FINISHED)]
|
||||
replies = {REFERENCE: [FINISHED, INTERRUPTED, UNFINISHED], CHEAP: [FINISHED, FINISHED], MIDDLE: [FINISHED, INTERRUPTED]}
|
||||
asked = []
|
||||
|
||||
def rule_all(on, rung):
|
||||
asked.append((len(on), rung))
|
||||
return replies[rung]
|
||||
|
||||
replay = replayed(cases, REFERENCE, [CHEAP, MIDDLE, Rung("claude-opus-5-5", "medium")], rule_all)
|
||||
self.assertEqual(replay, Replay([(cases[2], UNFINISHED)], [Tried(CHEAP, [(cases[1], FINISHED)]), Tried(MIDDLE, [])], MIDDLE))
|
||||
self.assertEqual(asked, [(3, REFERENCE), (2, CHEAP), (2, MIDDLE)])
|
||||
|
||||
def test_names_no_rung_when_every_candidate_misses_a_verdict(self):
|
||||
cases = [case("2026-10-14T10:00:00Z", FINISHED)]
|
||||
replay = replayed(cases, REFERENCE, [CHEAP], lambda on, rung: [FINISHED] if rung == REFERENCE else [INTERRUPTED])
|
||||
self.assertEqual(replay, Replay([], [Tried(CHEAP, [(cases[0], INTERRUPTED)])], None))
|
||||
|
||||
def test_tries_no_rung_when_the_reference_reproduces_no_verdict(self):
|
||||
cases = [case("2026-10-14T10:00:00Z", FINISHED)]
|
||||
self.assertEqual(replayed(cases, REFERENCE, [CHEAP], lambda on, rung: [INTERRUPTED] * len(on)), Replay([(cases[0], INTERRUPTED)], [], None))
|
||||
|
||||
def test_stops_at_a_usage_limit(self):
|
||||
limit = Limited(None, "limit")
|
||||
self.assertEqual(replayed([case("2026-10-14T10:00:00Z", FINISHED)], REFERENCE, [CHEAP], lambda on, rung: limit), limit)
|
||||
self.assertEqual(replayed([case("2026-10-14T10:00:00Z", FINISHED)], REFERENCE, [CHEAP], lambda on, rung: [FINISHED] if rung == REFERENCE else limit), limit)
|
||||
|
||||
|
||||
class Report(unittest.TestCase):
|
||||
def test_counts_the_verdicts_and_names_each_miss_and_the_cheapest_rung(self):
|
||||
cases = [case("2026-10-14T10:00:00Z", FINISHED), case("2026-10-14T10:00:01Z", INTERRUPTED)]
|
||||
replay = Replay([(cases[0], UNFINISHED)], [Tried(CHEAP, [(cases[1], FINISHED)]), Tried(MIDDLE, [])], MIDDLE)
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, replay).splitlines(), [
|
||||
"Replayed 2 verdicts of claude-opus-5-5 projects: 1 Verdict: finished, 1 Verdict: interrupt.",
|
||||
"The reference judge, claude-opus-5-5 at xhigh effort, reproduced 1. It returned another kind on:",
|
||||
" /code/p 2026-10-14T10:00:00Z: on record Verdict: finished, replayed Verdict: unfinished",
|
||||
"claude-haiku-5-5 at xhigh effort missed 1:",
|
||||
" /code/p 2026-10-14T10:00:01Z: on record Verdict: interrupt, replayed Verdict: finished",
|
||||
"claude-sonnet-5-5 at high effort missed 0.",
|
||||
"Cheapest rung that returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort",
|
||||
])
|
||||
missed_all = report("claude-opus-5-5", REFERENCE, cases, Replay([], [Tried(CHEAP, [(cases[1], FINISHED)])], None)).splitlines()
|
||||
self.assertEqual(missed_all[1], "The reference judge, claude-opus-5-5 at xhigh effort, reproduced 2.")
|
||||
self.assertEqual(missed_all[-1], "Cheapest rung that returns every verdict the reference judge reproduces: none up to the project's rung")
|
||||
self.assertEqual(report("claude-opus-5-5", REFERENCE, cases, Replay([(cases[0], UNFINISHED), (cases[1], FINISHED)], [], None)).splitlines()[-1], "No rung tried: the reference judge reproduced no verdict to hold one to.")
|
||||
|
||||
|
||||
def unused(*_):
|
||||
raise AssertionError("not called")
|
||||
|
||||
|
||||
class Main(unittest.TestCase):
|
||||
"""The tool on a temporary home: a project on Opus 5.5 with one finished and one interrupt verdict, and one on another model."""
|
||||
|
||||
def setUp(self):
|
||||
self.home = pathlib.Path(self.enterContext(tempfile.TemporaryDirectory()))
|
||||
(self.home / ".claude").mkdir()
|
||||
(self.home / ".claude" / "loop-claude-models.json").write_text(json.dumps({"proj": "claude-opus-5-5", "other": "claude-sonnet-5-5"}))
|
||||
(self.home / ".claude" / "CLAUDE.md").write_text("Be kind.\n")
|
||||
for key, verdicts in (("proj", (FINISHED, INTERRUPTED)), ("other", (FINISHED,))):
|
||||
(self.home / "code" / key).mkdir(parents=True)
|
||||
(self.home / "code" / key / "README.md").write_text("# p\n\n## Goals\n\n1. Ship.\n\n## Audience\n\nMe.\n")
|
||||
chunk = self.home / ".claude" / "loop-claude-transcripts" / key / "c1"
|
||||
chunk.mkdir(parents=True)
|
||||
for n, verdict in enumerate(verdicts):
|
||||
(chunk / f"t{n}.jsonl").write_text(json.dumps({"type": "assistant", "message": {"content": [{"type": "text", "text": f"said {verdict}"}]}}) + "\n")
|
||||
(chunk / f"t{n}.outcomes.md").write_text(f"Task: Do it.\n\n2026-10-14T10:00:0{n}Z {verdict}\n\nEvidence:\n> e\n")
|
||||
self.calls = []
|
||||
self.prompts = []
|
||||
|
||||
def run_main(self, argv, reply):
|
||||
def claude(args, cwd, prompt):
|
||||
self.calls.append((args[args.index("--model") + 1], args[args.index("--effort") + 1], cwd))
|
||||
self.prompts.append(prompt)
|
||||
result = reply(Rung(args[args.index("--model") + 1], args[args.index("--effort") + 1]), "said Verdict: interrupt" in prompt.rsplit("latest last:", 1)[-1])
|
||||
return subprocess.CompletedProcess(args, 0, json.dumps(result), "")
|
||||
|
||||
effects = Effects(claude=claude, claude_on_terminal=unused, execv=unused, git=unused, input=unused, local_time=unused, now=unused, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused)
|
||||
out, err = io.StringIO(), io.StringIO()
|
||||
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
code = main.main(argv, {}, self.home, effects)
|
||||
return code, out.getvalue(), err.getvalue()
|
||||
|
||||
@staticmethod
|
||||
def verdict(interrupt, evidence):
|
||||
output = {"answer": "", "bundled": 0, "evidence": evidence, "finished": not interrupt, "interrupt": interrupt, "question": "Which?" if interrupt else ""}
|
||||
return {"type": "result", "structured_output": output}
|
||||
|
||||
def test_replays_the_models_projects_on_the_reference_then_cheapest_first_and_prints_the_rung_that_returns_all(self):
|
||||
# Haiku misses the interrupt; Sonnet returns both.
|
||||
def reply(rung, interrupted):
|
||||
return self.verdict(interrupted and rung.model != "claude-haiku-5-5", f"said {INTERRUPTED if interrupted else FINISHED}")
|
||||
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], reply)
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual({cwd for _, _, cwd in self.calls}, {str(self.home / "code" / "proj")})
|
||||
self.assertEqual([(model, effort) for model, effort, _ in self.calls[::2]], [("claude-opus-5-5", "xhigh"), ("claude-haiku-5-5", "xhigh"), ("claude-haiku-5-5", "max"), ("claude-sonnet-5-5", "high")])
|
||||
self.assertIn("The reference judge, claude-opus-5-5 at xhigh effort, reproduced 2.", out)
|
||||
self.assertTrue(out.endswith("returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort\n"))
|
||||
for said in ("The task an agent was given:\nDo it.", "1. Ship.", "From ~/.claude/CLAUDE.md:\nBe kind."):
|
||||
self.assertIn(said, self.prompts[0])
|
||||
|
||||
def test_tries_rungs_up_to_the_models_cheapest_and_names_none_when_all_miss(self):
|
||||
code, out, _ = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted and rung == REFERENCE, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
self.assertEqual(code, 0)
|
||||
self.assertEqual(self.calls[-1][:2], ("claude-opus-5-5", "medium"))
|
||||
self.assertTrue(out.endswith(": none up to the project's rung\n"))
|
||||
|
||||
def test_replays_a_verdict_whose_transcript_is_gone_as_failed(self):
|
||||
(self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1" / "t0.jsonl").unlink()
|
||||
code, out, _ = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED}"))
|
||||
self.assertEqual(code, 0)
|
||||
self.assertIn("replayed failed: could not read the transcript", out)
|
||||
|
||||
def test_tries_up_to_the_models_cheapest_rung_and_retries_a_failed_reply_as_the_loop_does(self):
|
||||
def reply(rung, interrupted):
|
||||
return self.verdict(interrupted, "not in the transcript" if rung.model != "claude-opus-5-5" else f"said {INTERRUPTED if interrupted else FINISHED}")
|
||||
|
||||
code, out, _ = self.run_main(["claude-opus-5-5"], reply)
|
||||
self.assertEqual(code, 0)
|
||||
self.assertEqual(sum(model == "claude-sonnet-5-5" for model, _, _ in self.calls), 6)
|
||||
self.assertIn("claude-opus-5-5 at medium effort missed 0.", out)
|
||||
|
||||
def test_exits_1_on_a_usage_limit_bad_arguments_or_an_unreadable_models_file(self):
|
||||
code, _, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: {"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("usage limit: You've hit your limit", err)
|
||||
for argv in ([], ["off-ladder"], ["claude-opus-5-5", "x"]):
|
||||
self.assertEqual(self.run_main(argv, unused)[0], 1)
|
||||
code, _, err = self.run_main(["claude-haiku-5-5"], unused)
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("no archived verdicts of projects on claude-haiku-5-5", err)
|
||||
(self.home / ".claude" / "loop-claude-models.json").write_text("[")
|
||||
self.assertEqual(self.run_main(["claude-opus-5-5"], unused)[0], 1)
|
||||
|
||||
def test_prints_the_help(self):
|
||||
code, out, _ = self.run_main(["-h"], unused)
|
||||
self.assertEqual((code, out.splitlines()[2]), (0, "usage: judge-replay <model>"))
|
||||
|
||||
def test_the_launcher_prints_the_help(self):
|
||||
ran = subprocess.run([sys.executable, str(TOOL), "--help"], capture_output=True, text=True)
|
||||
self.assertEqual(ran.returncode, 0)
|
||||
self.assertIn("usage: judge-replay <model>", ran.stdout)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -9,6 +9,7 @@ from failure import Failure
|
||||
from claude_code.entries import hit_limit, limit_said, parse_entry, structured_output
|
||||
from loop_claude.project.records import HUMAN_RULE_NAMES
|
||||
from loop_claude.cycle.retry import Limited, retried
|
||||
from model_routing.ladder import Rung
|
||||
from loop_claude.stop import Stop
|
||||
from todo_table import TOP_SCORE
|
||||
|
||||
@@ -119,20 +120,28 @@ def verdict_from(output: object) -> Verdict | Failure:
|
||||
return Unfinished(output["evidence"], output["bundled"])
|
||||
|
||||
|
||||
def judge(root: str, task: str, premises: str, records: str, rendered_transcript: str, model: str, effects: Effects) -> Verdict | Stop:
|
||||
def judge(root: str, task: str, premises: str, records: str, rendered_transcript: str, rung: Rung, effects: Effects) -> Verdict | Stop:
|
||||
return retried("judge", lambda: judge_once(root, task, premises, records, rendered_transcript, rung, effects), effects)
|
||||
|
||||
|
||||
def judge_once(root: str, task: str, premises: str, records: str, rendered_transcript: str, rung: Rung, effects: Effects) -> Verdict | Failure | Limited:
|
||||
"""One judge run, as judge retries it; judge-replay replays it."""
|
||||
prompt, shown = judge_prompt(task, premises, records, rendered_transcript)
|
||||
return run_judge(root, judge_args(rung), prompt, shown, f"{premises}\n\n{records}", effects)
|
||||
|
||||
|
||||
def judge_args(rung: Rung) -> list[str]:
|
||||
# --json-schema has claude check its reply against the schema and return it as the json result's structured_output.
|
||||
# No tools, so the verdict rests on the transcript grounded() checks it against; no persistence, so its runs stay out of the session picker.
|
||||
args = [
|
||||
return [
|
||||
"-p",
|
||||
"--json-schema", json.dumps(JUDGE_SCHEMA),
|
||||
"--model", model,
|
||||
*rung.flags(),
|
||||
"--no-session-persistence",
|
||||
"--output-format", "json",
|
||||
"--permission-mode", "bypassPermissions",
|
||||
"--tools", "",
|
||||
]
|
||||
prompt, shown = judge_prompt(task, premises, records, rendered_transcript)
|
||||
return retried("judge", lambda: run_judge(root, args, prompt, shown, f"{premises}\n\n{records}", effects), effects)
|
||||
|
||||
|
||||
def run_judge(root: str, args: list[str], prompt: str, shown: str, quotable: str, effects: Effects) -> Verdict | Failure | Limited:
|
||||
|
||||
@@ -9,6 +9,7 @@ from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Interrupt, judge, QuotedAnswer, Unfinished
|
||||
from loop_claude.cycle.interrupt import interrupt
|
||||
from loop_claude.routing import judge_rung
|
||||
from transcript_archive import outcomes
|
||||
from loop_claude.paths import Paths
|
||||
from loop_claude.cycle import prompts
|
||||
@@ -58,6 +59,11 @@ def stopped(cycle: Cycle, stop: Stop, why: str) -> Stop:
|
||||
return Stop(1, f"{failure.reason}\n{stop.reason}".rstrip()) if failure and failure.reason != stop.reason else stop
|
||||
|
||||
|
||||
def judge_of(cycle: Cycle) -> Rung:
|
||||
"""The judge's rung, which the archive names beside each verdict."""
|
||||
return judge_rung(cycle.judge_model, cycle.route)
|
||||
|
||||
|
||||
def step(cycle: Cycle, state: Active) -> Event | Stop:
|
||||
match state:
|
||||
case Working(answered, _):
|
||||
@@ -94,7 +100,7 @@ def worker_start(cycle: Cycle, answered: QuotedAnswer | None) -> list[str]:
|
||||
|
||||
def keep_answer(cycle: Cycle, answered: QuotedAnswer) -> Stop | None:
|
||||
path = transcript_file(cycle.session)
|
||||
if path and (failure := archive(cycle.project_archive, path, cycle.task, outcomes.answered(answered.question, prompts.QUOTE_SOURCES, answered.answer, answered.evidence), cycle.effects.now())):
|
||||
if path and (failure := archive(cycle.project_archive, path, cycle.task, outcomes.answered(answered.question, prompts.QUOTE_SOURCES, answered.answer, answered.evidence, str(judge_of(cycle))), cycle.effects.now())):
|
||||
return Stop(1, failure.reason)
|
||||
return None
|
||||
|
||||
@@ -103,14 +109,14 @@ def rule(cycle: Cycle, transcript_path: str) -> Ruled | Stop:
|
||||
entries = read_transcript(transcript_path)
|
||||
if isinstance(entries, Failure):
|
||||
return Stop(1, entries.reason)
|
||||
verdict = judge(cycle.root, cycle.task, cycle.premises, cycle.records, render(entries), cycle.judge_model, cycle.effects)
|
||||
verdict = judge(cycle.root, cycle.task, cycle.premises, cycle.records, render(entries), judge_of(cycle), cycle.effects)
|
||||
return verdict if isinstance(verdict, Stop) else Ruled(verdict)
|
||||
|
||||
|
||||
def ask(cycle: Cycle, transcript_path: str, ruling: Interrupt | QuotedAnswer) -> HumanAnswered | Stop:
|
||||
question = question_for(ruling)
|
||||
# Archived now, so the transcript the question was asked on is kept with it; the human's answer then grows the transcript.
|
||||
if failure := archive(cycle.project_archive, transcript_path, cycle.task, outcomes.interrupted(question, ruling.evidence), cycle.effects.now()):
|
||||
if failure := archive(cycle.project_archive, transcript_path, cycle.task, outcomes.interrupted(question, ruling.evidence, str(judge_of(cycle))), cycle.effects.now()):
|
||||
return Stop(1, failure.reason)
|
||||
return interrupt(cycle.root, cycle.session.id, cycle.route.rung, transcript_path, question, cycle.paths, cycle.effects) or HumanAnswered()
|
||||
|
||||
@@ -123,7 +129,7 @@ def question_for(ruling: Interrupt | QuotedAnswer) -> str:
|
||||
|
||||
def close(cycle: Cycle, transcript_path: str, verdict: Finished | Unfinished) -> Archived | Stop:
|
||||
finished = isinstance(verdict, Finished)
|
||||
if failure := archive(cycle.project_archive, transcript_path, cycle.task, outcomes.finished(str(cycle.route), verdict.evidence) if finished else outcomes.unfinished(str(cycle.route), verdict.evidence), cycle.effects.now()):
|
||||
if failure := archive(cycle.project_archive, transcript_path, cycle.task, outcomes.finished(str(cycle.route), verdict.evidence, str(judge_of(cycle))) if finished else outcomes.unfinished(str(cycle.route), verdict.evidence, str(judge_of(cycle))), cycle.effects.now()):
|
||||
return Stop(1, failure.reason)
|
||||
if finished and (failure := delete_session(cycle.session)):
|
||||
return Stop(1, failure.reason)
|
||||
|
||||
@@ -28,16 +28,16 @@ interrupt, and as every cycle ends, Ctrl-C and a failure included, the session's
|
||||
copied to $LOOP_CLAUDE_TRANSCRIPTS/<key>/<chunk>/<sha256>.jsonl (default
|
||||
~/.claude/loop-claude-transcripts), named by its hash. <chunk> is the hash of the transcript's first
|
||||
line with a timestamp, the same for every cycle of a chunk. Beside the copy, <sha256>.outcomes.md
|
||||
holds the verdict, with the transcript passage the judge rested it on; each answer the judge
|
||||
quoted, with the question it answers; and why the loop stopped. Each of these opens with its UTC
|
||||
holds the verdict, with the transcript passage the judge rested it on and the judge's rung; each
|
||||
answer the judge quoted, with the question it answers; and why the loop stopped. Each of these opens with its UTC
|
||||
time. Claude Code keeps a folder per session beside its transcript, holding the subagents'
|
||||
transcripts among other files; that folder, and every other file named after the session except
|
||||
the transcript, are copied into <chunk>/session/. A file there is replaced only by a version that
|
||||
grew from it. A file Claude Code rewrote lands beside its copy as <name>.<hash of its content>.
|
||||
|
||||
Every claude loop-claude starts runs the project's model, saved per <key> in
|
||||
~/.claude/loop-claude-models.json and read each cycle, unless routing, below, puts the worker and
|
||||
the interrupt on another rung; a project without an entry, as on its first
|
||||
~/.claude/loop-claude-models.json and read each cycle, unless routing, below, puts the worker,
|
||||
the interrupt and the judge on another rung; a project without an entry, as on its first
|
||||
run, is asked for one. LOOP_CLAUDE_JUDGE_MODEL overrides it for the judge, LOOP_CLAUDE_WORKER_MODEL
|
||||
for the worker and the interrupt; with both set, nothing is asked.
|
||||
|
||||
@@ -60,6 +60,12 @@ and counts its unfinished verdicts afresh. A rung no longer on the ladder, or be
|
||||
item's R + C picks, is dropped, and the chunk is routed afresh from todo.md. A todo.md or an
|
||||
archive that cannot be read or parsed is said on stderr, and the chunk is routed without it.
|
||||
|
||||
The judge of a model listed in JUDGE_REPLAYED (bin/loop_claude/routing.py) runs on the rung the
|
||||
routed item's R + C picks from that model, never below the model's cheapest rung, and on the
|
||||
cheapest rung when no item routed the chunk. Its rung stays put when the worker steps up or takes in
|
||||
another item. A model is listed once judge-replay found its cheapest rung returning every verdict
|
||||
the reference judge reproduced. Any other model's judge runs at the effort its settings give.
|
||||
|
||||
Counting the cycles a verdict closed across all projects in the archive, the first loop to close
|
||||
one 20 or more cycles after the last measurement measures every finished chunk there, with no model
|
||||
run: tokens by project, subagent type, rung and the tool whose results they took in, and, for each
|
||||
|
||||
@@ -1,12 +1,14 @@
|
||||
"""Which ladder rung a chunk's worker runs on: docs/decisions.md → `routing:`."""
|
||||
"""Which ladder rung a chunk's worker and judge run on: docs/decisions.md → `routing:`."""
|
||||
import dataclasses
|
||||
|
||||
from failure import Failure
|
||||
from model_routing.ladder import highest, LADDER, picked, Rung, stepped_up
|
||||
from model_routing.ladder import highest, LADDER, picked, project_rung, Rung, stepped_up
|
||||
from todo_table import Item, items
|
||||
from model_routing.route import Route
|
||||
|
||||
UNFINISHED_PER_STEP = 2
|
||||
# The models `judge-replay` found returning, on their cheapest rung, every verdict the reference judge reproduced: docs/decisions.md → `routing: on a model judge-replay covered`.
|
||||
JUDGE_REPLAYED = ("claude-opus-5-5",)
|
||||
|
||||
|
||||
def routed_item(todo: str) -> Item | Failure | None:
|
||||
@@ -27,8 +29,13 @@ def first_route(anchor: Rung, item: Item | None) -> Route:
|
||||
|
||||
|
||||
def kept(route: Route, anchor: Rung) -> bool:
|
||||
"""Whether a chunk keeps route: its rung is on the ladder at or above the rung its item's R + C picks from anchor, the project's rung, or anchor itself when no item routed it."""
|
||||
return highest(route.rung, picked(anchor, route.risk_and_complexity) if route.item else anchor) == route.rung
|
||||
"""Whether a chunk keeps route: its rung is on the ladder at or above item_rung."""
|
||||
return highest(route.rung, item_rung(anchor, route)) == route.rung
|
||||
|
||||
|
||||
def item_rung(anchor: Rung, route: Route) -> Rung:
|
||||
"""The rung the route's item's R + C picks from anchor, the project's rung, or anchor itself when no item routed it. The worker may run on a rung below anchor; judge_rung lifts the judge's rung to anchor."""
|
||||
return picked(anchor, route.risk_and_complexity) if route.item else anchor
|
||||
|
||||
|
||||
def stepped(anchor: Rung, route: Route, bundled: int) -> Route:
|
||||
@@ -39,3 +46,16 @@ def stepped(anchor: Rung, route: Route, bundled: int) -> Route:
|
||||
if route.unfinished + 1 >= UNFINISHED_PER_STEP:
|
||||
return dataclasses.replace(route, rung=stepped_up(rung), unfinished=0)
|
||||
return dataclasses.replace(route, rung=rung, unfinished=route.unfinished + 1)
|
||||
|
||||
|
||||
def judge_rung(judge_model: str, route: Route) -> Rung:
|
||||
"""The rung the route's item picks from the judge model's cheapest rung, never below that rung, whatever rung the worker stepped to.
|
||||
|
||||
An item the worker takes in moves the worker's rung but not the judge's, until todo.md item 136 routes the chunk on the item the worker takes.
|
||||
|
||||
A model off the ladder, or not in JUDGE_REPLAYED, gets no effort flag, so claude runs it at the effort its settings give.
|
||||
"""
|
||||
anchor = project_rung(judge_model)
|
||||
if anchor is None or judge_model not in JUDGE_REPLAYED:
|
||||
return Rung(judge_model)
|
||||
return highest(anchor, item_rung(anchor, route))
|
||||
|
||||
@@ -364,8 +364,8 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual([path.name for path in checkout.transcripts()[0].parent.iterdir()], [checkout.transcripts()[0].name])
|
||||
self.assertIn("=== cycle 3, worker on worker-model ===", run.stdout)
|
||||
copies = {verdict.removeprefix(f"Task: {TASK}\n\n"): copy for copy, verdict in checkout.archived().items()}
|
||||
self.assertEqual(sorted(copies), ["<UTC> Stopped: Ctrl-C\n\nWorker: worker-model\n", "<UTC> Verdict: finished\n\nEvidence:\n> landed-it\n\nWorker: worker-model\n", "<UTC> Verdict: unfinished\n\nEvidence:\n> started\n\nWorker: worker-model\n"])
|
||||
self.assertIn('"text": "landed-it"', copies["<UTC> Verdict: finished\n\nEvidence:\n> landed-it\n\nWorker: worker-model\n"])
|
||||
self.assertEqual(sorted(copies), ["<UTC> Stopped: Ctrl-C\n\nWorker: worker-model\n", "<UTC> Verdict: finished\n\nEvidence:\n> landed-it\n\nJudge: judge-model\nWorker: worker-model\n", "<UTC> Verdict: unfinished\n\nEvidence:\n> started\n\nJudge: judge-model\nWorker: worker-model\n"])
|
||||
self.assertIn('"text": "landed-it"', copies["<UTC> Verdict: finished\n\nEvidence:\n> landed-it\n\nJudge: judge-model\nWorker: worker-model\n"])
|
||||
self.assertIn('"text": "next chunk"', copies["<UTC> Stopped: Ctrl-C\n\nWorker: worker-model\n"])
|
||||
self.assertEqual(len(list(checkout.archive().glob(f"proj/*/session/{session}.orphaned-1-ab.jsonl"))), 1)
|
||||
|
||||
@@ -392,17 +392,20 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual({(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] == "judge"}, {("judge-model", None)})
|
||||
self.assertIn("loop-claude runs you on claude-opus-5-5 at high effort, routed by item 4's R + C of 11. Name each todo.md item this chunk takes, with its R + C, whether or not it is item 4.", argument(workers[0], "-p"))
|
||||
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> three\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's R + C of 11\n\n<UTC> Stepped up\n\nWorker: claude-fable-5-1 at max effort, routed by item 4's R + C of 11\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: finished\n\nEvidence:\n> four\n\nWorker: claude-fable-5-1 at max effort, routed by item 4's R + C of 11\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> three\n\nJudge: judge-model\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's R + C of 11\n\n<UTC> Stepped up\n\nWorker: claude-fable-5-1 at max effort, routed by item 4's R + C of 11\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: finished\n\nEvidence:\n> four\n\nJudge: judge-model\nWorker: claude-fable-5-1 at max effort, routed by item 4's R + C of 11\n", outcomes)
|
||||
|
||||
def test_runs_a_cheap_items_worker_below_the_projects_rung_with_its_reviewers_on_it(self):
|
||||
def test_runs_a_cheap_items_worker_below_the_projects_rung_with_its_reviewers_and_judge_on_it(self):
|
||||
checkout = Checkout(self)
|
||||
checkout.env["LOOP_CLAUDE_WORKER_MODEL"] = "claude-opus-5-5"
|
||||
checkout.env["LOOP_CLAUDE_WORKER_MODEL"] = checkout.env["LOOP_CLAUDE_JUDGE_MODEL"] = "claude-opus-5-5"
|
||||
(checkout.root / "todo.md").write_text(TODO.replace("| 5 | 2 | 6 |", "| 2 | 2 | 5 |"))
|
||||
run = checkout.run([worker(say="one"), judge(finished=True, evidence="one"), worker(say="next chunk")])
|
||||
self.assertEqual(run.code, 130)
|
||||
first = next(call for call in run.calls if call["role"] == "worker")
|
||||
self.assertEqual((argument(first, "--model"), argument(first, "--effort")), ("claude-sonnet-5-5", "high"))
|
||||
ruling = next(call for call in run.calls if call["role"] == "judge")
|
||||
self.assertEqual((argument(ruling, "--model"), argument(ruling, "--effort")), ("claude-opus-5-5", "medium"))
|
||||
self.assertIn("Judge: claude-opus-5-5 at medium effort\nWorker: claude-sonnet-5-5 at high effort", "".join(checkout.archived().values()))
|
||||
self.assertIn("Your rung is below the project's, claude-opus-5-5 at medium effort: launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium", argument(first, "-p"))
|
||||
|
||||
def test_a_rerun_resumes_an_open_chunk_on_the_rung_it_last_ran(self):
|
||||
@@ -539,7 +542,7 @@ class Interrupt(unittest.TestCase):
|
||||
self.assertEqual(run.stderr.strip().splitlines()[-2:], ["A or B?", "the human is needed to answer the question above, but stdin or stdout is not a terminal; rerun loop-claude on a terminal"])
|
||||
[(copy, verdict)] = checkout.archived().items()
|
||||
self.assertEqual(copy, (checkout.transcripts()[0]).read_text())
|
||||
self.assertEqual(verdict, f"Task: {TASK}\n\n<UTC> Verdict: interrupt\n\nA or B?\n\nEvidence:\n> Shall I use A or B?\n\n<UTC> Stopped: the human is needed to answer the question above, but stdin or stdout is not a terminal; rerun loop-claude on a terminal\n\nWorker: worker-model\n")
|
||||
self.assertEqual(verdict, f"Task: {TASK}\n\n<UTC> Verdict: interrupt\n\nA or B?\n\nEvidence:\n> Shall I use A or B?\n\nJudge: judge-model\n\n<UTC> Stopped: the human is needed to answer the question above, but stdin or stdout is not a terminal; rerun loop-claude on a terminal\n\nWorker: worker-model\n")
|
||||
self.assertEqual(stat.S_IMODE((checkout.temp / "deep").stat().st_mode), 0o700)
|
||||
self.assertIn(f"transcript: {checkout.archive() / 'proj'}/", run.stderr)
|
||||
session = argument(run.calls[0], "--session-id")
|
||||
|
||||
@@ -183,7 +183,7 @@ class Cycles(unittest.TestCase):
|
||||
def test_a_finished_chunk_is_archived_and_its_session_deleted(self):
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("finished")])
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)), Finished("working"))
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: finished\n\nEvidence:\n> working\n\nWorker: worker-model\n"])
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: finished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n"])
|
||||
self.assertIsNone(session.transcript_file(self.session))
|
||||
|
||||
def test_an_unfinished_chunks_transcript_is_archived_and_the_next_cycle_resumes_its_session_with_the_judges_evidence(self):
|
||||
@@ -195,7 +195,7 @@ class Cycles(unittest.TestCase):
|
||||
self.assertIn("--session-id", workers[0])
|
||||
self.assertIn("--resume", workers[1])
|
||||
self.assertIn("as the work remaining:\n\n> working\n\nThe chunk is finished once", workers[1][workers[1].index("-p") + 1])
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: unfinished\n\nEvidence:\n> working\n\nWorker: worker-model\n"] * 2)
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: unfinished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n"] * 2)
|
||||
|
||||
def test_a_cycle_stopped_by_a_failure_or_ctrl_c_archives_what_its_session_holds(self):
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[ran(code=1)] * 3)
|
||||
@@ -248,12 +248,14 @@ class Cycles(unittest.TestCase):
|
||||
return 0
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("interrupt"), judged("unfinished")], on_terminal_claude=[answer], git=[ran()])
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)), Unfinished("working"))
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: interrupt\n\nWhich one?\n\nEvidence:\n> working\n", "Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: unfinished\n\nEvidence:\n> working\n\nWorker: worker-model\n"])
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: interrupt\n\nWhich one?\n\nEvidence:\n> working\n\nJudge: judge-model\n", "Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: unfinished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n"])
|
||||
self.assertIn(str(self.session.id), fake.made("claude_on_terminal")[0])
|
||||
judge_prompts = [stdin for args, stdin in fake.made("claude")]
|
||||
self.assertEqual(len(judge_prompts), 2)
|
||||
self.assertNotIn("the second", judge_prompts[0])
|
||||
self.assertIn("[human] the second", judge_prompts[1])
|
||||
self.assertIn("From README.md:\n## Goals", judge_prompts[0])
|
||||
self.assertIn("The task an agent was given:\nDo it.", judge_prompts[0])
|
||||
|
||||
def test_a_question_the_records_answer_resumes_the_worker_with_the_answer_three_times_a_cycle(self):
|
||||
answered = judged("quoted", answer="Use *A*.")
|
||||
@@ -265,8 +267,8 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual(len(fake.made("stream_claude")), 4)
|
||||
self.assertEqual(printed.getvalue().count("=== README.md's Goals and Audience, AGENTS.md, docs/decisions.md, ~/.claude/CLAUDE.md, ~/.claude/rules/ or ~/.claude/skills/larv-review/standards/ answered: Which one? ===\nUse *A*."), 3)
|
||||
self.assertIn("Which one?\n\nREADME.md's Goals and Audience, AGENTS.md, docs/decisions.md, ~/.claude/CLAUDE.md, ~/.claude/rules/ or ~/.claude/skills/larv-review/standards/ says:\nUse *A*.\n\nThose files have answered 3 questions this cycle", said.getvalue())
|
||||
quoted = "2026-09-21T14:13:20Z Verdict: quoted answer\n\nWhich one?\n\nREADME.md's Goals and Audience, AGENTS.md, docs/decisions.md, ~/.claude/CLAUDE.md, ~/.claude/rules/ or ~/.claude/skills/larv-review/standards/ says:\n> Use *A*.\n\nEvidence:\n> working\n"
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: interrupt\n\nWhich one?\n\nREADME.md's Goals and Audience, AGENTS.md, docs/decisions.md, ~/.claude/CLAUDE.md, ~/.claude/rules/ or ~/.claude/skills/larv-review/standards/ says:\nUse *A*.\n\nThose files have answered 3 questions this cycle, the most allowed, so this one comes to you.\n\nEvidence:\n> working\n\n2026-09-21T14:13:20Z Stopped: the human is needed to answer the question above, but the interrupt ended with nothing typed\n\nWorker: worker-model\n", *[f"Task: Do it.\n\n{quoted}"] * 3])
|
||||
quoted = "2026-09-21T14:13:20Z Verdict: quoted answer\n\nWhich one?\n\nREADME.md's Goals and Audience, AGENTS.md, docs/decisions.md, ~/.claude/CLAUDE.md, ~/.claude/rules/ or ~/.claude/skills/larv-review/standards/ says:\n> Use *A*.\n\nEvidence:\n> working\n\nJudge: judge-model\n"
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: interrupt\n\nWhich one?\n\nREADME.md's Goals and Audience, AGENTS.md, docs/decisions.md, ~/.claude/CLAUDE.md, ~/.claude/rules/ or ~/.claude/skills/larv-review/standards/ says:\nUse *A*.\n\nThose files have answered 3 questions this cycle, the most allowed, so this one comes to you.\n\nEvidence:\n> working\n\nJudge: judge-model\n\n2026-09-21T14:13:20Z Stopped: the human is needed to answer the question above, but the interrupt ended with nothing typed\n\nWorker: worker-model\n", *[f"Task: Do it.\n\n{quoted}"] * 3])
|
||||
|
||||
def test_stops_when_a_quoted_answer_cannot_be_archived(self):
|
||||
self.archive.parent.mkdir(parents=True)
|
||||
@@ -295,7 +297,7 @@ class Cycles(unittest.TestCase):
|
||||
stop = run.run_cycle(self.cycle(fake))
|
||||
self.assertEqual(stop, Stop(1, f"could not delete {folder}: Permission denied; delete every file and folder matching {folder}*, then rerun loop-claude; a rerun before that resumes the finished chunk's session"))
|
||||
self.assertTrue(session.session_open(self.session))
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: finished\n\nEvidence:\n> working\n\nWorker: worker-model\n\n2026-09-21T14:13:20Z Stopped: {stop.reason}\n\nWorker: worker-model\n"])
|
||||
self.assertEqual(self.archived_outcomes(), [f"Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: finished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n\n2026-09-21T14:13:20Z Stopped: {stop.reason}\n\nWorker: worker-model\n"])
|
||||
|
||||
def test_stops_when_the_worker_leaves_no_transcript(self):
|
||||
fake = FakeEffects(streams=[Streamed(0)])
|
||||
@@ -382,8 +384,8 @@ class RoutingCheck(unittest.TestCase):
|
||||
folder = self.root / project / f"c{number}"
|
||||
subagents = folder / "session" / "s" / "subagents"
|
||||
subagents.mkdir(parents=True)
|
||||
kept = "".join(f"2026-10-14T10:0{verdict}:00Z {outcomes.unfinished('w', 'e')}\n" for verdict in range(verdicts - 1))
|
||||
(folder / "copy.outcomes.md").write_text(f"Task: t\n\n{kept}2026-10-14T11:00:00Z {outcomes.finished('w', 'm')}\n")
|
||||
kept = "".join(f"2026-10-14T10:0{verdict}:00Z {outcomes.unfinished('w', 'e', 'j')}\n" for verdict in range(verdicts - 1))
|
||||
(folder / "copy.outcomes.md").write_text(f"Task: t\n\n{kept}2026-10-14T11:00:00Z {outcomes.finished('w', 'm', 'j')}\n")
|
||||
launch = {"type": "assistant", "timestamp": "2026-10-14T09:00:00Z", "message": {"id": "m", "content": [{"type": "tool_use", "id": "a", "name": "Agent", "input": {"subagent_type": "Explore"}}]}}
|
||||
report = {"type": "user", "message": {"content": [{"type": "tool_result", "tool_use_id": "a", "content": "long report"}]}}
|
||||
(folder / "copy.jsonl").write_text(json.dumps(launch) + "\n" + json.dumps(report) + "\n")
|
||||
@@ -533,6 +535,14 @@ class Routing(unittest.TestCase):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 0).rung, self.OPUS)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 2).rung, self.OPUS)
|
||||
|
||||
def test_runs_the_judge_on_the_rung_its_chunks_item_picks_never_below_the_projects(self):
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.HIGH, 3, 11)), self.HIGH)
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.XHIGH, 3, 11)), self.HIGH)
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(Rung("claude-haiku-5-5", "xhigh"), 3, 2)), self.OPUS)
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.HIGH)), self.OPUS)
|
||||
self.assertEqual(routing.judge_rung("judge-model", Route(self.HIGH, 3, 11)), Rung("judge-model"))
|
||||
self.assertEqual(routing.judge_rung("claude-sonnet-5-5", Route(self.HIGH, 3, 11)), Rung("claude-sonnet-5-5"))
|
||||
|
||||
def test_archives_the_route_in_a_line_it_reads_back(self):
|
||||
route = Route(self.HIGH, 3, 11)
|
||||
self.assertEqual(str(route), "claude-opus-5-5 at high effort, routed by item 3's R + C of 11")
|
||||
@@ -584,7 +594,7 @@ class RouteSources(unittest.TestCase):
|
||||
self.assertIsNone(main.archived_route(archive, opened))
|
||||
route = Route(Rung("claude-opus-5-5", "high"), 3, 11)
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(route), "working"), NOW))
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(route), "working", "j"), NOW))
|
||||
self.assertEqual(main.archived_route(archive, opened), route)
|
||||
(chunk,) = archive.iterdir()
|
||||
chunk.chmod(0)
|
||||
@@ -613,7 +623,7 @@ class Steps(unittest.TestCase):
|
||||
transcript.write_text(json.dumps({"type": "queue-operation", "timestamp": "1"}) + "\n")
|
||||
before = Route(self.OPUS, 3, 9)
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(before), "working"), NOW))
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(before), "working", "j"), NOW))
|
||||
after = main.after_verdict(main.LoopRun(cycle_number=1, root="/r", task="Do it.", unfinished_evidence="", route=before), self.OPUS, Unfinished("working", 13))
|
||||
self.assertIsNone(main.archive_step(archive, opened, after, before, FakeEffects().effects()))
|
||||
self.assertEqual(main.archived_route(archive, opened), Route(Rung("claude-opus-5-5", "xhigh"), 3, 9))
|
||||
|
||||
@@ -4,3 +4,9 @@ import pathlib
|
||||
def project_key(checkout_root: str, home: pathlib.Path) -> str:
|
||||
"""The checkout's path relative to ~/code, or to / outside it: what names a project's loop-claude session, its entry in the models file and its archive folder."""
|
||||
return checkout_root.removeprefix(f"{home / 'code'}/").lstrip("/")
|
||||
|
||||
|
||||
def checkout_of(key: str, home: pathlib.Path) -> str:
|
||||
"""The checkout a project key names: under ~/code where that folder exists, else under /."""
|
||||
under_code = home / "code" / key
|
||||
return str(under_code) if under_code.is_dir() else f"/{key}"
|
||||
|
||||
@@ -125,7 +125,7 @@ class ReadingCostTool(unittest.TestCase):
|
||||
transcript.write_bytes(jsonl([{"timestamp": str(at)}, *worker]))
|
||||
(transcript.with_suffix("") / "subagents" / "agent-a.jsonl").write_bytes(jsonl([said("1", "s1", tokens=1)]))
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(archive(self.archive, str(transcript), "Do it.", outcome("m", "merged"), at))
|
||||
self.assertIsNone(archive(self.archive, str(transcript), "Do it.", outcome("m", "merged", "j"), at))
|
||||
|
||||
def run_tool(self, cwd, *args):
|
||||
return subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, cwd=cwd, env=self.env, text=True)
|
||||
|
||||
@@ -15,5 +15,5 @@ class Types(unittest.TestCase):
|
||||
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
|
||||
if shutil.which("uv") is None:
|
||||
self.fail("uv on PATH fetches mypy; see README.md's Test line")
|
||||
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "rule_hook", "-p", "score_outcomes", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
|
||||
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "judge_replay", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "rule_hook", "-p", "score_outcomes", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
|
||||
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
|
||||
|
||||
@@ -7,9 +7,9 @@ import pathlib
|
||||
import re
|
||||
import typing
|
||||
|
||||
from transcript_archive.read import archived_outcomes, chunk_outcomes, finished_chunks
|
||||
from routing_figures.measure import Chunk, ChunkFigures, measured, MEASURED_AT, Subagent
|
||||
from failure import Failure
|
||||
from transcript_archive.read import finished_chunks
|
||||
from claude_code.entries import parse_entry
|
||||
from file_writes import replace_file
|
||||
from utc_stamp import STAMPED
|
||||
@@ -50,8 +50,11 @@ def locked(root: pathlib.Path) -> typing.Iterator[bool]:
|
||||
|
||||
def cycles(root: pathlib.Path) -> int | Failure:
|
||||
"""The cycles every project's loop has closed with a verdict."""
|
||||
listed = archived_outcomes(root)
|
||||
if isinstance(listed, Failure):
|
||||
return listed
|
||||
try:
|
||||
return sum(len(VERDICT_AT.findall(path.read_text(encoding="utf-8"))) for path in root.rglob("*.outcomes.md"))
|
||||
return sum(len(VERDICT_AT.findall(path.read_text(encoding="utf-8"))) for _, path in listed)
|
||||
except (OSError, UnicodeDecodeError) as failure:
|
||||
return Failure(f"could not count the cycles in {root}: {failure}")
|
||||
|
||||
@@ -76,10 +79,10 @@ class Measured:
|
||||
|
||||
def measure_archive(root: pathlib.Path, home: str, since: str) -> Measured | Failure:
|
||||
"""The figures of each chunk that finished after since, measured one chunk at a time because the archive outgrows memory. Only an archive that cannot be listed fails."""
|
||||
try:
|
||||
projects = sorted({path.parent.parent for path in root.rglob("*.outcomes.md")})
|
||||
except OSError as failure:
|
||||
return Failure(f"could not read the archive {root}: {failure}")
|
||||
listed = archived_outcomes(root)
|
||||
if isinstance(listed, Failure):
|
||||
return listed
|
||||
projects = sorted({root / key for key, _ in listed})
|
||||
found = []
|
||||
skipped = []
|
||||
for project in projects:
|
||||
@@ -90,7 +93,7 @@ def measure_archive(root: pathlib.Path, home: str, since: str) -> Measured | Fai
|
||||
for chunk in (chunk for chunk in chunks if chunk.finished > since):
|
||||
folder = project / chunk.folder
|
||||
try:
|
||||
kept = [path.read_text(encoding="utf-8") for path in sorted(folder.glob("*.outcomes.md"))]
|
||||
kept = [path.read_text(encoding="utf-8") for path in chunk_outcomes(folder)]
|
||||
subagents = [subagent(path) for path in chunk.subagents]
|
||||
worker = entries(chunk.transcript)
|
||||
except (OSError, UnicodeDecodeError):
|
||||
|
||||
@@ -49,7 +49,7 @@ class Measured(unittest.TestCase):
|
||||
message("m2", "2026-10-14T10:05:00.000Z", {"type": "text", "text": "done"}),
|
||||
message("m3", "2026-10-14T11:00:00.000Z", {"type": "text", "text": "on"}),
|
||||
]
|
||||
kept = stamped("2026-10-14T10:30:00Z", outcomes.unfinished(LOW_ROUTE, "e")) + stamped("2026-10-14T10:30:00Z", outcomes.stepped_up(str(HIGH))) + stamped("2026-10-14T12:00:00Z", outcomes.finished(str(HIGH), "merged"))
|
||||
kept = stamped("2026-10-14T10:30:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j")) + stamped("2026-10-14T10:30:00Z", outcomes.stepped_up(str(HIGH))) + stamped("2026-10-14T12:00:00Z", outcomes.finished(str(HIGH), "merged", "j"))
|
||||
found = measured(Chunk("log", [kept], worker, []), "/h")
|
||||
self.assertEqual(found.tokens["project"], {"log": 333})
|
||||
self.assertEqual(found.tokens["subagent type"], {"worker": 333})
|
||||
@@ -67,7 +67,7 @@ class Measured(unittest.TestCase):
|
||||
Subagent("Explore", "a1", [message("s1", "2026-10-14T10:00:01Z", model="claude-sonnet-5-5"), result("r", "z" * 20)]),
|
||||
Subagent("", "a2", [message("s2", "2026-10-14T10:00:01Z")]),
|
||||
]
|
||||
found = measured(Chunk("log", [stamped("2026-10-14T11:00:00Z", outcomes.finished(LOW_ROUTE, "m"))], worker, subagents), "/h")
|
||||
found = measured(Chunk("log", [stamped("2026-10-14T11:00:00Z", outcomes.finished(LOW_ROUTE, "m", "j"))], worker, subagents), "/h")
|
||||
self.assertEqual(found.delegated, {"claude-sonnet-5-5 at high effort": 1, str(LOW): 1})
|
||||
self.assertEqual(found.tokens["subagent type"], {"worker": 111, "Explore": 222})
|
||||
self.assertEqual(found.reports, {"Explore": Reports(2, 35, 20)})
|
||||
@@ -78,14 +78,14 @@ class Measured(unittest.TestCase):
|
||||
self.assertEqual(measured(Chunk("log", [], worker, []), "/h").rounds_past_first, 1)
|
||||
|
||||
def test_starts_no_route_for_a_chunk_no_item_routed(self):
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.finished(str(LOW), "m"))
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.finished(str(LOW), "m", "j"))
|
||||
found = measured(Chunk("log", [kept], [message("m1", "2026-10-14T09:00:00Z")], []), "/h")
|
||||
self.assertIsNone(found.start)
|
||||
self.assertEqual(found.tokens["rung"], {str(LOW): 111})
|
||||
|
||||
|
||||
def test_puts_a_subagent_on_the_model_it_ran_at_the_effort_of_the_cycle_that_launched_it(self):
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.unfinished(LOW_ROUTE, "e")) + stamped("2026-10-14T11:00:00Z", outcomes.finished(str(HIGH), "m"))
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j")) + stamped("2026-10-14T11:00:00Z", outcomes.finished(str(HIGH), "m", "j"))
|
||||
worker = [
|
||||
message("m1", "2026-10-14T10:00:00.900Z"),
|
||||
message("m2", "2026-10-14T10:30:00Z", call("Agent", "a1", subagent_type="Explore", model="sonnet")),
|
||||
@@ -172,14 +172,14 @@ class Archive(unittest.TestCase):
|
||||
(folder / "session" / "s" / "subagents" / "agent-a.meta.json").write_text(json.dumps(meta))
|
||||
|
||||
def test_counts_the_verdicts_across_every_projects_chunks(self):
|
||||
self.chunk("home/x", "c1", stamped("2026-10-14T10:00:00Z", outcomes.unfinished("w", "e")) + stamped("2026-10-14T10:01:00Z", outcomes.stopped("w", "Ctrl-C")) + stamped("2026-10-14T10:02:00Z", outcomes.finished("w", "e")), [])
|
||||
self.chunk("log", "c2", stamped("2026-10-14T10:00:00Z", outcomes.interrupted("q", "e")), [])
|
||||
self.chunk("home/x", "c1", stamped("2026-10-14T10:00:00Z", outcomes.unfinished("w", "e", "j")) + stamped("2026-10-14T10:01:00Z", outcomes.stopped("w", "Ctrl-C")) + stamped("2026-10-14T10:02:00Z", outcomes.finished("w", "e", "j")), [])
|
||||
self.chunk("log", "c2", stamped("2026-10-14T10:00:00Z", outcomes.interrupted("q", "e", "j")), [])
|
||||
self.assertEqual(archive.cycles(self.root), 2)
|
||||
self.assertEqual(archive.cycles(self.root / "missing"), 0)
|
||||
|
||||
def test_measures_each_finished_chunk_keyed_by_its_project(self):
|
||||
self.chunk("home/x", "c1", stamped("2026-10-14T10:00:00Z", outcomes.finished(LOW_ROUTE, "m")), [message("m1", "2026-10-14T09:00:00Z", call("Agent", "a1", subagent_type="Explore"))], ({"agentType": "Explore", "toolUseId": "a1"}, [message("s1", "2026-10-14T09:00:01Z")]))
|
||||
self.chunk("log", "c2", stamped("2026-10-14T10:00:00Z", outcomes.unfinished(LOW_ROUTE, "e")), [message("m1", "2026-10-14T09:00:00Z")])
|
||||
self.chunk("home/x", "c1", stamped("2026-10-14T10:00:00Z", outcomes.finished(LOW_ROUTE, "m", "j")), [message("m1", "2026-10-14T09:00:00Z", call("Agent", "a1", subagent_type="Explore"))], ({"agentType": "Explore", "toolUseId": "a1"}, [message("s1", "2026-10-14T09:00:01Z")]))
|
||||
self.chunk("log", "c2", stamped("2026-10-14T10:00:00Z", outcomes.unfinished(LOW_ROUTE, "e", "j")), [message("m1", "2026-10-14T09:00:00Z")])
|
||||
found = archive.measure_archive(self.root, "/h", "")
|
||||
self.assertEqual([chunk.tokens["project"] for chunk in found.chunks], [{"home/x": 222}])
|
||||
self.assertEqual(found.chunks[0].tokens["subagent type"], {"worker": 111, "Explore": 111})
|
||||
@@ -191,17 +191,17 @@ class Archive(unittest.TestCase):
|
||||
first = message("m1", "2026-09-21T00:00:00Z")
|
||||
transcript.write_text(json.dumps(first) + "\n")
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(kept / "log", str(transcript), "t", outcomes.unfinished(LOW_ROUTE, "e"), 1790000000))
|
||||
self.assertIsNone(write.archive(kept / "log", str(transcript), "t", outcomes.unfinished(LOW_ROUTE, "e", "j"), 1790000000))
|
||||
transcript.write_text(json.dumps(first) + "\n" + json.dumps(message("m2", "2099-01-01T00:00:00Z")) + "\n")
|
||||
self.assertIsNone(write.archive(kept / "log", str(transcript), "t", outcomes.finished(str(HIGH), "m"), 4070908800))
|
||||
self.assertIsNone(write.archive(kept / "log", str(transcript), "t", outcomes.finished(str(HIGH), "m", "j"), 4070908800))
|
||||
(found,) = archive.measure_archive(kept, "/h", "").chunks
|
||||
self.assertEqual((found.start, found.unfinished), (LOW, 1))
|
||||
self.assertEqual(found.tokens["rung"], {str(LOW): 111, str(HIGH): 111})
|
||||
|
||||
def test_skips_a_chunk_it_cannot_read_and_names_it(self):
|
||||
self.chunk("log", "c1", stamped("2026-10-14T10:00:00Z", outcomes.finished(LOW_ROUTE, "m")), [])
|
||||
self.chunk("log", "c1", stamped("2026-10-14T10:00:00Z", outcomes.finished(LOW_ROUTE, "m", "j")), [])
|
||||
(self.root / "log" / "c1" / "copy.jsonl").unlink()
|
||||
self.chunk("other", "c2", stamped("2026-10-14T10:00:00Z", outcomes.finished(LOW_ROUTE, "m")), [message("m1", "2026-10-14T09:00:00Z")])
|
||||
self.chunk("other", "c2", stamped("2026-10-14T10:00:00Z", outcomes.finished(LOW_ROUTE, "m", "j")), [message("m1", "2026-10-14T09:00:00Z")])
|
||||
self.chunk("broken", "c3", "", [])
|
||||
(self.root / "broken" / "c3" / "copy.outcomes.md").write_bytes(b"\xff")
|
||||
found = archive.measure_archive(self.root, "/h", "")
|
||||
|
||||
@@ -5,6 +5,9 @@ import typing
|
||||
from claude_code.transcript import first_timestamped_line
|
||||
from project_key import project_key
|
||||
|
||||
# What an outcomes file's name ends with, in place of its transcript copy's .jsonl.
|
||||
FILE_SUFFIX = ".outcomes.md"
|
||||
|
||||
|
||||
def archive_root(home: pathlib.Path, env: typing.Mapping[str, str]) -> pathlib.Path:
|
||||
"""Where loop-claude archives every project's chunks."""
|
||||
@@ -18,3 +21,13 @@ def project_archive(checkout_root: str, home: pathlib.Path, env: typing.Mapping[
|
||||
def chunk_folder(project_archive: pathlib.Path, transcript: bytes) -> pathlib.Path:
|
||||
# Each chunk's first timestamped line is its own: a finished chunk's session is deleted, so the next chunk's transcript starts afresh.
|
||||
return project_archive / hashlib.sha256(first_timestamped_line(transcript)).hexdigest()
|
||||
|
||||
|
||||
def outcomes_of(copy: pathlib.Path) -> pathlib.Path:
|
||||
"""The outcomes file beside a transcript copy; transcript_of is its inverse."""
|
||||
return copy.with_suffix(FILE_SUFFIX)
|
||||
|
||||
|
||||
def transcript_of(outcomes: pathlib.Path) -> pathlib.Path:
|
||||
"""The transcript copy an outcomes file sits beside; outcomes_of is its inverse."""
|
||||
return outcomes.with_name(outcomes.name.removesuffix(FILE_SUFFIX) + ".jsonl")
|
||||
|
||||
@@ -3,8 +3,14 @@ import re
|
||||
from markdown_text import quoted
|
||||
from utc_stamp import STAMPED
|
||||
|
||||
TASK = "Task: "
|
||||
FINISHED = "Verdict: finished"
|
||||
UNFINISHED = "Verdict: unfinished"
|
||||
INTERRUPTED = "Verdict: interrupt"
|
||||
ANSWERED = "Verdict: quoted answer"
|
||||
# The file opens with the task, then a blank line, then the first stamped outcome.
|
||||
TASK_AT = re.compile(rf"\A{re.escape(TASK)}(.*?)\n\n{STAMPED} ", re.DOTALL)
|
||||
ANY_VERDICT_AT = re.compile(rf"^({STAMPED}) ({'|'.join(re.escape(verdict) for verdict in (FINISHED, UNFINISHED, INTERRUPTED, ANSWERED))})$", re.MULTILINE)
|
||||
FINISHED_AT = re.compile(rf"^({STAMPED}) {re.escape(FINISHED)}$", re.MULTILINE)
|
||||
UNFINISHED_AT = re.compile(rf"^{STAMPED} {re.escape(UNFINISHED)}$", re.MULTILINE)
|
||||
# A verdict closes each cycle the judge ruled on; an interrupt or a stop closes none.
|
||||
@@ -13,22 +19,24 @@ VERDICT_AT = re.compile(rf"^{STAMPED} (?:{re.escape(FINISHED)}|{re.escape(UNFINI
|
||||
OUTCOME_AT = re.compile(rf"^({STAMPED}) ", re.MULTILINE)
|
||||
NAMES_A_WORKER = ("Verdict: finished\n", "Verdict: unfinished\n", "Stopped: ", "Stepped up\n")
|
||||
WORKER = "Worker: "
|
||||
# The rung that judged a verdict.
|
||||
JUDGE = "Judge: "
|
||||
|
||||
|
||||
def finished(worker: str, evidence: str) -> str:
|
||||
return f"{FINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{WORKER}{worker}"
|
||||
def finished(worker: str, evidence: str, judge: str) -> str:
|
||||
return f"{FINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{JUDGE}{judge}\n{WORKER}{worker}"
|
||||
|
||||
|
||||
def unfinished(worker: str, evidence: str) -> str:
|
||||
return f"{UNFINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{WORKER}{worker}"
|
||||
def unfinished(worker: str, evidence: str, judge: str) -> str:
|
||||
return f"{UNFINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{JUDGE}{judge}\n{WORKER}{worker}"
|
||||
|
||||
|
||||
def interrupted(question: str, evidence: str) -> str:
|
||||
return f"Verdict: interrupt\n\n{question}\n\nEvidence:\n{quoted(evidence)}"
|
||||
def interrupted(question: str, evidence: str, judge: str) -> str:
|
||||
return f"{INTERRUPTED}\n\n{question}\n\nEvidence:\n{quoted(evidence)}\n\n{JUDGE}{judge}"
|
||||
|
||||
|
||||
def answered(question: str, sources: str, answer: str, evidence: str) -> str:
|
||||
return f"Verdict: quoted answer\n\n{question}\n\n{sources} says:\n{quoted(answer)}\n\nEvidence:\n{quoted(evidence)}"
|
||||
def answered(question: str, sources: str, answer: str, evidence: str, judge: str) -> str:
|
||||
return f"{ANSWERED}\n\n{question}\n\n{sources} says:\n{quoted(answer)}\n\nEvidence:\n{quoted(evidence)}\n\n{JUDGE}{judge}"
|
||||
|
||||
|
||||
def stepped_up(worker: str) -> str:
|
||||
@@ -50,3 +58,15 @@ def workers(outcomes: str) -> list[tuple[str, str]]:
|
||||
if outcome.startswith(NAMES_A_WORKER) and last.startswith(WORKER):
|
||||
found.append((start[1], last.removeprefix(WORKER)))
|
||||
return found
|
||||
|
||||
|
||||
def task_of(outcomes: str) -> str | None:
|
||||
"""The task an outcomes file opens with; None when it opens with none."""
|
||||
found = TASK_AT.match(outcomes)
|
||||
return found[1] if found else None
|
||||
|
||||
|
||||
def first_verdict(outcomes: str) -> tuple[str, str] | None:
|
||||
"""The UTC stamp and the verdict, such as FINISHED, of the first verdict in an outcomes file; None when it holds none."""
|
||||
found = ANY_VERDICT_AT.search(outcomes)
|
||||
return (found[1], found[2]) if found else None
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import dataclasses
|
||||
import pathlib
|
||||
|
||||
from transcript_archive.location import chunk_folder
|
||||
from transcript_archive.location import chunk_folder, FILE_SUFFIX, transcript_of
|
||||
from failure import Failure
|
||||
from transcript_archive.outcomes import FINISHED_AT, workers
|
||||
|
||||
@@ -19,15 +19,28 @@ def finished_chunks(project_archive: pathlib.Path) -> list[FinishedChunk] | Fail
|
||||
found = []
|
||||
try:
|
||||
for folder in sorted(path for path in project_archive.glob("*") if path.is_dir()):
|
||||
copies = sorted((stamps[-1], outcomes) for outcomes in folder.glob("*.outcomes.md") if (stamps := FINISHED_AT.findall(outcomes.read_text(encoding="utf-8"))))
|
||||
copies = sorted((stamps[-1], outcomes) for outcomes in chunk_outcomes(folder) if (stamps := FINISHED_AT.findall(outcomes.read_text(encoding="utf-8"))))
|
||||
if copies:
|
||||
finished, outcomes = copies[-1]
|
||||
found.append(FinishedChunk(folder.name, finished, outcomes.with_name(outcomes.name.removesuffix(".outcomes.md") + ".jsonl"), sorted(folder.glob("session/*/subagents/*.jsonl"))))
|
||||
found.append(FinishedChunk(folder.name, finished, transcript_of(outcomes), sorted(folder.glob("session/*/subagents/*.jsonl"))))
|
||||
except (OSError, UnicodeDecodeError) as failure:
|
||||
return Failure(f"could not read the archive {project_archive}: {failure}")
|
||||
return sorted(found, key=lambda chunk: (chunk.finished, chunk.folder))
|
||||
|
||||
|
||||
def archived_outcomes(root: pathlib.Path) -> list[tuple[str, pathlib.Path]] | Failure:
|
||||
"""Each outcomes file in the archive at root, with the key of the project it belongs to, sorted."""
|
||||
try:
|
||||
return sorted((str(path.parent.parent.relative_to(root)), path) for path in root.rglob(f"*{FILE_SUFFIX}"))
|
||||
except OSError as failure:
|
||||
return Failure(f"could not read the archive {root}: {failure}")
|
||||
|
||||
|
||||
def chunk_outcomes(folder: pathlib.Path) -> list[pathlib.Path]:
|
||||
"""The outcomes file of each transcript copy in a chunk's folder, sorted."""
|
||||
return sorted(folder.glob(f"*{FILE_SUFFIX}"))
|
||||
|
||||
|
||||
def read_threads(chunk: FinishedChunk) -> list[bytes] | Failure:
|
||||
"""The worker's transcript, then each subagent's."""
|
||||
try:
|
||||
@@ -46,7 +59,7 @@ def latest_worker(project_archive: pathlib.Path, transcript: str) -> str | Failu
|
||||
folder = chunk_folder(project_archive, handle.read())
|
||||
if not folder.is_dir():
|
||||
return None
|
||||
files = sorted((path for path in folder.iterdir() if path.name.endswith(".outcomes.md")), key=lambda path: path.stat().st_mtime_ns, reverse=True)
|
||||
files = sorted((path for path in folder.iterdir() if path.name.endswith(FILE_SUFFIX)), key=lambda path: path.stat().st_mtime_ns, reverse=True)
|
||||
for path in files:
|
||||
if found := workers(path.read_text(encoding="utf-8")):
|
||||
return found[-1][1]
|
||||
|
||||
@@ -8,10 +8,10 @@ import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from project_key import checkout_of, project_key
|
||||
from failure import Failure
|
||||
from transcript_archive import outcomes, read, write
|
||||
from transcript_archive.location import project_archive
|
||||
from project_key import project_key
|
||||
from utc_stamp import utc
|
||||
|
||||
NOW = 1790000000
|
||||
@@ -24,7 +24,7 @@ class Archive(unittest.TestCase):
|
||||
transcript = pathlib.Path(root, "t.jsonl")
|
||||
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n')
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.unfinished("m", "working"), NOW))
|
||||
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.unfinished("m", "working", "j"), NOW))
|
||||
(kept_outcomes,) = kept.glob("*/*.outcomes.md")
|
||||
kept_outcomes.unlink()
|
||||
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.stopped("m", "Ctrl-C"), NOW))
|
||||
@@ -45,14 +45,14 @@ class CopyFor(unittest.TestCase):
|
||||
|
||||
class Outcomes(unittest.TestCase):
|
||||
def test_the_archive_stamps_what_it_reads_back(self):
|
||||
self.assertRegex(f"{utc(NOW)} {outcomes.finished('m', 'merged')}", outcomes.FINISHED_AT)
|
||||
self.assertRegex(f"{utc(NOW)} {outcomes.finished('m', 'merged', 'j')}", outcomes.FINISHED_AT)
|
||||
|
||||
def test_reads_back_the_worker_of_each_outcome_that_closes_a_cycle(self):
|
||||
kept = "Task: Do it.\n\n" + "\n".join(f"{utc(NOW + offset)} {outcome}\n" for offset, outcome in enumerate([
|
||||
outcomes.unfinished("w1", "Worker: not this"),
|
||||
outcomes.interrupted("Which one?\nWorker: nor this", "e"),
|
||||
outcomes.unfinished("w1", "Worker: not this", "j"),
|
||||
outcomes.interrupted("Which one?\nWorker: nor this", "e", "j"),
|
||||
outcomes.stopped("w2", "the judge failed\nthree times"),
|
||||
outcomes.finished("w3", "merged"),
|
||||
outcomes.finished("w3", "merged", "j"),
|
||||
]))
|
||||
self.assertEqual(outcomes.workers(kept), [(utc(NOW), "w1"), (utc(NOW + 2), "w2"), (utc(NOW + 3), "w3")])
|
||||
|
||||
@@ -68,7 +68,7 @@ class Outcomes(unittest.TestCase):
|
||||
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n')
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
for worker in ("z-first", "a-last"):
|
||||
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.unfinished(worker, "working"), NOW))
|
||||
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.unfinished(worker, "working", "j"), NOW))
|
||||
self.assertEqual(read.latest_worker(kept, str(transcript)), "a-last")
|
||||
|
||||
def test_names_the_worker_the_latest_outcome_of_the_transcripts_chunk_records(self):
|
||||
@@ -78,7 +78,7 @@ class Outcomes(unittest.TestCase):
|
||||
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n')
|
||||
self.assertIsNone(read.latest_worker(kept, str(transcript)))
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.unfinished("w1", "working"), NOW))
|
||||
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.unfinished("w1", "working", "j"), NOW))
|
||||
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n{"more": 1}\n')
|
||||
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.stopped("w2", "Ctrl-C"), NOW + 1))
|
||||
self.assertEqual(read.latest_worker(kept, str(transcript)), "w2")
|
||||
@@ -94,6 +94,12 @@ class Location(unittest.TestCase):
|
||||
self.assertEqual(project_archive("/h/code/log", home, {}), pathlib.Path("/h/.claude/loop-claude-transcripts/log"))
|
||||
self.assertEqual(project_archive("/h/code/log", home, {"LOOP_CLAUDE_TRANSCRIPTS": "/a"}), pathlib.Path("/a/log"))
|
||||
|
||||
def test_finds_the_checkout_a_key_names_under_code_or_else_under_the_root(self):
|
||||
with tempfile.TemporaryDirectory() as home:
|
||||
(pathlib.Path(home) / "code" / "log").mkdir(parents=True)
|
||||
self.assertEqual(checkout_of("log", pathlib.Path(home)), f"{home}/code/log")
|
||||
self.assertEqual(checkout_of("home/me/.claude", pathlib.Path(home)), "/home/me/.claude")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -3,10 +3,11 @@ import os
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
from transcript_archive.location import chunk_folder
|
||||
from transcript_archive.location import chunk_folder, outcomes_of
|
||||
from failure import Failure
|
||||
from file_writes import make_private, replace_file
|
||||
from claude_code.session import session_files
|
||||
from transcript_archive.outcomes import TASK
|
||||
from utc_stamp import utc
|
||||
|
||||
|
||||
@@ -22,12 +23,12 @@ def archive(project_archive: pathlib.Path, path: str, task: str, outcome: str, a
|
||||
if sibling != path and (unkept := mirror(pathlib.Path(sibling), folder / "session" / os.path.basename(sibling))):
|
||||
return unkept
|
||||
copy = folder / f"{hashlib.sha256(raw).hexdigest()}.jsonl"
|
||||
outcomes = copy.with_suffix(".outcomes.md")
|
||||
outcomes = outcomes_of(copy)
|
||||
stamped = f"{utc(at)} {outcome}"
|
||||
if copy.exists() and outcomes.exists():
|
||||
return add_outcome(outcomes, stamped)
|
||||
# The copy lands last, so a copy on disk always has its outcomes and its session's files beside it.
|
||||
for dest, data in ((outcomes, f"Task: {task}\n\n{stamped}\n".encode()), (copy, raw)):
|
||||
for dest, data in ((outcomes, f"{TASK}{task}\n\n{stamped}\n".encode()), (copy, raw)):
|
||||
if unwritten := replace_file(dest, data):
|
||||
return unarchived(dest, unwritten.reason)
|
||||
print(f"transcript: {copy}", file=sys.stderr)
|
||||
|
||||
@@ -1685,3 +1685,30 @@ Inherited architect, decided by:
|
||||
- Shape: `bin/routing_figures/measure.py:74` `measured`: one 234-line module holds six concerns, and `measured`, `Measured` and `measure_archive` sit side by side; `kept` and `calls` each name two things
|
||||
- Navigation: `bin/model_routing/route.py:11` `Route`: moved out of loop_claude while still carrying loop-only state, so "routing" now spans five places
|
||||
- Self-sufficiency: `bin/routing_figures/measure.py:130` `spend`: the docstring says it counts as `tokens_in` does, but it adds output tokens
|
||||
|
||||
## 2026-10-15T07:36:58Z, PR #266 at 6cc3ded, against 8412a48
|
||||
|
||||
Ruling: better
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Mid A | same | better | same | same | better |
|
||||
| Inherited architect | better | same | same | same | better |
|
||||
|
||||
Mid A, decided by:
|
||||
|
||||
- Locality: `bin/transcript_archive/location.py:9` `FILE_SUFFIX`, with `outcomes_of` and `transcript_of`: the one place that maps a transcript copy to its outcomes file and back, where the base wrote the suffix and the inverse step out in write.py, read.py and routing_figures/archive.py
|
||||
- Locality: `bin/transcript_archive/read.py:31` `archived_outcomes` and `chunk_outcomes`: replace the raw globs routing_figures made, so it no longer knows the archive's folder depth
|
||||
- Locality: `bin/transcript_archive/outcomes.py:6` `TASK`, `INTERRUPTED`, `ANSWERED`, `TASK_AT`, `ANY_VERDICT_AT`: writer and readers share one set of constants, so the outcomes format reads and changes in one file
|
||||
- Locality: `bin/loop_claude/routing.py:36` `item_rung`: `kept` and `judge_rung` share one named helper where the base held the rule in an inline conditional
|
||||
- Locality: `bin/loop_claude/cycle/judge.py:127` `judge_once` and `judge_args`: one judge run and its flags stand alone, so the replay reuses them without copying
|
||||
|
||||
Inherited architect, decided by:
|
||||
|
||||
- Navigation, Overall: `bin/transcript_archive/outcomes.py:22` `JUDGE`, `interrupted`, `answered`: each archived verdict carries a `Judge:` line, so a reader paged about a wrong ruling reads which rung judged it from the outcomes file
|
||||
- Navigation, Locality, Overall: `bin/transcript_archive/location.py:9` `FILE_SUFFIX`, `outcomes_of`, `transcript_of`, with `bin/transcript_archive/read.py:31` `archived_outcomes`: the archive layout lives in one module, though `archived_outcomes` still derives the key from `path.parent.parent`
|
||||
- Locality: `bin/judge_replay/main.py:124` `first_reply`: re-implements the loop's retry count beside `retried`, two places that can drift
|
||||
- Locality: `bin/project_key.py:9` `checkout_of`: a guessing inverse of `project_key` that maps a project archived from `/x` to `~/code/x` when that folder exists, silently
|
||||
- Shape: `bin/judge_replay/main.py:31` imports: one launcher reaching into five `loop_claude` internals, balanced by `judge`, `judge_once` and `judge_args` in `bin/loop_claude/cycle/judge.py:123` now three units with truthful names
|
||||
- Locality: `bin/loop_claude/routing.py:51` `judge_rung`: derives its own anchor from the judge model while `Cycle.anchor` is the worker's, so "anchor" means two rungs in one cycle when the models differ
|
||||
- Self-sufficiency: `bin/loop_claude/routing.py:54` `judge_rung` docstring: points into the plan at item 136, and `JUDGE_REPLAYED` and `REFERENCE_EFFORT` rest on a past replay and a past settings value the code cannot show
|
||||
|
||||
+13
-1
@@ -172,7 +172,7 @@ Valid while a mutation run on a chunk's changed lines stays cheap beside the rev
|
||||
|
||||
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model, and a chunk's worker may run below that rung, launching its reviewers on that rung; the worker routes each subagent it delegates to by the R + C of the subagent's task
|
||||
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude, by the worker through `bin/subagent-rung`, and by larv-review once item 90 lands. `bin/model_routing/ladder.py` holds the rungs and the bands.
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude, by the worker through `bin/subagent-rung`, and by larv-review once item 90 lands. The judge's rung is the next entry's. `bin/model_routing/ladder.py` holds the rungs and the bands.
|
||||
|
||||
The rungs, least competent first: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models, read 2026-10-14), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). That cut drops Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index falls among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scores it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price.
|
||||
|
||||
@@ -187,3 +187,15 @@ The worker delegates a task only when it needs just the task's conclusion: a sea
|
||||
The worker changes rung only as the chunk starts, after every second unfinished verdict, or when the judge reports a taken-in item whose R + C picks a higher rung, because a model switch inside a session discards its prompt cache. loop-claude routes on the first row of the next release in `todo.md`'s table that waits on no question. It reads neither an order the plan states in prose nor the task, so a chunk may start on another item's rung: a higher one costs money, and a lower one holds until the judge reports the item the worker took in, after the first cycle. `todo.md` item 136 ends this, routing on the item the worker says it takes.
|
||||
|
||||
Valid while the index ranks these models, and while no newer model joins the catalog.
|
||||
|
||||
## routing: on a model judge-replay covered, the judge runs on the rung its chunk's item picks, never below the project's rung
|
||||
|
||||
2026-10-15, lilleman's plan, settled by the replay below. Serves goals 1.2 and 1.4, under the ladder and bands of `routing: one ladder`.
|
||||
|
||||
On a model listed in `JUDGE_REPLAYED` in `bin/loop_claude/routing.py`, the judge runs on the rung its chunk's item picks, never below the project's rung, and keeps that rung when the worker steps up. Without an item, it runs on the project's rung. Each archived verdict names the judge's rung on a `Judge:` line.
|
||||
|
||||
On 2026-10-15, `bin/judge-replay claude-opus-5-5` replayed the latest 15 archived verdicts of each kind from the Opus 5.5 projects, 39 in all. The reference judge, Opus 5.5 at the xhigh effort `settings.json` gave every judge before, reproduced 31 of them. The other 8 may rest on records or judge rules that have changed since; either way no rung is held to them. On the 31, Haiku 5.5 missed 7 at xhigh and 8 at max. That run tried each judge once. 4 of Haiku's misses, 1 at xhigh and 3 at max, misquoted the transcript, which the loop and `judge-replay` now retry; without them it missed 6 and 5. Sonnet 5.5 at high returned another kind on 3 interrupts, and Opus 5.5 at medium missed none. A missed interrupt lets the worker settle a decision that is lilleman's (goal 1.2), so the judge never runs below the project's rung.
|
||||
|
||||
Only Opus 5.5 was replayed, so the judge of a project on another model keeps the effort its settings give until `judge-replay` covers that model (goal 1.1). Every routed chunk on record had an item of R + C 9, so the replay could not show that the project's rung suffices for harder items, and the judge climbs with the routed item as the worker does. For R + C 11 to 14 it still runs at or below xhigh (goal 1.4); from 15, `routing: one ladder` records that lilleman wants the highest quality at any price. An item the worker takes in later lifts the worker, not the judge, until `todo.md` item 136 routes a chunk on the item its worker takes.
|
||||
|
||||
Valid while the archived verdicts show what a judge should return, and until a new model joins the ladder: replay then.
|
||||
|
||||
@@ -316,3 +316,20 @@ Read most (opens and searches):
|
||||
- `AGENTS.md` 1
|
||||
- `CLAUDE.md` 1
|
||||
- `bin/larv-review` 1
|
||||
|
||||
## Reading cost, chunk 1c06e6295656, finished 2026-10-15T05:26:15Z
|
||||
|
||||
147391 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `bin/loop_claude/test_loop_claude.py` 6
|
||||
- `todo.md` 5
|
||||
- `README.md` 4
|
||||
- `.gitignore` 3
|
||||
- `bin/loop_claude/loop/main.py` 3
|
||||
- `docs/decisions.md` 2
|
||||
- `CHANGELOG.md` 1
|
||||
- `bin/claude_code/entries.py` 1
|
||||
- `bin/claude_code/trust.py` 1
|
||||
- `bin/git_command.py` 1
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
`Bar = 9`
|
||||
|
||||
`Next ID = 140`
|
||||
`Next ID = 141`
|
||||
|
||||
| Goal | W |
|
||||
|---|---|
|
||||
@@ -26,8 +26,7 @@ Release 0.2.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
|
||||
|
||||
| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 90 | 0.1.0 | | **Route each reviewer by the chunk's R + C, once a replay of past chunks shows the routed reviewer finds the 🔴 and 🟡 findings on record.** | 4 | 5 | 8 | 6 | 7 | 1.4 | 20.2 |
|
||||
| 89 | 0.1.0 | | **Run the judge on the rung its chunk's R + C picks, once a replay of archived transcripts shows it returns the verdicts on record.** | 3 | 4 | 7 | 5 | 6 | 1.4 | 17.5 |
|
||||
| 90 | 0.1.0 | question | **Route each reviewer by the chunk's R + C, once a replay of past chunks shows the routed reviewer finds the 🔴 and 🟡 findings on record.** | 4 | 5 | 8 | 6 | 7 | 1.4 | 20.2 |
|
||||
| 136 | 0.1.0 | decision | **Route a chunk on the item its worker says it takes, in place of the next release's first row.** | 2 | 3 | 4 | 4 | 5 | 1.4 | 14.8 |
|
||||
| 105 | 0.2.0 | decision | **Compare review rounds per pass and reading cost for the five chunks before and after the `worker:` decision moved the principles out of the worker's context, and file an item if either rose.** | 1 | 2 | 3 | 5 | 6 | 1.4 | 20.0 |
|
||||
| 112 | 0.2.0 | | **Count an item as shipped only where its chunk built it, so an item a chunk dropped from the plan gets no outcome.** | 2 | 3 | 5 | 5 | 6 | 1.1 | 19.1 |
|
||||
@@ -47,6 +46,7 @@ Release 0.2.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
|
||||
| 131 | 0.2.0 | | **Describe the transcript archive's layout in `bin/transcript_archive/`, which owns it, and have loop-claude's help point there.** | 1 | 2 | 1 | 3 | 4 | 1.1 | 12.4 |
|
||||
| 133 | 0.2.0 | | **Show, beside `bin/rule_hook/override_log.py`'s `this_chunks`, one example per case it decides: an amended or rebased commit, a renamed branch, and a commit on the default branch.** | 1 | 2 | 2 | 3 | 4 | 1.1 | 12.4 |
|
||||
| 134 | 0.2.0 | | **Name in `bin/shell_command.py`'s `segments` the shell forms it does not parse, so a caller knows what it misses.** | 1 | 2 | 2 | 3 | 4 | 1.1 | 12.4 |
|
||||
| 140 | 0.2.0 | | **Split `docs/decisions.md` → `routing: one ladder` into one entry per decision, so a reader finds the rules for the ladder, the worker's rung, the routing figures and delegation by their titles.** | 1 | 2 | 1 | 3 | 4 | 1.1 | 12.4 |
|
||||
| 114 | 0.2.0 | decision | **Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.** | 3 | 4 | 4 | 3 | 5 | 1.1 | 11.9 |
|
||||
| 117 | 0.2.0 | | **Give `todo_table.items` one way to say a plan has no items table, so no caller tells it from an empty plan by truthiness.** | 2 | 2 | 3 | 3 | 4 | 1.1 | 11.4 |
|
||||
| 124 | 0.2.0 | decision | **Group the shared modules at `bin/`'s root into packages named by purpose, so the root holds only launchers and stand-alone tools.** | 3 | 6 | 3 | 3 | 5 | 1.1 | 11.4 |
|
||||
@@ -89,14 +89,14 @@ Release 0.2.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
|
||||
|
||||
larv-review passes the rung through the Agent tool's `model` and `effort`. A reviewer never steps down when the worker has stepped down on the same chunk, so a cheap worker always meets a reviewer on the project model or above.
|
||||
|
||||
### 89. Run the judge on the rung its chunk's R + C picks, once a replay of archived transcripts shows it returns the verdicts on record.
|
||||
|
||||
loop-claude starts the judge itself (`bin/loop_claude/cycle/judge.py`), so it routes without any skill change. Replay the judge on archived cycles at the candidate rung, and switch only when its interrupts and verdicts match those on record.
|
||||
Question: drop this item, or say what it should route? Under `docs/decisions.md` → `routing:`, reviewers already run where this item would put them. Below the project's rung, the worker launches every reviewer on the project's rung. Above it, reviewers inherit the worker's rung, which the same R + C picks. The item would change only a chunk whose worker climbed past that rung, after two unfinished verdicts or on a taken-in item. None of the 3 routed chunks on record on 2026-10-15 did.
|
||||
|
||||
### 136. Route a chunk on the item its worker says it takes, in place of the next release's first row.
|
||||
|
||||
loop-claude routes on the first row of the next release, while the worker takes the item the plan's prose orders first, or a prerequisite, as the chunk that shipped item 88, a prerequisite, took item 88 in place of the first row. A first row of higher R + C pays for a dearer rung all chunk; one of lower R + C holds until the judge reports the item taken in. The judge reports the item the worker takes only after the first cycle, so routing the first cycle right needs the item before the worker starts.
|
||||
|
||||
Once this lands, the judge follows the item the worker takes too, since `judge_rung` routes on the chunk's item.
|
||||
|
||||
### 105. Compare review rounds per pass and reading cost for the five chunks before and after the `worker:` decision moved the principles out of the worker's context, and file an item if either rose.
|
||||
|
||||
The `worker:` decision is valid while review rounds per pass do not rise. Count the rounds from each PR's `Answer <pass> round <n>:` commits, and take reading cost from `docs/reading-cost.md`. Run it once five chunks have merged after the one that recorded the decision.
|
||||
@@ -171,6 +171,10 @@ Junior A, Mid A and the maintainability senior of PR #257's scoring run named `b
|
||||
|
||||
Junior A and Mid A of PR #257's scoring run named `bin/shell_command.py:31` `segments`: a hand-rolled tokenizer with dense regexes and a second lexing path for unbalanced quotes, which the rule hook and `reading-cost` both depend on. Its comments state each quirk but not its limits. Make `segments` easier to read.
|
||||
|
||||
### 140. Split `docs/decisions.md` → `routing: one ladder` into one entry per decision, so a reader finds the rules for the ladder, the worker's rung, the routing figures and delegation by their titles.
|
||||
|
||||
The entry holds 7 paragraphs under one title of about 70 words chaining three decisions: the ladder and bands, the worker below the project's rung with its reviewers, the routing figures and calls, delegation, and when the worker changes rung. A reader after one rule reads the whole entry. The prose pass of the chunk that split out the judge's rung asked for the rest.
|
||||
|
||||
### 114. Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.
|
||||
|
||||
Shape is the lowest dimension, so this item goes before 0.2.0's other items, per `docs/decisions.md` → `larv-review: a project lifting itself to the comprehension floor starts at the lowest dimension`.
|
||||
|
||||
Reference in New Issue
Block a user