Merge pull request 'Measure routing figures every 20 cycles and raise each call for a routing change once' (#265) from routing-figures into main
This commit was merged in pull request #265.
This commit is contained in:
@@ -46,6 +46,8 @@
|
||||
!bin/repo_checks/*.py
|
||||
!bin/round_titles.py
|
||||
!bin/rule-hook
|
||||
!bin/routing_figures/
|
||||
!bin/routing_figures/*.py
|
||||
!bin/rule_hook/
|
||||
!bin/rule_hook/*.py
|
||||
!bin/score-outcomes
|
||||
|
||||
@@ -37,6 +37,7 @@ Decisions live in `docs/decisions.md`:
|
||||
|
||||
## Working rules
|
||||
|
||||
- `routing-calls-raised.md` sits at the root of loop-claude's archive (`$LOOP_CLAUDE_TRANSCRIPTS`, default `~/.claude/loop-claude-transcripts`). Each of its lines is a call or a `Reset`, with a stamp. A call is new when it is stamped after the latest `Reset` and after the latest stamp that `todo.md` or `docs/decisions.md` → `routing:` cites. When there are new calls, file one `todo.md` item to settle them, or refresh the item already filed. Its detail quotes those calls with their stamps, and the routed-chunks table of `routing-figures.md` beside it, never the project names its other tables hold. A change to the ladder, its bands or the delegation rule is the human's. The chunk that settles the calls cites the latest stamp it settled in `docs/decisions.md` → `routing:`, whatever the human decided, so they are not filed again. Once a chunk that changes the ladder, its bands or the delegation rule has merged, append `- <UTC time> Reset` to `routing-calls-raised.md`: the figures then count only chunks finished after it, and its calls may be raised again.
|
||||
- When `~/.claude/settings.json` has uncommitted changes, commit them on a branch of their own and merge them through a PR: Claude Code writes that file from any session. Read their diff first. When it holds a credential, such as a token in `env` or `mcpServers`, ask the human to move it out of the file, and commit only once it is gone. A token in `env` moves to the shell profile; a server in `mcpServers` moves to `~/.claude.json` with `claude mcp add --scope user`.
|
||||
|
||||
## Scoring run
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Every 20 cycles across all projects, `loop-claude` measures every finished chunk in its archive, with no model run, and writes the figures to `routing-figures.md` at the archive's root: tokens by project, subagent type, rung and tool, and, for each chunk a `todo.md` item routed, grouped by the rung it started on: its review rounds past each pass's first, its unfinished verdicts and its subagents' rungs. It interrupts you, with `claude` on the project's model on the terminal in `~/.claude`, when chunks routed onto a rung took more review rounds or unfinished verdicts than chunks routed onto the next rung up, each rung with 5 chunks or more, or when 5 or more subagents of one type wrote reports longer than the tool results they read. Each call is raised once, then listed in `routing-calls-raised.md` beside the figures; the next chunk of this repo files a `todo.md` item for it. Off a terminal, the calls are said on stderr and the loop goes on. They are raised at the next measurement on a terminal. After a routing change, a `- <UTC time> Reset` line in that file makes the figures count only chunks finished after it, and lets its calls be raised again. Expect a pause of about ten seconds at each measurement.
|
||||
- A session delegates a task to a subagent only when it needs just the task's conclusion, such as a search across many files, and keeps edits in its own thread. It scores the task's R and C as for a `todo.md` item, and launches the subagent on the model and effort that the new `bin/subagent-rung <model> <R+C>` picks from the project's model, such as `model sonnet, effort high`. `CLAUDE.md` states the rule. Expect searches to run on cheaper models, and less of each session's context to fill with search output.
|
||||
- `loop-claude` routes each chunk's worker on a ladder of model and effort rungs, from Haiku 5.5 at low effort to Fable 5.1 at max, when the project's model is on it. As a chunk starts, the R + C of `todo.md`'s top item in the next release picks the rung: 11–12 picks one rung above the project's rung, so Opus 5.5 at high for an Opus 5.5 project, and so on up to four rungs. An item of R + C 10 or less runs the worker at or below the project's model's cheapest rung, which for Opus 5.5 is medium effort, below the xhigh `settings.json` sets. While the worker runs below the project's model's cheapest rung, it launches every reviewer, comprehension seat and security pass, the full scan included, in a subagent on that rung. While it runs above that rung, it launches each comprehension seat and the security pass on that rung. It steps up one rung after every second unfinished verdict on a chunk, and to a higher rung when it takes in an item of higher R + C. Each cycle's header, and each finished or unfinished verdict, stop and step-up in the archive, name the worker's rung; a rerun resumes an open chunk on the latest rung archived for it. `docs/decisions.md` → `routing:` gives the ladder, its source and the bands. Expect a chunk on an easy item to cost much less than under xhigh, and perhaps to take more review rounds, and one on an item of R + C 15 or more to cost more.
|
||||
- A session reads `todo.md` up to `## Details`, and the details of only the items its chunk takes. It reads a `docs/decisions.md` entry only where the entry's title leaves open what it needs. Both rules are in `CLAUDE.md`, so they hold from a session's start. The decisions rule replaces one in `rules/decisions.md`, which loads only once a session has opened `docs/decisions.md` or `AGENTS.md`. A chunk's opening line now counts the unreleased changes instead of listing them. Expect fewer tokens read before a chunk's first commit.
|
||||
|
||||
@@ -92,7 +92,7 @@ pytest_add_cli_args=-k
|
||||
|
||||
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
|
||||
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, and `score_outcomes` imports `reading_cost`'s record path. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, and `score_outcomes` imports `reading_cost`'s record path. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
|
||||
## Tracking a new file
|
||||
|
||||
|
||||
@@ -119,3 +119,10 @@ def rejected_until(event: dict) -> int | None:
|
||||
return None
|
||||
resets_at = info.get("resetsAt")
|
||||
return resets_at if isinstance(resets_at, int) and not isinstance(resets_at, bool) else None
|
||||
|
||||
|
||||
# A count Claude Code left out or wrote as another type adds nothing, so one odd entry leaves the rest of a figure standing.
|
||||
def usage_count(usage: dict, field: str) -> int:
|
||||
"""One token count of a message's usage: {"input_tokens": 2, "cache_creation_input_tokens": 14541, "output_tokens": 8, ...}"""
|
||||
value = usage.get(field)
|
||||
return value if isinstance(value, int) else 0
|
||||
|
||||
@@ -17,16 +17,23 @@ def interrupt(root: str, session_id: uuid.UUID, worker: Rung, path: str, questio
|
||||
print(f"\n{question}", file=sys.stderr)
|
||||
if stop := needs_terminal(ANSWER_QUESTION, effects.on_terminal()):
|
||||
return stop
|
||||
trust(paths.claude_json, worktrees(root, effects))
|
||||
entries = read_transcript(path)
|
||||
if isinstance(entries, Failure):
|
||||
return Stop(1, entries.reason)
|
||||
before = human_turns(entries)
|
||||
try:
|
||||
effects.claude_on_terminal([*worker.flags(), "--permission-mode", "bypassPermissions", "--resume", str(session_id)], root)
|
||||
except OSError as failure:
|
||||
return Stop(1, f"could not start claude for the interrupt: {failure.strerror}")
|
||||
if unstarted := hand_over(worker, ["--resume", str(session_id)], root, paths, effects):
|
||||
return Stop(1, f"could not start claude for the interrupt: {unstarted.reason}")
|
||||
entries = read_transcript(path)
|
||||
if isinstance(entries, Failure):
|
||||
return Stop(1, f"could not tell whether you typed anything: {entries.reason}")
|
||||
return None if human_turns(entries) > before else needs_human(ANSWER_QUESTION, "the interrupt ended with nothing typed")
|
||||
|
||||
|
||||
def hand_over(rung: Rung, args: list[str], root: str, paths: Paths, effects: Effects) -> Failure | None:
|
||||
"""Runs claude on the terminal in root on rung, the checkout and each of its worktrees first marked trusted; the failure is why claude could not start."""
|
||||
trust(paths.claude_json, worktrees(root, effects))
|
||||
try:
|
||||
effects.claude_on_terminal([*rung.flags(), "--permission-mode", "bypassPermissions", *args], root)
|
||||
except OSError as failure:
|
||||
return Failure(str(failure.strerror))
|
||||
return None
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
from model_routing.ladder import highest, Rung
|
||||
from loop_claude.project.records import HUMAN_RULE_NAMES, RECORD_FILES
|
||||
from markdown_text import quoted
|
||||
from loop_claude.routing import Route
|
||||
from model_routing.route import Route
|
||||
|
||||
def either(names: tuple[str, ...]) -> str:
|
||||
"""The names as `a, b or c`."""
|
||||
|
||||
@@ -14,7 +14,7 @@ from loop_claude.paths import Paths
|
||||
from loop_claude.cycle import prompts
|
||||
from claude_code.transcript import read_transcript, render
|
||||
from loop_claude.cycle.retry import retried
|
||||
from loop_claude.routing import Route
|
||||
from model_routing.route import Route
|
||||
from loop_claude.cycle.worker import run_worker, worker_args
|
||||
from model_routing.ladder import Rung
|
||||
from loop_claude.stop import Stop
|
||||
|
||||
@@ -60,6 +60,22 @@ and counts its unfinished verdicts afresh. A rung no longer on the ladder, or be
|
||||
item's R + C picks, is dropped, and the chunk is routed afresh from todo.md. A todo.md or an
|
||||
archive that cannot be read or parsed is said on stderr, and the chunk is routed without it.
|
||||
|
||||
Counting the cycles a verdict closed across all projects in the archive, the first loop to close
|
||||
one 20 or more cycles after the last measurement measures every finished chunk there, with no model
|
||||
run: tokens by project, subagent type, rung and the tool whose results they took in, and, for each
|
||||
chunk a todo.md item routed, grouped by the rung it started on: its review rounds past each pass's
|
||||
first, its unfinished verdicts and its subagents' rungs. It writes them to routing-figures.md at the archive's root. They call for a
|
||||
routing change when chunks routed onto a rung took more review rounds or unfinished verdicts than
|
||||
chunks routed onto the next rung up, each rung with 5 chunks or more, or when the reports of 5 or
|
||||
more subagents of one type were longer than the tool results they read. Each call not raised before
|
||||
is an interrupt: claude, on the project's model, on the terminal in ~/.claude, asked to put the
|
||||
calls to the human. Once that claude ends, the calls are listed in routing-calls-raised.md beside the
|
||||
figures, and not raised again. Off a terminal, the calls are said on stderr, the loop goes on, and
|
||||
they are raised again at the next measurement. A line `- <UTC time> Reset` in
|
||||
routing-calls-raised.md restarts both: the figures count only chunks that finished after it, and a
|
||||
call raised before it may be raised again. A chunk that cannot be read is skipped and named in the
|
||||
figures; figures that cannot be measured or written are said on stderr, and the loop goes on.
|
||||
|
||||
A change to bin/loop-claude or any bin/**/*.py takes effect at the next cycle boundary, once
|
||||
loop-claude --help succeeds: the loop reloads into the edited script.
|
||||
|
||||
@@ -101,23 +117,25 @@ import sys
|
||||
import typing
|
||||
|
||||
from transcript_archive.write import archive
|
||||
from transcript_archive.location import archive_root, project_archive
|
||||
from loop_claude.loop.routing_check import check_routing
|
||||
from loop_claude.project.checkout import checkout_root
|
||||
from loop_claude.cycle.run import Cycle, run_cycle
|
||||
from loop_claude.loop.reload import decode_handed, exec_if_edited, fingerprint, LoopRun, RELOAD_FLAG
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.judge import Finished, Unfinished
|
||||
from loop_claude.routing import first_route, kept, Route, route_from, routed_item, stepped
|
||||
from loop_claude.routing import first_route, kept, routed_item, stepped
|
||||
from loop_claude.project.model_choice import get_or_ask_models
|
||||
from todo_table import Item
|
||||
from transcript_archive.read import latest_worker
|
||||
from transcript_archive import outcomes
|
||||
from loop_claude.paths import Paths
|
||||
from transcript_archive.location import project_archive
|
||||
from project_key import project_key
|
||||
from model_routing.ladder import project_rung, Rung
|
||||
from loop_claude.project.premises import read_premises
|
||||
from loop_claude.project.records import read_records
|
||||
from model_routing.route import Route, route_from
|
||||
from claude_code.session import Session, session_named, transcript_file
|
||||
from loop_claude.stop import Stop
|
||||
|
||||
@@ -165,6 +183,8 @@ def run_loop(argv: list[str], env: typing.Mapping[str, str], paths: Paths, launc
|
||||
before, loop_run = loop_run.route, after_verdict(loop_run, anchor, verdict)
|
||||
if stop := archive_step(archive_folder, session, loop_run, before, effects):
|
||||
return stop
|
||||
if stop := check_routing(archive_root(paths.home, env), anchor or Rung(models.worker), paths, effects):
|
||||
return stop
|
||||
|
||||
|
||||
def routed(loop_run: LoopRun, anchor: Rung | None, archived: typing.Callable[[], Route | None], item: typing.Callable[[], Item | None]) -> LoopRun:
|
||||
|
||||
@@ -7,7 +7,7 @@ import types
|
||||
import typing
|
||||
|
||||
from loop_claude.effects import Effects
|
||||
from loop_claude.routing import Route
|
||||
from model_routing.route import Route
|
||||
from loop_claude.stop import Stop
|
||||
|
||||
RELOAD_FLAG = "--reloaded"
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Every CYCLES_PER_CHECK cycles across all projects, measure the archive's routing figures, keep them at its root, and interrupt the human with each call for a routing change not raised before: docs/decisions.md → `routing:`."""
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
from routing_figures import archive
|
||||
from routing_figures.measure import calls, due, render, totals
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.cycle.interrupt import hand_over
|
||||
from loop_claude.paths import Paths
|
||||
from model_routing.ladder import Rung
|
||||
from loop_claude.stop import Stop
|
||||
from utc_stamp import utc
|
||||
|
||||
|
||||
def check_routing(root: pathlib.Path, rung: Rung, paths: Paths, effects: Effects) -> Stop | None:
|
||||
"""Stops only when claude cannot start for the calls; a figure it cannot measure or keep is said on stderr, and the loop goes on. rung is the rung of the claude that puts the calls to the human."""
|
||||
with archive.locked(root) as held:
|
||||
new = new_calls(root, paths, effects) if held else []
|
||||
if isinstance(new, Failure):
|
||||
print(f"routing figures: {new.reason}", file=sys.stderr)
|
||||
return None
|
||||
return raise_calls(root, new, rung, paths, effects) if new else None
|
||||
|
||||
|
||||
def new_calls(root: pathlib.Path, paths: Paths, effects: Effects) -> list[str] | Failure:
|
||||
"""Once due, measures and keeps the figures, and returns their calls not raised since the latest reset; none while not due."""
|
||||
cycle = archive.cycles(root)
|
||||
last = archive.last_measured(root)
|
||||
if isinstance(cycle, Failure) or isinstance(last, Failure) or not due(cycle, last):
|
||||
return cycle if isinstance(cycle, Failure) else last if isinstance(last, Failure) else []
|
||||
kept = archive.raised(root)
|
||||
if isinstance(kept, Failure):
|
||||
return kept
|
||||
print(f"\n=== cycle {cycle} across loop-claude's archive: measuring the routing figures ===", flush=True)
|
||||
measured = archive.measure_archive(root, str(paths.home), kept.since)
|
||||
if isinstance(measured, Failure):
|
||||
return measured
|
||||
figures = totals(measured.chunks, measured.skipped)
|
||||
found = calls(figures)
|
||||
if unwritten := archive.write_figures(root, render(figures, found, utc(effects.now()), cycle, kept.since)):
|
||||
return unwritten
|
||||
return [call for call in found if call not in kept.calls]
|
||||
|
||||
|
||||
def raise_calls(root: pathlib.Path, new: list[str], rung: Rung, paths: Paths, effects: Effects) -> Stop | None:
|
||||
"""Puts the calls to the human through a claude in this setup's checkout, where the routing decision lives. Off a terminal they are only said, and stay in the figures, raised at the next measurement on a terminal."""
|
||||
listed = "".join(f"- {call}\n" for call in new)
|
||||
print(f"\nloop-claude's routing figures call for a routing change:\n{listed}The figures: {root / archive.FIGURES}", file=sys.stderr)
|
||||
if not effects.on_terminal():
|
||||
print("Not on a terminal: the calls are raised again at the next measurement.", file=sys.stderr)
|
||||
return None
|
||||
prompt = f"Put these calls for a routing change from loop-claude's figures to the human, and change nothing until they answer:\n{listed}The figures are in {root / archive.FIGURES}. The ladder, its bands and the delegation rule are docs/decisions.md → `routing:`, and a change to them is the human's. Record what they decide as ~/.claude/CLAUDE.md says; the next chunk of this repo files the plan item for these calls."
|
||||
if unstarted := hand_over(rung, [prompt], str(paths.claude_dir), paths, effects):
|
||||
return Stop(1, f"could not start claude for the routing calls: {unstarted.reason}")
|
||||
# Raised once the human has seen them, answered or not: an unanswered call reaches the plan through AGENTS.md, so it is not asked twice.
|
||||
if unmarked := archive.mark_raised(root, new, utc(effects.now())):
|
||||
print(f"routing figures: {unmarked.reason}", file=sys.stderr)
|
||||
return None
|
||||
@@ -1,37 +1,12 @@
|
||||
"""Which ladder rung a chunk's worker runs on: docs/decisions.md → `routing:`."""
|
||||
import dataclasses
|
||||
import re
|
||||
|
||||
from failure import Failure
|
||||
from model_routing.ladder import highest, LADDER, picked, Rung, stepped_up
|
||||
from todo_table import Item, items
|
||||
from model_routing.route import Route
|
||||
|
||||
UNFINISHED_PER_STEP = 2
|
||||
ROUTED = re.compile(r"(?P<model>\S+) at (?P<effort>\S+) effort(?:, routed by item (?P<item>[0-9]+)'s R \+ C of (?P<sum>[0-9]+))?")
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Route:
|
||||
"""A chunk's worker rung and the plan item whose R + C picked it, 0 for none.
|
||||
|
||||
unfinished counts the unfinished verdicts towards the next one-rung step; a reload hands it over with the rung.
|
||||
"""
|
||||
rung: Rung
|
||||
item: int = 0
|
||||
risk_and_complexity: int = 0
|
||||
unfinished: int = 0
|
||||
|
||||
def __str__(self) -> str:
|
||||
"""What the archive records as each verdict's worker; ROUTED reads it back."""
|
||||
return f"{self.rung}, routed by item {self.item}'s R + C of {self.risk_and_complexity}" if self.item else str(self.rung)
|
||||
|
||||
|
||||
def route_from(worker: str) -> Route | None:
|
||||
"""The route an archived worker line names; None for a model run off the ladder."""
|
||||
found = ROUTED.fullmatch(worker)
|
||||
if not found:
|
||||
return None
|
||||
return Route(Rung(found["model"], found["effort"]), int(found["item"] or 0), int(found["sum"] or 0))
|
||||
|
||||
|
||||
def routed_item(todo: str) -> Item | Failure | None:
|
||||
|
||||
@@ -5,6 +5,7 @@ import io
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import shutil
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
@@ -15,6 +16,7 @@ import uuid
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from routing_figures import archive
|
||||
from loop_claude.project import checkout, model_choice, records
|
||||
from loop_claude.effects import Effects
|
||||
from claude_code import entries, session
|
||||
@@ -23,11 +25,13 @@ from loop_claude.cycle.judge import Finished, Interrupt, QuotedAnswer, Unfinishe
|
||||
from loop_claude.cycle import interrupt, judge, prompts, retry, run, states, worker
|
||||
from model_routing.ladder import LADDER, Rung
|
||||
from loop_claude.cycle.retry import Limited
|
||||
from loop_claude.loop import main, reload
|
||||
from loop_claude.loop import main, reload, routing_check
|
||||
from transcript_archive import outcomes, write
|
||||
from loop_claude import paths, routing
|
||||
from transcript_archive.location import project_archive
|
||||
from model_routing.route import Route, route_from
|
||||
from loop_claude.stop import Stop
|
||||
from utc_stamp import utc
|
||||
|
||||
CWD = pathlib.Path("/c")
|
||||
NOW = 1790000000
|
||||
@@ -171,7 +175,7 @@ class Cycles(unittest.TestCase):
|
||||
self.archive = project_archive(str(self.home / "code" / "proj"), self.home, {})
|
||||
|
||||
def cycle(self, fake, records="", unfinished_evidence=""):
|
||||
return run.Cycle(anchor=None, effects=fake.effects(), judge_model="judge-model", paths=self.paths, premises="## Goals", project_archive=self.archive, records=records, root=str(self.home), route=routing.Route(Rung("worker-model")), session=self.session, task="Do it.", unfinished_evidence=unfinished_evidence)
|
||||
return run.Cycle(anchor=None, effects=fake.effects(), judge_model="judge-model", paths=self.paths, premises="## Goals", project_archive=self.archive, records=records, root=str(self.home), route=Route(Rung("worker-model")), session=self.session, task="Do it.", unfinished_evidence=unfinished_evidence)
|
||||
|
||||
def archived_outcomes(self):
|
||||
return sorted(path.read_text() for path in self.archive.glob("*/*.outcomes.md"))
|
||||
@@ -365,6 +369,98 @@ class Interrupts(unittest.TestCase):
|
||||
self.assertEqual(stop, Stop(2, "the human is needed to answer the question above, but the interrupt ended with nothing typed"))
|
||||
|
||||
|
||||
class RoutingCheck(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.home = pathlib.Path(tempfile.mkdtemp())
|
||||
self.addCleanup(shutil.rmtree, self.home)
|
||||
self.root = self.home / "archive"
|
||||
self.paths = paths.paths_from(self.home)
|
||||
|
||||
def chunks(self, count, verdicts, project="log"):
|
||||
"""count finished chunks, each closed by verdicts verdicts, each with one Explore subagent whose report outgrew its searches."""
|
||||
for number in range(count):
|
||||
folder = self.root / project / f"c{number}"
|
||||
subagents = folder / "session" / "s" / "subagents"
|
||||
subagents.mkdir(parents=True)
|
||||
kept = "".join(f"2026-10-14T10:0{verdict}:00Z {outcomes.unfinished('w', 'e')}\n" for verdict in range(verdicts - 1))
|
||||
(folder / "copy.outcomes.md").write_text(f"Task: t\n\n{kept}2026-10-14T11:00:00Z {outcomes.finished('w', 'm')}\n")
|
||||
launch = {"type": "assistant", "timestamp": "2026-10-14T09:00:00Z", "message": {"id": "m", "content": [{"type": "tool_use", "id": "a", "name": "Agent", "input": {"subagent_type": "Explore"}}]}}
|
||||
report = {"type": "user", "message": {"content": [{"type": "tool_result", "tool_use_id": "a", "content": "long report"}]}}
|
||||
(folder / "copy.jsonl").write_text(json.dumps(launch) + "\n" + json.dumps(report) + "\n")
|
||||
(subagents / "agent-a.jsonl").write_text("")
|
||||
(subagents / "agent-a.meta.json").write_text(json.dumps({"agentType": "Explore", "toolUseId": "a"}))
|
||||
|
||||
def check(self, fake):
|
||||
with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()) as said:
|
||||
stop = routing_check.check_routing(self.root, Rung("m", "high"), self.paths, fake.effects())
|
||||
return stop, said.getvalue()
|
||||
|
||||
def test_measures_once_20_cycles_have_closed_since_the_last_measurement(self):
|
||||
self.chunks(5, 3)
|
||||
self.assertEqual(self.check(FakeEffects()), (None, ""))
|
||||
self.assertFalse((self.root / "routing-figures.md").exists())
|
||||
self.chunks(6, 1, "other")
|
||||
self.assertEqual(self.check(FakeEffects(on_terminal=False))[0], None)
|
||||
self.assertIn("at cycle 21 across", (self.root / "routing-figures.md").read_text())
|
||||
|
||||
def test_hands_the_terminal_over_once_for_each_new_call(self):
|
||||
self.chunks(5, 4)
|
||||
fake = FakeEffects(on_terminal_claude=[0], git=[ran()])
|
||||
stop, said = self.check(fake)
|
||||
self.assertIsNone(stop)
|
||||
call = "Explore subagents' reports took more of the worker's context than their searches would have."
|
||||
self.assertIn(f"- {call}", said)
|
||||
self.assertIn(f"- {call}", (self.root / "routing-figures.md").read_text())
|
||||
((args),) = fake.made("claude_on_terminal")
|
||||
self.assertEqual(args[:6], ["--model", "m", "--effort", "high", "--permission-mode", "bypassPermissions"])
|
||||
self.assertIn(f"change nothing until they answer:\n- {call}", args[6])
|
||||
(self.root / "routing-figures.md").write_text("Measured x, at cycle 0 across")
|
||||
self.assertEqual(self.check(FakeEffects())[0], None)
|
||||
self.assertNotIn(call, self.check(FakeEffects())[1])
|
||||
|
||||
def test_stamps_the_figures_and_the_calls_raised_when_it_measures(self):
|
||||
self.chunks(5, 4)
|
||||
(self.root / "routing-calls-raised.md").write_text("# Routing calls raised\n\n- 2026-10-14T10:02:00Z Reset\n")
|
||||
self.check(FakeEffects(on_terminal_claude=[0], git=[ran()]))
|
||||
figures = (self.root / "routing-figures.md").read_text()
|
||||
self.assertIn(f"Measured {utc(NOW)}, at cycle 20 across", figures)
|
||||
self.assertIn("that finished after the reset of 2026-10-14T10:02:00Z", figures)
|
||||
self.assertIn(f"- {utc(NOW)} Explore subagents'", (self.root / "routing-calls-raised.md").read_text())
|
||||
|
||||
def test_says_a_figure_it_cannot_read(self):
|
||||
self.chunks(5, 4)
|
||||
(self.root / "routing-figures.md").mkdir()
|
||||
self.assertIn("routing figures: could not read", self.check(FakeEffects())[1])
|
||||
|
||||
def test_goes_on_off_a_terminal_and_raises_the_call_again_next_time(self):
|
||||
self.chunks(5, 4)
|
||||
stop, said = self.check(FakeEffects(on_terminal=False))
|
||||
self.assertIsNone(stop)
|
||||
self.assertIn("Not on a terminal: the calls are raised again at the next measurement.", said)
|
||||
(self.root / "routing-figures.md").write_text("Measured x, at cycle 0 across")
|
||||
self.assertIn("Not on a terminal", self.check(FakeEffects(on_terminal=False))[1])
|
||||
|
||||
def test_leaves_the_measuring_to_a_loop_holding_the_lock(self):
|
||||
self.chunks(5, 4)
|
||||
with archive.locked(self.root):
|
||||
self.assertEqual(self.check(FakeEffects()), (None, ""))
|
||||
|
||||
def test_stops_when_claude_cannot_start_for_the_calls(self):
|
||||
self.chunks(5, 4)
|
||||
stop, _ = self.check(FakeEffects(on_terminal_claude=[OSError(2, "No such file or directory")], git=[ran()]))
|
||||
self.assertEqual(stop, Stop(1, "could not start claude for the routing calls: No such file or directory"))
|
||||
|
||||
@unittest.skipIf(os.geteuid() == 0, "root writes into a read-only directory")
|
||||
def test_says_a_figure_it_cannot_write_and_goes_on(self):
|
||||
self.chunks(5, 4)
|
||||
(self.root / "routing-figures.lock").touch()
|
||||
self.root.chmod(0o555)
|
||||
self.addCleanup(self.root.chmod, 0o755)
|
||||
stop, said = self.check(FakeEffects())
|
||||
self.assertIsNone(stop)
|
||||
self.assertIn("routing figures: could not write", said)
|
||||
|
||||
|
||||
class ModelPrompt(unittest.TestCase):
|
||||
OFFERED = [("claude-opus-5-5", "Opus 5.5"), ("claude-haiku-4-5", "Haiku 4.5")]
|
||||
|
||||
@@ -411,56 +507,56 @@ class Routing(unittest.TestCase):
|
||||
self.assertEqual(routing.routed_item(TODO.replace("| 5 | 3 |", "| x | 3 |")), Failure("item 3's R 'x' is not a whole number from 1 to 10"))
|
||||
|
||||
def test_starts_a_chunk_on_the_rung_its_items_r_plus_c_picks_below_the_projects_too(self):
|
||||
self.assertEqual(routing.first_route(self.OPUS, routing.routed_item(TODO)), routing.Route(self.HIGH, 3, 11))
|
||||
self.assertEqual(routing.first_route(self.OPUS, routing.routed_item(TODO)), Route(self.HIGH, 3, 11))
|
||||
cheap = routing.routed_item(TODO.replace("| 5 | 3 | 6 |", "| 1 | 3 | 1 |"))
|
||||
self.assertEqual(routing.first_route(self.OPUS, cheap), routing.Route(Rung("claude-haiku-5-5", "xhigh"), 3, 2))
|
||||
self.assertEqual(routing.first_route(self.OPUS, None), routing.Route(self.OPUS))
|
||||
self.assertEqual(routing.first_route(self.OPUS, cheap), Route(Rung("claude-haiku-5-5", "xhigh"), 3, 2))
|
||||
self.assertEqual(routing.first_route(self.OPUS, None), Route(self.OPUS))
|
||||
|
||||
def test_steps_up_a_rung_after_every_second_unfinished_verdict(self):
|
||||
once = routing.stepped(self.OPUS, routing.Route(self.OPUS, 3, 9), 0)
|
||||
self.assertEqual(once, routing.Route(self.OPUS, 3, 9, 1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, once, 0), routing.Route(self.HIGH, 3, 9, 0))
|
||||
self.assertEqual(routing.stepped(self.OPUS, routing.Route(LADDER[-1], unfinished=1), 0).rung, LADDER[-1])
|
||||
once = routing.stepped(self.OPUS, Route(self.OPUS, 3, 9), 0)
|
||||
self.assertEqual(once, Route(self.OPUS, 3, 9, 1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, once, 0), Route(self.HIGH, 3, 9, 0))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(LADDER[-1], unfinished=1), 0).rung, LADDER[-1])
|
||||
|
||||
def test_steps_up_to_the_rung_a_bundled_items_r_plus_c_picks_and_never_down(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, routing.Route(self.OPUS), 13).rung, self.XHIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, routing.Route(self.XHIGH), 11).rung, self.XHIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, routing.Route(self.OPUS, unfinished=1), 13).rung, Rung("claude-opus-5-5", "max"))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS), 13).rung, self.XHIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 11).rung, self.XHIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS, unfinished=1), 13).rung, Rung("claude-opus-5-5", "max"))
|
||||
|
||||
def test_steps_up_from_a_rung_below_the_projects_one_rung_at_a_time(self):
|
||||
low = routing.Route(Rung("claude-haiku-5-5", "xhigh"), 3, 2, 1)
|
||||
low = Route(Rung("claude-haiku-5-5", "xhigh"), 3, 2, 1)
|
||||
self.assertEqual(routing.stepped(self.OPUS, low, 0).rung, Rung("claude-haiku-5-5", "max"))
|
||||
self.assertEqual(routing.stepped(self.OPUS, routing.Route(low.rung, 3, 2), 0).rung, low.rung)
|
||||
self.assertEqual(routing.stepped(self.OPUS, routing.Route(low.rung, 3, 2), 7).rung, Rung("claude-sonnet-5-5", "high"))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 2), 0).rung, low.rung)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 2), 7).rung, Rung("claude-sonnet-5-5", "high"))
|
||||
|
||||
def test_a_rung_an_edit_took_off_the_ladder_counts_as_the_projects(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, routing.Route(Rung("claude-opus-5-5", "low")), 0).rung, self.OPUS)
|
||||
self.assertEqual(routing.stepped(self.OPUS, routing.Route(Rung("claude-opus-5-5", "low")), 2).rung, self.OPUS)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 0).rung, self.OPUS)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 2).rung, self.OPUS)
|
||||
|
||||
def test_archives_the_route_in_a_line_it_reads_back(self):
|
||||
route = routing.Route(self.HIGH, 3, 11)
|
||||
route = Route(self.HIGH, 3, 11)
|
||||
self.assertEqual(str(route), "claude-opus-5-5 at high effort, routed by item 3's R + C of 11")
|
||||
self.assertEqual(routing.route_from(str(route)), route)
|
||||
self.assertEqual(routing.route_from(str(routing.Route(self.HIGH))), routing.Route(self.HIGH))
|
||||
self.assertEqual(str(routing.Route(Rung("worker-model"))), "worker-model")
|
||||
self.assertIsNone(routing.route_from("worker-model"))
|
||||
self.assertEqual(route_from(str(route)), route)
|
||||
self.assertEqual(route_from(str(Route(self.HIGH))), Route(self.HIGH))
|
||||
self.assertEqual(str(Route(Rung("worker-model"))), "worker-model")
|
||||
self.assertIsNone(route_from("worker-model"))
|
||||
|
||||
def test_keeps_a_chunks_route_or_its_archived_one_at_or_above_the_rung_its_item_picks_and_else_routes_afresh(self):
|
||||
def run_with(route=None):
|
||||
return main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=route)
|
||||
item = routing.routed_item(TODO)
|
||||
archived = routing.Route(self.XHIGH, 3, 11)
|
||||
archived = Route(self.XHIGH, 3, 11)
|
||||
def never():
|
||||
raise AssertionError("not asked")
|
||||
self.assertIsNone(main.routed(run_with(archived), None, never, never).route)
|
||||
self.assertEqual(main.routed(run_with(archived), self.OPUS, never, never).route, archived)
|
||||
self.assertEqual(main.routed(run_with(), self.OPUS, lambda: archived, never).route, archived)
|
||||
self.assertEqual(main.routed(run_with(), self.OPUS, lambda: None, lambda: item).route, routing.Route(self.HIGH, 3, 11))
|
||||
below = routing.Route(Rung("claude-sonnet-5-5", "high"))
|
||||
self.assertEqual(main.routed(run_with(below), self.OPUS, never, lambda: None).route, routing.Route(self.OPUS))
|
||||
cheap = routing.Route(Rung("claude-sonnet-5-5", "high"), 4, 6)
|
||||
self.assertEqual(main.routed(run_with(), self.OPUS, lambda: None, lambda: item).route, Route(self.HIGH, 3, 11))
|
||||
below = Route(Rung("claude-sonnet-5-5", "high"))
|
||||
self.assertEqual(main.routed(run_with(below), self.OPUS, never, lambda: None).route, Route(self.OPUS))
|
||||
cheap = Route(Rung("claude-sonnet-5-5", "high"), 4, 6)
|
||||
self.assertEqual(main.routed(run_with(cheap), self.OPUS, never, never).route, cheap)
|
||||
self.assertEqual(main.routed(run_with(routing.Route(Rung("claude-haiku-5-5", "max"), 4, 7)), self.OPUS, never, lambda: None).route, routing.Route(self.OPUS))
|
||||
self.assertEqual(main.routed(run_with(Route(Rung("claude-haiku-5-5", "max"), 4, 7)), self.OPUS, never, lambda: None).route, Route(self.OPUS))
|
||||
|
||||
|
||||
class RouteSources(unittest.TestCase):
|
||||
@@ -486,7 +582,7 @@ class RouteSources(unittest.TestCase):
|
||||
transcript.parent.mkdir(parents=True)
|
||||
transcript.write_text(json.dumps({"type": "queue-operation", "timestamp": "1"}) + "\n")
|
||||
self.assertIsNone(main.archived_route(archive, opened))
|
||||
route = routing.Route(Rung("claude-opus-5-5", "high"), 3, 11)
|
||||
route = Route(Rung("claude-opus-5-5", "high"), 3, 11)
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(route), "working"), NOW))
|
||||
self.assertEqual(main.archived_route(archive, opened), route)
|
||||
@@ -504,9 +600,9 @@ class Steps(unittest.TestCase):
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
|
||||
def test_a_finished_verdict_clears_the_route_and_the_evidence(self):
|
||||
run_with = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="old", route=routing.Route(self.OPUS))
|
||||
run_with = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="old", route=Route(self.OPUS))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Finished("x")), dataclasses.replace(run_with, route=None, unfinished_evidence=""))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Unfinished("new", 13)), dataclasses.replace(run_with, route=routing.Route(Rung("claude-opus-5-5", "xhigh"), unfinished=1), unfinished_evidence="new"))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Unfinished("new", 13)), dataclasses.replace(run_with, route=Route(Rung("claude-opus-5-5", "xhigh"), unfinished=1), unfinished_evidence="new"))
|
||||
|
||||
def test_archives_a_step_up_as_it_happens_so_a_rerun_resumes_on_it(self):
|
||||
with tempfile.TemporaryDirectory() as home:
|
||||
@@ -515,12 +611,12 @@ class Steps(unittest.TestCase):
|
||||
transcript = pathlib.Path(home, "projects", "-fake", f"{opened.id}.jsonl")
|
||||
transcript.parent.mkdir(parents=True)
|
||||
transcript.write_text(json.dumps({"type": "queue-operation", "timestamp": "1"}) + "\n")
|
||||
before = routing.Route(self.OPUS, 3, 9)
|
||||
before = Route(self.OPUS, 3, 9)
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(before), "working"), NOW))
|
||||
after = main.after_verdict(main.LoopRun(cycle_number=1, root="/r", task="Do it.", unfinished_evidence="", route=before), self.OPUS, Unfinished("working", 13))
|
||||
self.assertIsNone(main.archive_step(archive, opened, after, before, FakeEffects().effects()))
|
||||
self.assertEqual(main.archived_route(archive, opened), routing.Route(Rung("claude-opus-5-5", "xhigh"), 3, 9))
|
||||
self.assertEqual(main.archived_route(archive, opened), Route(Rung("claude-opus-5-5", "xhigh"), 3, 9))
|
||||
self.assertIsNone(main.archive_step(archive, opened, after, after.route, FakeEffects().effects()))
|
||||
self.assertIsNone(main.archive_step(archive, opened, after, None, FakeEffects().effects()))
|
||||
blocked = pathlib.Path(home, "blocked")
|
||||
@@ -531,32 +627,32 @@ class Steps(unittest.TestCase):
|
||||
|
||||
class WorkerArgs(unittest.TestCase):
|
||||
def test_names_the_route_in_a_fresh_prompt_only_where_an_item_routed_it(self):
|
||||
routed = prompts.fresh("Do it.", "## Goals", routing.Route(Rung("m", "high"), 3, 11), None)
|
||||
routed = prompts.fresh("Do it.", "## Goals", Route(Rung("m", "high"), 3, 11), None)
|
||||
self.assertIn("Continue any open draft PR or worktree for the chunk before creating a new one.\n\nloop-claude runs you on m at high effort, routed by item 3's R + C of 11. Name each todo.md item this chunk takes, with its R + C, whether or not it is item 3.\n\nThe chunk's", routed)
|
||||
unrouted = prompts.fresh("Do it.", "## Goals", routing.Route(Rung("m")), None)
|
||||
unrouted = prompts.fresh("Do it.", "## Goals", Route(Rung("m")), None)
|
||||
self.assertIn("before creating a new one.\n\nThe chunk's", unrouted)
|
||||
self.assertNotIn("loop-claude runs you on", unrouted)
|
||||
|
||||
def test_keeps_reviewers_on_the_projects_rung_while_the_worker_runs_below_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
below = prompts.fresh("Do it.", "## Goals", routing.Route(Rung("claude-sonnet-5-5", "high"), 3, 5), opus)
|
||||
below = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 5), opus)
|
||||
self.assertIn("whether or not it is item 3. Your rung is below the project's, claude-opus-5-5 at medium effort: launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them.\n\n", below)
|
||||
self.assertNotIn("the project's", prompts.fresh("Do it.", "## Goals", routing.Route(opus, 3, 9), opus))
|
||||
self.assertNotIn("the project's", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 9), opus))
|
||||
|
||||
def test_keeps_seats_and_the_security_pass_on_the_projects_rung_while_the_worker_runs_above_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
above = prompts.fresh("Do it.", "## Goals", routing.Route(Rung("claude-opus-5-5", "high"), 3, 11), opus)
|
||||
stepped_up = prompts.resumed("", routing.Route(Rung("claude-opus-5-5", "high"), 3, 9), opus)
|
||||
above = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-opus-5-5", "high"), 3, 11), opus)
|
||||
stepped_up = prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 9), opus)
|
||||
self.assertTrue(stepped_up.endswith(" Your rung is above the project's, claude-opus-5-5 at medium effort: launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them."), stepped_up)
|
||||
self.assertNotIn("the project's", prompts.resumed("", routing.Route(opus, 3, 9), opus))
|
||||
self.assertNotIn("the project's", prompts.resumed("", Route(opus, 3, 9), opus))
|
||||
self.assertIn("whether or not it is item 3. Your rung is above the project's, claude-opus-5-5 at medium effort: launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them.\n\n", above)
|
||||
|
||||
def test_opens_a_fresh_session_by_name_and_resumes_an_open_one(self):
|
||||
opened = session.session_named("A proj loop-claude", pathlib.Path("/h/.claude/projects"))
|
||||
fresh = worker.worker_args(opened, Rung("m"), prompts.fresh("Do it.", "## Goals\n1. g\n\n## Audience\nme", routing.Route(Rung("m")), None), resume=False)
|
||||
fresh = worker.worker_args(opened, Rung("m"), prompts.fresh("Do it.", "## Goals\n1. g\n\n## Audience\nme", Route(Rung("m")), None), resume=False)
|
||||
self.assertEqual(fresh[fresh.index("--session-id") + 1], str(opened.id))
|
||||
self.assertEqual(fresh[fresh.index("--name") + 1], "A proj loop-claude")
|
||||
resumed = worker.worker_args(opened, Rung("m"), prompts.resumed("", routing.Route(Rung("m")), None), resume=True)
|
||||
resumed = worker.worker_args(opened, Rung("m"), prompts.resumed("", Route(Rung("m")), None), resume=True)
|
||||
self.assertEqual(resumed[resumed.index("--resume") + 1], str(opened.id))
|
||||
self.assertNotIn("--session-id", resumed)
|
||||
self.assertTrue(resumed[resumed.index("-p") + 1].startswith("Continue the task; the previous run ended before the chunk was finished. The chunk is finished once"))
|
||||
@@ -671,7 +767,7 @@ class LoopRuns(unittest.TestCase):
|
||||
self.assertIsNone(reload.decode_handed('{"count": "1", "name": "n"}', Stringly))
|
||||
|
||||
def test_decodes_a_route_inside_the_state_or_none(self):
|
||||
state = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=routing.Route(Rung("claude-opus-5-5", "high"), 3, 11, 1))
|
||||
state = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=Route(Rung("claude-opus-5-5", "high"), 3, 11, 1))
|
||||
self.assertEqual(reload.decode_handed(json.dumps(dataclasses.asdict(state)), main.LoopRun), state)
|
||||
unrouted = dataclasses.replace(state, route=None)
|
||||
self.assertEqual(reload.decode_handed(json.dumps(dataclasses.asdict(unrouted)), main.LoopRun), unrouted)
|
||||
@@ -725,6 +821,7 @@ class ExitContract(unittest.TestCase):
|
||||
("loop/main.py", "loop_run_from", "0 if task else 1", "__doc__.strip()", "no task is given"),
|
||||
("loop/main.py", "loop_run_from", "1", 'f"the edited loop-claude cannot read the state', "a reload cannot read the state the running script handed it"),
|
||||
("loop/reload.py", "exec_if_edited", "1", "f'could not reload loop-claude:", "python cannot start for a reload"),
|
||||
("loop/routing_check.py", "raise_calls", "1", "f'could not start claude for the routing calls:", "claude cannot start for an interrupt"),
|
||||
("project/checkout.py", "checkout_root", "1", "f'could not start git:", git),
|
||||
("project/checkout.py", "checkout_root", "1", "found.stderr.strip()", git),
|
||||
("project/model_choice.py", "get_or_ask_model", "1", "f'could not save the model in", models_),
|
||||
|
||||
@@ -15,7 +15,7 @@ class Rung:
|
||||
"""The rung in the Agent tool's words. The tool takes a family name, such as opus, and runs that family's latest model, which is the model each rung names."""
|
||||
return f"model {self.model.split('-')[1]}, effort {self.effort}"
|
||||
|
||||
# The archive records this wording, and loop_claude.routing.ROUTED reads it back to resume a chunk on its rung.
|
||||
# The archive records this wording, and model_routing.route.ROUTED reads it back to resume a chunk on its rung.
|
||||
def __str__(self) -> str:
|
||||
return f"{self.model} at {self.effort} effort" if self.effort else self.model
|
||||
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
"""A chunk's route: its worker's rung and the plan item that picked it, in the words the archive records each verdict's worker in."""
|
||||
import dataclasses
|
||||
import re
|
||||
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
ROUTED = re.compile(r"(?P<model>\S+) at (?P<effort>\S+) effort(?:, routed by item (?P<item>[0-9]+)'s R \+ C of (?P<sum>[0-9]+))?")
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Route:
|
||||
"""A chunk's worker rung and the plan item whose R + C picked it, 0 for none.
|
||||
|
||||
unfinished counts the unfinished verdicts towards the next one-rung step; a reload hands it over with the rung.
|
||||
"""
|
||||
rung: Rung
|
||||
item: int = 0
|
||||
risk_and_complexity: int = 0
|
||||
unfinished: int = 0
|
||||
|
||||
def __str__(self) -> str:
|
||||
"""What the archive records as each verdict's worker; ROUTED reads it back."""
|
||||
return f"{self.rung}, routed by item {self.item}'s R + C of {self.risk_and_complexity}" if self.item else str(self.rung)
|
||||
|
||||
|
||||
def route_from(worker: str) -> Route | None:
|
||||
"""The route an archived worker line names; None for a model run off the ladder."""
|
||||
found = ROUTED.fullmatch(worker)
|
||||
if not found:
|
||||
return None
|
||||
return Route(Rung(found["model"], found["effort"]), int(found["item"] or 0), int(found["sum"] or 0))
|
||||
@@ -1,7 +1,7 @@
|
||||
import collections
|
||||
import dataclasses
|
||||
|
||||
from claude_code.entries import assistant_parts, parse_entry, tool_call
|
||||
from claude_code.entries import assistant_parts, parse_entry, tool_call, usage_count
|
||||
from git_command import commits
|
||||
from reading_cost.shell_reads import Reads, shell_reads
|
||||
from shell_command import resolved, simple_commands
|
||||
@@ -52,13 +52,7 @@ def commit_time(worker: list[dict], home: str) -> str | None:
|
||||
def tokens_in(entries: list[dict]) -> int:
|
||||
# Claude Code writes one entry per content block of a message, each with the message's usage.
|
||||
usages = {str(message.get("id") or entry.get("uuid")): message.get("usage") for entry in entries if isinstance(message := entry.get("message"), dict)}
|
||||
return sum(count(usage, "input_tokens") + count(usage, "cache_creation_input_tokens") for usage in usages.values() if isinstance(usage, dict))
|
||||
|
||||
|
||||
# A count Claude Code left out or wrote as another type adds nothing, so one odd entry leaves the rest of the figure standing.
|
||||
def count(usage: dict, field: str) -> int:
|
||||
value = usage.get(field)
|
||||
return value if isinstance(value, int) else 0
|
||||
return sum(usage_count(usage, "input_tokens") + usage_count(usage, "cache_creation_input_tokens") for usage in usages.values() if isinstance(usage, dict))
|
||||
|
||||
|
||||
def call_of(part: object) -> tuple[object, dict]:
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
"""Reads the chunks of every project in loop-claude's archive, and keeps the routing figures and the calls raised at the archive's root."""
|
||||
import contextlib
|
||||
import dataclasses
|
||||
import fcntl
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import typing
|
||||
|
||||
from routing_figures.measure import Chunk, ChunkFigures, measured, MEASURED_AT, Subagent
|
||||
from failure import Failure
|
||||
from transcript_archive.read import finished_chunks
|
||||
from claude_code.entries import parse_entry
|
||||
from file_writes import replace_file
|
||||
from utc_stamp import STAMPED
|
||||
from transcript_archive.outcomes import VERDICT_AT
|
||||
|
||||
FIGURES = "routing-figures.md"
|
||||
LOCK = "routing-figures.lock"
|
||||
RAISED = "routing-calls-raised.md"
|
||||
RAISED_HEADING = "# Routing calls raised\n"
|
||||
# Each line is a call and when it was raised, or a reset: `- <UTC time> <call>` or `- <UTC time> Reset`.
|
||||
RAISED_LINE = re.compile(rf"^- ({STAMPED}) (.+)$", re.MULTILINE)
|
||||
RESET = "Reset"
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Raised:
|
||||
"""since is the latest reset's stamp, empty for none; calls are those raised after it."""
|
||||
since: str
|
||||
calls: frozenset[str]
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def locked(root: pathlib.Path) -> typing.Iterator[bool]:
|
||||
"""Whether this loop holds the archive's routing lock; while another loop holds it, measuring is that loop's."""
|
||||
try:
|
||||
handle = open(root / LOCK, "a")
|
||||
except OSError:
|
||||
yield False
|
||||
return
|
||||
with handle:
|
||||
try:
|
||||
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except OSError:
|
||||
yield False
|
||||
return
|
||||
yield True
|
||||
|
||||
|
||||
def cycles(root: pathlib.Path) -> int | Failure:
|
||||
"""The cycles every project's loop has closed with a verdict."""
|
||||
try:
|
||||
return sum(len(VERDICT_AT.findall(path.read_text(encoding="utf-8"))) for path in root.rglob("*.outcomes.md"))
|
||||
except (OSError, UnicodeDecodeError) as failure:
|
||||
return Failure(f"could not count the cycles in {root}: {failure}")
|
||||
|
||||
|
||||
def last_measured(root: pathlib.Path) -> int | Failure:
|
||||
"""The cycle the kept figures were measured at; 0 before any."""
|
||||
try:
|
||||
found = MEASURED_AT.search((root / FIGURES).read_text(encoding="utf-8"))
|
||||
except FileNotFoundError:
|
||||
return 0
|
||||
except (OSError, UnicodeDecodeError) as failure:
|
||||
return Failure(f"could not read {root / FIGURES}: {failure}")
|
||||
return int(found.group(1)) if found else 0
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Measured:
|
||||
"""skipped names each chunk folder that could not be read whole, such as one another loop is writing."""
|
||||
chunks: list[ChunkFigures]
|
||||
skipped: tuple[str, ...]
|
||||
|
||||
|
||||
def measure_archive(root: pathlib.Path, home: str, since: str) -> Measured | Failure:
|
||||
"""The figures of each chunk that finished after since, measured one chunk at a time because the archive outgrows memory. Only an archive that cannot be listed fails."""
|
||||
try:
|
||||
projects = sorted({path.parent.parent for path in root.rglob("*.outcomes.md")})
|
||||
except OSError as failure:
|
||||
return Failure(f"could not read the archive {root}: {failure}")
|
||||
found = []
|
||||
skipped = []
|
||||
for project in projects:
|
||||
chunks = finished_chunks(project)
|
||||
if isinstance(chunks, Failure):
|
||||
skipped.append(str(project.relative_to(root)))
|
||||
continue
|
||||
for chunk in (chunk for chunk in chunks if chunk.finished > since):
|
||||
folder = project / chunk.folder
|
||||
try:
|
||||
kept = [path.read_text(encoding="utf-8") for path in sorted(folder.glob("*.outcomes.md"))]
|
||||
subagents = [subagent(path) for path in chunk.subagents]
|
||||
worker = entries(chunk.transcript)
|
||||
except (OSError, UnicodeDecodeError):
|
||||
skipped.append(str(folder.relative_to(root)))
|
||||
continue
|
||||
found.append(measured(Chunk(str(project.relative_to(root)), kept, worker, subagents), home))
|
||||
return Measured(found, tuple(skipped))
|
||||
|
||||
|
||||
def entries(path: pathlib.Path) -> list[dict]:
|
||||
with open(path, "rb") as handle:
|
||||
return [entry for line in handle if (entry := parse_entry(line))]
|
||||
|
||||
|
||||
def subagent(transcript: pathlib.Path) -> Subagent:
|
||||
"""The subagent, named by the .meta.json Claude Code writes beside its transcript: {"agentType": "Explore", "toolUseId": "toolu_01…", ...}. A meta missing or unreadable names nothing."""
|
||||
path = transcript.with_name(transcript.name.removesuffix(".jsonl") + ".meta.json")
|
||||
try:
|
||||
meta = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (FileNotFoundError, ValueError):
|
||||
meta = {}
|
||||
meta = meta if isinstance(meta, dict) else {}
|
||||
return Subagent(str(meta.get("agentType") or ""), str(meta.get("toolUseId") or ""), entries(transcript))
|
||||
|
||||
|
||||
def write_figures(root: pathlib.Path, text: str) -> Failure | None:
|
||||
unwritten = replace_file(root / FIGURES, text.encode())
|
||||
return Failure(f"could not write {root / FIGURES}: {unwritten.reason}") if unwritten else None
|
||||
|
||||
|
||||
def raised(root: pathlib.Path) -> Raised | Failure:
|
||||
text = read_raised(root)
|
||||
if isinstance(text, Failure):
|
||||
return text
|
||||
lines = RAISED_LINE.findall(text)
|
||||
since = max((stamp for stamp, what in lines if what == RESET), default="")
|
||||
return Raised(since, frozenset(what for stamp, what in lines if stamp > since and what != RESET))
|
||||
|
||||
|
||||
def mark_raised(root: pathlib.Path, calls: list[str], at: str) -> Failure | None:
|
||||
kept = read_raised(root)
|
||||
if isinstance(kept, Failure):
|
||||
return kept
|
||||
# The first line opens a list below the heading; each later one joins it.
|
||||
gap = "\n" if RAISED_LINE.search(kept) else "\n\n"
|
||||
unwritten = replace_file(root / RAISED, (kept.rstrip("\n") + gap + "".join(f"- {at} {call}\n" for call in calls)).encode())
|
||||
return Failure(f"could not write {root / RAISED}: {unwritten.reason}") if unwritten else None
|
||||
|
||||
|
||||
def read_raised(root: pathlib.Path) -> str | Failure:
|
||||
try:
|
||||
return (root / RAISED).read_text(encoding="utf-8")
|
||||
except FileNotFoundError:
|
||||
return RAISED_HEADING
|
||||
except (OSError, UnicodeDecodeError) as failure:
|
||||
return Failure(f"could not read {root / RAISED}: {failure}")
|
||||
@@ -0,0 +1,233 @@
|
||||
"""Token spend and routing outcomes of the chunks loop-claude finished, and the routing changes they call for: docs/decisions.md → `routing:`."""
|
||||
import collections
|
||||
import dataclasses
|
||||
import re
|
||||
import statistics
|
||||
import typing
|
||||
|
||||
from claude_code.entries import assistant_parts, parts, tool_call, usage_count
|
||||
from git_command import commits
|
||||
from model_routing.ladder import position, Rung
|
||||
from round_titles import round_counts
|
||||
from model_routing.route import route_from
|
||||
from shell_command import simple_commands
|
||||
from transcript_archive.outcomes import UNFINISHED_AT, workers
|
||||
|
||||
CYCLES_PER_CHECK = 20
|
||||
MEASURED_AT = re.compile(r"^Measured .+, at cycle ([0-9]+) across", re.MULTILINE)
|
||||
# A rung or a subagent type is compared only once this many of its chunks or subagents are measured, so a single odd chunk calls for no change.
|
||||
MEASURED_AT_LEAST = 5
|
||||
DIMENSIONS = ("project", "subagent type", "rung", "tool")
|
||||
# The tool label of a message that follows no tool call, such as the first message after a prompt.
|
||||
PROMPT = "prompt"
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Subagent:
|
||||
"""kind is the agentType and launched_by the toolUseId of the launching Agent call, as the subagent's .meta.json names them, each empty where it names none; entries are its transcript's entries."""
|
||||
kind: str
|
||||
launched_by: str
|
||||
entries: list[dict]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Chunk:
|
||||
"""outcomes holds each outcomes file archived for the chunk, one per transcript copy, each opening with its task; worker is its finished transcript's entries."""
|
||||
project: str
|
||||
outcomes: list[str]
|
||||
worker: list[dict]
|
||||
subagents: list[Subagent]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Reports:
|
||||
"""How many subagents of a type reported, and the characters of their reports and of the tool results they read."""
|
||||
subagents: int
|
||||
report: int
|
||||
searches: int
|
||||
|
||||
def __add__(self, other: "Reports") -> "Reports":
|
||||
return Reports(self.subagents + other.subagents, self.report + other.report, self.searches + other.searches)
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class ChunkFigures:
|
||||
"""tokens maps each of DIMENSIONS to its labels' tokens; start is the rung a routed chunk started on, None where no item routed it."""
|
||||
tokens: dict[str, collections.Counter[str]]
|
||||
start: Rung | None
|
||||
rounds_past_first: int
|
||||
unfinished: int
|
||||
delegated: collections.Counter[str]
|
||||
reports: dict[str, Reports]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Figures:
|
||||
"""skipped names each chunk folder the archive could not read whole."""
|
||||
chunks: int
|
||||
tokens: dict[str, collections.Counter[str]]
|
||||
routed: dict[Rung, list[ChunkFigures]]
|
||||
reports: dict[str, Reports]
|
||||
skipped: tuple[str, ...] = ()
|
||||
|
||||
|
||||
def measured(chunk: Chunk, home: str) -> ChunkFigures:
|
||||
cycles = sorted((cycle for kept in chunk.outcomes for cycle in workers(kept)), key=lambda cycle: cycle[0])
|
||||
tokens: dict[str, collections.Counter[str]] = {dimension: collections.Counter() for dimension in DIMENSIONS}
|
||||
for at, tools, spent in spend(chunk.worker):
|
||||
add(tokens, chunk.project, "worker", str(rung_at(cycles, at)), tools, spent)
|
||||
launches = agent_calls(chunk.worker)
|
||||
reported = results(chunk.worker)
|
||||
delegated: collections.Counter[str] = collections.Counter()
|
||||
reports: dict[str, Reports] = {}
|
||||
for subagent in chunk.subagents:
|
||||
launched_at, arguments = launches.get(subagent.launched_by, ("", {}))
|
||||
kind = subagent.kind or str(arguments.get("subagent_type") or "unknown")
|
||||
rung = str(subagent_rung(subagent.entries, arguments, rung_at(cycles, launched_at)))
|
||||
delegated[rung] += 1
|
||||
reports[kind] = reports.get(kind, Reports(0, 0, 0)) + Reports(1, len(reported.get(subagent.launched_by, "")), sum(map(len, results(subagent.entries).values())))
|
||||
for _, tools, spent in spend(subagent.entries):
|
||||
add(tokens, chunk.project, kind, rung, tools, spent)
|
||||
start = route_from(cycles[0][1]) if cycles else None
|
||||
return ChunkFigures(tokens, start.rung if start and start.item else None, rounds_past_first(chunk.worker, home), sum(len(UNFINISHED_AT.findall(kept)) for kept in chunk.outcomes), delegated, reports)
|
||||
|
||||
|
||||
def agent_calls(worker: list[dict]) -> dict[str, tuple[str, dict]]:
|
||||
"""The timestamp and the arguments of each Agent call, by the call's id."""
|
||||
found = {}
|
||||
for entry in worker:
|
||||
for part in assistant_parts(entry):
|
||||
call = tool_call(part)
|
||||
if call and call[0] == "Agent" and isinstance(part, dict) and isinstance(call[1], dict):
|
||||
found[str(part.get("id"))] = (str(entry.get("timestamp", "")), call[1])
|
||||
return found
|
||||
|
||||
|
||||
def add(tokens: dict[str, collections.Counter[str]], project: str, kind: str, rung: str, tools: list[str], spent: int) -> None:
|
||||
tokens["project"][project] += spent
|
||||
tokens["subagent type"][kind] += spent
|
||||
tokens["rung"][rung] += spent
|
||||
# Split whole, the first tools taking the remainder, so the tool table adds up to the others.
|
||||
for place, tool in enumerate(tools):
|
||||
tokens["tool"][tool] += spent // len(tools) + (place < spent % len(tools))
|
||||
|
||||
|
||||
def rung_at(cycles: list[tuple[str, str]], timestamp: str) -> Rung | str:
|
||||
"""The rung of the cycle running at timestamp: the worker named by the first outcome stamped at or after it, or else by the last outcome. A worker off the ladder is returned as its text."""
|
||||
second = timestamp[:19] + "Z"
|
||||
found = next((rung for stamp, rung in cycles if stamp >= second), cycles[-1][1] if cycles else "unrouted")
|
||||
route = route_from(found)
|
||||
return route.rung if route else found
|
||||
|
||||
|
||||
def subagent_rung(entries: list[dict], arguments: dict, worker: Rung | str) -> Rung:
|
||||
"""The model the subagent ran, at the effort its Agent call set, or else at the worker's."""
|
||||
model = next((message["model"] for entry in entries if entry.get("type") == "assistant" and isinstance(message := entry.get("message"), dict) and isinstance(message.get("model"), str)), str(arguments.get("model") or "unknown"))
|
||||
effort = arguments.get("effort") or (worker.effort if isinstance(worker, Rung) else "")
|
||||
return Rung(model, str(effort))
|
||||
|
||||
|
||||
def spend(entries: list[dict]) -> list[tuple[str, list[str], int]]:
|
||||
"""Each message's timestamp, the tools whose results it took in, and its input, cache-written and output tokens, counted once per message, as reading_cost's tokens_in dedupes them, though tokens_in leaves output tokens out."""
|
||||
messages: dict[str, tuple[str, object, list[str]]] = {}
|
||||
for entry in entries:
|
||||
if entry.get("type") != "assistant" or not isinstance(message := entry.get("message"), dict):
|
||||
continue
|
||||
key = str(message.get("id") or entry.get("uuid"))
|
||||
kept = messages.setdefault(key, (str(entry.get("timestamp", "")), message.get("usage"), []))
|
||||
kept[2].extend(name for part in assistant_parts(entry) if (found := tool_call(part)) and isinstance(name := found[0], str))
|
||||
spent = []
|
||||
before = [PROMPT]
|
||||
for at, usage, called in messages.values():
|
||||
tokens = sum(usage_count(usage, field) for field in ("input_tokens", "cache_creation_input_tokens", "output_tokens")) if isinstance(usage, dict) else 0
|
||||
spent.append((at, before, tokens))
|
||||
before = called or [PROMPT]
|
||||
return spent
|
||||
|
||||
|
||||
def results(entries: list[dict]) -> dict[str, str]:
|
||||
"""Each tool result's text by the id of the call it answers."""
|
||||
found = {}
|
||||
for entry in entries:
|
||||
for part in parts(entry):
|
||||
if isinstance(part, dict) and part.get("type") == "tool_result":
|
||||
found[str(part.get("tool_use_id"))] = result_text(part.get("content"))
|
||||
return found
|
||||
|
||||
|
||||
# {"type": "tool_result", "content": "hi"} or {"type": "tool_result", "content": [{"type": "text", "text": "hi"}]}
|
||||
def result_text(body: object) -> str:
|
||||
if isinstance(body, str):
|
||||
return body
|
||||
return "".join(block["text"] for block in body if isinstance(block, dict) and isinstance(block.get("text"), str)) if isinstance(body, list) else ""
|
||||
|
||||
|
||||
def rounds_past_first(worker: list[dict], home: str) -> int:
|
||||
# A commit retried after a failed hook repeats its title, so each title counts once.
|
||||
titles = dict.fromkeys(
|
||||
commit.messages[0].split("\n", 1)[0]
|
||||
for entry in worker for part in assistant_parts(entry)
|
||||
if (found := tool_call(part)) and found[0] == "Bash" and isinstance(found[1], dict)
|
||||
for commit in commits(simple_commands(str(found[1].get("command", "")), str(entry.get("cwd") or "/"), home), home) if commit.messages
|
||||
)
|
||||
return sum(max(0, rounds - 1) for rounds in round_counts(list(titles)).values())
|
||||
|
||||
|
||||
def due(cycle: int, last_measured: int) -> bool:
|
||||
"""Whether CYCLES_PER_CHECK cycles have closed since the figures were last measured. Counting from the last measurement still measures when another loop's cycles carried the count past a multiple of CYCLES_PER_CHECK."""
|
||||
return cycle >= last_measured + CYCLES_PER_CHECK
|
||||
|
||||
|
||||
def totals(chunks: list[ChunkFigures], skipped: tuple[str, ...] = ()) -> Figures:
|
||||
tokens = {dimension: sum((chunk.tokens.get(dimension, collections.Counter()) for chunk in chunks), collections.Counter()) for dimension in DIMENSIONS}
|
||||
routed: dict[Rung, list[ChunkFigures]] = {}
|
||||
reports: dict[str, Reports] = {}
|
||||
for chunk in chunks:
|
||||
if chunk.start:
|
||||
routed.setdefault(chunk.start, []).append(chunk)
|
||||
for kind, found in chunk.reports.items():
|
||||
reports[kind] = reports.get(kind, Reports(0, 0, 0)) + found
|
||||
return Figures(len(chunks), tokens, routed, reports, skipped)
|
||||
|
||||
|
||||
def calls(figures: Figures) -> list[str]:
|
||||
"""Each routing change the figures call for, worded alike whatever the numbers, so a call already raised is known again."""
|
||||
compared = sorted((rung for rung, chunks in figures.routed.items() if len(chunks) >= MEASURED_AT_LEAST and position(rung) >= 0), key=position)
|
||||
found = []
|
||||
for lower, upper in zip(compared, compared[1:]):
|
||||
for what, of in (("review rounds past each pass's first", lambda chunk: chunk.rounds_past_first), ("unfinished verdicts", lambda chunk: chunk.unfinished)):
|
||||
if mean(figures.routed[lower], of) > mean(figures.routed[upper], of):
|
||||
found.append(f"Chunks routed onto {lower} took more {what} than chunks routed onto {upper}.")
|
||||
# Characters stand in for tokens: reports and searches are both text the worker would read, so their sums compare their shares of its context.
|
||||
found += [f"{kind} subagents' reports took more of the worker's context than their searches would have." for kind, reports in sorted(figures.reports.items()) if reports.subagents >= MEASURED_AT_LEAST and reports.report > reports.searches]
|
||||
return found
|
||||
|
||||
|
||||
def mean(chunks: list[ChunkFigures], of: typing.Callable[[ChunkFigures], int]) -> float:
|
||||
return statistics.mean(map(of, chunks))
|
||||
|
||||
|
||||
def render(figures: Figures, found: list[str], at: str, cycle: int, since: str) -> str:
|
||||
"""The figures as markdown; MEASURED_AT reads the cycle back. since is the latest reset's stamp, empty for none."""
|
||||
sections = [
|
||||
"# Routing figures",
|
||||
f"Measured {at}, at cycle {cycle} across loop-claude's archive, from {figures.chunks} finished chunk{'' if figures.chunks == 1 else 's'}{f' that finished after the reset of {since}' if since else ''}. Tokens are input, cache-written and output tokens; cache reads are left out. A message's tokens count towards the tools whose results it took in, split evenly; a message that took in no tool result counts towards `prompt`.",
|
||||
"## Calls for a routing change",
|
||||
"".join(f"- {call}\n" for call in found).rstrip("\n") or "None.",
|
||||
*[part for dimension in DIMENSIONS for part in (f"## Tokens by {dimension}", table((dimension.capitalize(), "Tokens"), [(label, str(tokens)) for label, tokens in figures.tokens[dimension].most_common()]))],
|
||||
"## Routed chunks by the rung they started on",
|
||||
table(("Rung", "Chunks", "Review rounds past each pass's first, mean", "Unfinished verdicts, mean", "Subagent rungs"), [
|
||||
(str(rung), str(len(chunks)), f"{mean(chunks, lambda chunk: chunk.rounds_past_first):.1f}", f"{mean(chunks, lambda chunk: chunk.unfinished):.1f}", ", ".join(f"{label} {times}" for label, times in sum((chunk.delegated for chunk in chunks), collections.Counter()).most_common()))
|
||||
for rung, chunks in sorted(figures.routed.items(), key=lambda item: position(item[0]))
|
||||
]),
|
||||
"## Subagent reports against the searches they ran, in characters",
|
||||
table(("Subagent type", "Subagents", "Reports", "Searches"), [(kind, str(found.subagents), str(found.report), str(found.searches)) for kind, found in sorted(figures.reports.items())]),
|
||||
*(["## Chunks skipped, unreadable", "".join(f"- {folder}\n" for folder in figures.skipped).rstrip("\n")] if figures.skipped else []),
|
||||
]
|
||||
return "\n\n".join(sections) + "\n"
|
||||
|
||||
|
||||
def table(header: tuple[str, ...], rows: list[tuple[str, ...]]) -> str:
|
||||
if not rows:
|
||||
return "None."
|
||||
return "\n".join(f"| {' | '.join(row)} |" for row in [header, tuple("---" for _ in header), *rows])
|
||||
@@ -0,0 +1,244 @@
|
||||
import collections
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from routing_figures import archive
|
||||
from routing_figures.measure import calls, Chunk, ChunkFigures, due, measured, render, Reports, Subagent, totals
|
||||
from failure import Failure
|
||||
from model_routing.ladder import LADDER, Rung
|
||||
from transcript_archive import outcomes, write
|
||||
|
||||
LOW = Rung("claude-opus-5-5", "medium")
|
||||
HIGH = Rung("claude-opus-5-5", "high")
|
||||
LOW_ROUTE = f"{LOW}, routed by item 7's R + C of 9"
|
||||
|
||||
|
||||
def message(id_, at, *parts, model="claude-opus-5-5", tokens=(1, 10, 100)):
|
||||
return {"type": "assistant", "timestamp": at, "cwd": "/c", "message": {"id": id_, "model": model, "content": list(parts), "usage": {"input_tokens": tokens[0], "cache_creation_input_tokens": tokens[1], "output_tokens": tokens[2], "cache_read_input_tokens": 5000}}}
|
||||
|
||||
|
||||
def call(name, id_="t1", **arguments):
|
||||
return {"type": "tool_use", "id": id_, "name": name, "input": arguments}
|
||||
|
||||
|
||||
def result(id_, text):
|
||||
return {"type": "user", "message": {"content": [{"type": "tool_result", "tool_use_id": id_, "content": [{"type": "text", "text": text}]}]}}
|
||||
|
||||
|
||||
def stamped(at, outcome):
|
||||
return f"{at} {outcome}\n"
|
||||
|
||||
|
||||
def figures(start=LOW, rounds=0, unfinished=0, reports=None):
|
||||
return ChunkFigures({}, start, rounds, unfinished, collections.Counter(), reports or {})
|
||||
|
||||
|
||||
class Measured(unittest.TestCase):
|
||||
def test_counts_each_messages_tokens_once_by_project_type_rung_and_the_tool_it_followed(self):
|
||||
worker = [
|
||||
message("m1", "2026-10-14T10:00:00.100Z", {"type": "text", "text": "reading"}),
|
||||
message("m1", "2026-10-14T10:00:00.200Z", call("Read", file_path="/c/a")),
|
||||
message("m2", "2026-10-14T10:05:00.000Z", {"type": "text", "text": "done"}),
|
||||
message("m3", "2026-10-14T11:00:00.000Z", {"type": "text", "text": "on"}),
|
||||
]
|
||||
kept = stamped("2026-10-14T10:30:00Z", outcomes.unfinished(LOW_ROUTE, "e")) + stamped("2026-10-14T10:30:00Z", outcomes.stepped_up(str(HIGH))) + stamped("2026-10-14T12:00:00Z", outcomes.finished(str(HIGH), "merged"))
|
||||
found = measured(Chunk("log", [kept], worker, []), "/h")
|
||||
self.assertEqual(found.tokens["project"], {"log": 333})
|
||||
self.assertEqual(found.tokens["subagent type"], {"worker": 333})
|
||||
self.assertEqual(found.tokens["rung"], {str(LOW): 222, str(HIGH): 111})
|
||||
self.assertEqual(found.tokens["tool"], {"prompt": 222, "Read": 111})
|
||||
self.assertEqual((found.start, found.unfinished), (LOW, 1))
|
||||
|
||||
def test_reads_a_subagents_rung_from_its_model_and_the_effort_its_call_set_or_else_the_workers(self):
|
||||
worker = [
|
||||
message("m1", "2026-10-14T10:00:00Z", call("Agent", "a1", subagent_type="Explore", model="sonnet", effort="high", prompt="find"), call("Agent", "a2", subagent_type="Explore", prompt="find")),
|
||||
result("a1", "x" * 30),
|
||||
result("a2", "y" * 5),
|
||||
]
|
||||
subagents = [
|
||||
Subagent("Explore", "a1", [message("s1", "2026-10-14T10:00:01Z", model="claude-sonnet-5-5"), result("r", "z" * 20)]),
|
||||
Subagent("", "a2", [message("s2", "2026-10-14T10:00:01Z")]),
|
||||
]
|
||||
found = measured(Chunk("log", [stamped("2026-10-14T11:00:00Z", outcomes.finished(LOW_ROUTE, "m"))], worker, subagents), "/h")
|
||||
self.assertEqual(found.delegated, {"claude-sonnet-5-5 at high effort": 1, str(LOW): 1})
|
||||
self.assertEqual(found.tokens["subagent type"], {"worker": 111, "Explore": 222})
|
||||
self.assertEqual(found.reports, {"Explore": Reports(2, 35, 20)})
|
||||
|
||||
def test_counts_review_rounds_past_each_passs_first_once_per_title(self):
|
||||
commit = lambda id_, title: message(id_, "2026-10-14T10:00:00Z", call("Bash", command=f"git commit -m '{title}'"))
|
||||
worker = [commit("m1", "Answer stability round 1: 1 🔴, 0 🟡"), commit("m2", "Answer stability round 2: 0 🔴, 0 🟡"), commit("m3", "Answer stability round 2: 0 🔴, 0 🟡"), commit("m4", "Answer prose round 1: 0 🔴, 1 🟡")]
|
||||
self.assertEqual(measured(Chunk("log", [], worker, []), "/h").rounds_past_first, 1)
|
||||
|
||||
def test_starts_no_route_for_a_chunk_no_item_routed(self):
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.finished(str(LOW), "m"))
|
||||
found = measured(Chunk("log", [kept], [message("m1", "2026-10-14T09:00:00Z")], []), "/h")
|
||||
self.assertIsNone(found.start)
|
||||
self.assertEqual(found.tokens["rung"], {str(LOW): 111})
|
||||
|
||||
|
||||
def test_puts_a_subagent_on_the_model_it_ran_at_the_effort_of_the_cycle_that_launched_it(self):
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.unfinished(LOW_ROUTE, "e")) + stamped("2026-10-14T11:00:00Z", outcomes.finished(str(HIGH), "m"))
|
||||
worker = [
|
||||
message("m1", "2026-10-14T10:00:00.900Z"),
|
||||
message("m2", "2026-10-14T10:30:00Z", call("Agent", "a1", subagent_type="Explore", model="sonnet")),
|
||||
message("m3", "2026-10-14T10:31:00Z", call("Agent", "a2", subagent_type="Explore")),
|
||||
]
|
||||
subagents = [Subagent("Explore", "a1", [result("r", "x")]), Subagent("Explore", "a2", [message("s2", "2026-10-14T10:31:01Z", model="claude-opus-5-5")])]
|
||||
found = measured(Chunk("log", [kept], worker, subagents), "/h")
|
||||
self.assertEqual(found.delegated, {"sonnet at high effort": 1, str(HIGH): 1})
|
||||
self.assertEqual(found.tokens["rung"], {str(LOW): 111, str(HIGH): 333})
|
||||
|
||||
def test_names_a_subagent_by_its_meta_or_else_its_call_or_else_unknown(self):
|
||||
worker = [message("m1", "2026-10-14T10:00:00Z", call("Agent", "a1", subagent_type="Explore"), call("Agent", "a2", subagent_type="Plan"))]
|
||||
subagents = [Subagent("stability-reviewer", "a1", []), Subagent("", "a2", []), Subagent("", "", [])]
|
||||
self.assertEqual(measured(Chunk("log", [], worker, subagents), "/h").reports, {"stability-reviewer": Reports(1, 0, 0), "Plan": Reports(1, 0, 0), "unknown": Reports(1, 0, 0)})
|
||||
|
||||
def test_counts_a_message_without_usage_as_nothing_and_splits_tokens_whole_across_tools(self):
|
||||
worker = [
|
||||
{"type": "assistant", "uuid": "u1", "message": {"content": [call("Read"), call("Grep")], "usage": {"input_tokens": 3, "output_tokens": "many"}}},
|
||||
{"type": "user", "message": {"content": [{"type": "tool_result", "tool_use_id": "t1", "content": "plain"}, "odd"]}},
|
||||
{"type": "assistant", "uuid": "u2", "message": {"content": [], "usage": {"input_tokens": 5}}},
|
||||
{"type": "assistant", "uuid": "u3", "message": {"content": []}},
|
||||
]
|
||||
found = measured(Chunk("log", [], worker, []), "/h")
|
||||
self.assertEqual(found.tokens["tool"], {"prompt": 3, "Read": 3, "Grep": 2})
|
||||
self.assertEqual(found.tokens["project"], {"log": 8})
|
||||
|
||||
|
||||
class Calls(unittest.TestCase):
|
||||
def test_calls_when_chunks_on_a_rung_took_more_rounds_or_unfinished_verdicts_than_the_rung_above(self):
|
||||
chunks = [figures(LOW, rounds=2)] * 5 + [figures(HIGH, unfinished=1)] * 5
|
||||
self.assertEqual(calls(totals(chunks)), [
|
||||
f"Chunks routed onto {LOW} took more review rounds past each pass's first than chunks routed onto {HIGH}.",
|
||||
])
|
||||
self.assertEqual(calls(totals([figures(LOW, unfinished=1)] * 5 + [figures(HIGH)] * 5)), [
|
||||
f"Chunks routed onto {LOW} took more unfinished verdicts than chunks routed onto {HIGH}.",
|
||||
])
|
||||
|
||||
def test_compares_the_bottom_rungs_and_calls_nothing_on_a_tie(self):
|
||||
self.assertEqual(calls(totals([figures(LADDER[0], rounds=1)] * 5 + [figures(LADDER[1])] * 5)), [f"Chunks routed onto {LADDER[0]} took more review rounds past each pass's first than chunks routed onto {LADDER[1]}."])
|
||||
self.assertEqual(calls(totals([figures(LOW, rounds=1)] * 5 + [figures(HIGH, rounds=1)] * 5)), [])
|
||||
self.assertEqual(calls(totals([figures(reports={"Explore": Reports(1, 20, 20)})] * 5)), [])
|
||||
|
||||
def test_compares_only_rungs_with_five_chunks_each(self):
|
||||
self.assertEqual(calls(totals([figures(LOW, rounds=2)] * 5 + [figures(HIGH)] * 4)), [])
|
||||
|
||||
def test_calls_when_a_subagent_types_reports_outgrow_the_searches_they_ran(self):
|
||||
self.assertEqual(calls(totals([figures(reports={"Explore": Reports(1, 30, 20)})] * 5)), [
|
||||
"Explore subagents' reports took more of the worker's context than their searches would have.",
|
||||
])
|
||||
self.assertEqual(calls(totals([figures(reports={"Explore": Reports(1, 30, 20)})] * 4)), [])
|
||||
self.assertEqual(calls(totals([figures(reports={"Explore": Reports(1, 20, 30)})] * 5)), [])
|
||||
|
||||
def test_renders_each_dimension_largest_first_and_the_calls(self):
|
||||
chunk = ChunkFigures({"project": collections.Counter({"a": 2, "b": 20})}, LOW, 1, 0, collections.Counter({str(HIGH): 2}), {"Explore": Reports(2, 3, 4)})
|
||||
text = render(totals([chunk]), ["Call."], "2026-10-15T00:00:00Z", 40, "")
|
||||
self.assertIn("Measured 2026-10-15T00:00:00Z, at cycle 40 across loop-claude's archive, from 1 finished chunk.", text)
|
||||
self.assertIn("## Calls for a routing change\n\n- Call.\n", text)
|
||||
self.assertIn("| b | 20 |\n| a | 2 |", text)
|
||||
self.assertIn(f"| {LOW} | 1 | 1.0 | 0.0 | {HIGH} 2 |", text)
|
||||
two = ChunkFigures({}, HIGH, 0, 1, collections.Counter({str(HIGH): 2, str(LOW): 1}), {})
|
||||
self.assertIn(f"| {LOW} | 1 | 1.0 | 0.0 | {HIGH} 2 |\n| {HIGH} | 1 | 0.0 | 1.0 | {HIGH} 2, {LOW} 1 |", render(totals([two, chunk]), [], "2026-10-15T00:00:00Z", 40, ""))
|
||||
self.assertIn("| Explore | 2 | 3 | 4 |", text)
|
||||
self.assertIn("## Calls for a routing change\n\nNone.\n", render(totals([]), [], "2026-10-15T00:00:00Z", 20, ""))
|
||||
self.assertIn("from 0 finished chunks that finished after the reset of 2026-10-14T00:00:00Z.", render(totals([]), [], "2026-10-15T00:00:00Z", 20, "2026-10-14T00:00:00Z"))
|
||||
|
||||
def test_measures_once_the_count_runs_20_past_the_last_measurement(self):
|
||||
self.assertEqual([due(cycle, 20) for cycle in (39, 40, 41)], [False, True, True])
|
||||
self.assertTrue(due(20, 0))
|
||||
|
||||
|
||||
class Archive(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.root = pathlib.Path(tempfile.mkdtemp())
|
||||
self.addCleanup(shutil.rmtree, self.root)
|
||||
|
||||
def chunk(self, key, name, kept, worker, subagent=None):
|
||||
folder = self.root / key / name
|
||||
(folder / "session" / "s" / "subagents").mkdir(parents=True)
|
||||
(folder / "copy.outcomes.md").write_text(f"Task: t\n\n{kept}")
|
||||
(folder / "copy.jsonl").write_text("".join(json.dumps(entry) + "\n" for entry in worker))
|
||||
if subagent:
|
||||
meta, entries = subagent
|
||||
(folder / "session" / "s" / "subagents" / "agent-a.jsonl").write_text("".join(json.dumps(entry) + "\n" for entry in entries))
|
||||
(folder / "session" / "s" / "subagents" / "agent-a.meta.json").write_text(json.dumps(meta))
|
||||
|
||||
def test_counts_the_verdicts_across_every_projects_chunks(self):
|
||||
self.chunk("home/x", "c1", stamped("2026-10-14T10:00:00Z", outcomes.unfinished("w", "e")) + stamped("2026-10-14T10:01:00Z", outcomes.stopped("w", "Ctrl-C")) + stamped("2026-10-14T10:02:00Z", outcomes.finished("w", "e")), [])
|
||||
self.chunk("log", "c2", stamped("2026-10-14T10:00:00Z", outcomes.interrupted("q", "e")), [])
|
||||
self.assertEqual(archive.cycles(self.root), 2)
|
||||
self.assertEqual(archive.cycles(self.root / "missing"), 0)
|
||||
|
||||
def test_measures_each_finished_chunk_keyed_by_its_project(self):
|
||||
self.chunk("home/x", "c1", stamped("2026-10-14T10:00:00Z", outcomes.finished(LOW_ROUTE, "m")), [message("m1", "2026-10-14T09:00:00Z", call("Agent", "a1", subagent_type="Explore"))], ({"agentType": "Explore", "toolUseId": "a1"}, [message("s1", "2026-10-14T09:00:01Z")]))
|
||||
self.chunk("log", "c2", stamped("2026-10-14T10:00:00Z", outcomes.unfinished(LOW_ROUTE, "e")), [message("m1", "2026-10-14T09:00:00Z")])
|
||||
found = archive.measure_archive(self.root, "/h", "")
|
||||
self.assertEqual([chunk.tokens["project"] for chunk in found.chunks], [{"home/x": 222}])
|
||||
self.assertEqual(found.chunks[0].tokens["subagent type"], {"worker": 111, "Explore": 111})
|
||||
self.assertEqual(archive.measure_archive(self.root, "/h", "2026-10-14T10:00:00Z"), archive.Measured([], ()))
|
||||
|
||||
def test_reads_each_cycles_route_from_its_own_outcomes_file(self):
|
||||
transcript = self.root / "t.jsonl"
|
||||
kept = self.root / "archive"
|
||||
first = message("m1", "2026-09-21T00:00:00Z")
|
||||
transcript.write_text(json.dumps(first) + "\n")
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(kept / "log", str(transcript), "t", outcomes.unfinished(LOW_ROUTE, "e"), 1790000000))
|
||||
transcript.write_text(json.dumps(first) + "\n" + json.dumps(message("m2", "2099-01-01T00:00:00Z")) + "\n")
|
||||
self.assertIsNone(write.archive(kept / "log", str(transcript), "t", outcomes.finished(str(HIGH), "m"), 4070908800))
|
||||
(found,) = archive.measure_archive(kept, "/h", "").chunks
|
||||
self.assertEqual((found.start, found.unfinished), (LOW, 1))
|
||||
self.assertEqual(found.tokens["rung"], {str(LOW): 111, str(HIGH): 111})
|
||||
|
||||
def test_skips_a_chunk_it_cannot_read_and_names_it(self):
|
||||
self.chunk("log", "c1", stamped("2026-10-14T10:00:00Z", outcomes.finished(LOW_ROUTE, "m")), [])
|
||||
(self.root / "log" / "c1" / "copy.jsonl").unlink()
|
||||
self.chunk("other", "c2", stamped("2026-10-14T10:00:00Z", outcomes.finished(LOW_ROUTE, "m")), [message("m1", "2026-10-14T09:00:00Z")])
|
||||
self.chunk("broken", "c3", "", [])
|
||||
(self.root / "broken" / "c3" / "copy.outcomes.md").write_bytes(b"\xff")
|
||||
found = archive.measure_archive(self.root, "/h", "")
|
||||
self.assertEqual((len(found.chunks), found.skipped), (1, ("broken", "log/c1")))
|
||||
self.assertIn("## Chunks skipped, unreadable\n\n- log/c1\n", render(totals([], ("log/c1",)), [], "2026-10-15T00:00:00Z", 20, ""))
|
||||
|
||||
def test_keeps_the_figures_and_reads_back_the_cycle_they_were_measured_at(self):
|
||||
self.assertEqual(archive.last_measured(self.root), 0)
|
||||
self.assertIsNone(archive.write_figures(self.root, render(totals([]), [], "2026-10-15T00:00:00Z", 40, "")))
|
||||
self.assertEqual(archive.last_measured(self.root), 40)
|
||||
|
||||
def test_reads_the_calls_raised_since_the_latest_reset(self):
|
||||
self.assertEqual(archive.raised(self.root), archive.Raised("", frozenset()))
|
||||
self.assertIsNone(archive.mark_raised(self.root, ["A."], "2026-10-15T00:00:00Z"))
|
||||
self.assertIsNone(archive.mark_raised(self.root, ["B."], "2026-10-15T01:00:00Z"))
|
||||
self.assertEqual((self.root / archive.RAISED).read_text(), "# Routing calls raised\n\n- 2026-10-15T00:00:00Z A.\n- 2026-10-15T01:00:00Z B.\n")
|
||||
self.assertEqual(archive.raised(self.root), archive.Raised("", frozenset({"A.", "B."})))
|
||||
self.assertIsNone(archive.mark_raised(self.root, ["Reset", "C."], "2026-10-15T00:30:00Z"))
|
||||
self.assertEqual(archive.raised(self.root), archive.Raised("2026-10-15T00:30:00Z", frozenset({"B."})))
|
||||
|
||||
def test_reads_no_measured_cycle_from_figures_that_name_none(self):
|
||||
(self.root / archive.FIGURES).write_text("# Routing figures\n")
|
||||
self.assertEqual(archive.last_measured(self.root), 0)
|
||||
(self.root / archive.FIGURES).unlink()
|
||||
(self.root / archive.FIGURES).mkdir()
|
||||
self.assertIsInstance(archive.last_measured(self.root), Failure)
|
||||
|
||||
def test_holds_the_lock_while_no_other_loop_does(self):
|
||||
with archive.locked(self.root) as held, archive.locked(self.root) as other:
|
||||
self.assertEqual((held, other), (True, False))
|
||||
with archive.locked(self.root) as again:
|
||||
self.assertTrue(again)
|
||||
|
||||
def test_fails_on_an_archive_it_cannot_read(self):
|
||||
(self.root / archive.RAISED).mkdir()
|
||||
self.assertIsInstance(archive.raised(self.root), Failure)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -6,9 +6,13 @@ from claude_code.transcript import first_timestamped_line
|
||||
from project_key import project_key
|
||||
|
||||
|
||||
def archive_root(home: pathlib.Path, env: typing.Mapping[str, str]) -> pathlib.Path:
|
||||
"""Where loop-claude archives every project's chunks."""
|
||||
return pathlib.Path(env["LOOP_CLAUDE_TRANSCRIPTS"]).expanduser() if env.get("LOOP_CLAUDE_TRANSCRIPTS") else home / ".claude" / "loop-claude-transcripts"
|
||||
|
||||
|
||||
def project_archive(checkout_root: str, home: pathlib.Path, env: typing.Mapping[str, str]) -> pathlib.Path:
|
||||
archive = pathlib.Path(env["LOOP_CLAUDE_TRANSCRIPTS"]).expanduser() if env.get("LOOP_CLAUDE_TRANSCRIPTS") else home / ".claude" / "loop-claude-transcripts"
|
||||
return archive / project_key(checkout_root, home)
|
||||
return archive_root(home, env) / project_key(checkout_root, home)
|
||||
|
||||
|
||||
def chunk_folder(project_archive: pathlib.Path, transcript: bytes) -> pathlib.Path:
|
||||
|
||||
@@ -4,7 +4,11 @@ from markdown_text import quoted
|
||||
from utc_stamp import STAMPED
|
||||
|
||||
FINISHED = "Verdict: finished"
|
||||
UNFINISHED = "Verdict: unfinished"
|
||||
FINISHED_AT = re.compile(rf"^({STAMPED}) {re.escape(FINISHED)}$", re.MULTILINE)
|
||||
UNFINISHED_AT = re.compile(rf"^{STAMPED} {re.escape(UNFINISHED)}$", re.MULTILINE)
|
||||
# A verdict closes each cycle the judge ruled on; an interrupt or a stop closes none.
|
||||
VERDICT_AT = re.compile(rf"^{STAMPED} (?:{re.escape(FINISHED)}|{re.escape(UNFINISHED)})$", re.MULTILINE)
|
||||
# An outcome opens with its UTC stamp; a verdict that closes a cycle, a stop, or a step up ends with the worker line.
|
||||
OUTCOME_AT = re.compile(rf"^({STAMPED}) ", re.MULTILINE)
|
||||
NAMES_A_WORKER = ("Verdict: finished\n", "Verdict: unfinished\n", "Stopped: ", "Stepped up\n")
|
||||
@@ -16,7 +20,7 @@ def finished(worker: str, evidence: str) -> str:
|
||||
|
||||
|
||||
def unfinished(worker: str, evidence: str) -> str:
|
||||
return f"Verdict: unfinished\n\nEvidence:\n{quoted(evidence)}\n\n{WORKER}{worker}"
|
||||
return f"{UNFINISHED}\n\nEvidence:\n{quoted(evidence)}\n\n{WORKER}{worker}"
|
||||
|
||||
|
||||
def interrupted(question: str, evidence: str) -> str:
|
||||
|
||||
@@ -1659,3 +1659,29 @@ Inherited architect, decided by:
|
||||
- Locality: `CLAUDE.md:55` "Working with the codebase": which model a subagent gets now spans this prose rule, `prompts.py`'s `off_the_projects_rung` and `ladder.py`'s `Rung.agent_args`; that reviewers above the project's rung inherit the worker's rung is said in neither file
|
||||
- Locality: `bin/model_routing/ladder.py:16` `Rung.agent_args`: the family name comes from `self.model.split('-')[1]`, tying the ladder's ids to the Agent tool's vocabulary with nothing checking it
|
||||
- Locality: `bin/loop_claude/routing.py:54` `kept` and `:59` `stepped`: whether a rerun resumes now depends on the R + C the archived route line holds, and the order of the checks is more to hold
|
||||
|
||||
## 2026-10-14T23:10:29Z, PR #265 at ac41116, against 8bc0d88
|
||||
|
||||
Ruling: worse
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Mid A | same | worse | worse | same | worse |
|
||||
| Inherited architect | worse | worse | worse | worse | worse |
|
||||
|
||||
Mid A, decided by:
|
||||
|
||||
- Locality: `bin/loop_claude/loop/main.py:186` `run_loop`: every cycle now ends with a cross-project check that counts every project's cycles, takes a lock and may hand the terminal to a claude in ~/.claude, so the loop no longer reads alone
|
||||
- Locality: `bin/loop_claude/loop/routing_check.py:26` `new_calls` and `bin/routing_figures/archive.py:59` `last_measured`: whether a measurement is due is read back by regex from the markdown `render` writes, so rewording it silently breaks `due`; the raised-calls file's lifecycle ends in a manual rule in AGENTS.md
|
||||
- Locality: `bin/routing_figures/measure.py:130` `spend`: repeats `reading_cost/measure.py` `tokens_in`'s dedupe-by-message rule, adding output tokens, so a counting change has two places to land
|
||||
- Shape: `bin/routing_figures/measure.py:74` `measured`: dense single expressions building `Reports` and a six-argument `ChunkFigures`, and nested comprehensions in `render`
|
||||
- Shape: `bin/routing_figures/measure.py:43` `Reports`: `searches` holds characters of tool results, not a count of searches; generic names `calls`, `due`, `add`, `totals`, `kept`
|
||||
|
||||
Inherited architect, decided by:
|
||||
|
||||
- Locality, Self-sufficiency: `bin/loop_claude/loop/routing_check.py:46` `raise_calls`: how a routing call is raised, deduplicated and retired is split across the code, the prompt and a working rule in AGENTS.md about cited stamps and a hand-appended Reset
|
||||
- Locality: `bin/routing_figures/measure.py:193` `calls` and `bin/routing_figures/archive.py:124` `raised`: a call's identity is its English sentence, and the measured cycle is read back from rendered markdown, so the report's prose is load-bearing state
|
||||
- Locality: `bin/loop_claude/loop/main.py:186` `run_loop`: each project's loop now runs a cross-project step holding a lock and scanning the whole archive, so one loop depends on what other loops wrote
|
||||
- Shape: `bin/routing_figures/measure.py:74` `measured`: one 234-line module holds six concerns, and `measured`, `Measured` and `measure_archive` sit side by side; `kept` and `calls` each name two things
|
||||
- Navigation: `bin/model_routing/route.py:11` `Route`: moved out of loop_claude while still carrying loop-only state, so "routing" now spans five places
|
||||
- Self-sufficiency: `bin/routing_figures/measure.py:130` `spend`: the docstring says it counts as `tokens_in` does, but it adds output tokens
|
||||
|
||||
+3
-1
@@ -176,7 +176,9 @@ Valid while a mutation run on a chunk's changed lines stays cheap beside the rev
|
||||
|
||||
The rungs, least competent first: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models, read 2026-10-14), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). That cut drops Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index falls among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scores it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price.
|
||||
|
||||
A chunk's worker runs on the rung its item picks, below the project's rung too. Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks spends on work nothing ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each reviewer, each comprehension seat and the security pass's subagent on the project's rung, through the Agent tool's model and effort. It runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. The cost of a cheaper worker shows as extra review rounds and unfinished verdicts, which `todo.md` item 94 is to measure, interrupting lilleman when a rung takes more of either than the rung above. A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. lilleman chose medium knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
|
||||
A chunk's worker runs on the rung its item picks, below the project's rung too. Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks spends on work nothing ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each reviewer, each comprehension seat and the security pass's subagent on the project's rung, through the Agent tool's model and effort. It runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. lilleman chose medium knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
|
||||
|
||||
The cost of a cheaper worker shows as extra review rounds and unfinished verdicts. Every 20 cycles across the archive, loop-claude measures them in code, with no model run (goal 1.4), and interrupts lilleman with the calls not raised before, when chunks routed onto a rung took more of either than chunks routed onto the next rung up, each rung with 5 chunks or more. Five keeps one odd chunk from calling for a change (goal 1.2). It also calls when the reports of 5 or more subagents of one type were longer than the tool results they read, so a delegated search cost the worker more context than it saved. `bin/routing_figures/` measures and words the calls. A loop on another project cannot commit here, so it writes the figures and the calls raised at the archive's root, and this repo's next chunk files the plan item, as `AGENTS.md` says. Off a terminal, a call is said on stderr and the loop goes on: it is about this setup, not the running project's chunk. A reset line in the calls file restarts the figures after a routing change, so they show whether the change worked.
|
||||
|
||||
A model off the ladder routes nothing: claude runs it at the effort its settings give. R + C steps a task from the project's rung: 2–4 three rungs down, 5–6 two down, 7–8 one down, 9–10 none, 11–12 one up, 13–14 two up, 15–16 three up, 17 and above four up. A step past either end stops there. The bands spread the R + C that `todo.md`'s items held on 2026-10-14, 2 to 12, across the rungs, so most tasks step down. S never moves a rung, so a huge task with low R and C runs cheap. While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers inherit the worker's higher rung until item 90 routes them. `score-outcomes` had too few finished chunks to ground the bands, and a change to the ladder or the bands is lilleman's.
|
||||
|
||||
|
||||
@@ -299,3 +299,20 @@ Read most (opens and searches):
|
||||
- `skills/larv-review/SKILL.md` 2
|
||||
- `AGENTS.md` 1
|
||||
- `bin/loop_claude/cycle/prompts.py` 1
|
||||
|
||||
## Reading cost, chunk 44c0242e1b7d, finished 2026-10-14T22:22:35Z
|
||||
|
||||
128734 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `todo.md` 8
|
||||
- `CHANGELOG.md` 5
|
||||
- `.gitignore` 4
|
||||
- `docs/decisions.md` 4
|
||||
- `bin/score_outcomes/main.py` 3
|
||||
- `README.md` 2
|
||||
- `skills/larv-review/SKILL.md` 2
|
||||
- `AGENTS.md` 1
|
||||
- `CLAUDE.md` 1
|
||||
- `bin/larv-review` 1
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
`Bar = 9`
|
||||
|
||||
`Next ID = 139`
|
||||
`Next ID = 140`
|
||||
|
||||
| Goal | W |
|
||||
|---|---|
|
||||
@@ -26,7 +26,6 @@ Release 0.2.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
|
||||
|
||||
| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 94 | 0.1.0 | decision | **Every 20th cycle, have loop-claude measure token spend and routing outcomes, file them here, and interrupt the human only when they call for a routing change.** | 3 | 5 | 6 | 6 | 7 | 1.1, 1.4 | 21.6 |
|
||||
| 90 | 0.1.0 | | **Route each reviewer by the chunk's R + C, once a replay of past chunks shows the routed reviewer finds the 🔴 and 🟡 findings on record.** | 4 | 5 | 8 | 6 | 7 | 1.4 | 20.2 |
|
||||
| 89 | 0.1.0 | | **Run the judge on the rung its chunk's R + C picks, once a replay of archived transcripts shows it returns the verdicts on record.** | 3 | 4 | 7 | 5 | 6 | 1.4 | 17.5 |
|
||||
| 136 | 0.1.0 | decision | **Route a chunk on the item its worker says it takes, in place of the next release's first row.** | 2 | 3 | 4 | 4 | 5 | 1.4 | 14.8 |
|
||||
@@ -41,6 +40,7 @@ Release 0.2.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
|
||||
| 113 | 0.2.0 | | **Compute an item's S from the code it names, where it names any, and show it beside the finished items `score-outcomes` lists.** | 3 | 4 | 6 | 4 | 5 | 1.1, 1.4 | 13.9 |
|
||||
| 122 | 0.2.0 | | **Assemble the judge's prompt in `bin/loop_claude/cycle/judge.py`, importing only wording the worker also sees, and call the worker "the worker" in it.** | 3 | 4 | 4 | 4 | 5 | 1.1 | 13.9 |
|
||||
| 123 | 0.2.0 | | **Have `bin/comprehension_history.py` parse each history entry once into seats, votes and places, so `latest_places`, `worse_places` and `worse_votes` neither depend on table order nor split a label to find a seat.** | 3 | 5 | 5 | 4 | 5 | 1.1 | 13.7 |
|
||||
| 139 | 0.2.0 | | **Split `bin/routing_figures/measure.py` by concern, and keep the figures' measured cycle and each raised call as fields loop-claude reads back, not as the prose it renders.** | 2 | 3 | 3 | 3 | 5 | 1.1 | 13.2 |
|
||||
| 127 | 0.2.0 | | **Define seat, place, paired ruling, scoring run and the judge's records in `CLAUDE.md` → "Vocabulary".** | 1 | 1 | 1 | 3 | 4 | 1.1 | 12.7 |
|
||||
| 125 | 0.2.0 | decision | **Give "record", "outcome" and "places" one meaning each across `bin/`, and rename `bin/larv_review/git.py` by what it holds.** | 2 | 5 | 3 | 3 | 5 | 1.1 | 12.7 |
|
||||
| 126 | 0.2.0 | | **State when work has landed towards a release once, in `bin/larv_review/plan.py`, and have larv-review's help point to it.** | 1 | 2 | 2 | 3 | 4 | 1.1 | 12.4 |
|
||||
@@ -85,10 +85,6 @@ Release 0.2.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
|
||||
|
||||
## Details
|
||||
|
||||
### 94. Every 20th cycle, have loop-claude measure token spend and routing outcomes, file them here, and interrupt the human only when they call for a routing change.
|
||||
|
||||
Count the cycles across all projects in loop-claude's archive, which holds every project's transcripts on the machine. Measure in code, as `bin/reading-cost` does, so the check itself costs no model run (goal 1.4): tokens by project, subagent type, rung and tool, and per routed chunk its review rounds and unfinished verdicts, and the rungs its worker delegated subagents on, so the figures show a cheap subagent whose conclusions cost review rounds. A loop on another project cannot commit here, so it writes the figures to the archive, and this repo's next chunk files or refreshes one plan item from them. The ladder's thresholds and the delegation rule are decisions, so a change to them is the human's. Interrupt when chunks on a rung took more review rounds or unfinished verdicts than chunks on the rung above, or when subagent reports fill more of the worker's peak context than the searches they replaced would have; never twice for the same figures (goal 1.2).
|
||||
|
||||
### 90. Route each reviewer by the chunk's R + C, once a replay of past chunks shows the routed reviewer finds the 🔴 and 🟡 findings on record.
|
||||
|
||||
larv-review passes the rung through the Agent tool's `model` and `effort`. A reviewer never steps down when the worker has stepped down on the same chunk, so a cheap worker always meets a reviewer on the project model or above.
|
||||
@@ -145,6 +141,10 @@ All four seats of PR #257's scoring run named `bin/loop_claude/cycle/judge.py:16
|
||||
|
||||
Three seats of PR #257's scoring run named `bin/comprehension_history.py:106` `latest_places`; the logic it rests on is `:114` `worse_places` and `:126` `worse_votes`. They regex the markdown the module writes. `worse_votes` keeps a seat's first row through `setdefault`, because the votes table comes first. `worse_places` recovers the seat with `label.split(",")[0]`. `Entry` still holds scoring seats' places for rulings recorded before PR #255. Make these functions easier to read.
|
||||
|
||||
### 139. Split `bin/routing_figures/measure.py` by concern, and keep the figures' measured cycle and each raised call as fields loop-claude reads back, not as the prose it renders.
|
||||
|
||||
The paired ruling on PR #265 ruled worse. Make these places easier to read: `measure.py` `measured` and `render` (token attribution, subagent rungs, round counting, calls and rendering in one module; `measured`, `Measured` and `measure_archive` side by side); `calls` with `archive.py` `raised` and `last_measured`, where a call's identity is its sentence and the measured cycle is read back from rendered markdown; and `Reports.searches`, which holds characters, not searches.
|
||||
|
||||
### 127. Define seat, place, paired ruling, scoring run and the judge's records in `CLAUDE.md` → "Vocabulary".
|
||||
|
||||
Two seats of PR #257's scoring run assembled these terms from four files: `bin/larv_review/main.py`'s help, `skills/larv-review/SKILL.md:99`, `bin/comprehension_history.py:82` and `skills/comprehension-panel/seat-prompts.md`. "Records" is defined only in a comment at `bin/loop_claude/project/records.py:6`.
|
||||
@@ -189,7 +189,7 @@ PR #257's scoring run put Shape at 6.25, the lowest dimension. `bin/` holds abou
|
||||
|
||||
### 118. Have `score-outcomes` count review rounds only from chunks merged after 2026-10-13, when review passes were narrowed, so C's medians compare chunks reviewed under one rule.
|
||||
|
||||
Since 2026-10-13 (`docs/decisions.md` → "review: stability and product-owner report only…"), reviewers report fewer findings and later rounds review only the answers to earlier ones, to cut rounds. A chunk merged before it took more rounds for the same C, so mixing both eras skews C's medians and item 94's comparison of rungs.
|
||||
Since 2026-10-13 (`docs/decisions.md` → "review: stability and product-owner report only…"), reviewers report fewer findings and later rounds review only the answers to earlier ones, to cut rounds. A chunk merged before that date took more rounds for the same C, so mixing both eras skews C's medians.
|
||||
|
||||
### 130. Move the vote counting out of `bin/larv_review/ruling.py`'s `votes`, so reading a votes file decides nothing.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user