Answer prose round 1: 0 🔴, 10 🟡; split the AGENTS rule's conditions, say one interrupt carries the calls, unfold the figures' clauses, and drop a false README claim
This commit is contained in:
@@ -37,7 +37,7 @@ Decisions live in `docs/decisions.md`:
|
||||
|
||||
## Working rules
|
||||
|
||||
- When `routing-calls-raised.md`, at the root of loop-claude's archive (`$LOOP_CLAUDE_TRANSCRIPTS`, default `~/.claude/loop-claude-transcripts`), lists a call, any line but a `Reset`, stamped after its latest `Reset` and after the latest stamp `todo.md` or `docs/decisions.md` → `routing:` cites from it, file one `todo.md` item to settle the calls stamped since, or refresh the one already filed. Its detail quotes those calls with their stamps, and the routed-chunks table of `routing-figures.md` beside it, never the project names its other tables hold. A change to the ladder, its bands or the delegation rule is the human's. The chunk that settles the calls cites the latest stamp it settled in `docs/decisions.md` → `routing:`, whatever the human decided, so they are not filed again. Once the chunk that ships one has merged, append `- <UTC time> Reset` to `routing-calls-raised.md`, so the figures count only chunks finished after it, and its calls may be raised again.
|
||||
- `routing-calls-raised.md` sits at the root of loop-claude's archive (`$LOOP_CLAUDE_TRANSCRIPTS`, default `~/.claude/loop-claude-transcripts`). Each of its lines is a call or a `Reset`, with a stamp. A call is new when it is stamped after the latest `Reset` and after the latest stamp that `todo.md` or `docs/decisions.md` → `routing:` cites. When there are new calls, file one `todo.md` item to settle them, or refresh the item already filed. Its detail quotes those calls with their stamps, and the routed-chunks table of `routing-figures.md` beside it, never the project names its other tables hold. A change to the ladder, its bands or the delegation rule is the human's. The chunk that settles the calls cites the latest stamp it settled in `docs/decisions.md` → `routing:`, whatever the human decided, so they are not filed again. Once a chunk that changes the ladder, its bands or the delegation rule has merged, append `- <UTC time> Reset` to `routing-calls-raised.md`: the figures then count only chunks finished after it, and its calls may be raised again.
|
||||
- When `~/.claude/settings.json` has uncommitted changes, commit them on a branch of their own and merge them through a PR: Claude Code writes that file from any session. Read their diff first. When it holds a credential, such as a token in `env` or `mcpServers`, ask the human to move it out of the file, and commit only once it is gone. A token in `env` moves to the shell profile; a server in `mcpServers` moves to `~/.claude.json` with `claude mcp add --scope user`.
|
||||
|
||||
## Scoring run
|
||||
|
||||
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Every 20 cycles across all projects, `loop-claude` measures every finished chunk in its archive, with no model run, and writes the figures to `routing-figures.md` at the archive's root: tokens by project, subagent type, rung and tool, and for each chunk an item routed, by the rung it started on, its review rounds past each pass's first, its unfinished verdicts and its subagents' rungs. It interrupts you, with `claude` on the project's model on the terminal in `~/.claude`, when chunks routed onto a rung took more review rounds or unfinished verdicts than chunks routed onto the next rung up, each rung with 5 chunks or more, or when 5 or more subagents of one type wrote reports longer than the tool results they read. Each call is raised once, then listed in `routing-calls-raised.md` beside the figures; the next chunk of this repo files a `todo.md` item for it. Off a terminal, the calls are only said on stderr and stay in the figures, the loop goes on, and they are raised, and so reach the plan, at the next measurement on a terminal. After a routing change, a `- <UTC time> Reset` line in that file makes the figures count only chunks finished after it. Expect a pause of about ten seconds at each measurement.
|
||||
- Every 20 cycles across all projects, `loop-claude` measures every finished chunk in its archive, with no model run, and writes the figures to `routing-figures.md` at the archive's root: tokens by project, subagent type, rung and tool, and, for each chunk a `todo.md` item routed, grouped by the rung it started on: its review rounds past each pass's first, its unfinished verdicts and its subagents' rungs. It interrupts you, with `claude` on the project's model on the terminal in `~/.claude`, when chunks routed onto a rung took more review rounds or unfinished verdicts than chunks routed onto the next rung up, each rung with 5 chunks or more, or when 5 or more subagents of one type wrote reports longer than the tool results they read. Each call is raised once, then listed in `routing-calls-raised.md` beside the figures; the next chunk of this repo files a `todo.md` item for it. Off a terminal, the calls are said on stderr and the loop goes on. They are raised at the next measurement on a terminal. After a routing change, a `- <UTC time> Reset` line in that file makes the figures count only chunks finished after it, and lets its calls be raised again. Expect a pause of about ten seconds at each measurement.
|
||||
- A session delegates a task to a subagent only when it needs just the task's conclusion, such as a search across many files, and keeps edits in its own thread. It scores the task's R and C as for a `todo.md` item, and launches the subagent on the model and effort that the new `bin/subagent-rung <model> <R+C>` picks from the project's model, such as `model sonnet, effort high`. `CLAUDE.md` states the rule. Expect searches to run on cheaper models, and less of each session's context to fill with search output.
|
||||
- `loop-claude` routes each chunk's worker on a ladder of model and effort rungs, from Haiku 5.5 at low effort to Fable 5.1 at max, when the project's model is on it. As a chunk starts, the R + C of `todo.md`'s top item in the next release picks the rung: 11–12 picks one rung above the project's rung, so Opus 5.5 at high for an Opus 5.5 project, and so on up to four rungs. An item of R + C 10 or less runs the worker at or below the project's model's cheapest rung, which for Opus 5.5 is medium effort, below the xhigh `settings.json` sets. While the worker runs below the project's model's cheapest rung, it launches every reviewer, comprehension seat and security pass, the full scan included, in a subagent on that rung. While it runs above that rung, it launches each comprehension seat and the security pass on that rung. It steps up one rung after every second unfinished verdict on a chunk, and to a higher rung when it takes in an item of higher R + C. Each cycle's header, and each finished or unfinished verdict, stop and step-up in the archive, name the worker's rung; a rerun resumes an open chunk on the latest rung archived for it. `docs/decisions.md` → `routing:` gives the ladder, its source and the bands. Expect a chunk on an easy item to cost much less than under xhigh, and perhaps to take more review rounds, and one on an item of R + C 15 or more to cost more.
|
||||
- A session reads `todo.md` up to `## Details`, and the details of only the items its chunk takes. It reads a `docs/decisions.md` entry only where the entry's title leaves open what it needs. Both rules are in `CLAUDE.md`, so they hold from a session's start. The decisions rule replaces one in `rules/decisions.md`, which loads only once a session has opened `docs/decisions.md` or `AGENTS.md`. A chunk's opening line now counts the unreleased changes instead of listing them. Expect fewer tokens read before a chunk's first commit.
|
||||
|
||||
@@ -92,7 +92,7 @@ pytest_add_cli_args=-k
|
||||
|
||||
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
|
||||
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, and `score_outcomes` imports `reading_cost`'s record path. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls, reading tokens as `reading_cost` does. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, and `score_outcomes` imports `reading_cost`'s record path. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `routing_figures/` measures loop-claude's archive for loop-claude's routing calls. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
|
||||
## Tracking a new file
|
||||
|
||||
|
||||
@@ -60,11 +60,11 @@ and counts its unfinished verdicts afresh. A rung no longer on the ladder, or be
|
||||
item's R + C picks, is dropped, and the chunk is routed afresh from todo.md. A todo.md or an
|
||||
archive that cannot be read or parsed is said on stderr, and the chunk is routed without it.
|
||||
|
||||
Every 20 cycles across all projects in the archive, counting the cycles a verdict closed, the loop
|
||||
that closes the 20th measures every finished chunk there, with no model run: tokens by project,
|
||||
subagent type, rung and the tool whose results they took in, and for each chunk an item routed, by
|
||||
the rung it started on, its review rounds past each pass's first, its unfinished verdicts and its
|
||||
subagents' rungs. It writes them to routing-figures.md at the archive's root. They call for a
|
||||
Counting the cycles a verdict closed across all projects in the archive, the first loop to close
|
||||
one 20 or more cycles after the last measurement measures every finished chunk there, with no model
|
||||
run: tokens by project, subagent type, rung and the tool whose results they took in, and, for each
|
||||
chunk a todo.md item routed, grouped by the rung it started on: its review rounds past each pass's
|
||||
first, its unfinished verdicts and its subagents' rungs. It writes them to routing-figures.md at the archive's root. They call for a
|
||||
routing change when chunks routed onto a rung took more review rounds or unfinished verdicts than
|
||||
chunks routed onto the next rung up, each rung with 5 chunks or more, or when the reports of 5 or
|
||||
more subagents of one type were longer than the tool results they read. Each call not raised before
|
||||
|
||||
@@ -14,7 +14,7 @@ from utc_stamp import utc
|
||||
|
||||
|
||||
def check_routing(root: pathlib.Path, rung: Rung, paths: Paths, effects: Effects) -> Stop | None:
|
||||
"""Stops only when claude cannot start for the calls; a figure it cannot measure or keep is said on stderr, and the loop goes on. rung runs the claude the calls are put to."""
|
||||
"""Stops only when claude cannot start for the calls; a figure it cannot measure or keep is said on stderr, and the loop goes on. rung is the rung of the claude that puts the calls to the human."""
|
||||
with archive.locked(root) as held:
|
||||
new = new_calls(root, paths, effects) if held else []
|
||||
if isinstance(new, Failure):
|
||||
|
||||
@@ -75,7 +75,7 @@ class Measured:
|
||||
|
||||
|
||||
def measure_archive(root: pathlib.Path, home: str, since: str) -> Measured | Failure:
|
||||
"""The figures of each chunk that finished after since, measured one chunk at a time, since the archive outgrows memory. Only an archive that cannot be listed fails."""
|
||||
"""The figures of each chunk that finished after since, measured one chunk at a time because the archive outgrows memory. Only an archive that cannot be listed fails."""
|
||||
try:
|
||||
projects = sorted({path.parent.parent for path in root.rglob("*.outcomes.md")})
|
||||
except OSError as failure:
|
||||
|
||||
@@ -24,7 +24,7 @@ PROMPT = "prompt"
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Subagent:
|
||||
"""What the subagent's .meta.json names, its agentType and the toolUseId of the worker's Agent call that launched it, each empty where it names none; and its transcript's entries."""
|
||||
"""kind is the agentType and launched_by the toolUseId of the launching Agent call, as the subagent's .meta.json names them, each empty where it names none; entries are its transcript's entries."""
|
||||
kind: str
|
||||
launched_by: str
|
||||
entries: list[dict]
|
||||
@@ -113,7 +113,7 @@ def add(tokens: dict[str, collections.Counter[str]], project: str, kind: str, ru
|
||||
|
||||
|
||||
def rung_at(cycles: list[tuple[str, str]], timestamp: str) -> Rung | str:
|
||||
"""The rung the worker line stamped first at or after timestamp names: the rung of the cycle that ran then. A line off the ladder names itself."""
|
||||
"""The rung of the cycle running at timestamp: the worker named by the first outcome stamped at or after it, or else by the last outcome. A worker off the ladder is returned as its text."""
|
||||
second = timestamp[:19] + "Z"
|
||||
found = next((rung for stamp, rung in cycles if stamp >= second), cycles[-1][1] if cycles else "unrouted")
|
||||
route = route_from(found)
|
||||
@@ -128,10 +128,7 @@ def subagent_rung(entries: list[dict], arguments: dict, worker: Rung | str) -> R
|
||||
|
||||
|
||||
def spend(entries: list[dict]) -> list[tuple[str, list[str], int]]:
|
||||
"""Each message's timestamp, the tools whose results it took in, and its input, cache-written and output tokens.
|
||||
|
||||
Claude Code writes one entry per content block of a message, each with the message's usage, so a message counts once.
|
||||
"""
|
||||
"""Each message's timestamp, the tools whose results it took in, and its input, cache-written and output tokens, counted once per message, as reading_cost's tokens_in counts them."""
|
||||
messages: dict[str, tuple[str, object, list[str]]] = {}
|
||||
for entry in entries:
|
||||
if entry.get("type") != "assistant" or not isinstance(message := entry.get("message"), dict):
|
||||
@@ -177,7 +174,7 @@ def rounds_past_first(worker: list[dict], home: str) -> int:
|
||||
|
||||
|
||||
def due(cycle: int, last_measured: int) -> bool:
|
||||
"""Whether the archive's cycle count has run CYCLES_PER_CHECK past the cycle the figures were last measured at: counting from the last measurement, a count another loop pushed past a multiple still measures."""
|
||||
"""Whether CYCLES_PER_CHECK cycles have closed since the figures were last measured. Counting from the last measurement still measures when another loop's cycles carried the count past a multiple of CYCLES_PER_CHECK."""
|
||||
return cycle >= last_measured + CYCLES_PER_CHECK
|
||||
|
||||
|
||||
@@ -201,7 +198,7 @@ def calls(figures: Figures) -> list[str]:
|
||||
for what, of in (("review rounds past each pass's first", lambda chunk: chunk.rounds_past_first), ("unfinished verdicts", lambda chunk: chunk.unfinished)):
|
||||
if mean(figures.routed[lower], of) > mean(figures.routed[upper], of):
|
||||
found.append(f"Chunks routed onto {lower} took more {what} than chunks routed onto {upper}.")
|
||||
# Characters stand in for tokens: both sides are text the worker would read. Both would fill the same peak context, so comparing the sums compares their shares of it.
|
||||
# Characters stand in for tokens: reports and searches are both text the worker would read, so their sums compare their shares of its context.
|
||||
found += [f"{kind} subagents' reports took more of the worker's context than their searches would have." for kind, reports in sorted(figures.reports.items()) if reports.subagents >= MEASURED_AT_LEAST and reports.report > reports.searches]
|
||||
return found
|
||||
|
||||
@@ -214,7 +211,7 @@ def render(figures: Figures, found: list[str], at: str, cycle: int, since: str)
|
||||
"""The figures as markdown; MEASURED_AT reads the cycle back. since is the latest reset's stamp, empty for none."""
|
||||
sections = [
|
||||
"# Routing figures",
|
||||
f"Measured {at}, at cycle {cycle} across loop-claude's archive, from {figures.chunks} finished chunk{'' if figures.chunks == 1 else 's'}{f' that finished after the reset of {since}' if since else ''}. Tokens are input, cache-written and output tokens; cache reads are left out. A message's tokens count towards the tools whose results it took in, split evenly.",
|
||||
f"Measured {at}, at cycle {cycle} across loop-claude's archive, from {figures.chunks} finished chunk{'' if figures.chunks == 1 else 's'}{f' that finished after the reset of {since}' if since else ''}. Tokens are input, cache-written and output tokens; cache reads are left out. A message's tokens count towards the tools whose results it took in, split evenly; a message that took in no tool result counts towards `prompt`.",
|
||||
"## Calls for a routing change",
|
||||
"".join(f"- {call}\n" for call in found).rstrip("\n") or "None.",
|
||||
*[part for dimension in DIMENSIONS for part in (f"## Tokens by {dimension}", table((dimension.capitalize(), "Tokens"), [(label, str(tokens)) for label, tokens in figures.tokens[dimension].most_common()]))],
|
||||
|
||||
+3
-1
@@ -176,7 +176,9 @@ Valid while a mutation run on a chunk's changed lines stays cheap beside the rev
|
||||
|
||||
The rungs, least competent first: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models, read 2026-10-14), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). That cut drops Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index falls among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scores it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price.
|
||||
|
||||
A chunk's worker runs on the rung its item picks, below the project's rung too. Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks spends on work nothing ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each reviewer, each comprehension seat and the security pass's subagent on the project's rung, through the Agent tool's model and effort. It runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. The cost of a cheaper worker shows as extra review rounds and unfinished verdicts. Every 20 cycles across the archive, loop-claude measures them in code, with no model run (goal 1.4), and interrupts lilleman once per call when chunks routed onto a rung took more of either than chunks routed onto the next rung up, each rung with 5 chunks or more. Five keeps one odd chunk from calling for a change (goal 1.2). It also calls when the reports of 5 or more subagents of one type were longer than the tool results they read, so a delegated search cost the worker more context than it saved. `bin/routing_figures/` measures and words the calls. A loop on another project cannot commit here, so it writes the figures and the calls raised at the archive's root, and this repo's next chunk files the plan item, as `AGENTS.md` says. Off a terminal, a call is said on stderr and the loop goes on: it is about this setup, not the running project's chunk. A reset line in the calls file restarts the figures after a routing change, so they show whether the change worked. A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. lilleman chose medium knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
|
||||
A chunk's worker runs on the rung its item picks, below the project's rung too. Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks spends on work nothing ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each reviewer, each comprehension seat and the security pass's subagent on the project's rung, through the Agent tool's model and effort. It runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. lilleman chose medium knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
|
||||
|
||||
The cost of a cheaper worker shows as extra review rounds and unfinished verdicts. Every 20 cycles across the archive, loop-claude measures them in code, with no model run (goal 1.4), and interrupts lilleman with the calls not raised before, when chunks routed onto a rung took more of either than chunks routed onto the next rung up, each rung with 5 chunks or more. Five keeps one odd chunk from calling for a change (goal 1.2). It also calls when the reports of 5 or more subagents of one type were longer than the tool results they read, so a delegated search cost the worker more context than it saved. `bin/routing_figures/` measures and words the calls. A loop on another project cannot commit here, so it writes the figures and the calls raised at the archive's root, and this repo's next chunk files the plan item, as `AGENTS.md` says. Off a terminal, a call is said on stderr and the loop goes on: it is about this setup, not the running project's chunk. A reset line in the calls file restarts the figures after a routing change, so they show whether the change worked.
|
||||
|
||||
A model off the ladder routes nothing: claude runs it at the effort its settings give. R + C steps a task from the project's rung: 2–4 three rungs down, 5–6 two down, 7–8 one down, 9–10 none, 11–12 one up, 13–14 two up, 15–16 three up, 17 and above four up. A step past either end stops there. The bands spread the R + C that `todo.md`'s items held on 2026-10-14, 2 to 12, across the rungs, so most tasks step down. S never moves a rung, so a huge task with low R and C runs cheap. While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers inherit the worker's higher rung until item 90 routes them. `score-outcomes` had too few finished chunks to ground the bands, and a change to the ladder or the bands is lilleman's.
|
||||
|
||||
|
||||
@@ -184,7 +184,7 @@ PR #257's scoring run put Shape at 6.25, the lowest dimension. `bin/` holds abou
|
||||
|
||||
### 118. Have `score-outcomes` count review rounds only from chunks merged after 2026-10-13, when review passes were narrowed, so C's medians compare chunks reviewed under one rule.
|
||||
|
||||
Since 2026-10-13 (`docs/decisions.md` → "review: stability and product-owner report only…"), reviewers report fewer findings and later rounds review only the answers to earlier ones, to cut rounds. A chunk merged before it took more rounds for the same C, so mixing both eras skews C's medians.
|
||||
Since 2026-10-13 (`docs/decisions.md` → "review: stability and product-owner report only…"), reviewers report fewer findings and later rounds review only the answers to earlier ones, to cut rounds. A chunk merged before that date took more rounds for the same C, so mixing both eras skews C's medians.
|
||||
|
||||
### 130. Move the vote counting out of `bin/larv_review/ruling.py`'s `votes`, so reading a votes file decides nothing.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user