Answer prose round 1: 0 🔴, 6 🟡; correct what the help, changelog, decision and item 136 claim, and tighten the routing comments

This commit is contained in:
2026-10-14 23:32:04 +02:00
parent 3820657506
commit f52228d9ad
13 changed files with 29 additions and 29 deletions
+1 -1
View File
@@ -33,7 +33,7 @@ Decisions live in `docs/decisions.md`:
- worker: a chunk's worker writes the simplest change that solves its item, test first wherever a test can check the change, and reads a technical principle only where the item, a finding or its own record names it; larv-review's reviewers hold the change to the principles and to the standards in `skills/larv-review/standards/`
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C steps it from its project's model's cheapest rung by fixed bands, and a chunk's worker never goes below that rung
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model, and a chunk's worker never runs below that rung
## Working rules
+1 -1
View File
@@ -2,7 +2,7 @@
## Unreleased
- `loop-claude` routes each chunk's worker on a ladder of model and effort rungs, from Haiku 5.5 at low effort to Fable 5.1 at max, when the project's model is on it. As a chunk starts, the R + C of `todo.md`'s top item in the next release picks the rung: 11–12 one rung above the project's model, which for Opus 5.5 is Opus 5.5 at high effort, and so on up to four rungs. The worker never runs below the project's model's cheapest rung, which for Opus 5.5 is medium effort, below the xhigh `settings.json` sets. It steps up one rung after every second unfinished verdict on a chunk, and to a higher rung when it takes in an item of higher R + C. Each cycle's header, and each verdict and stop in the archive, name the worker's rung; a rerun resumes an open chunk on the rung it last ran. `docs/decisions.md` → `routing:` gives the ladder, its source and the bands. Expect a chunk on an easy item to cost less than under xhigh, and one on a hard item to cost more.
- `loop-claude` routes each chunk's worker on a ladder of model and effort rungs, from Haiku 5.5 at low effort to Fable 5.1 at max, when the project's model is on it. As a chunk starts, the R + C of `todo.md`'s top item in the next release picks the rung: 11–12 one rung above the project's model, which for Opus 5.5 is Opus 5.5 at high effort, and so on up to four rungs. The worker never runs below the project's model's cheapest rung, which for Opus 5.5 is medium effort, below the xhigh `settings.json` sets. It steps up one rung after every second unfinished verdict on a chunk, and to a higher rung when it takes in an item of higher R + C. Each cycle's header, and each finished or unfinished verdict, stop and step-up in the archive, name the worker's rung; a rerun resumes an open chunk on the latest rung archived for it. `docs/decisions.md` → `routing:` gives the ladder, its source and the bands. Expect a chunk on an easy item to cost less than under xhigh, and one on an item of R + C 15 or more to cost more.
- A session reads `todo.md` up to `## Details`, and the details of only the items its chunk takes. It reads a `docs/decisions.md` entry only where the entry's title leaves open what it needs. Both rules are in `CLAUDE.md`, so they hold from a session's start. The decisions rule replaces one in `rules/decisions.md`, which loads only once a session has opened `docs/decisions.md` or `AGENTS.md`. A chunk's opening line now counts the unreleased changes instead of listing them. Expect fewer tokens read before a chunk's first commit.
- larv-review's stability and product-owner reviewers report only a defect, a requirement of the plan or the goals that the change breaks, a breach of a technical principle, or for stability a hard place the change touches; the stability reviewer leaves the style of comments to the prose reviewer, which now also reads help and error text. Each architect finding names a concrete later change the PR makes harder, and one that names none is dropped. The prose pass runs only on a chunk that changes prose a reader reads, such as docs, comments, help text or `todo.md`, and skips a code-only chunk. Expect fewer findings and fewer review rounds.
- Every `todo.md` item carries a C score, how hard the change is to get right, from 1 to 10, in a column after S. C stays out of the score. `rank_todo.py` fails on an items table without the column, and the rule hook then blocks committing that `todo.md`. Before the next commit that changes a project's `todo.md`, add the column and score every item's C, as `rank_todo.py --help` shows.
+1 -1
View File
@@ -50,7 +50,7 @@ JUDGE_RULES = (
@dataclasses.dataclass(frozen=True)
class Reply:
answer: str = dataclasses.field(metadata={"description": "With interrupt false, the passage of README.md or the records that answers the agent's question, quoted verbatim, without elisions. Empty otherwise."})
# Routes the next cycle's worker; loop-claude never steps it down, so the item the chunk was routed on may count too.
# Routes the next cycle's worker. Counting the item the chunk was routed on is harmless: loop-claude never steps the worker down.
bundled: int = dataclasses.field(metadata={"description": "The highest R + C the agent states for a todo.md item it took into this chunk; 0 when it states none."})
evidence: str = dataclasses.field(metadata={"description": "A verbatim quote of the one transcript passage the verdict rests on, without elisions: what triggers the interrupt; for a quoted answer, the agent's question; else what shows the chunk finished or the work remaining."})
finished: bool = dataclasses.field(metadata={"description": "Whether the chunk is finished."})
+1 -1
View File
@@ -20,7 +20,7 @@ def fresh(task: str, premises: str, route: Route) -> str:
def routed_by(route: Route) -> str:
return f"loop-claude runs you on {route.rung}, the rung item {route.item}'s R + C of {route.risk_and_complexity} picks. Name each todo.md item this chunk takes, with its R + C, whether or not it is item {route.item}."
return f"loop-claude runs you on {route.rung}, routed by item {route.item}'s R + C of {route.risk_and_complexity}. Name each todo.md item this chunk takes, with its R + C, whether or not it is item {route.item}."
def resumed(unfinished_evidence: str) -> str:
+13 -12
View File
@@ -41,15 +41,16 @@ run, is asked for one. LOOP_CLAUDE_JUDGE_MODEL overrides it for the judge, LOOP_
for the worker and the interrupt; with both set, nothing is asked.
Where that model is on the routing ladder (bin/model_routing/ladder.py), the worker and the
interrupt run on a rung of it instead: a model at an effort. As a chunk starts, the R + C of
todo.md's top item in the next release, skipping a question, picks the rung from the model's
cheapest one, and never below it; without such an item, or with a C missing, the chunk runs on the
model's cheapest rung. The worker's first prompt names the rung and the item, and asks it to name
each item the chunk takes, with its R + C. The judge reports the highest R + C it sees the worker
state, and the next cycle steps up to the rung that picks, never down. Every second unfinished
verdict on a chunk steps it up one rung. Each verdict, stop and step up archived names the worker's
rung, with the item and R + C that routed the chunk; a rerun resumes an open chunk on the latest
rung named there, and counts its unfinished verdicts afresh. A rung no longer on the ladder, or
interrupt run on one of the ladder's rungs: a model at an effort. As a chunk starts, the R + C of
todo.md's top item in the next release, skipping an item marked question, picks the rung. It
counts from the model's cheapest rung and never goes below it; without such an item, or with a C
missing, the chunk runs on the model's cheapest rung. The worker's first prompt names the rung and
the item, and asks the worker to name each item the chunk takes, with its R + C. The judge reports
the highest R + C the worker states, and the next cycle steps up to the rung that R + C picks,
never down. Every second unfinished verdict on a chunk steps it up one rung. Each finished or
unfinished verdict, stop and step-up in the archive names the worker's rung, with the item and
R + C that routed the chunk; a rerun resumes an open chunk on the latest rung named there, and
counts its unfinished verdicts afresh. A rung no longer on the ladder, or
below the model's cheapest, is dropped, and the chunk is routed afresh from todo.md. A todo.md or
an archive that cannot be read or parsed is said on stderr, and the chunk is routed without it.
@@ -73,9 +74,9 @@ Exit 1, saying why, when:
- the worker or the judge fails three times running, retried a minute apart within the cycle. A
worker fails when claude cannot start, exits non-zero, is killed, or ends without a result event;
a judge when claude cannot start or exits non-zero, or its reply misses or mistypes a field,
interrupts or quotes an answer without a question, gives a bundled R + C outside 0 to 20, or
quotes a passage the transcript does not hold; it holds an elided passage when each fragment
appears in order.
interrupts or quotes an answer without a question, gives an unfinished verdict an R + C outside
0 to 20, or quotes a passage the transcript does not hold; it holds an elided passage when each
fragment appears in order.
A usage limit is not a failure, and restarts the count of three: the worker or the judge is
retried 2 minutes after the limit resets, or every 10 minutes when the reset time is unknown or
more than 2 minutes past. The judge's reply gives its reset time only as text, so the judge is
+1 -1
View File
@@ -14,7 +14,7 @@ ROUTED = re.compile(r"(?P<model>\S+) at (?P<effort>\S+) effort(?:, routed by ite
class Route:
"""A chunk's worker rung and the plan item whose R + C picked it, 0 for none.
unfinished is the loop's count of unfinished verdicts since the rung last stepped up, kept here so a reload hands it over with the rung.
unfinished counts the unfinished verdicts towards the next one-rung step; a reload hands it over with the rung.
"""
rung: Rung
item: int = 0
+1 -1
View File
@@ -390,7 +390,7 @@ class Cycles(unittest.TestCase):
("claude-opus-5-5", "high"),
])
self.assertEqual({(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] == "judge"}, {("judge-model", None)})
self.assertIn("loop-claude runs you on claude-opus-5-5 at high effort, the rung item 4's R + C of 11 picks. Name each todo.md item this chunk takes, with its R + C, whether or not it is item 4.", argument(workers[0], "-p"))
self.assertIn("loop-claude runs you on claude-opus-5-5 at high effort, routed by item 4's R + C of 11. Name each todo.md item this chunk takes, with its R + C, whether or not it is item 4.", argument(workers[0], "-p"))
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> three\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's R + C of 11\n\n<UTC> Stepped up\n\nWorker: claude-fable-5-1 at max effort, routed by item 4's R + C of 11\n", outcomes)
self.assertIn("<UTC> Verdict: finished\n\nEvidence:\n> four\n\nWorker: claude-fable-5-1 at max effort, routed by item 4's R + C of 11\n", outcomes)
+2 -2
View File
@@ -523,7 +523,7 @@ class Steps(unittest.TestCase):
class WorkerArgs(unittest.TestCase):
def test_names_the_route_in_a_fresh_prompt_only_where_an_item_routed_it(self):
routed = prompts.fresh("Do it.", "## Goals", routing.Route(Rung("m", "high"), 3, 11))
self.assertIn("Continue any open draft PR or worktree for the chunk before creating a new one.\n\nloop-claude runs you on m at high effort, the rung item 3's R + C of 11 picks. Name each todo.md item this chunk takes, with its R + C, whether or not it is item 3.\n\nThe chunk's", routed)
self.assertIn("Continue any open draft PR or worktree for the chunk before creating a new one.\n\nloop-claude runs you on m at high effort, routed by item 3's R + C of 11. Name each todo.md item this chunk takes, with its R + C, whether or not it is item 3.\n\nThe chunk's", routed)
unrouted = prompts.fresh("Do it.", "## Goals", routing.Route(Rung("m")))
self.assertIn("before creating a new one.\n\nThe chunk's", unrouted)
self.assertNotIn("loop-claude runs you on", unrouted)
@@ -736,7 +736,7 @@ class JudgeFailures(unittest.TestCase):
("verdict_from", 'f"the judge\'s reply has no {name}', field),
("verdict_from", "'the judge interrupted without a question'", unasked),
("verdict_from", "'the judge quoted an answer without a question'", unasked),
("verdict_from", 'f"the judge\'s bundled R + C', "gives a bundled R + C outside 0 to 20"),
("verdict_from", 'f"the judge\'s bundled R + C', "gives an unfinished verdict an R + C outside 0 to 20"),
("run_judge", "f'could not start claude:", started),
("run_judge", "proc.stderr.strip() or f'the judge exited", started),
("run_judge", 'f"{verdict.reason}; it replied:', field),
+1 -2
View File
@@ -16,8 +16,7 @@ class Rung:
return f"{self.model} at {self.effort} effort" if self.effort else self.model
# Least competent first. The Artificial Analysis Intelligence Index orders every rung but the top one, keeping a rung only
# where no cheaper rung scores as high; Fable 5.1 tops the ladder as Anthropic's most capable model, which the index does not show.
# Least competent first.
LADDER = (
Rung("claude-haiku-5-5", "low"),
Rung("claude-haiku-5-5", "medium"),
+1 -1
View File
@@ -23,7 +23,7 @@ class Item:
@property
def risk_and_complexity(self) -> int:
"""R + C, which routes the item's chunk to a model; 0 without a C."""
"""R + C, which routes the item's chunk to a rung; 0 without a C."""
return self.scores["R"] + self.scores["C"] if "C" in self.scores else 0
+1 -1
View File
@@ -37,7 +37,7 @@ def read_threads(chunk: FinishedChunk) -> list[bytes] | Failure:
def latest_worker(project_archive: pathlib.Path, transcript: str) -> str | Failure | None:
"""The worker the latest outcome archived for the transcript's chunk names; None before its first verdict.
"""The worker named by the latest outcome archived for the transcript's chunk; None before any outcome names one.
Stamps run to the second, so the newest outcomes file decides, and in it the last outcome naming a worker.
"""
+4 -4
View File
@@ -170,14 +170,14 @@ The run has no time cap. Limiting it to the changed lines already bounds it by t
Valid while a mutation run on a chunk's changed lines stays cheap beside the review itself.
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C steps it from its project's model's cheapest rung by fixed bands, and a chunk's worker never goes below that rung
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model, and a chunk's worker never runs below that rung
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude now, and by larv-review and the worker once the router's later items land. `bin/model_routing/ladder.py` holds the rungs and the bands.
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude now, and by larv-review and the worker once items 90 and 92 land. `bin/model_routing/ladder.py` holds the rungs and the bands.
The rungs, least competent first: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models, read 2026-10-14), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). That cut drops Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index falls among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scores it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price.
A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. lilleman chose medium knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes. A model off the ladder routes nothing: claude runs it at its default effort. R + C steps a task from the project's rung: 2–4 three rungs down, 5–6 two down, 7–8 one down, 9–10 none, 11–12 one up, 13–14 two up, 15–16 three up, 17 and above four up. A step past either end stops there. The bands spread the R + C that `todo.md`'s items held on 2026-10-14, 2 to 12, across the rungs, so most tasks step down. S never moves a rung, so a huge task with low R and C runs cheap. The comprehension panel and the security pass stay on the project's model. `score-outcomes` had too few finished chunks to ground the bands; `todo.md` item 94 measures the routes' outcomes, and a change to the ladder or the bands is lilleman's.
A chunk's worker never runs below the project's rung, until `todo.md` item 93 shows a cheaper worker takes no more review rounds (goal 1.1). A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. lilleman chose medium knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes. A model off the ladder routes nothing: claude runs it at the effort its settings give. R + C steps a task from the project's rung: 2–4 three rungs down, 5–6 two down, 7–8 one down, 9–10 none, 11–12 one up, 13–14 two up, 15–16 three up, 17 and above four up. A step past either end stops there. The bands spread the R + C that `todo.md`'s items held on 2026-10-14, 2 to 12, across the rungs, so most tasks would step down; a chunk's worker does not, until item 93 lands. S never moves a rung, so a huge task with low R and C runs cheap. The comprehension panel and the security pass stay on the project's model. `score-outcomes` had too few finished chunks to ground the bands; `todo.md` item 94 measures the routes' outcomes, and a change to the ladder or the bands is lilleman's.
A chunk's worker starts on the rung its item's R + C picks, never below the project's rung, until `todo.md` item 93 shows a cheaper worker costs no review rounds (goal 1.1). The project's rung itself sits below xhigh, by lilleman's choice above. It steps up only as the chunk starts, after every second unfinished verdict, or when the judge reports an item taken into the chunk whose R + C picks a higher rung. A model switch inside a session discards its prompt cache. loop-claude routes on the first row of the next release in `todo.md`'s table that waits on no question. It reads neither an order the plan states in prose nor the task, so a chunk may start on another item's rung: a higher one costs money, and a lower one holds until the judge reports the item the worker took in, after the first cycle. `todo.md` item 136 ends this, routing on the item the worker says it takes.
The worker changes rung only as the chunk starts, after every second unfinished verdict, or when the judge reports a taken-in item whose R + C picks a higher rung, because a model switch inside a session discards its prompt cache. loop-claude routes on the first row of the next release in `todo.md`'s table that waits on no question. It reads neither an order the plan states in prose nor the task, so a chunk may start on another item's rung: a higher one costs money, and a lower one holds until the judge reports the item the worker took in, after the first cycle. `todo.md` item 136 ends this, routing on the item the worker says it takes.
Valid while the index ranks these models, and while no newer model joins the catalog.
+1 -1
View File
@@ -110,7 +110,7 @@ loop-claude starts the judge itself (`bin/loop_claude/cycle/judge.py`), so it ro
### 136. Route a chunk on the item its worker says it takes, in place of the next release's first row.
loop-claude routes on the first row of the next release, while the worker takes the item the plan's prose orders first, or a prerequisite, as this chunk took 88 over 93. A first row of higher R + C pays for a dearer rung all chunk; one of lower R + C holds until the judge reports the item taken in. The worker names its item in its first reply, so the judge can report that item's R + C after the first cycle; routing the first cycle right needs the item before the worker starts.
loop-claude routes on the first row of the next release, while the worker takes the item the plan's prose orders first, or a prerequisite, as the chunk that shipped 88 took it over 93. A first row of higher R + C pays for a dearer rung all chunk; one of lower R + C holds until the judge reports the item taken in. The judge reports the item the worker takes only after the first cycle, so routing the first cycle right needs the item before the worker starts.
### 105. Compare review rounds per pass and reading cost for the five chunks before and after the `worker:` decision moved the principles out of the worker's context, and file an item if either rose.