Answer prose round 1: 0 🔴, 6 🟡 (0 misread); the comments, the presentation and the routing decision state the new steps
CI / tests (pull_request) Failing after 3m51s

This commit is contained in:
2026-11-17 08:41:04 +01:00
parent 7b9e43e2bd
commit bfde3ffecb
8 changed files with 8 additions and 8 deletions
+1 -1
View File
@@ -43,7 +43,7 @@ class Cycle:
# The rung routing.judge_rung picks for the project's judge model.
judge: Rung
premises: str
# The project's rung on the ladder model_routing/ladder_file.py read as the cycle started; None for a model off the ladder.
# The project's rung on the ladder that model_routing/ladder_file.py read as the cycle started; None for a model off the ladder.
project: ProjectRung | None
project_archive: pathlib.Path
records: str
+1 -1
View File
@@ -9,7 +9,7 @@ The worker, and the session /talk opens, run on a Route (model_routing/route.py)
A step-up weighs five rungs:
- the project's rung, a ProjectRung (model_routing/ladder.py) that also holds the ladder: the cheapest rung of the project's model, which each L's rung is counted from, up or down;
- the project's rung: the cheapest rung of the project's model, which each L's rung is counted from, up or down;
- item_rung: the rung the chunk started on, which the routed item's L picked;
- the route's rung: where the worker runs now, at or above item_rung;
- floor_rung: the lowest rung the worker may run on next, which is the route's rung, raised where an item the worker took in has an L that picks a higher one;
+1 -1
View File
@@ -150,7 +150,7 @@ if os.path.basename(sys.argv[0]) == "loop-claude":
"""
PREMISES = "# Project\n\n## Goals\n\n1. Work.\n\n## Audience\n\nMe.\n"
# Item 4's L of 4 steps one rung up from the project's.
# Item 4's L of 4 steps two rungs up from the project's.
TODO = "## Items\n\n| ID | Release | Exempt | Item | L | E | I | R | S | Moves | Score |\n|---|---|---|---|---|---|---|---|---|---|---|\n| 4 | 0.1.0 | | **Do it.** | 4 | 2 | 3 | 7 | 2 | 1 by 9 | 25 |\n"
# Item 4's L of 2 steps one rung down.
CHEAP = TODO.replace("| 4 | 2 | 3 | 7 |", "| 2 | 1 | 2 | 3 |")
+1 -1
View File
@@ -60,7 +60,7 @@ class Ladder:
return Rung(family.model, effort) if effort else family
def project_rung(self, model: str) -> "ProjectRung | None":
"""The model's cheapest rung, which for Opus 5.5 is its default effort, on this ladder; None for a model off the ladder."""
"""The model's cheapest rung on this ladder, which for Opus 5.5 is its default effort; None for a model off the ladder."""
rung = next((rung for rung in self.rungs if rung.model == model), None)
return ProjectRung(self, rung) if rung else None
+1 -1
View File
@@ -23,7 +23,7 @@ def hands_rungs(environ: typing.Mapping[str, str]) -> bool:
def handed(environ: typing.Mapping[str, str], ladder: Ladder) -> tuple[ProjectRung | None, Rung | None, int]:
"""The project's rung, on ladder, and the worker's rung, each None when unset or not on ladder, and the reviews' E + I, 0 when unset or not a whole number. ladder is the project's. An older loop-claude hands no E + I, and 0 steps no review up."""
"""The project's rung and the worker's rung, each None when unset or not on ladder, and the reviews' E + I, 0 when unset or not a whole number. ladder is the project's. An older loop-claude hands no E + I, and 0 steps no review up."""
escape_and_impact = environ.get(ESCAPE_AND_IMPACT, "")
project = on_the_ladder(environ.get(PROJECT_RUNG), ladder)
return ProjectRung(ladder, project) if project else None, on_the_ladder(environ.get(WORKER_RUNG), ladder), int(escape_and_impact) if escape_and_impact.isdecimal() else 0
+1 -1
View File
@@ -36,7 +36,7 @@ class SubagentRung(unittest.TestCase):
def test_picks_the_rung_on_the_ladder_of_the_project_it_runs_in(self):
checkout = self.project('[{"model": "claude-fable-5-2", "effort": "low"}, {"model": "claude-fable-5-2", "effort": "max"}]')
(checkout / "nested").mkdir()
# On the shipped ladder an L of 1 steps Fable 5.2 down to Opus 5.5, and an L of 4 up to its medium effort.
# On the shipped ladder an L of 1 steps Fable 5.2 down to Opus 5.5, and an L of 4 up to Opus 5.5 at xhigh.
self.assertEqual([self.run_tool("claude-fable-5-2", level, cwd=checkout / "nested").stdout for level in ("1", "4")], ["model fable, effort low\n", "model fable, effort max\n"])
off = self.run_tool("claude-opus-5-5", "3", cwd=checkout)
own = self.home / ".claude-projects" / "acme" / "routing-ladder.json"
+1 -1
View File
@@ -644,7 +644,7 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
2026-11-11, lilleman approved the rungs for Fable 5.2 from the index read that day. Its score and cost per task, for each effort from low to max: Sonnet 5.5 26 $0.51, 31 $0.78, 42 $1.64, 50 $4.09, 55 $10.02; Opus 5.5 35 $0.98, 48 $2.45, 52 $3.46, 57 $6.05, 59 $10.04; Fable 5.2 54 $3.10, 56 $3.80, 62 $9.08, 62 $14.83, 65 $26.54. The cut drops Opus 5.5 at high, since Fable 5.2 at low scores more for less; Opus 5.5 at max, since Fable 5.2 at high scores more for less; and Fable 5.2 at xhigh, which ties Fable 5.2 at high at a higher cost. Sonnet 5.5 at high stays the bottom rung, as the 2026-10-29 note chose. With Haiku 5.5 off, the cut alone would also keep Sonnet 5.5 at low and medium and Opus 5.5 at low, below it. On an Opus 5.5 project, an L 4 worker, an E + I 5 review and a worker's first escalation now run on Fable 5.2 at low, where they ran on Opus 5.5 at high; E + I 8 runs on Fable 5.2 at high, where it ran on Fable 5.1 at max. Three steps land on a rung the index scores lower than before: E + I 6 on Fable 5.2 at medium, 56 against Opus 5.5 at xhigh's 57; E + I 7 on Opus 5.5 at xhigh, 57 against Opus 5.5 at max's 59; and an L 3 worker's escalation stops, without lilleman's yes, at Fable 5.2 at medium, where it stopped at Opus 5.5 at xhigh. On a Sonnet 5.5 project, E + I 6 runs on Fable 5.2 at low, 54 against Opus 5.5 at high's 52; E + I 7 on Fable 5.2 at medium, 56 against Opus 5.5 at xhigh's 57; and E + I 8 on Opus 5.5 at xhigh, 57 against Opus 5.5 at max's 59. Fable 5.2 at max, on top, is reached only one rung up from Fable 5.2 at high, as by a reviewer's rerun. A Fable 5.2 project's rung is Fable 5.2 at low, so its L 1 and L 2 workers run on Opus 5.5 at medium.
2026-11-14, lilleman ruled that L 3 is to step one rung up and L 4 two up, in every project and for delegated subagents. The steps took effect in 0.12.0, as lilleman chose on 2026-11-16. On an Opus 5.5 project, an L 3 worker or subagent runs on Fable 5.2 at low, and an escalation stops without lilleman's yes at Opus 5.5 at xhigh. An L 4 one runs on Fable 5.2 at medium, and stops at Fable 5.2 at high. The ruling settles the two calls of the routing figures measured at 2026-11-14T00:59:50Z, which `routing-calls-raised.md` stamps 2026-11-14T13:57:34Z. The figures came from 276 chunks finished since the 2026-10-29 reset. Chunks started on Opus 5.5 at medium, 79 of them, took a mean 0.8 review rounds past each pass's first, 0.3 unfinished verdicts and 0.2 later fixes. Chunks started on Fable 5.2 at low, 5 of them, took 0.4, 0.0 and 0.0. On an Opus 5.5 project only L 4 items start on Fable 5.2 at low, so it did better on harder items. Two things weaken the evidence: 5 chunks is the fewest a call allows, and the Opus 5.5 figures include chunks from before Fable 5.2 joined the ladder. On the 2026-11-11 index, an L 3 task's cost rises about 1.3 times, from $2.45 to $3.10, for fewer rounds and fewer unfinished verdicts (goal 1.4). The chunks started on Sonnet 5.5 at high, 164 of the 276, don't move.
2026-11-14, lilleman ruled that L 3 is to step one rung up and L 4 two up, in every project and for delegated subagents. The steps take effect in 0.12.0, as lilleman chose on 2026-11-16. On an Opus 5.5 project, an L 3 worker or subagent runs on Fable 5.2 at low, and an escalation stops without lilleman's yes at Opus 5.5 at xhigh. An L 4 one runs on Fable 5.2 at medium, and stops at Fable 5.2 at high. The ruling settles the two calls that `routing-calls-raised.md` stamps 2026-11-14T13:57:34Z, raised on the routing figures measured at 2026-11-14T00:59:50Z. The figures came from 276 chunks finished since the 2026-10-29 reset. Chunks started on Opus 5.5 at medium, 79 of them, took a mean 0.8 review rounds past each pass's first, 0.3 unfinished verdicts and 0.2 later fixes. Chunks started on Fable 5.2 at low, 5 of them, took 0.4, 0.0 and 0.0. Under the old steps, only L 4 items started on Fable 5.2 at low on an Opus 5.5 project, so it did better on harder items. Two things weaken the evidence: 5 chunks is the fewest a call allows, and the Opus 5.5 figures include chunks from before Fable 5.2 joined the ladder. On the 2026-11-11 index, an L 3 task's cost rises about 1.3 times, from $2.45 to $3.10, for fewer rounds and fewer unfinished verdicts (goal 1.4). The chunks started on Sonnet 5.5 at high, 164 of the 276, don't move.
2026-11-15, lilleman asked for the ladder as a file that a project can replace, on a day when an outage left only Fable 5.2 answering: the rungs were a constant in `bin/model_routing/ladder.py`, so no project could leave Sonnet 5.5 and Opus 5.5 without a release. The shipped ladder is `bin/model_routing/routing-ladder.json`, a JSON list of rungs, least competent first. A project that keeps a `routing-ladder.json` in its config's folder, `~/.claude-projects/<key>/`, runs on that ladder whole, and the steps count on it as on the shipped one. Every project still runs the same routing rules and steps, so the workflow of goal 1.3 holds; the ladder joins what a project's config folder may supply, which that goal's list does not name yet. lilleman chose a file per project over one per machine, so a single project can differ. A ladder holds one model of each family, since the Agent tool launches a subagent or a review by its family's name. The project's file is lilleman's to write, as any change to the ladder is, and the loop never commits it. loop-claude reads the project's ladder as each cycle starts; `subagent-rung` and rule-hook's `reviewer-rung` read it each time they run, from the checkout they run in. A ladder file that holds a fault stops each of them, since work routed on the shipped ladder in its place could run on a model the project cannot reach. The routing figures, `judge-replay` and `review-replay` span every project, so they read the shipped ladder: a rung only a project's own ladder holds is left out of the figures' calls, and no replay qualifies it. lilleman shipped the change as 0.10.1, a patch cut from the tag v0.10.0, on the day of the outage: it adds a file format and an exit 1 to `subagent-rung`, which the technical principles' semantic versioning gives a minor, and lilleman asked for the patch "even if that breaks semver", so that the release in use could run again without taking the unreleased work on `main`.
+1 -1
View File
@@ -815,7 +815,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
<div>
<h3>How a rung is picked</h3>
<ul class="tight">
<li><b>Worker:</b> L 1–2 one rung down, L 3 the project's rung, L 4 one up.</li>
<li><b>Worker:</b> L 1–2 one rung down, L 3 one up, L 4 two up.</li>
<li><b>Reviewers:</b> never below the project's rung or the worker's. E + I of 5 or more steps them up, one rung per point.</li>
<li><b>Readers and security:</b> always the project's rung. <b>The judge:</b> never below it.</li>
</ul>