Answer stability round 1: 0 🔴, 7 🟡 (0 misread); placeholder rungs in docstrings, rung-free test names, the decision note and changelog name every move

This commit is contained in:
2026-11-12 00:20:18 +01:00
parent 071d6cf1cd
commit 6e67cc59fa
10 changed files with 64 additions and 65 deletions
+1 -1
View File
@@ -2,7 +2,7 @@
## Unreleased
- The routing ladder now runs on Fable 5.2, ordered by the Artificial Analysis Intelligence Index read 2026-11-11: Sonnet 5.5 at high, Opus 5.5 at medium, Fable 5.2 at low and medium, Opus 5.5 at xhigh, Fable 5.2 at high and max. On an Opus 5.5 project, L 4 workers and E + I 5 reviews now run on Fable 5.2 at low, where they ran on Opus 5.5 at high, and E + I 8 reviews on Fable 5.2 at high, where they ran on Fable 5.1 at max. A Fable 5.2 project starts on Fable 5.2 at low.
- The routing ladder now tops out at Fable 5.2, ordered by the Artificial Analysis Intelligence Index read 2026-11-11: Sonnet 5.5 at high, Opus 5.5 at medium, Fable 5.2 at low and medium, Opus 5.5 at xhigh, Fable 5.2 at high and max. On an Opus 5.5 project, L 4 workers, E + I 5 reviews and a worker's first escalation now run on Fable 5.2 at low, where they ran on Opus 5.5 at high; E + I 6 reviews on Fable 5.2 at medium, where they ran on Opus 5.5 at xhigh; E + I 7 on Opus 5.5 at xhigh, where they ran on Opus 5.5 at max; and E + I 8 on Fable 5.2 at high, where they ran on Fable 5.1 at max. On a Sonnet 5.5 project, E + I 6, 7 and 8 reviews now run on Fable 5.2 at low, Fable 5.2 at medium and Opus 5.5 at xhigh, where they ran on Opus 5.5 at high, xhigh and max. The judge climbs with the reviews. A Fable 5.2 project starts on Fable 5.2 at low. Update `~/.claude` while no loop is mid-chunk: a chunk that stepped up to Opus 5.5 at high or max, or Fable 5.1 at max, finds its rung gone and restarts on its item's rung.
- loop-claude now sends you a Mattermost DM from a bot for each question it asks, each release ready to cut, each routing call and each stop but Ctrl-C, where `~/.claude/.env` names the server, the bot's token and your username. `bin/mattermost-dm` sends one by hand; `mattermost-dm --help` names the keys. A DM that fails is one line on loop-claude's terminal, and the loop goes on.
- larv-review's worker now runs the project's tests and static analysis, and checks the docs for each name its change touches, before each stability round, and hands stability-reviewer the commit that passed; the reviewer doesn't run them again on it.
- stability-reviewer, product-owner-reviewer and systems-architect have one-line descriptions, which shrinks the agent listing every session carries.
+4 -4
View File
@@ -88,7 +88,7 @@ def markdown_launch(choice: ReviewChoice) -> list[str]:
def shared_launches(choice: ReviewChoice, route: Route, project: Rung) -> list[str]:
"""A sentence for each rung, other than the worker's, that the reviewers, the seats or the security pass run on.
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project, routed by item 3 of E + I 5, they read: `Launch each reviewer with the Agent tool set to model fable, effort low, the rung item 3's E + I of 5 picks.` and `Launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.`
For a worker one rung below the project's, routed by item 3 of E + I 5, they read: `Launch each reviewer with the Agent tool set to <rung>, the rung item 3's E + I of 5 picks.` and `Launch each comprehension seat and the security pass's subagent with the Agent tool set to <rung>, the project's rung.`
"""
together = {EACH_REVIEWER: choice.reviewers, EACH_SEAT: choice.seat, THE_SECURITY_PASS: choice.security}
launched: dict[Rung, list[str]] = {}
@@ -101,7 +101,7 @@ def shared_launches(choice: ReviewChoice, route: Route, project: Rung) -> list[s
def taken_in_launches(choice: ReviewChoice) -> list[str]:
"""The sentence naming the reviewers' rung each higher E + I picks, so the worker reviews an item it takes in on that rung in the same run, since loop-claude reads a taken-in item's E + I only as the cycle ends, and routes on it from the next cycle. None where no higher E + I picks a higher rung.
For a worker on Opus 5.5 at medium, on an Opus 5.5 project, routed by an item of E + I 6, it reads: `Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8.`
For a worker on the project's rung, routed by an item of E + I 6, it reads: `Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `<rung>` from 7 and `<rung>` from 8.`
"""
if not choice.higher:
return []
@@ -122,7 +122,7 @@ def escalation(route: Route, project: Rung | None) -> list[str]:
"""The prompt's escalation sentences, the marker's among them. None off the ladder or on its top rung. Where escalation_rungs opens none, the worker escalates only with the human's yes.
Else they name the rungs escalation_rungs opens to a subagent, and offer to resume the worker one rung up for the rest of the chunk, up to the highest of those rungs.
For a worker on Fable 5.2 at low, on an Opus 5.5 project, routed by item 3 of L 4, they read: `When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort medium`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first.`, `Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.` and `Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`.`
For a worker one rung above the project's, routed by item 3 of L 4, they read: `When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `<rung>`, one rung above yours, and on each further trigger one rung higher, up to `<rung>`; above that, ask the human first.`, `Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.` and `Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `<rung>`.`
"""
if project is None or route.rung not in LADDER[:-1]:
return []
@@ -144,7 +144,7 @@ def escalation(route: Route, project: Rung | None) -> list[str]:
def reruns(route: Route, project: Rung | None) -> list[str]:
"""The sentence naming the rung one above a reviewer's and the one above a seat's, on which each reruns when it asks, leaving out either that is already on the top rung. None off the ladder.
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project, routed by an item of E + I 4, it reads: `A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort low`.`
For a worker one rung below the project's, routed by an item of E + I 4, it reads: `A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `<rung>`.`
"""
if project is None or route.rung not in LADDER:
return []
+42 -42
View File
@@ -503,7 +503,7 @@ class Cycles(unittest.TestCase):
for ruling in (Interrupt("e", "q", entries_read=3), QuotedAnswer("e", "q", "A.", entries_read=3)):
with self.subTest(ruling=ruling):
self.assertIsNone(run.ruling_step_up(cycle, ruling))
self.assertEqual(run.ruling_step_up(cycle, Interrupt("e", "q", "round 2: 🔴", TakenIn(highest_likelihood=4), entries_read=3)), routing.StepUp(cycle.route, Route(Routing.XHIGH, 3, 3, 4, escalated_at=3)))
self.assertEqual(run.ruling_step_up(cycle, Interrupt("e", "q", "round 2: 🔴", TakenIn(highest_likelihood=4), entries_read=3)), routing.StepUp(cycle.route, Route(Routing.TWO_UP, 3, 3, 4, escalated_at=3)))
def test_an_escalation_after_an_interrupt_stepped_the_worker_steps_it_again_in_the_same_cycle(self):
streams = [Worker(self.paths.claude_code_transcripts.parent, text="round 2: 🔴"), Worker(self.paths.claude_code_transcripts.parent), Worker(self.paths.claude_code_transcripts.parent)]
@@ -512,7 +512,7 @@ class Cycles(unittest.TestCase):
cycle = dataclasses.replace(self.cycle(fake, records="From AGENTS.md:\nUse A."), project=Routing.OPUS, route=Route(Routing.SONNET, 179, 2, 5))
with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
concluded = run.run_cycle(cycle)
self.assertEqual(concluded.route, Route(Routing.HIGH, 179, 2, 5, escalated_at=4))
self.assertEqual(concluded.route, Route(Routing.ONE_UP, 179, 2, 5, escalated_at=4))
self.assertEqual([args[args.index("--effort") + 1] for args in fake.made("stream_claude")], ["high", "medium", "low"])
def test_an_escalation_quoted_from_before_the_workers_last_step_up_steps_nothing_and_a_later_one_reaches_the_verdict(self):
@@ -1039,9 +1039,9 @@ def interrupted(likelihood=0, escalated_at=None):
class Routing(unittest.TestCase):
OPUS = Rung("claude-opus-5-5", "medium")
HIGH = Rung("claude-fable-5-2", "low")
XHIGH = Rung("claude-fable-5-2", "medium")
MAX = Rung("claude-opus-5-5", "xhigh")
ONE_UP = Rung("claude-fable-5-2", "low")
TWO_UP = Rung("claude-fable-5-2", "medium")
THREE_UP = Rung("claude-opus-5-5", "xhigh")
SONNET = Rung("claude-sonnet-5-5", "high")
def test_routes_on_the_next_releases_top_item_that_waits_on_no_question(self):
@@ -1069,7 +1069,7 @@ class Routing(unittest.TestCase):
self.assertEqual(routing.routed_item(todo.replace(first, "") + first, NO_CUT).id, 3)
def test_starts_a_chunk_on_the_rung_its_items_l_picks_below_the_projects_too(self):
self.assertEqual(routing.first_route(self.OPUS, routing.routed_item(TODO, NO_CUT)), Route(self.HIGH, 3, 4, 5))
self.assertEqual(routing.first_route(self.OPUS, routing.routed_item(TODO, NO_CUT)), Route(self.ONE_UP, 3, 4, 5))
cheap = routing.routed_item(TODO.replace("| 4 | 2 | 3 |", "| 1 | 2 | 3 |"), NO_CUT)
self.assertEqual(routing.first_route(self.OPUS, cheap), Route(Rung("claude-sonnet-5-5", "high"), 3, 1, 5))
self.assertEqual(routing.first_route(self.OPUS, None), Route(self.OPUS))
@@ -1077,13 +1077,13 @@ class Routing(unittest.TestCase):
def test_steps_up_a_rung_after_every_second_unfinished_verdict(self):
once = routing.next_route(self.OPUS, Route(self.OPUS, 3, 3, 4), unfinished())
self.assertEqual(once, Route(self.OPUS, 3, 3, 4, unfinished=1))
self.assertEqual(routing.next_route(self.OPUS, once, unfinished()), Route(self.HIGH, 3, 3, 4))
self.assertEqual(routing.next_route(self.OPUS, once, unfinished()), Route(self.ONE_UP, 3, 3, 4))
self.assertEqual(routing.next_route(self.OPUS, Route(LADDER[-1], unfinished=1), unfinished()).rung, LADDER[-1])
def test_steps_up_to_the_rung_a_taken_in_items_l_picks_and_never_down(self):
self.assertEqual(routing.next_route(self.OPUS, Route(self.OPUS), unfinished(4)).rung, self.HIGH)
self.assertEqual(routing.next_route(self.OPUS, Route(self.XHIGH), unfinished(4)).rung, self.XHIGH)
self.assertEqual(routing.next_route(self.OPUS, Route(self.OPUS, unfinished=1), unfinished(4)).rung, self.XHIGH)
self.assertEqual(routing.next_route(self.OPUS, Route(self.OPUS), unfinished(4)).rung, self.ONE_UP)
self.assertEqual(routing.next_route(self.OPUS, Route(self.TWO_UP), unfinished(4)).rung, self.TWO_UP)
self.assertEqual(routing.next_route(self.OPUS, Route(self.OPUS, unfinished=1), unfinished(4)).rung, self.TWO_UP)
def test_steps_up_from_a_rung_below_the_projects_one_rung_at_a_time(self):
low = Route(self.SONNET, 3, 1, 3, unfinished=1)
@@ -1092,30 +1092,30 @@ class Routing(unittest.TestCase):
self.assertEqual(routing.next_route(self.OPUS, Route(low.rung, 3, 1, 3), unfinished(3)).rung, self.OPUS)
def test_steps_up_a_rung_on_an_escalation_the_judge_found_up_to_two_above_the_items_rung_marking_the_transcript_entries_it_read(self):
self.assertEqual(routing.next_route(self.OPUS, Route(self.HIGH, 3, 4, 5), unfinished(escalated_at=7)), Route(self.XHIGH, 3, 4, 5, escalated_at=7))
self.assertEqual(routing.next_route(self.OPUS, Route(self.XHIGH, 3, 4, 5, escalated_at=7), unfinished(escalated_at=9)), Route(self.MAX, 3, 4, 5, escalated_at=9))
self.assertEqual(routing.next_route(self.OPUS, Route(self.MAX, 3, 4, 5, escalated_at=9), unfinished(escalated_at=12)), Route(self.MAX, 3, 4, 5, unfinished=1, escalated_at=9))
self.assertEqual(routing.next_route(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1), unfinished(escalated_at=7)), Route(self.XHIGH, 3, 4, 5, escalated_at=7))
self.assertEqual(routing.next_route(self.OPUS, Route(self.XHIGH), unfinished(escalated_at=7)), Route(self.XHIGH, unfinished=1))
self.assertEqual(routing.next_route(self.OPUS, Route(self.SONNET, 3, 2, 4), unfinished(4, escalated_at=7)), Route(self.HIGH, 3, 2, 4, unfinished=1))
self.assertEqual(routing.next_route(self.OPUS, Route(self.ONE_UP, 3, 4, 5), unfinished(escalated_at=7)), Route(self.TWO_UP, 3, 4, 5, escalated_at=7))
self.assertEqual(routing.next_route(self.OPUS, Route(self.TWO_UP, 3, 4, 5, escalated_at=7), unfinished(escalated_at=9)), Route(self.THREE_UP, 3, 4, 5, escalated_at=9))
self.assertEqual(routing.next_route(self.OPUS, Route(self.THREE_UP, 3, 4, 5, escalated_at=9), unfinished(escalated_at=12)), Route(self.THREE_UP, 3, 4, 5, unfinished=1, escalated_at=9))
self.assertEqual(routing.next_route(self.OPUS, Route(self.ONE_UP, 3, 4, 5, unfinished=1), unfinished(escalated_at=7)), Route(self.TWO_UP, 3, 4, 5, escalated_at=7))
self.assertEqual(routing.next_route(self.OPUS, Route(self.TWO_UP), unfinished(escalated_at=7)), Route(self.TWO_UP, unfinished=1))
self.assertEqual(routing.next_route(self.OPUS, Route(self.SONNET, 3, 2, 4), unfinished(4, escalated_at=7)), Route(self.ONE_UP, 3, 2, 4, unfinished=1))
def test_keeps_the_mark_through_a_step_no_escalation_took(self):
self.assertEqual(routing.next_route(self.OPUS, Route(self.OPUS, 3, 3, 4, unfinished=1, escalated_at=5), unfinished()), Route(self.HIGH, 3, 3, 4, escalated_at=5))
self.assertEqual(routing.next_route(self.OPUS, Route(self.OPUS, 3, 3, 4, unfinished=1, escalated_at=5), unfinished()), Route(self.ONE_UP, 3, 3, 4, escalated_at=5))
def test_steps_up_a_rung_as_the_judge_interrupts_up_to_two_above_the_items_rung_without_counting_an_unfinished_verdict(self):
self.assertEqual(routing.next_route(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1), interrupted(escalated_at=7)), Route(self.XHIGH, 3, 4, 5, escalated_at=7))
self.assertEqual(routing.next_route(self.OPUS, Route(self.XHIGH, 3, 4, 5, escalated_at=7), interrupted(escalated_at=9)), Route(self.MAX, 3, 4, 5, escalated_at=9))
self.assertEqual(routing.next_route(self.OPUS, Route(self.MAX, 3, 4, 5, unfinished=1, escalated_at=9), interrupted(escalated_at=12)), Route(self.MAX, 3, 4, 5, unfinished=1, escalated_at=9))
self.assertEqual(routing.next_route(self.OPUS, Route(self.ONE_UP, 3, 4, 5, unfinished=1), interrupted(escalated_at=7)), Route(self.TWO_UP, 3, 4, 5, escalated_at=7))
self.assertEqual(routing.next_route(self.OPUS, Route(self.TWO_UP, 3, 4, 5, escalated_at=7), interrupted(escalated_at=9)), Route(self.THREE_UP, 3, 4, 5, escalated_at=9))
self.assertEqual(routing.next_route(self.OPUS, Route(self.THREE_UP, 3, 4, 5, unfinished=1, escalated_at=9), interrupted(escalated_at=12)), Route(self.THREE_UP, 3, 4, 5, unfinished=1, escalated_at=9))
self.assertEqual(routing.next_route(self.OPUS, Route(self.SONNET, 3, 2, 4), interrupted(escalated_at=7)), Route(self.OPUS, 3, 2, 4, escalated_at=7))
self.assertEqual(routing.next_route(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1, escalated_at=2), interrupted(4)), Route(self.HIGH, 3, 4, 5, unfinished=1, escalated_at=2))
self.assertEqual(routing.next_route(self.OPUS, Route(self.ONE_UP, 3, 4, 5, unfinished=1, escalated_at=2), interrupted(4)), Route(self.ONE_UP, 3, 4, 5, unfinished=1, escalated_at=2))
def test_raises_the_worker_to_a_taken_in_items_rung_above_the_escalation_cap_without_a_step_as_the_judge_interrupts(self):
self.assertEqual(routing.next_route(self.OPUS, Route(self.OPUS, 3, 1, 3, escalated_at=2), interrupted(4, escalated_at=7)), Route(self.HIGH, 3, 1, 3, escalated_at=2))
self.assertEqual(routing.next_route(self.OPUS, Route(self.OPUS, 3, 1, 3, escalated_at=2), interrupted(4, escalated_at=7)), Route(self.ONE_UP, 3, 1, 3, escalated_at=2))
def test_steps_up_from_the_rung_a_taken_in_item_picks_as_the_judge_interrupts(self):
self.assertEqual(routing.next_route(self.OPUS, Route(self.OPUS, 3, 3, 4), interrupted(4, escalated_at=7)), Route(self.XHIGH, 3, 3, 4, escalated_at=7))
self.assertEqual(routing.next_route(self.OPUS, Route(self.XHIGH, 3, 4, 5), interrupted(4, escalated_at=7)), Route(self.MAX, 3, 4, 5, escalated_at=7))
self.assertEqual(routing.next_route(self.OPUS, Route(self.MAX, 3, 4, 5, unfinished=1), interrupted(4, escalated_at=7)), Route(self.MAX, 3, 4, 5, unfinished=1))
self.assertEqual(routing.next_route(self.OPUS, Route(self.OPUS, 3, 3, 4), interrupted(4, escalated_at=7)), Route(self.TWO_UP, 3, 3, 4, escalated_at=7))
self.assertEqual(routing.next_route(self.OPUS, Route(self.TWO_UP, 3, 4, 5), interrupted(4, escalated_at=7)), Route(self.THREE_UP, 3, 4, 5, escalated_at=7))
self.assertEqual(routing.next_route(self.OPUS, Route(self.THREE_UP, 3, 4, 5, unfinished=1), interrupted(4, escalated_at=7)), Route(self.THREE_UP, 3, 4, 5, unfinished=1))
def test_keeps_an_e_plus_i_taken_in_only_where_it_picks_a_higher_review_rung(self):
taken = Route(self.OPUS, 3, 3, 4).taking_in(6)
@@ -1134,50 +1134,50 @@ class Routing(unittest.TestCase):
self.assertEqual(routing.next_route(self.OPUS, Route(Rung("claude-opus-5-5", "low")), unfinished(2)).rung, self.OPUS)
def test_runs_the_judge_on_the_rung_its_chunks_items_e_plus_i_picks_never_below_the_projects(self):
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.OPUS, 3, 3, 5)), self.HIGH)
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.XHIGH, 3, 4, 5)), self.HIGH)
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.SONNET, 3, 2, 7)), self.MAX)
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.OPUS, 3, 3, 5)), self.ONE_UP)
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.TWO_UP, 3, 4, 5)), self.ONE_UP)
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.SONNET, 3, 2, 7)), self.THREE_UP)
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(Rung("claude-haiku-5-5", "max"), 3, 1, 3)), self.OPUS)
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.HIGH)), self.OPUS)
self.assertEqual(routing.judge_rung("judge-model", Route(self.HIGH, 3, 4, 5)), Rung("judge-model"))
self.assertEqual(routing.judge_rung("claude-sonnet-5-5", Route(self.HIGH, 3, 4, 5)), Rung("claude-sonnet-5-5"))
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.ONE_UP)), self.OPUS)
self.assertEqual(routing.judge_rung("judge-model", Route(self.ONE_UP, 3, 4, 5)), Rung("judge-model"))
self.assertEqual(routing.judge_rung("claude-sonnet-5-5", Route(self.ONE_UP, 3, 4, 5)), Rung("claude-sonnet-5-5"))
def test_runs_the_judge_on_the_rung_the_highest_e_plus_i_taken_in_picks(self):
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.OPUS, 3, 3, 4, escape_and_impact_taken_in=6)), self.XHIGH)
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.OPUS, escape_and_impact_taken_in=5)), self.HIGH)
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.OPUS, 3, 3, 4, escape_and_impact_taken_in=6)), self.TWO_UP)
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.OPUS, escape_and_impact_taken_in=5)), self.ONE_UP)
def test_refuses_a_route_whose_item_has_levels_off_their_scales_or_levels_without_an_item(self):
for item, likelihood, escape_and_impact in ((3, 0, 5), (3, 5, 5), (3, 4, 1), (3, 4, 9), (0, 4, 5), (0, 4, 0), (0, 0, 5)):
with self.subTest(item=item, likelihood=likelihood, escape_and_impact=escape_and_impact):
with self.assertRaises(ValueError):
Route(self.HIGH, item, likelihood, escape_and_impact)
Route(self.ONE_UP, item, likelihood, escape_and_impact)
def test_refuses_a_route_taking_in_an_e_plus_i_off_its_scale_or_not_above_its_items(self):
for taken_in in (1, 9, -1, 5, 4):
with self.subTest(taken_in=taken_in):
with self.assertRaises(ValueError):
Route(self.HIGH, 3, 4, 5, escape_and_impact_taken_in=taken_in)
Route(self.ONE_UP, 3, 4, 5, escape_and_impact_taken_in=taken_in)
def test_takes_a_route_whose_item_holds_each_levels_lowest_value(self):
self.assertEqual(Route(self.HIGH, 3, 1, 2).escape_and_impact, 2)
self.assertEqual(Route(self.ONE_UP, 3, 1, 2).escape_and_impact, 2)
def test_routes_nothing_on_an_item_without_e_or_i(self):
without_i = TODO.replace("| E | I | R |", "| E | R |").replace("|---|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|---|").replace("| 4 | 3 | 5 | 10 |", "| 4 | 3 | 10 |").replace("| 4 | 2 | 3 | 7 |", "| 4 | 2 | 7 |").replace("| 1 | 1 | 1 | 1 |", "| 1 | 1 | 1 |")
self.assertIsNone(routing.routed_item(without_i, NO_CUT))
def test_archives_the_route_as_a_json_worker_line_it_reads_back_field_by_field(self):
route = Route(self.HIGH, 3, 4, 5)
route = Route(self.ONE_UP, 3, 4, 5)
self.assertEqual(route.worker_line(), '{"model": "claude-fable-5-2", "effort": "low", "item": 3, "likelihood": 4, "escape_and_impact": 5}')
self.assertEqual(route_from(route.worker_line()), route)
self.assertEqual(Route(Rung("worker-model")).worker_line(), '{"model": "worker-model"}')
self.assertEqual(route_from('{"model": "worker-model"}'), Route(Rung("worker-model")))
marked = Route(self.HIGH, 3, 4, 5, escape_and_impact_taken_in=7, escalated_at=12)
marked = Route(self.ONE_UP, 3, 4, 5, escape_and_impact_taken_in=7, escalated_at=12)
self.assertEqual(marked.worker_line(), '{"model": "claude-fable-5-2", "effort": "low", "item": 3, "likelihood": 4, "escape_and_impact": 5, "escape_and_impact_taken_in": 7, "escalated_at": 12}')
self.assertEqual(route_from(marked.worker_line()), marked)
self.assertEqual(route_from(Route(self.HIGH, escalated_at=3).worker_line()), Route(self.HIGH, escalated_at=3))
self.assertEqual(route_from(Route(self.ONE_UP, escalated_at=3).worker_line()), Route(self.ONE_UP, escalated_at=3))
def test_leaves_the_count_of_unfinished_verdicts_out_of_the_worker_line_so_a_rerun_restarts_it(self):
self.assertEqual(route_from(Route(self.HIGH, 3, 4, 5, unfinished=1).worker_line()), Route(self.HIGH, 3, 4, 5))
self.assertEqual(route_from(Route(self.ONE_UP, 3, 4, 5, unfinished=1).worker_line()), Route(self.ONE_UP, 3, 4, 5))
def test_reads_no_route_from_a_worker_line_that_route_worker_line_could_not_have_written(self):
for line in ('{"effort": "high"}', '{"model": "claude-opus-5-5", "effort": "high", "unfinished": 1}', '{"model": "claude-opus-5-5", "item": "3"}', '{"model": "claude-opus-5-5", "effort": 3}', '{"model": 5}', '{"model": "claude-opus-5-5", "effort": "high", "escalated_at": -3}', '{"model": "claude-opus-5-5", "item": true, "likelihood": 4, "escape_and_impact": 5}', '{"model": "claude-opus-5-5", "effort": "high", "item": 3, "likelihood": 5, "escape_and_impact": 5}', '{"model": "claude-opus-5-5", "effort": "high", "escape_and_impact_taken_in": 9}', '["claude-opus-5-5"]', '{"model": ', "worker-model", "claude-fable-5-2 at low effort, routed by item 3's L of 4 and E + I of 5"):
@@ -1186,13 +1186,13 @@ class Routing(unittest.TestCase):
def test_keeps_a_chunks_route_or_its_archived_one_at_or_above_the_rung_its_item_picks_and_else_routes_afresh(self):
item = routing.routed_item(TODO, NO_CUT)
archived = Route(self.XHIGH, 3, 4, 5)
archived = Route(self.TWO_UP, 3, 4, 5)
def never():
raise AssertionError("not asked")
self.assertEqual(chunk_start.routed(archived, None, "worker-model", never, never), Route(Rung("worker-model")))
self.assertEqual(chunk_start.routed(archived, self.OPUS, "claude-opus-5-5", never, never), archived)
self.assertEqual(chunk_start.routed(None, self.OPUS, "claude-opus-5-5", lambda: archived, never), archived)
self.assertEqual(chunk_start.routed(None, self.OPUS, "claude-opus-5-5", lambda: None, lambda: item), Route(self.HIGH, 3, 4, 5))
self.assertEqual(chunk_start.routed(None, self.OPUS, "claude-opus-5-5", lambda: None, lambda: item), Route(self.ONE_UP, 3, 4, 5))
self.assertEqual(chunk_start.routed(Route(self.SONNET), self.OPUS, "claude-opus-5-5", never, lambda: None), Route(self.OPUS))
cheap = Route(self.SONNET, 4, 2, 3)
self.assertEqual(chunk_start.routed(cheap, self.OPUS, "claude-opus-5-5", never, never), cheap)
+2 -10
View File
@@ -13,16 +13,8 @@ FABLE_LOW = Rung("claude-fable-5-2", "low")
class Ladder(unittest.TestCase):
def test_holds_the_rungs_lilleman_approved_from_the_index_least_competent_first(self):
self.assertEqual(LADDER, (
Rung("claude-sonnet-5-5", "high"),
OPUS,
FABLE_LOW,
Rung("claude-fable-5-2", "medium"),
Rung("claude-opus-5-5", "xhigh"),
Rung("claude-fable-5-2", "high"),
Rung("claude-fable-5-2", "max"),
))
def test_holds_each_rung_once_since_a_rung_s_position_sets_every_step(self):
self.assertEqual(len(set(LADDER)), len(LADDER))
def test_puts_a_project_on_its_models_cheapest_rung_and_none_off_the_ladder(self):
self.assertEqual(project_rung("claude-opus-5-5"), OPUS)
+1 -1
View File
@@ -20,7 +20,7 @@ class Price:
return self.input * 2
# Anthropic's list prices as the claude-api skill lists them (cached 2026-10-06; Fable 5.2's 2026-11-03), with Opus 5.5's cache prices from `docs/decisions.md` → `loop-claude: the worker's claude compacts…`. Haiku 5.5's cache read is assumed at a tenth of its input, and its dearer prompts above 200k tokens are left out. Where a chunk records its cost, only the proportions between one model's messages matter, so these may be a little off.
# Anthropic's list prices as the claude-api skill lists them (cached 2026-10-06, and Fable 5.2's on 2026-11-03), with Opus 5.5's cache prices from `docs/decisions.md` → `loop-claude: the worker's claude compacts…`. Haiku 5.5's cache read is assumed at a tenth of its input, and its dearer prompts above 200k tokens are left out. Where a chunk records its cost, only the proportions between one model's messages matter, so these may be a little off.
PRICES = {
# Archived chunks that ran on Fable 5.1 are still priced from here.
"claude-fable-5-1": Price(10, 0.25, 50),
+1 -1
View File
@@ -35,7 +35,7 @@ LOW_ROUTE = Route(LOW, 7, 3, 4).worker_line()
def message(id_, at, *parts, model="claude-opus-5-5", usage=None):
"""Costs $2 on Opus 5.5, whose output is $20 per million tokens, $1 on Sonnet 5.5, whose output is $10, and $4 on Fable 5.2, whose output is $40."""
"""Writes 100k output tokens: $2 on Opus 5.5, $1 on Sonnet 5.5 and $4 on Fable 5.2, at $20, $10 and $40 per million."""
return {"type": "assistant", "timestamp": at, "cwd": "/c", "message": {"id": id_, "model": model, "content": list(parts), "usage": usage or {"output_tokens": 100_000}}}
+1 -1
View File
@@ -216,7 +216,7 @@ class OverrideLog(unittest.TestCase):
class ReviewerRung(unittest.TestCase):
# A worker routed to Sonnet 5.5 at high, on a project whose rung is Opus 5.5 at medium, with its reviewers on the project's rung.
ENVIRON = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-sonnet-5-5 at high effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "4"}
# The same project, with a worker, and so its reviewers, routed above it to Fable 5.1 at max.
# The same project, with a worker, and so its reviewers, routed above it to Fable 5.2 at max.
ABOVE = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-fable-5-2 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "4"}
def refused(self, given: dict, environ: dict[str, str] = ENVIRON) -> list[str]:
+3 -3
View File
@@ -462,11 +462,11 @@ Only Opus 5.5 was replayed, so the judge of a project on another model keeps the
Valid while the archived verdicts show what a judge should return, and until a new model joins the ladder: replay then.
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.2 at max on top; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude and by the worker through `bin/subagent-rung`. The judge's rung is set by `routing: on a model judge-replay covered`. `bin/model_routing/ladder.py` holds the rungs and the steps.
The rungs, least competent first: Sonnet 5.5 at high; Opus 5.5 at medium; Fable 5.2 at low and medium; Opus 5.5 at xhigh; Fable 5.2 at high and max. The 2026-11-11 note below put them so. Before it they were Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max, after the 2026-10-29 note took Haiku 5.5 off. They first were: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models, read 2026-10-14), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). That cut drops Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index falls among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scored it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price. Fable 5.2 at max tops the index outright, so that exception has ended.
The rungs, least competent first: Sonnet 5.5 at high; Opus 5.5 at medium; Fable 5.2 at low and medium; Opus 5.5 at xhigh; Fable 5.2 at high and max. The 2026-11-11 note below put them so. Before it they were Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max, after the 2026-10-29 note took Haiku 5.5 off. They first were: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). The first cut, from the index read 2026-10-14, dropped Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index fell among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scored it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price. Fable 5.2 at max tops the index outright, so that exception has ended.
A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. A model off the ladder routes nothing: claude runs it at the effort its settings give. A task's L steps its worker from the project's rung: 1 and 2 one down, 3 none, 4 one up. Its E + I steps its reviewers up from the project's rung: 4 or less none, 5 one up, 6 two up, 7 three up, 8 four up. A step past either end stops there. A worker errs more the more it changes and the newer that is, which L scores; review must look harder the further a defect would escape and the more it would break, which E + I scores (goals 1.1 and 1.4). Size raises L, so a large change runs its worker higher. The steps spread the values this repo's plan held when rescored on 2026-10-16, L 1 to 4 and E + I 2 to 7, so a worker on most items runs at or below the project's rung, and a review climbs only where a defect would escape far or break much. lilleman approved the steps on 2026-10-16 as a guess. `score-outcomes` reports how well L, E and I each order finished chunks by review rounds and 🔴 findings. A change to the ladder or the steps is lilleman's.
@@ -480,7 +480,7 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
2026-11-08, lilleman kept the steps, starting the re-tune with 54 `item-*` branches merged since the steps took effect, 36 of them chunks that each shipped one item scored with L, E and I, in place of waiting for 50 such chunks. Mean review rounds past each pass's first rose with L: 1.57 at L 1, 2.15 at L 2, 2.75 at L 3 and 3.50 at L 4. E + I ordered neither rounds nor 🔴 findings between 3 and 5, and no chunk scored an E + I of 7 or more, so nothing supports changing the steps for E + I 6 to 8. L 1 and L 2 share the Sonnet 5.5 rung yet differ in 🔴 (1 of 7 chunks against 6 of 12), so a 🔴 follows the item more than the rung, and nothing moves L 2. `todo.md` item 273 re-tunes the E + I steps once 10 chunks score an E + I of 6 or more.
2026-11-11, lilleman approved the rungs for Fable 5.2, from the index read that day, as score and cost per task. Sonnet 5.5: low 26 $0.51, medium 31 $0.78, high 42 $1.64, xhigh 50 $4.09, max 55 $10.02. Opus 5.5: low 35 $0.98, medium 48 $2.45, high 52 $3.46, xhigh 57 $6.05, max 59 $10.04. Fable 5.2: low 54 $3.10, medium 56 $3.80, high 62 $9.08, xhigh 62 $14.83, max 65 $26.54. The same cut drops Opus 5.5 at high, since Fable 5.2 at low scores more for less; Opus 5.5 at max, since Fable 5.2 at high scores more for less; and Fable 5.2 at xhigh, which ties Fable 5.2 at high at a higher cost. Sonnet 5.5 at high stays the bottom rung, as the 2026-10-29 note chose, though with Haiku 5.5 off the cut alone would keep cheaper rungs below it. On an Opus 5.5 project the steps now reach: an L 4 worker, an E + I 5 review and the first escalation run on Fable 5.2 at low; E + I 6 on Fable 5.2 at medium; E + I 7 on Opus 5.5 at xhigh; E + I 8 on Fable 5.2 at high. Only a climb past that reaches Fable 5.2 at max. A Fable 5.2 project's rung is Fable 5.2 at low, so its L 1 and L 2 workers run on Opus 5.5 at medium.
2026-11-11, lilleman approved the rungs for Fable 5.2 from the index read that day. Its score and cost per task, for each effort from low to max: Sonnet 5.5 26 $0.51, 31 $0.78, 42 $1.64, 50 $4.09, 55 $10.02; Opus 5.5 35 $0.98, 48 $2.45, 52 $3.46, 57 $6.05, 59 $10.04; Fable 5.2 54 $3.10, 56 $3.80, 62 $9.08, 62 $14.83, 65 $26.54. The cut drops Opus 5.5 at high, since Fable 5.2 at low scores more for less; Opus 5.5 at max, since Fable 5.2 at high scores more for less; and Fable 5.2 at xhigh, which ties Fable 5.2 at high at a higher cost. Sonnet 5.5 at high stays the bottom rung, as the 2026-10-29 note chose. With Haiku 5.5 off, the cut alone would also keep Sonnet 5.5 at low and medium and Opus 5.5 at low, below it. On an Opus 5.5 project, an L 4 worker, an E + I 5 review and a worker's first escalation now run on Fable 5.2 at low, where they ran on Opus 5.5 at high; E + I 8 runs on Fable 5.2 at high, where it ran on Fable 5.1 at max. Three steps land on a rung the index scores lower than before: E + I 6 on Fable 5.2 at medium, 56 against Opus 5.5 at xhigh's 57; E + I 7 on Opus 5.5 at xhigh, 57 against Opus 5.5 at max's 59; and an L 3 worker's escalation stops, without lilleman's yes, at Fable 5.2 at medium, where it stopped at Opus 5.5 at xhigh. On a Sonnet 5.5 project, E + I 6 runs on Fable 5.2 at low, 54 against Opus 5.5 at high's 52; E + I 7 on Fable 5.2 at medium, 56 against Opus 5.5 at xhigh's 57; and E + I 8 on Opus 5.5 at xhigh, 57 against Opus 5.5 at max's 59. Fable 5.2 at max, on top, is reached only one rung up from Fable 5.2 at high, as by a reviewer's rerun. A Fable 5.2 project's rung is Fable 5.2 at low, so its L 1 and L 2 workers run on Opus 5.5 at medium.
Valid while the index ranks these models, and while no newer model joins the catalog.
+1 -1
View File
@@ -1006,7 +1006,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
</tbody></table></div>
<h3 class="group">Models</h3>
<div class="tablewrap"><table class="src-t"><colgroup><col style="width:4%"><col style="width:24%"><col style="width:72%"></colgroup><tbody>
<tr id="src-25"><td>25</td><td><b>The model ladder, and no Haiku</b></td><td><span class="kind r">Benchmark</span> The rungs follow the Artificial Analysis Intelligence Index: a rung stays only where no cheaper one scores as high. Fable 5.2 at max scores highest, so it is on top.<br><span class="kind m">Own measurement</span> A chunk on Haiku 5.5 ran 2 h 17 min and wrote 3.1M tokens. The chunk before it, on Sonnet 5.5, took 29 min and 111k. Chunks on Haiku took 7.8 extra review rounds, against 2.8 on Sonnet. So Haiku left the ladder.</td></tr>
<tr id="src-25"><td>25</td><td><b>The model ladder, and no Haiku</b></td><td><span class="kind r">Benchmark</span> The rungs follow the Artificial Analysis Intelligence Index: a rung stays only where no cheaper one scores as high.<br><span class="kind m">Own measurement</span> A chunk on Haiku 5.5 ran 2 h 17 min and wrote 3.1M tokens. The chunk before it, on Sonnet 5.5, took 29 min and 111k. Chunks on Haiku took 7.8 extra review rounds, against 2.8 on Sonnet. So Haiku left the ladder.</td></tr>
<tr id="src-26"><td>26</td><td><b>Judge and reviewers never run below the project's model</b></td><td><span class="kind m">Own measurement</span> 39 judge verdicts were replayed. The reference judge reproduced 31; on those, Opus 5.5 at medium missed none, and Sonnet 5.5 mislabelled 3 interrupts. A replay of review rounds: Sonnet 5.5 missed 44–55% of what the project's model found.</td></tr>
</tbody></table></div>
<h3 class="group">The machine</h3>
+8 -1
View File
@@ -6,12 +6,13 @@
`Bar = 0.1`
`Next ID = 330`
`Next ID = 331`
## Items
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 330 | 0.8.0 | | | **Replay the judge and every review pass on the Fable 5.2 ladder, and record what each replay shows.** | 2 | 2 | 2 | 4 | 3 | 1.2 by 4, 1.1 by 3 | 2.9 |
| 320 | 0.8.0 | | | **Bench a worker below the project's rung with the advisor tool against the same worker alone and the project's rung, and file the switch if it wins.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3, 1.1 by 2 | 2.0 |
| 315 | 0.8.0 | | | **Start each reviewer under `agents/` without `CLAUDE.md`, and hand it the vocabulary, the prose rules and the project's decision titles.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 6 | 2.0 |
| 312 | 0.8.0 | | | **Research the tooling agents read, search and run commands with, and file an item for each tool that cuts tokens a replay shows keep quality.** | 1 | 2 | 1 | 2 | 3 | 1.4 by 3, 1.5 by 2 | 2.0 |
@@ -72,6 +73,12 @@
## Details
### 330. Replay the judge and every review pass on the Fable 5.2 ladder, and record what each replay shows.
Item 292 put Fable 5.2 on the routing ladder. Two decisions expire when a new model joins it: `routing: on a model judge-replay covered, the judge runs …` ("replay then") and `routing: a review pass runs below the project's rung only on a rung that a replay …` ("a replay of every pass"). On an Opus 5.5 project the judge now climbs onto Fable 5.2 at low, medium and high at E + I 5, 6 and 8, rungs `judge-replay` never covered.
Run `bin/judge-replay claude-opus-5-5` on the new ladder, and `bin/review-replay` for each pass the 2026-10-15 to 2026-10-18 runs covered. Record each result in its decision, and renew or change each "Valid" line. Where a judge rung on Fable 5.2 misses an interrupt the project's rung returns, keep the judge off it until a decision says otherwise.
### 320. Bench a worker below the project's rung with the advisor tool against the same worker alone and the project's rung, and file the switch if it wins.
Anthropic reports Sonnet 4.6 with an Opus 4.6 advisor scored 2.7 points more on SWE-bench Multilingual at 11.9% less cost than Sonnet alone (the vendor's own run; https://claude.com/blog/the-advisor-strategy). The docs say `/advisor` works with `-p`, but not outright that the advisor is called during a `-p` run, so probe that first. Each advisor call hands the advisor the whole conversation, so count its cost. Bench on `workflow-bench` with archived L 1 and L 2 items: Sonnet with the advisor, Sonnet alone, Opus at medium. lilleman approved on 2026-11-11.