Rebuild the routing ladder for Fable 5.2 from the index read 2026-11-11
CI / tests (pull_request) Successful in 2m36s
CI / tests (pull_request) Successful in 2m36s
This commit is contained in:
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- The routing ladder now runs on Fable 5.2, ordered by the Artificial Analysis Intelligence Index read 2026-11-11: Sonnet 5.5 at high, Opus 5.5 at medium, Fable 5.2 at low and medium, Opus 5.5 at xhigh, Fable 5.2 at high and max. On an Opus 5.5 project, L 4 workers and E + I 5 reviews now run on Fable 5.2 at low, where they ran on Opus 5.5 at high, and E + I 8 reviews on Fable 5.2 at high, where they ran on Fable 5.1 at max. A Fable 5.2 project starts on Fable 5.2 at low.
|
||||
- loop-claude now sends you a Mattermost DM from a bot for each question it asks, each release ready to cut, each routing call and each stop but Ctrl-C, where `~/.claude/.env` names the server, the bot's token and your username. `bin/mattermost-dm` sends one by hand; `mattermost-dm --help` names the keys. A DM that fails is one line on loop-claude's terminal, and the loop goes on.
|
||||
- larv-review's worker now runs the project's tests and static analysis, and checks the docs for each name its change touches, before each stability round, and hands stability-reviewer the commit that passed; the reviewer doesn't run them again on it.
|
||||
- stability-reviewer, product-owner-reviewer and systems-architect have one-line descriptions, which shrinks the agent listing every session carries.
|
||||
|
||||
@@ -151,13 +151,13 @@ class Main(unittest.TestCase):
|
||||
self.assertEqual([(model, effort) for model, effort, _ in self.calls[::2]][-1], ("claude-sonnet-5-5", "high"))
|
||||
self.assertTrue(out.endswith("returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort\n"), out)
|
||||
|
||||
def test_a_fable_project_tries_no_rung_below_opus_at_max_and_its_report_names_fable(self):
|
||||
(self.home / ".claude" / "loop-claude-models.json").write_text(json.dumps({"proj": "claude-opus-5-5", "other": "claude-fable-5-1"}))
|
||||
code, out, err = self.run_main(["claude-fable-5-1"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
def test_a_fable_project_tries_no_rung_below_opus_at_medium_and_its_report_names_fable(self):
|
||||
(self.home / ".claude" / "loop-claude-models.json").write_text(json.dumps({"proj": "claude-opus-5-5", "other": "claude-fable-5-2"}))
|
||||
code, out, err = self.run_main(["claude-fable-5-2"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual([(model, effort) for model, effort, _ in self.calls][-1], ("claude-opus-5-5", "max"))
|
||||
self.assertTrue(out.splitlines()[0].startswith("Replayed 1 verdicts of claude-fable-5-1 projects"), out)
|
||||
self.assertTrue(out.endswith("returns every verdict the reference judge reproduces: claude-opus-5-5 at max effort\n"), out)
|
||||
self.assertEqual([(model, effort) for model, effort, _ in self.calls][-1], ("claude-opus-5-5", "medium"))
|
||||
self.assertTrue(out.splitlines()[0].startswith("Replayed 1 verdicts of claude-fable-5-2 projects"), out)
|
||||
self.assertTrue(out.endswith("returns every verdict the reference judge reproduces: claude-opus-5-5 at medium effort\n"), out)
|
||||
|
||||
def test_stops_at_the_projects_rung_and_names_none_when_it_misses(self):
|
||||
code, out, _ = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted and rung == REFERENCE, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
@@ -192,9 +192,9 @@ class Main(unittest.TestCase):
|
||||
self.assertEqual(self.run_main(argv, unused)[0], 1)
|
||||
|
||||
def test_exits_1_for_a_model_no_project_runs_on(self):
|
||||
code, _, err = self.run_main(["claude-fable-5-1"], unused)
|
||||
code, _, err = self.run_main(["claude-fable-5-2"], unused)
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("no archived verdicts of projects on claude-fable-5-1", err)
|
||||
self.assertIn("no archived verdicts of projects on claude-fable-5-2", err)
|
||||
|
||||
def test_exits_1_on_an_unreadable_models_file(self):
|
||||
(self.home / ".claude" / "loop-claude-models.json").write_text("[")
|
||||
|
||||
@@ -88,7 +88,7 @@ def markdown_launch(choice: ReviewChoice) -> list[str]:
|
||||
def shared_launches(choice: ReviewChoice, route: Route, project: Rung) -> list[str]:
|
||||
"""A sentence for each rung, other than the worker's, that the reviewers, the seats or the security pass run on.
|
||||
|
||||
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project, routed by item 3 of E + I 5, they read: `Launch each reviewer with the Agent tool set to model opus, effort high, the rung item 3's E + I of 5 picks.` and `Launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.`
|
||||
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project, routed by item 3 of E + I 5, they read: `Launch each reviewer with the Agent tool set to model fable, effort low, the rung item 3's E + I of 5 picks.` and `Launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.`
|
||||
"""
|
||||
together = {EACH_REVIEWER: choice.reviewers, EACH_SEAT: choice.seat, THE_SECURITY_PASS: choice.security}
|
||||
launched: dict[Rung, list[str]] = {}
|
||||
@@ -122,7 +122,7 @@ def escalation(route: Route, project: Rung | None) -> list[str]:
|
||||
"""The prompt's escalation sentences, the marker's among them. None off the ladder or on its top rung. Where escalation_rungs opens none, the worker escalates only with the human's yes.
|
||||
Else they name the rungs escalation_rungs opens to a subagent, and offer to resume the worker one rung up for the rest of the chunk, up to the highest of those rungs.
|
||||
|
||||
For a worker on Opus 5.5 at high, on an Opus 5.5 project, routed by item 3 of L 4, they read: `When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort xhigh`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort max`; above that, ask the human first.`, `Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.` and `Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort max`.`
|
||||
For a worker on Fable 5.2 at low, on an Opus 5.5 project, routed by item 3 of L 4, they read: `When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort medium`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first.`, `Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.` and `Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`.`
|
||||
"""
|
||||
if project is None or route.rung not in LADDER[:-1]:
|
||||
return []
|
||||
@@ -144,7 +144,7 @@ def escalation(route: Route, project: Rung | None) -> list[str]:
|
||||
def reruns(route: Route, project: Rung | None) -> list[str]:
|
||||
"""The sentence naming the rung one above a reviewer's and the one above a seat's, on which each reruns when it asks, leaving out either that is already on the top rung. None off the ladder.
|
||||
|
||||
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project, routed by an item of E + I 4, it reads: `A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`.`
|
||||
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project, routed by an item of E + I 4, it reads: `A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort low`.`
|
||||
"""
|
||||
if project is None or route.rung not in LADDER:
|
||||
return []
|
||||
|
||||
@@ -444,15 +444,15 @@ class Cycles(unittest.TestCase):
|
||||
("claude-sonnet-5-5", "high"),
|
||||
("claude-sonnet-5-5", "high"),
|
||||
("claude-opus-5-5", "medium"),
|
||||
("claude-opus-5-5", "high"),
|
||||
("claude-fable-5-2", "low"),
|
||||
("claude-sonnet-5-5", "high"),
|
||||
])
|
||||
self.assertEqual({(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] == "judge"}, {("judge-model", None)})
|
||||
self.assertIn("loop-claude runs you on claude-sonnet-5-5 at high effort, routed by item 4's L of 2.", argument(workers[0], "-p"))
|
||||
self.assertIn("on one line, as in `Chunk items: ", argument(workers[0], "-p"))
|
||||
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
|
||||
self.assertIn('<UTC> Verdict: unfinished\n\nEvidence:\n> three\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "medium", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n\n<UTC> Stepped up\n\nWorker: {"model": "claude-opus-5-5", "effort": "high", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n', outcomes)
|
||||
self.assertIn('<UTC> Verdict: finished\n\nEvidence:\n> four\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "high", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n', outcomes)
|
||||
self.assertIn('<UTC> Verdict: unfinished\n\nEvidence:\n> three\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "medium", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n\n<UTC> Stepped up\n\nWorker: {"model": "claude-fable-5-2", "effort": "low", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n', outcomes)
|
||||
self.assertIn('<UTC> Verdict: finished\n\nEvidence:\n> four\n\nJudge: judge-model\nWorker: {"model": "claude-fable-5-2", "effort": "low", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n', outcomes)
|
||||
|
||||
def test_routes_on_and_states_the_same_item_from_the_fetched_default_branch_whatever_the_checkout_holds(self):
|
||||
checkout = Checkout(self)
|
||||
@@ -501,14 +501,14 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual(run.code, 130)
|
||||
workers = [call for call in run.calls if call["role"] == "worker"]
|
||||
self.assertEqual([(argument(call, "--model"), argument(call, "--effort")) for call in workers], [
|
||||
("claude-opus-5-5", "high"),
|
||||
("claude-opus-5-5", "xhigh"),
|
||||
("claude-opus-5-5", "xhigh"),
|
||||
("claude-opus-5-5", "high"),
|
||||
("claude-fable-5-2", "low"),
|
||||
("claude-fable-5-2", "medium"),
|
||||
("claude-fable-5-2", "medium"),
|
||||
("claude-fable-5-2", "low"),
|
||||
])
|
||||
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
|
||||
self.assertIn('<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "high", "item": 4, "likelihood": 4, "escape_and_impact": 5}\n\n<UTC> Stepped up\n\nWorker: {"model": "claude-opus-5-5", "effort": "xhigh", "item": 4, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 4}\n', outcomes)
|
||||
self.assertIn('<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "xhigh", "item": 4, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 4}\n', outcomes)
|
||||
self.assertIn('<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: {"model": "claude-fable-5-2", "effort": "low", "item": 4, "likelihood": 4, "escape_and_impact": 5}\n\n<UTC> Stepped up\n\nWorker: {"model": "claude-fable-5-2", "effort": "medium", "item": 4, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 4}\n', outcomes)
|
||||
self.assertIn('<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nJudge: judge-model\nWorker: {"model": "claude-fable-5-2", "effort": "medium", "item": 4, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 4}\n', outcomes)
|
||||
|
||||
def test_steps_the_worker_up_once_as_the_judge_interrupts_when_the_run_also_cites_an_escalation_trigger(self):
|
||||
checkout = Checkout(self)
|
||||
@@ -522,11 +522,11 @@ class Cycles(unittest.TestCase):
|
||||
], terminal_input="/talk\n", before_typing=ANSWERING)
|
||||
self.assertEqual(run.code, 130)
|
||||
self.assertEqual(run.roles(), ["worker", "judge", "interrupt", "judge", "worker", "judge"])
|
||||
self.assertEqual([(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] != "judge"], [("claude-opus-5-5", "high"), ("claude-opus-5-5", "xhigh"), ("claude-opus-5-5", "xhigh")])
|
||||
self.assertEqual([(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] != "judge"], [("claude-fable-5-2", "low"), ("claude-fable-5-2", "medium"), ("claude-fable-5-2", "medium")])
|
||||
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
|
||||
self.assertIn('<UTC> Stepped up\n\nWorker: {"model": "claude-opus-5-5", "effort": "xhigh", "item": 4, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 4}\n', "".join(outcomes))
|
||||
self.assertIn('<UTC> Stepped up\n\nWorker: {"model": "claude-fable-5-2", "effort": "medium", "item": 4, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 4}\n', "".join(outcomes))
|
||||
self.assertNotIn("at max effort", "".join(outcomes))
|
||||
self.assertIn('Verdict: interrupt\n\nA or B?\n\nEvidence:\n> Shall I use A or B?\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: {"model": "claude-opus-5-5", "effort": "high", "item": 4, "likelihood": 4, "escape_and_impact": 5}\n\n<UTC> Stepped up\n\nWorker: {"model": "claude-opus-5-5", "effort": "xhigh", "item": 4, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 4}\n', "".join(outcomes))
|
||||
self.assertIn('Verdict: interrupt\n\nA or B?\n\nEvidence:\n> Shall I use A or B?\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: {"model": "claude-fable-5-2", "effort": "low", "item": 4, "likelihood": 4, "escape_and_impact": 5}\n\n<UTC> Stepped up\n\nWorker: {"model": "claude-fable-5-2", "effort": "medium", "item": 4, "likelihood": 4, "escape_and_impact": 5, "escalated_at": 4}\n', "".join(outcomes))
|
||||
|
||||
def test_runs_a_cheap_items_worker_below_the_projects_rung_with_its_reviewers_and_judge_on_it(self):
|
||||
checkout = Checkout(self)
|
||||
@@ -551,13 +551,13 @@ class Cycles(unittest.TestCase):
|
||||
worker(say="next chunk"),
|
||||
])
|
||||
self.assertEqual(run.code, 130)
|
||||
self.assertEqual([argument(call, "--effort") for call in run.calls if call["role"] == "judge"], ["medium", "xhigh", "medium"])
|
||||
self.assertEqual([(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] == "judge"], [("claude-opus-5-5", "medium"), ("claude-fable-5-2", "medium"), ("claude-opus-5-5", "medium")])
|
||||
workers = [call for call in run.calls if call["role"] == "worker"]
|
||||
self.assertEqual({argument(call, "--effort") for call in workers}, {"high"})
|
||||
self.assertIn("Launch each reviewer with the Agent tool set to model opus, effort xhigh, the rung picked by an item of E + I 6 taken into the chunk.", argument(workers[1], "-p"))
|
||||
self.assertIn("Launch each reviewer with the Agent tool set to model fable, effort medium, the rung picked by an item of E + I 6 taken into the chunk.", argument(workers[1], "-p"))
|
||||
outcomes = "".join(checkout.archived().values())
|
||||
self.assertIn('<UTC> Stepped up\n\nWorker: {"model": "claude-sonnet-5-5", "effort": "high", "item": 4, "likelihood": 2, "escape_and_impact": 3, "escape_and_impact_taken_in": 6}\n', outcomes)
|
||||
self.assertIn('Judge: claude-opus-5-5 at xhigh effort\nWorker: {"model": "claude-sonnet-5-5", "effort": "high", "item": 4, "likelihood": 2, "escape_and_impact": 3, "escape_and_impact_taken_in": 6}\n', outcomes)
|
||||
self.assertIn('Judge: claude-fable-5-2 at medium effort\nWorker: {"model": "claude-sonnet-5-5", "effort": "high", "item": 4, "likelihood": 2, "escape_and_impact": 3, "escape_and_impact_taken_in": 6}\n', outcomes)
|
||||
|
||||
def test_a_rerun_resumes_an_open_chunk_on_the_rung_it_last_ran(self):
|
||||
checkout = Checkout(self)
|
||||
@@ -565,12 +565,12 @@ class Cycles(unittest.TestCase):
|
||||
checkout.commit_plan(TODO)
|
||||
run = checkout.run([worker(), judge(), worker(), judge()])
|
||||
self.assertEqual(run.code, 130)
|
||||
self.assertEqual(argument(run.calls[-1], "--effort"), "xhigh")
|
||||
self.assertEqual((argument(run.calls[-1], "--model"), argument(run.calls[-1], "--effort")), ("claude-fable-5-2", "medium"))
|
||||
checkout.git("rm", "--quiet", "todo.md")
|
||||
checkout.git("commit", "--quiet", "-m", "drop the plan")
|
||||
run = checkout.run([worker()])
|
||||
self.assertIsNotNone(argument(run.calls[0], "--resume"))
|
||||
self.assertEqual((argument(run.calls[0], "--model"), argument(run.calls[0], "--effort")), ("claude-opus-5-5", "xhigh"))
|
||||
self.assertEqual((argument(run.calls[0], "--model"), argument(run.calls[0], "--effort")), ("claude-fable-5-2", "medium"))
|
||||
|
||||
def test_exits_1_when_the_transcript_cannot_be_archived_and_resumes_on_the_next_run(self):
|
||||
checkout = Checkout(self)
|
||||
@@ -702,10 +702,10 @@ class Cycles(unittest.TestCase):
|
||||
checkout.commit_plan(CHEAP)
|
||||
run = checkout.run([step for _ in range(6) for step in (worker(), judge())] + [worker()])
|
||||
self.assertEqual(run.code, 2)
|
||||
stepped = '<UTC> Stepped up\n\nWorker: {"model": "claude-opus-5-5", "effort": "xhigh", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n\n<UTC> Stopped: '
|
||||
stepped = '<UTC> Stepped up\n\nWorker: {"model": "claude-fable-5-2", "effort": "medium", "item": 4, "likelihood": 2, "escape_and_impact": 3}\n\n<UTC> Stopped: '
|
||||
self.assertEqual(len([outcome for outcome in checkout.archived().values() if stepped in outcome]), 1)
|
||||
run = checkout.run([worker()])
|
||||
self.assertEqual([(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] == "worker"], [("claude-opus-5-5", "xhigh")])
|
||||
self.assertEqual([(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] == "worker"], [("claude-fable-5-2", "medium")])
|
||||
|
||||
def test_five_unfinished_verdicts_and_a_finished_one_do_not_add_up_to_six(self):
|
||||
checkout = Checkout(self)
|
||||
|
||||
@@ -248,7 +248,7 @@ class Transitions(unittest.TestCase):
|
||||
states.transition(state, event)
|
||||
|
||||
def test_a_rulings_step_up_reaches_the_worker_it_resumes_through_the_answer_the_human_types_and_no_further(self):
|
||||
stepped_to = Route(Rung("claude-opus-5-5", "high"), escalated_at=3)
|
||||
stepped_to = Route(Rung("claude-fable-5-2", "low"), escalated_at=3)
|
||||
interrupt, quoted = Interrupt("e", "q", "round 2: 🔴"), QuotedAnswer("e", "q", "A.", "round 2: 🔴")
|
||||
for state, event, after in (
|
||||
(states.Judging("/t"), states.Ruled(quoted, stepped_to), states.Working(quoted, 1, True)),
|
||||
@@ -513,7 +513,7 @@ class Cycles(unittest.TestCase):
|
||||
with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
|
||||
concluded = run.run_cycle(cycle)
|
||||
self.assertEqual(concluded.route, Route(Routing.HIGH, 179, 2, 5, escalated_at=4))
|
||||
self.assertEqual([args[args.index("--effort") + 1] for args in fake.made("stream_claude")], ["high", "medium", "high"])
|
||||
self.assertEqual([args[args.index("--effort") + 1] for args in fake.made("stream_claude")], ["high", "medium", "low"])
|
||||
|
||||
def test_an_escalation_quoted_from_before_the_workers_last_step_up_steps_nothing_and_a_later_one_reaches_the_verdict(self):
|
||||
for escalation, verdict in (("round 2: 🔴", Unfinished("working", entries_read=4)), ("working", Unfinished("working", "working", entries_read=4))):
|
||||
@@ -554,7 +554,7 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual([args[args.index("--effort") + 1] for args in (first, resumed)], ["high", "medium"])
|
||||
typed = resumed[resumed.index("-p") + 1]
|
||||
self.assertIn("Your request for a higher rung went through: loop-claude runs you on claude-opus-5-5 at medium effort.", typed)
|
||||
self.assertIn(" Launch each reviewer with the Agent tool set to model opus, effort high, the rung item 179's E + I of 5 picks.", typed)
|
||||
self.assertIn(" Launch each reviewer with the Agent tool set to model fable, effort low, the rung item 179's E + I of 5 picks.", typed)
|
||||
self.assertNotIn("went through", first[first.index("-p") + 1])
|
||||
|
||||
def test_a_typed_answer_past_the_quoted_answers_cap_reaches_the_worker_with_its_own_question(self):
|
||||
@@ -1039,9 +1039,9 @@ def interrupted(likelihood=0, escalated_at=None):
|
||||
|
||||
class Routing(unittest.TestCase):
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
HIGH = Rung("claude-opus-5-5", "high")
|
||||
XHIGH = Rung("claude-opus-5-5", "xhigh")
|
||||
MAX = Rung("claude-opus-5-5", "max")
|
||||
HIGH = Rung("claude-fable-5-2", "low")
|
||||
XHIGH = Rung("claude-fable-5-2", "medium")
|
||||
MAX = Rung("claude-opus-5-5", "xhigh")
|
||||
SONNET = Rung("claude-sonnet-5-5", "high")
|
||||
|
||||
def test_routes_on_the_next_releases_top_item_that_waits_on_no_question(self):
|
||||
@@ -1167,12 +1167,12 @@ class Routing(unittest.TestCase):
|
||||
|
||||
def test_archives_the_route_as_a_json_worker_line_it_reads_back_field_by_field(self):
|
||||
route = Route(self.HIGH, 3, 4, 5)
|
||||
self.assertEqual(route.worker_line(), '{"model": "claude-opus-5-5", "effort": "high", "item": 3, "likelihood": 4, "escape_and_impact": 5}')
|
||||
self.assertEqual(route.worker_line(), '{"model": "claude-fable-5-2", "effort": "low", "item": 3, "likelihood": 4, "escape_and_impact": 5}')
|
||||
self.assertEqual(route_from(route.worker_line()), route)
|
||||
self.assertEqual(Route(Rung("worker-model")).worker_line(), '{"model": "worker-model"}')
|
||||
self.assertEqual(route_from('{"model": "worker-model"}'), Route(Rung("worker-model")))
|
||||
marked = Route(self.HIGH, 3, 4, 5, escape_and_impact_taken_in=7, escalated_at=12)
|
||||
self.assertEqual(marked.worker_line(), '{"model": "claude-opus-5-5", "effort": "high", "item": 3, "likelihood": 4, "escape_and_impact": 5, "escape_and_impact_taken_in": 7, "escalated_at": 12}')
|
||||
self.assertEqual(marked.worker_line(), '{"model": "claude-fable-5-2", "effort": "low", "item": 3, "likelihood": 4, "escape_and_impact": 5, "escape_and_impact_taken_in": 7, "escalated_at": 12}')
|
||||
self.assertEqual(route_from(marked.worker_line()), marked)
|
||||
self.assertEqual(route_from(Route(self.HIGH, escalated_at=3).worker_line()), Route(self.HIGH, escalated_at=3))
|
||||
|
||||
@@ -1180,7 +1180,7 @@ class Routing(unittest.TestCase):
|
||||
self.assertEqual(route_from(Route(self.HIGH, 3, 4, 5, unfinished=1).worker_line()), Route(self.HIGH, 3, 4, 5))
|
||||
|
||||
def test_reads_no_route_from_a_worker_line_that_route_worker_line_could_not_have_written(self):
|
||||
for line in ('{"effort": "high"}', '{"model": "claude-opus-5-5", "effort": "high", "unfinished": 1}', '{"model": "claude-opus-5-5", "item": "3"}', '{"model": "claude-opus-5-5", "effort": 3}', '{"model": 5}', '{"model": "claude-opus-5-5", "effort": "high", "escalated_at": -3}', '{"model": "claude-opus-5-5", "item": true, "likelihood": 4, "escape_and_impact": 5}', '{"model": "claude-opus-5-5", "effort": "high", "item": 3, "likelihood": 5, "escape_and_impact": 5}', '{"model": "claude-opus-5-5", "effort": "high", "escape_and_impact_taken_in": 9}', '["claude-opus-5-5"]', '{"model": ', "worker-model", "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 5"):
|
||||
for line in ('{"effort": "high"}', '{"model": "claude-opus-5-5", "effort": "high", "unfinished": 1}', '{"model": "claude-opus-5-5", "item": "3"}', '{"model": "claude-opus-5-5", "effort": 3}', '{"model": 5}', '{"model": "claude-opus-5-5", "effort": "high", "escalated_at": -3}', '{"model": "claude-opus-5-5", "item": true, "likelihood": 4, "escape_and_impact": 5}', '{"model": "claude-opus-5-5", "effort": "high", "item": 3, "likelihood": 5, "escape_and_impact": 5}', '{"model": "claude-opus-5-5", "effort": "high", "escape_and_impact_taken_in": 9}', '["claude-opus-5-5"]', '{"model": ', "worker-model", "claude-fable-5-2 at low effort, routed by item 3's L of 4 and E + I of 5"):
|
||||
with self.subTest(line=line):
|
||||
self.assertIsNone(route_from(line))
|
||||
|
||||
@@ -1224,7 +1224,7 @@ class RouteSources(unittest.TestCase):
|
||||
opened = session.session_named("A proj loop-claude", pathlib.Path(home, "projects"))
|
||||
archive = pathlib.Path(home, "archive")
|
||||
transcript = pathlib.Path(home, "projects", "-fake", f"{opened.id}.jsonl")
|
||||
route = Route(Rung("claude-opus-5-5", "high"), 3, 4, 5)
|
||||
route = Route(Rung("claude-fable-5-2", "low"), 3, 4, 5)
|
||||
with contextlib.redirect_stderr(io.StringIO()) as stderr:
|
||||
self.assertIsNone(chunk_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)))
|
||||
transcript.parent.mkdir(parents=True)
|
||||
@@ -1257,7 +1257,7 @@ class Steps(unittest.TestCase):
|
||||
run_with = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="old", route=Route(self.OPUS))
|
||||
self.assertEqual(close.after_verdict(run_with, self.OPUS, run.Concluded(Finished("x"), run_with.route)), dataclasses.replace(run_with, route=None, unfinished_evidence=""))
|
||||
self.assertEqual(close.after_verdict(dataclasses.replace(run_with, unfinished_run=3), self.OPUS, run.Concluded(Finished("x"), run_with.route)).unfinished_run, 0)
|
||||
self.assertEqual(close.after_verdict(run_with, self.OPUS, run.Concluded(Unfinished("new", taken_in=TakenIn(highest_likelihood=4)), run_with.route)), dataclasses.replace(run_with, route=Route(Rung("claude-opus-5-5", "high"), unfinished=1), unfinished_evidence="new", unfinished_run=1))
|
||||
self.assertEqual(close.after_verdict(run_with, self.OPUS, run.Concluded(Unfinished("new", taken_in=TakenIn(highest_likelihood=4)), run_with.route)), dataclasses.replace(run_with, route=Route(Rung("claude-fable-5-2", "low"), unfinished=1), unfinished_evidence="new", unfinished_run=1))
|
||||
|
||||
def test_an_unfinished_chunk_keeps_the_items_its_verdict_found_taken_in_and_a_finished_one_drops_them(self):
|
||||
run_with = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=Route(self.OPUS), items_taken_in=(3,))
|
||||
@@ -1266,7 +1266,7 @@ class Steps(unittest.TestCase):
|
||||
|
||||
def test_steps_from_the_route_its_cycle_ended_on_which_a_ruling_may_have_raised(self):
|
||||
run_with = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=Route(self.OPUS, 3, 3, 4))
|
||||
raised = Route(Rung("claude-opus-5-5", "high"), 3, 3, 4, escalated_at=5)
|
||||
raised = Route(Rung("claude-fable-5-2", "low"), 3, 3, 4, escalated_at=5)
|
||||
self.assertEqual(close.after_verdict(run_with, self.OPUS, run.Concluded(Unfinished("new"), raised)).route, dataclasses.replace(raised, unfinished=1))
|
||||
|
||||
def test_keeps_the_route_its_cycle_ended_on_off_the_ladder(self):
|
||||
@@ -1276,7 +1276,7 @@ class Steps(unittest.TestCase):
|
||||
|
||||
def test_an_unfinished_verdicts_escalation_marks_the_transcript_entries_the_judge_read(self):
|
||||
run_with = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=Route(self.OPUS, 3, 3, 4, escalated_at=2))
|
||||
self.assertEqual(close.after_verdict(run_with, self.OPUS, run.Concluded(Unfinished("new", "round 2: 🔴", entries_read=9), run_with.route)).route, Route(Rung("claude-opus-5-5", "high"), 3, 3, 4, escalated_at=9))
|
||||
self.assertEqual(close.after_verdict(run_with, self.OPUS, run.Concluded(Unfinished("new", "round 2: 🔴", entries_read=9), run_with.route)).route, Route(Rung("claude-fable-5-2", "low"), 3, 3, 4, escalated_at=9))
|
||||
self.assertEqual(close.after_verdict(run_with, self.OPUS, run.Concluded(Unfinished("new", entries_read=9), run_with.route)).route, Route(self.OPUS, 3, 3, 4, unfinished=1, escalated_at=2))
|
||||
|
||||
def test_archives_a_step_up_as_it_happens_so_a_rerun_resumes_on_it(self):
|
||||
@@ -1290,10 +1290,10 @@ class Steps(unittest.TestCase):
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(before.worker_line(), "working", "j", ""), NOW))
|
||||
after = close.closed_cycle(main.LoopRun(cycle_number=1, root="/r", task="Do it.", unfinished_evidence=""), self.OPUS, run.Concluded(Unfinished("working", taken_in=TakenIn(highest_likelihood=4)), before), archive, a_loop(FakeEffects().effects(), session=opened))
|
||||
self.assertEqual(chunk_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)), Route(Rung("claude-opus-5-5", "high"), 3, 3, 4))
|
||||
self.assertEqual(chunk_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)), Route(Rung("claude-fable-5-2", "low"), 3, 3, 4))
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
close.closed_cycle(after, self.OPUS, run.Concluded(Finished("done"), after.route), archive, a_loop(FakeEffects().effects(), session=opened))
|
||||
self.assertEqual(chunk_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)), Route(Rung("claude-opus-5-5", "high"), 3, 3, 4))
|
||||
self.assertEqual(chunk_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)), Route(Rung("claude-fable-5-2", "low"), 3, 3, 4))
|
||||
blocked = pathlib.Path(home, "blocked")
|
||||
blocked.write_text("")
|
||||
stop = close.closed_cycle(main.LoopRun(cycle_number=1, root="/r", task="Do it.", unfinished_evidence=""), self.OPUS, run.Concluded(Unfinished("working", taken_in=TakenIn(highest_likelihood=4)), before), blocked, a_loop(FakeEffects().effects(), session=opened))
|
||||
@@ -1410,80 +1410,80 @@ class WorkerArgs(unittest.TestCase):
|
||||
def test_keeps_every_review_on_the_projects_rung_while_the_worker_runs_below_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
below = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 4), opus, self.FACTS)
|
||||
self.assertIn(f"item 3's L of 2. Launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.{self.SCAN} Your rung is below the project's: launch each subagent that files or scores todo.md items on the rung `~/.claude/bin/subagent-rung --plan claude-opus-5-5 <L>` prints. Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5, `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8. When an escalation trigger", below)
|
||||
self.assertIn(f"item 3's L of 2. Launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.{self.SCAN} Your rung is below the project's: launch each subagent that files or scores todo.md items on the rung `~/.claude/bin/subagent-rung --plan claude-opus-5-5 <L>` prints. Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model fable, effort low` from 5, `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8. When an escalation trigger", below)
|
||||
self.assertNotIn("Launch each", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus, self.FACTS))
|
||||
|
||||
def test_launches_reviewers_on_the_rung_the_items_e_plus_i_picks_above_the_workers(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
level = prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 6), opus, self.FACTS)
|
||||
self.assertIn(f"item 3's L of 3. Launch each reviewer with the Agent tool set to model opus, effort xhigh, the rung item 3's E + I of 6 picks. Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", level)
|
||||
self.assertIn(f"item 3's L of 3. Launch each reviewer with the Agent tool set to model fable, effort medium, the rung item 3's E + I of 6 picks. Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.{self.MARKDOWN} When an escalation trigger", level)
|
||||
below = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 5), opus, self.FACTS)
|
||||
self.assertIn(f"item 3's L of 2. Launch each reviewer with the Agent tool set to model opus, effort high, the rung item 3's E + I of 5 picks.{self.SEATS}{self.SCAN} Your rung is below the project's:", below)
|
||||
self.assertNotIn("Launch each reviewer", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 6), opus))
|
||||
self.assertIn(f"item 3's L of 2. Launch each reviewer with the Agent tool set to model fable, effort low, the rung item 3's E + I of 5 picks.{self.SEATS}{self.SCAN} Your rung is below the project's:", below)
|
||||
self.assertNotIn("Launch each reviewer", prompts.resumed("", Route(Rung("claude-fable-5-2", "medium"), 3, 4, 6), opus))
|
||||
|
||||
def test_names_the_reviewers_rung_each_higher_e_plus_i_an_item_taken_in_picks(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
taking_in = " Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5, `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8."
|
||||
taking_in = " Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model fable, effort low` from 5, `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8."
|
||||
self.assertIn(f"item 3's L of 3.{taking_in} When an escalation trigger", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus, self.FACTS))
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8.", prompts.resumed("", Route(opus, 3, 3, 6), opus))
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8.", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 4), opus))
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.", prompts.resumed("", Route(opus, 3, 3, 6), opus))
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.", prompts.resumed("", Route(Rung("claude-fable-5-2", "medium"), 3, 4, 4), opus))
|
||||
self.assertNotIn("Where an item you take into", prompts.resumed("", Route(opus, 3, 3, 8), opus))
|
||||
self.assertNotIn("Where an item you take into", prompts.resumed("", Route(Rung("m")), None))
|
||||
|
||||
def test_launches_reviewers_on_the_rung_the_highest_e_plus_i_taken_in_picks(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
self.assertIn(f"loop-claude runs you on claude-opus-5-5 at medium effort. Launch each reviewer with the Agent tool set to model opus, effort max, the rung picked by an item of E + I 7 taken into the chunk. Where an item you take into the chunk has an E + I of 8 or more, launch each reviewer with the Agent tool set to `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", prompts.resumed("", Route(opus, 3, 3, 5, escape_and_impact_taken_in=7), opus))
|
||||
self.assertIn(" Launch each reviewer with the Agent tool set to model opus, effort high, the rung picked by an item of E + I 5 taken into the chunk.", prompts.resumed("", Route(opus, escape_and_impact_taken_in=5), opus))
|
||||
self.assertIn(f"loop-claude runs you on claude-opus-5-5 at medium effort. Launch each reviewer with the Agent tool set to model opus, effort xhigh, the rung picked by an item of E + I 7 taken into the chunk. Where an item you take into the chunk has an E + I of 8 or more, launch each reviewer with the Agent tool set to `model fable, effort high` from 8.{self.MARKDOWN} When an escalation trigger", prompts.resumed("", Route(opus, 3, 3, 5, escape_and_impact_taken_in=7), opus))
|
||||
self.assertIn(" Launch each reviewer with the Agent tool set to model fable, effort low, the rung picked by an item of E + I 5 taken into the chunk.", prompts.resumed("", Route(opus, escape_and_impact_taken_in=5), opus))
|
||||
|
||||
def test_keeps_seats_and_the_security_pass_on_the_projects_rung_while_the_worker_runs_above_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
above = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-opus-5-5", "high"), 3, 4, 4), opus, self.FACTS)
|
||||
stepped_up = prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus)
|
||||
self.assertIn(f"{prompts.ANSWERS_RECORDED}. {prompts.NAME_THE_ITEMS} loop-claude runs you on claude-opus-5-5 at high effort.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", stepped_up)
|
||||
above = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-fable-5-2", "low"), 3, 4, 4), opus, self.FACTS)
|
||||
stepped_up = prompts.resumed("", Route(Rung("claude-fable-5-2", "low"), 3, 3, 4), opus)
|
||||
self.assertIn(f"{prompts.ANSWERS_RECORDED}. {prompts.NAME_THE_ITEMS} loop-claude runs you on claude-fable-5-2 at low effort.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.{self.MARKDOWN} When an escalation trigger", stepped_up)
|
||||
self.assertNotIn("the project's", prompts.resumed("", Route(opus, 3, 3, 4), opus))
|
||||
self.assertNotIn("runs you on", prompts.resumed("", Route(Rung("m")), None))
|
||||
self.assertIn(f"item 3's L of 4.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", above)
|
||||
self.assertIn(f"item 3's L of 4.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.{self.MARKDOWN} When an escalation trigger", above)
|
||||
|
||||
def test_names_the_rungs_an_escalation_trigger_lets_the_worker_hand_a_task_to_at_most_two_above_its_items(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
fresh = prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus, self.FACTS)
|
||||
self.assertIn("item 3's L of 3. Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5, `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", fresh)
|
||||
self.assertIn(" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort xhigh`, one rung above yours; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus))
|
||||
self.assertIn(" An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `model opus, effort max`, one rung above yours, only once the human agrees; loop-claude will not resume you higher for a trigger either, so ask the human for that yes. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 3, 4), opus))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("claude-fable-5-1", "max"), 3, 4, 8), opus))
|
||||
self.assertIn("only once the human agrees", prompts.resumed("", Route(Rung("claude-opus-5-5", "max"), 3, 4, 5), opus))
|
||||
self.assertIn("hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort max`, one rung above yours;", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 5), opus))
|
||||
self.assertIn("item 3's L of 3. Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model fable, effort low` from 5, `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort low`, one rung above yours, and on each further trigger one rung higher, up to `model fable, effort medium`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", fresh)
|
||||
self.assertIn(" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort medium`, one rung above yours; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-fable-5-2", "low"), 3, 3, 4), opus))
|
||||
self.assertIn(" An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `model opus, effort xhigh`, one rung above yours, only once the human agrees; loop-claude will not resume you higher for a trigger either, so ask the human for that yes. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-fable-5-2", "medium"), 3, 3, 4), opus))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("claude-fable-5-2", "max"), 3, 4, 8), opus))
|
||||
self.assertIn("only once the human agrees", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 5), opus))
|
||||
self.assertIn("hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort xhigh`, one rung above yours;", prompts.resumed("", Route(Rung("claude-fable-5-2", "medium"), 3, 4, 5), opus))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("m")), None))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("claude-opus-5-5", "low"), 3, 3, 4), opus))
|
||||
self.assertIn(f"{prompts.READING_RULE} {prompts.NAME_THE_ITEMS}\n\nWhere an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5, `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number. Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`. A reviewer", prompts.fresh("Do it.", "## Goals", Route(opus), opus, self.FACTS))
|
||||
self.assertIn(f"{prompts.READING_RULE} {prompts.NAME_THE_ITEMS}\n\nWhere an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model fable, effort low` from 5, `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort low`, one rung above yours, and on each further trigger one rung higher, up to `model fable, effort medium`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number. Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model fable, effort medium`. A reviewer", prompts.fresh("Do it.", "## Goals", Route(opus), opus, self.FACTS))
|
||||
|
||||
def test_offers_the_worker_a_higher_rung_for_the_rest_of_the_chunk_up_to_the_escalation_cap(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
offer = " Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`."
|
||||
offer = " Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model fable, effort medium`."
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus, self.FACTS))
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus))
|
||||
self.assertNotIn("Where the hard part", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 3, 4), opus))
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.resumed("", Route(Rung("claude-fable-5-2", "low"), 3, 3, 4), opus))
|
||||
self.assertNotIn("Where the hard part", prompts.resumed("", Route(Rung("claude-fable-5-2", "medium"), 3, 3, 4), opus))
|
||||
|
||||
def test_lets_a_markdown_only_branch_review_on_the_projects_rung_where_the_reviewers_run_above_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
markdown = " Where your branch changes only Markdown, none of it `CLAUDE.md`, `AGENTS.md`, `workflow-bench/*-prompt.md` or under `agents/`, `principles/`, `rules/` or `skills/`, launch each reviewer on the project's rung instead, with the Agent tool set to model opus, effort medium."
|
||||
self.assertIn(markdown, prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 7), opus, self.FACTS))
|
||||
self.assertIn(markdown, prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 4, 4), opus))
|
||||
self.assertIn(markdown, prompts.resumed("", Route(Rung("claude-fable-5-2", "low"), 3, 4, 4), opus))
|
||||
self.assertNotIn("only Markdown", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus, self.FACTS))
|
||||
|
||||
def test_names_the_rung_a_reviewer_or_seat_reruns_on_one_above_its_own(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
both = " A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`."
|
||||
both = " A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort low`."
|
||||
self.assertTrue(prompts.resumed("", Route(opus, 3, 3, 4), opus).endswith(both))
|
||||
self.assertTrue(prompts.resumed("", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 4), opus).endswith(both))
|
||||
self.assertTrue(prompts.fresh("Do it.", "## Goals", Route(opus), opus, self.FACTS).split(f"\n\n{self.FACTS}")[0].endswith(both))
|
||||
self.assertTrue(prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 4, 4), opus).endswith(" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort xhigh`, and a comprehension seat with `model opus, effort high`."))
|
||||
self.assertTrue(prompts.resumed("", Route(opus, 3, 3, 7), opus).endswith(" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort max`, and a comprehension seat with `model opus, effort high`."))
|
||||
self.assertTrue(prompts.resumed("", Route(Rung("claude-fable-5-1", "max"), 3, 4, 8), opus).endswith(" A comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`."))
|
||||
self.assertTrue(prompts.resumed("", Route(Rung("claude-fable-5-2", "low"), 3, 4, 4), opus).endswith(" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort medium`, and a comprehension seat with `model fable, effort low`."))
|
||||
self.assertTrue(prompts.resumed("", Route(opus, 3, 3, 7), opus).endswith(" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort high`, and a comprehension seat with `model fable, effort low`."))
|
||||
self.assertTrue(prompts.resumed("", Route(Rung("claude-fable-5-2", "max"), 3, 4, 8), opus).endswith(" A comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort low`."))
|
||||
self.assertNotIn("rerun", prompts.resumed("", Route(Rung("m")), None))
|
||||
self.assertNotIn("rerun", prompts.fresh("Do it.", "## Goals", Route(Rung("m", "high"), 3, 4, 5), None, self.FACTS))
|
||||
self.assertIn(f"{both} After your first read", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus, self.FACTS))
|
||||
fable = Rung("claude-fable-5-1", "max")
|
||||
fable = Rung("claude-fable-5-2", "max")
|
||||
self.assertNotIn("rerun", prompts.resumed("", Route(fable), fable))
|
||||
|
||||
def test_opens_a_fresh_session_by_name_and_resumes_an_open_one(self):
|
||||
@@ -1740,12 +1740,12 @@ class LoopRuns(unittest.TestCase):
|
||||
self.assertEqual(main.loop_run_from([main.RELOAD_FLAG, *handed], CWD, FakeEffects().effects()), Stop(1, f"the edited loop-claude cannot read the state the running one handed it, likely because the edit changed that handover; rerun loop-claude <task>. Handed: {' '.join(handed)}"))
|
||||
|
||||
def test_decodes_every_field_of_the_state_and_its_route(self):
|
||||
state = main.LoopRun(cycle_number=2, root="/r", task="do it", unfinished_evidence="old", route=Route(Rung("claude-opus-5-5", "high"), 3, 4, 5, 1, 7, 9), unfinished_run=2, items_taken_in=(12, 14))
|
||||
state = main.LoopRun(cycle_number=2, root="/r", task="do it", unfinished_evidence="old", route=Route(Rung("claude-fable-5-2", "low"), 3, 4, 5, 1, 7, 9), unfinished_run=2, items_taken_in=(12, 14))
|
||||
self.assertEqual(reload.decode_handed(json.dumps(dataclasses.asdict(state))), state)
|
||||
self.assertEqual(set(reload.LOOP_RUN_FIELDS), {field.name for field in dataclasses.fields(main.LoopRun)})
|
||||
|
||||
def test_rejects_a_field_of_another_type_or_an_unknown_field(self):
|
||||
state = dataclasses.asdict(main.LoopRun(cycle_number=2, root="/r", task="do it", unfinished_evidence="", route=Route(Rung("claude-opus-5-5", "high"), 3, 4, 5, 1, 7, 9), unfinished_run=2))
|
||||
state = dataclasses.asdict(main.LoopRun(cycle_number=2, root="/r", task="do it", unfinished_evidence="", route=Route(Rung("claude-fable-5-2", "low"), 3, 4, 5, 1, 7, 9), unfinished_run=2))
|
||||
for path, wrong in ((("unfinished_run",), "2"), (("unfinished_run",), True), (("route", "unfinished"), 1.0), (("route", "escalated_at"), "9"), (("route", "item"), True), (("route", "rung"), "claude-opus-5-5"), (("route", "rung", "model"), None), (("route",), "r"), (("route", "extra"), 1), (("route", "model"), "claude-opus-5-5"), (("route", "effort"), "high")):
|
||||
with self.subTest(path=path, wrong=wrong):
|
||||
handed = json.loads(json.dumps(state))
|
||||
@@ -1756,7 +1756,7 @@ class LoopRuns(unittest.TestCase):
|
||||
self.assertIsNone(reload.decode_handed(json.dumps(handed)))
|
||||
|
||||
def test_rejects_a_state_or_rung_missing_a_required_field(self):
|
||||
state = dataclasses.asdict(main.LoopRun(cycle_number=2, root="/r", task="do it", unfinished_evidence="", route=Route(Rung("claude-opus-5-5", "high"))))
|
||||
state = dataclasses.asdict(main.LoopRun(cycle_number=2, root="/r", task="do it", unfinished_evidence="", route=Route(Rung("claude-fable-5-2", "low"))))
|
||||
for path in (("cycle_number",), ("root",), ("task",), ("unfinished_evidence",), ("route", "rung"), ("route", "rung", "model")):
|
||||
with self.subTest(path=path):
|
||||
handed = json.loads(json.dumps(state))
|
||||
@@ -1774,7 +1774,7 @@ class LoopRuns(unittest.TestCase):
|
||||
self.assertIsNone(reload.decode_handed("not json"))
|
||||
|
||||
def test_decodes_a_route_inside_the_state_or_none(self):
|
||||
state = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=Route(Rung("claude-opus-5-5", "high"), 3, 4, 5, 1))
|
||||
state = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=Route(Rung("claude-fable-5-2", "low"), 3, 4, 5, 1))
|
||||
self.assertEqual(reload.decode_handed(json.dumps(dataclasses.asdict(state))), state)
|
||||
unrouted = dataclasses.replace(state, route=None)
|
||||
self.assertEqual(reload.decode_handed(json.dumps(dataclasses.asdict(unrouted))), unrouted)
|
||||
|
||||
@@ -51,10 +51,11 @@ class Rung:
|
||||
LADDER = (
|
||||
Rung("claude-sonnet-5-5", "high"),
|
||||
Rung("claude-opus-5-5", "medium"),
|
||||
Rung("claude-opus-5-5", "high"),
|
||||
Rung("claude-fable-5-2", "low"),
|
||||
Rung("claude-fable-5-2", "medium"),
|
||||
Rung("claude-opus-5-5", "xhigh"),
|
||||
Rung("claude-opus-5-5", "max"),
|
||||
Rung("claude-fable-5-1", "max"),
|
||||
Rung("claude-fable-5-2", "high"),
|
||||
Rung("claude-fable-5-2", "max"),
|
||||
)
|
||||
# The rungs a task's L steps its worker from the project's rung.
|
||||
WORKER_STEPS = {1: -1, 2: -1, 3: 0, 4: 1}
|
||||
|
||||
@@ -9,19 +9,25 @@ from release_plan.todo_table import LEVELS
|
||||
from review_rules.review_passes import REVIEWERS
|
||||
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
FABLE_LOW = Rung("claude-fable-5-2", "low")
|
||||
|
||||
|
||||
class Ladder(unittest.TestCase):
|
||||
def test_runs_from_sonnet_at_high_effort_to_fable_at_max_with_opus_at_medium_between(self):
|
||||
self.assertEqual(LADDER[0], Rung("claude-sonnet-5-5", "high"))
|
||||
self.assertEqual(LADDER[1], OPUS)
|
||||
self.assertEqual(LADDER[-1], Rung("claude-fable-5-1", "max"))
|
||||
self.assertEqual(len(LADDER), 6)
|
||||
self.assertEqual(len(set(LADDER)), len(LADDER))
|
||||
def test_holds_the_rungs_lilleman_approved_from_the_index_least_competent_first(self):
|
||||
self.assertEqual(LADDER, (
|
||||
Rung("claude-sonnet-5-5", "high"),
|
||||
OPUS,
|
||||
FABLE_LOW,
|
||||
Rung("claude-fable-5-2", "medium"),
|
||||
Rung("claude-opus-5-5", "xhigh"),
|
||||
Rung("claude-fable-5-2", "high"),
|
||||
Rung("claude-fable-5-2", "max"),
|
||||
))
|
||||
|
||||
def test_puts_a_project_on_its_models_cheapest_rung_and_none_off_the_ladder(self):
|
||||
self.assertEqual(project_rung("claude-opus-5-5"), OPUS)
|
||||
self.assertEqual(project_rung("claude-sonnet-5-5"), Rung("claude-sonnet-5-5", "high"))
|
||||
self.assertEqual(project_rung("claude-fable-5-2"), FABLE_LOW)
|
||||
self.assertIsNone(project_rung("claude-opus-5"))
|
||||
self.assertIsNone(project_rung("opus"))
|
||||
|
||||
@@ -39,7 +45,7 @@ class Ladder(unittest.TestCase):
|
||||
self.assertEqual(worker_rung(LADDER[0], 1), LADDER[0])
|
||||
self.assertEqual(review_floor(Rung("claude-opus-5-5", "xhigh"), 8), LADDER[-1])
|
||||
self.assertEqual(stepped_up(LADDER[-1]), LADDER[-1])
|
||||
self.assertEqual(stepped_up(OPUS), Rung("claude-opus-5-5", "high"))
|
||||
self.assertEqual(stepped_up(OPUS), FABLE_LOW)
|
||||
|
||||
def test_steps_a_worker_for_every_level_on_ls_scale(self):
|
||||
self.assertEqual(set(WORKER_STEPS), set(range(1, LEVELS["L"] + 1)))
|
||||
@@ -49,7 +55,7 @@ class Ladder(unittest.TestCase):
|
||||
|
||||
def test_plans_on_the_rung_l_picks_but_never_below_the_projects(self):
|
||||
self.assertEqual(planning_rung(OPUS, 1), OPUS)
|
||||
self.assertEqual(planning_rung(OPUS, 4), Rung("claude-opus-5-5", "high"))
|
||||
self.assertEqual(planning_rung(OPUS, 4), FABLE_LOW)
|
||||
|
||||
def test_names_each_rung_by_a_family_the_agent_tool_takes(self):
|
||||
self.assertEqual({rung.agent_args().split(",")[0] for rung in LADDER}, {"model sonnet", "model opus", "model fable"})
|
||||
@@ -89,7 +95,7 @@ class Ladder(unittest.TestCase):
|
||||
self.assertEqual(Rung.from_agent("sonnet", ""), Rung("claude-sonnet-5-5", "high"))
|
||||
|
||||
def test_reads_an_unknown_family_or_an_effort_off_the_ladder_as_a_rung_off_it(self):
|
||||
for model, effort in (("gpt", "high"), ("gpt", ""), ("opus", "low"), ("haiku", "bogus")):
|
||||
for model, effort in (("gpt", "high"), ("gpt", ""), ("opus", "low"), ("opus", "high"), ("haiku", "bogus")):
|
||||
with self.subTest(model=model, effort=effort):
|
||||
self.assertNotIn(Rung.from_agent(model, effort), LADDER)
|
||||
self.assertEqual(Rung.from_agent("opus", "low"), Rung("claude-opus-5-5", "low"))
|
||||
@@ -97,9 +103,9 @@ class Ladder(unittest.TestCase):
|
||||
def test_puts_reviewers_on_the_highest_of_the_projects_rung_the_workers_and_the_one_e_plus_i_picks(self):
|
||||
sonnet, fable = LADDER[0], LADDER[-1]
|
||||
self.assertEqual(review_rung(OPUS, sonnet, 4, "stability"), OPUS)
|
||||
self.assertEqual(review_rung(OPUS, sonnet, 5, "stability"), Rung("claude-opus-5-5", "high"))
|
||||
self.assertEqual(review_rung(OPUS, sonnet, 5, "stability"), FABLE_LOW)
|
||||
self.assertEqual(review_rung(OPUS, fable, 5, "stability"), fable)
|
||||
self.assertEqual(review_rung(OPUS, Rung("claude-opus-5-5", "high"), 7, "stability"), Rung("claude-opus-5-5", "max"))
|
||||
self.assertEqual(review_rung(OPUS, FABLE_LOW, 7, "stability"), Rung("claude-opus-5-5", "xhigh"))
|
||||
|
||||
def test_keeps_the_seats_and_the_security_pass_on_the_projects_rung_whatever_the_worker_or_e_plus_i(self):
|
||||
self.assertEqual([review_rung(OPUS, LADDER[-1], 8, kind) for kind in (*REVIEWERS, "seat", "security")], [LADDER[-1]] * len(REVIEWERS) + [OPUS, OPUS])
|
||||
|
||||
@@ -5,7 +5,7 @@ from model_routing.route import Route
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
FABLE = Rung("claude-fable-5-1", "max")
|
||||
FABLE = Rung("claude-fable-5-2", "max")
|
||||
|
||||
|
||||
class Chosen(unittest.TestCase):
|
||||
|
||||
@@ -20,9 +20,11 @@ class Price:
|
||||
return self.input * 2
|
||||
|
||||
|
||||
# Anthropic's list prices as the claude-api skill lists them (cached 2026-10-06), with Opus 5.5's cache prices from `docs/decisions.md` → `loop-claude: the worker's claude compacts…`. Haiku 5.5's cache read is assumed at a tenth of its input, and its dearer prompts above 200k tokens are left out. Where a chunk records its cost, only the proportions between one model's messages matter, so these may be a little off.
|
||||
# Anthropic's list prices as the claude-api skill lists them (cached 2026-10-06; Fable 5.2's 2026-11-03), with Opus 5.5's cache prices from `docs/decisions.md` → `loop-claude: the worker's claude compacts…`. Haiku 5.5's cache read is assumed at a tenth of its input, and its dearer prompts above 200k tokens are left out. Where a chunk records its cost, only the proportions between one model's messages matter, so these may be a little off.
|
||||
PRICES = {
|
||||
# Archived chunks that ran on Fable 5.1 are still priced from here.
|
||||
"claude-fable-5-1": Price(10, 0.25, 50),
|
||||
"claude-fable-5-2": Price(8, 0.2, 40),
|
||||
"claude-haiku-5-5": Price(0.1, 0.01, 0.5),
|
||||
"claude-opus-5-5": Price(4, 0.2, 20),
|
||||
"claude-sonnet-5-5": Price(2, 0.2, 10),
|
||||
|
||||
@@ -30,12 +30,12 @@ from routing_figures.render import render
|
||||
from model_routing.route import Route
|
||||
|
||||
LOW = Rung("claude-opus-5-5", "medium")
|
||||
HIGH = Rung("claude-opus-5-5", "high")
|
||||
HIGH = Rung("claude-fable-5-2", "low")
|
||||
LOW_ROUTE = Route(LOW, 7, 3, 4).worker_line()
|
||||
|
||||
|
||||
def message(id_, at, *parts, model="claude-opus-5-5", usage=None):
|
||||
"""Costs $2 on Opus 5.5, whose output is $20 per million tokens, and $1 on Sonnet 5.5, whose output is $10."""
|
||||
"""Costs $2 on Opus 5.5, whose output is $20 per million tokens, $1 on Sonnet 5.5, whose output is $10, and $4 on Fable 5.2, whose output is $40."""
|
||||
return {"type": "assistant", "timestamp": at, "cwd": "/c", "message": {"id": id_, "model": model, "content": list(parts), "usage": usage or {"output_tokens": 100_000}}}
|
||||
|
||||
|
||||
@@ -164,10 +164,10 @@ class Measured(unittest.TestCase):
|
||||
message("m2", "2026-10-14T10:30:00Z", call("Agent", "a1", subagent_type="Explore", model="sonnet")),
|
||||
message("m3", "2026-10-14T10:31:00Z", call("Agent", "a2", subagent_type="Explore")),
|
||||
]
|
||||
subagents = [subagent_of("Explore", "a1", [result("r", "x")]), subagent_of("Explore", "a2", [message("s2", "2026-10-14T10:31:01Z", model="claude-opus-5-5")])]
|
||||
subagents = [subagent_of("Explore", "a1", [result("r", "x")]), subagent_of("Explore", "a2", [message("s2", "2026-10-14T10:31:01Z", model="claude-fable-5-2")])]
|
||||
found = chunk_figures(chunk_of("log", [kept], worker, subagents), "/h")
|
||||
self.assertEqual(found.delegated, {"sonnet at high effort": 1, str(HIGH): 1})
|
||||
self.assertEqual(found.dollars["rung"], {str(LOW): 2.0, str(HIGH): 6.0})
|
||||
self.assertEqual(found.delegated, {"sonnet at low effort": 1, str(HIGH): 1})
|
||||
self.assertEqual(found.dollars["rung"], {str(LOW): 2.0, str(HIGH): 8.0})
|
||||
|
||||
def test_names_a_subagent_by_its_meta_or_else_its_call_or_else_unknown(self):
|
||||
worker = [message("m1", "2026-10-14T10:00:00Z", call("Agent", "a1", subagent_type="Explore"), call("Agent", "a2", subagent_type="Plan"))]
|
||||
|
||||
@@ -217,7 +217,7 @@ class ReviewerRung(unittest.TestCase):
|
||||
# A worker routed to Sonnet 5.5 at high, on a project whose rung is Opus 5.5 at medium, with its reviewers on the project's rung.
|
||||
ENVIRON = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-sonnet-5-5 at high effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "4"}
|
||||
# The same project, with a worker, and so its reviewers, routed above it to Fable 5.1 at max.
|
||||
ABOVE = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-fable-5-1 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "4"}
|
||||
ABOVE = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-fable-5-2 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "4"}
|
||||
|
||||
def refused(self, given: dict, environ: dict[str, str] = ENVIRON) -> list[str]:
|
||||
return rules(reviewer_breaches(Call("Agent", given, "/r"), environ))
|
||||
@@ -260,7 +260,7 @@ class ReviewerRung(unittest.TestCase):
|
||||
|
||||
def test_a_reviewer_on_the_project_rung_or_a_rerun_above_it_passes(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "opus", "effort": "medium"}), [])
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "opus", "effort": "high"}), [])
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "fable", "effort": "low"}), [])
|
||||
|
||||
def test_an_agent_calls_prompt_is_the_text_scanned_for_a_marker(self):
|
||||
call = Call("Agent", {"subagent_type": "prose-reviewer", "prompt": "Review. # rule-override reviewer-rung: trust me"}, "/r")
|
||||
|
||||
@@ -64,7 +64,7 @@ class FirstReply(unittest.TestCase):
|
||||
|
||||
class DownFrom(unittest.TestCase):
|
||||
def test_runs_from_the_projects_rung_down_to_the_floor_highest_first(self):
|
||||
self.assertEqual(down_from(Rung("claude-opus-5-5", "high"), MIDDLE), [Rung("claude-opus-5-5", "high"), REFERENCE, MIDDLE])
|
||||
self.assertEqual(down_from(Rung("claude-fable-5-2", "low"), MIDDLE), [Rung("claude-fable-5-2", "low"), REFERENCE, MIDDLE])
|
||||
self.assertEqual(down_from(REFERENCE, MIDDLE), [REFERENCE, MIDDLE])
|
||||
|
||||
|
||||
|
||||
@@ -17,13 +17,13 @@ class SubagentRung(unittest.TestCase):
|
||||
def test_prints_the_agent_tools_model_and_effort_for_the_rung_an_l_picks(self):
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 3), "model opus, effort medium")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 2), "model sonnet, effort high")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 4), "model opus, effort high")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 4), "model fable, effort low")
|
||||
self.assertEqual(main.agent_args("claude-sonnet-5-5", 1), "model sonnet, effort high")
|
||||
self.assertIsNone(main.agent_args("claude-haiku-5-5", 1))
|
||||
|
||||
def test_a_planning_task_never_runs_below_the_projects_rung(self):
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 1, planning=True), "model opus, effort medium")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 4, planning=True), "model opus, effort high")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 4, planning=True), "model fable, effort low")
|
||||
self.assertEqual(self.run_tool("--plan", "claude-opus-5-5", "2").stdout, "model opus, effort medium\n")
|
||||
|
||||
def test_picks_nothing_for_a_model_off_the_ladder(self):
|
||||
@@ -38,7 +38,7 @@ class SubagentRung(unittest.TestCase):
|
||||
|
||||
def test_the_launcher_prints_the_rung_and_exits_0_on_the_ladder(self):
|
||||
on = self.run_tool("claude-opus-5-5", "4")
|
||||
self.assertEqual((on.returncode, on.stdout, on.stderr), (0, "model opus, effort high\n", ""))
|
||||
self.assertEqual((on.returncode, on.stdout, on.stderr), (0, "model fable, effort low\n", ""))
|
||||
|
||||
def test_the_launcher_prints_nothing_and_says_why_and_exits_0_off_the_ladder(self):
|
||||
off = self.run_tool("claude-opus-5", "4")
|
||||
|
||||
+4
-2
@@ -462,11 +462,11 @@ Only Opus 5.5 was replayed, so the judge of a project on another model keeps the
|
||||
|
||||
Valid while the archived verdicts show what a judge should return, and until a new model joins the ladder: replay then.
|
||||
|
||||
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
|
||||
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.2 at max on top; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
|
||||
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude and by the worker through `bin/subagent-rung`. The judge's rung is set by `routing: on a model judge-replay covered`. `bin/model_routing/ladder.py` holds the rungs and the steps.
|
||||
|
||||
The rungs, least competent first: Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The 2026-10-29 note below took Haiku 5.5 off. They first were: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models, read 2026-10-14), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). That cut drops Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index falls among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scores it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price.
|
||||
The rungs, least competent first: Sonnet 5.5 at high; Opus 5.5 at medium; Fable 5.2 at low and medium; Opus 5.5 at xhigh; Fable 5.2 at high and max. The 2026-11-11 note below put them so. Before it they were Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max, after the 2026-10-29 note took Haiku 5.5 off. They first were: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models, read 2026-10-14), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). That cut drops Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index falls among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scored it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price. Fable 5.2 at max tops the index outright, so that exception has ended.
|
||||
|
||||
A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. A model off the ladder routes nothing: claude runs it at the effort its settings give. A task's L steps its worker from the project's rung: 1 and 2 one down, 3 none, 4 one up. Its E + I steps its reviewers up from the project's rung: 4 or less none, 5 one up, 6 two up, 7 three up, 8 four up. A step past either end stops there. A worker errs more the more it changes and the newer that is, which L scores; review must look harder the further a defect would escape and the more it would break, which E + I scores (goals 1.1 and 1.4). Size raises L, so a large change runs its worker higher. The steps spread the values this repo's plan held when rescored on 2026-10-16, L 1 to 4 and E + I 2 to 7, so a worker on most items runs at or below the project's rung, and a review climbs only where a defect would escape far or break much. lilleman approved the steps on 2026-10-16 as a guess. `score-outcomes` reports how well L, E and I each order finished chunks by review rounds and 🔴 findings. A change to the ladder or the steps is lilleman's.
|
||||
|
||||
@@ -480,6 +480,8 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
|
||||
|
||||
2026-11-08, lilleman kept the steps, starting the re-tune with 54 `item-*` branches merged since the steps took effect, 36 of them chunks that each shipped one item scored with L, E and I, in place of waiting for 50 such chunks. Mean review rounds past each pass's first rose with L: 1.57 at L 1, 2.15 at L 2, 2.75 at L 3 and 3.50 at L 4. E + I ordered neither rounds nor 🔴 findings between 3 and 5, and no chunk scored an E + I of 7 or more, so nothing supports changing the steps for E + I 6 to 8. L 1 and L 2 share the Sonnet 5.5 rung yet differ in 🔴 (1 of 7 chunks against 6 of 12), so a 🔴 follows the item more than the rung, and nothing moves L 2. `todo.md` item 273 re-tunes the E + I steps once 10 chunks score an E + I of 6 or more.
|
||||
|
||||
2026-11-11, lilleman approved the rungs for Fable 5.2, from the index read that day, as score and cost per task. Sonnet 5.5: low 26 $0.51, medium 31 $0.78, high 42 $1.64, xhigh 50 $4.09, max 55 $10.02. Opus 5.5: low 35 $0.98, medium 48 $2.45, high 52 $3.46, xhigh 57 $6.05, max 59 $10.04. Fable 5.2: low 54 $3.10, medium 56 $3.80, high 62 $9.08, xhigh 62 $14.83, max 65 $26.54. The same cut drops Opus 5.5 at high, since Fable 5.2 at low scores more for less; Opus 5.5 at max, since Fable 5.2 at high scores more for less; and Fable 5.2 at xhigh, which ties Fable 5.2 at high at a higher cost. Sonnet 5.5 at high stays the bottom rung, as the 2026-10-29 note chose, though with Haiku 5.5 off the cut alone would keep cheaper rungs below it. On an Opus 5.5 project the steps now reach: an L 4 worker, an E + I 5 review and the first escalation run on Fable 5.2 at low; E + I 6 on Fable 5.2 at medium; E + I 7 on Opus 5.5 at xhigh; E + I 8 on Fable 5.2 at high. Only a climb past that reaches Fable 5.2 at max. A Fable 5.2 project's rung is Fable 5.2 at low, so its L 1 and L 2 workers run on Opus 5.5 at medium.
|
||||
|
||||
Valid while the index ranks these models, and while no newer model joins the catalog.
|
||||
|
||||
## routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds
|
||||
|
||||
+11
-10
@@ -794,18 +794,19 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<div class="eyebrow">Right model for the job</div>
|
||||
<h2>The model ladder</h2>
|
||||
<p class="lead">Easy work runs on a cheaper model. Risky work gets a stronger one, and stronger reviewers.</p>
|
||||
<div class="diagram" role="img" aria-label="Six rungs from cheapest to strongest: Sonnet 5.5 high, Opus 5.5 medium, high, xhigh, max, Fable 5.1 max.">
|
||||
<div class="diagram" role="img" aria-label="Seven rungs from cheapest to strongest: Sonnet 5.5 high, Opus 5.5 medium, Fable 5.2 low, Fable 5.2 medium, Opus 5.5 xhigh, Fable 5.2 high, Fable 5.2 max.">
|
||||
<svg viewBox="0 0 960 300">
|
||||
<defs><marker id="ah7" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path class="head" d="M0,0 L10,5 L0,10 z"/></marker></defs>
|
||||
<g>
|
||||
<rect class="box" x="20" y="230" width="140" height="50" rx="5"/><text class="t" x="34" y="252" style="font-size:15px">Sonnet 5.5</text><text class="s" x="34" y="270">high</text>
|
||||
<rect class="worker" x="174" y="192" width="140" height="88" rx="5"/><text class="t" x="188" y="214" style="font-size:15px">Opus 5.5</text><text class="s" x="188" y="232">medium</text>
|
||||
<rect class="box" x="328" y="154" width="140" height="126" rx="5"/><text class="t" x="342" y="176" style="font-size:15px">Opus 5.5</text><text class="s" x="342" y="194">high</text>
|
||||
<rect class="box" x="482" y="116" width="140" height="164" rx="5"/><text class="t" x="496" y="138" style="font-size:15px">Opus 5.5</text><text class="s" x="496" y="156">xhigh</text>
|
||||
<rect class="box" x="636" y="78" width="140" height="202" rx="5"/><text class="t" x="650" y="100" style="font-size:15px">Opus 5.5</text><text class="s" x="650" y="118">max</text>
|
||||
<rect class="box" x="790" y="40" width="150" height="240" rx="5"/><text class="t" x="804" y="62" style="font-size:15px">Fable 5.1</text><text class="s" x="804" y="80">max</text>
|
||||
<rect class="box" x="20" y="230" width="120" height="50" rx="5"/><text class="t" x="32" y="252" style="font-size:15px">Sonnet 5.5</text><text class="s" x="32" y="270">high</text>
|
||||
<rect class="worker" x="154" y="198" width="120" height="82" rx="5"/><text class="t" x="166" y="220" style="font-size:15px">Opus 5.5</text><text class="s" x="166" y="238">medium</text>
|
||||
<rect class="box" x="288" y="166" width="120" height="114" rx="5"/><text class="t" x="300" y="188" style="font-size:15px">Fable 5.2</text><text class="s" x="300" y="206">low</text>
|
||||
<rect class="box" x="422" y="134" width="120" height="146" rx="5"/><text class="t" x="434" y="156" style="font-size:15px">Fable 5.2</text><text class="s" x="434" y="174">medium</text>
|
||||
<rect class="box" x="556" y="102" width="120" height="178" rx="5"/><text class="t" x="568" y="124" style="font-size:15px">Opus 5.5</text><text class="s" x="568" y="142">xhigh</text>
|
||||
<rect class="box" x="690" y="70" width="120" height="210" rx="5"/><text class="t" x="702" y="92" style="font-size:15px">Fable 5.2</text><text class="s" x="702" y="110">high</text>
|
||||
<rect class="box" x="824" y="38" width="120" height="242" rx="5"/><text class="t" x="836" y="60" style="font-size:15px">Fable 5.2</text><text class="s" x="836" y="78">max</text>
|
||||
</g>
|
||||
<text class="lbl" x="188" y="266" style="font-weight:700;fill:var(--worker)">project's rung</text>
|
||||
<text class="lbl" x="166" y="266" style="font-weight:700;fill:var(--worker)">project's rung</text>
|
||||
<text class="lbl" x="20" y="24" style="font-weight:700">CHEAPER</text>
|
||||
<text class="lbl" x="940" y="24" text-anchor="end" style="font-weight:700">STRONGER</text>
|
||||
<path class="edge" d="M100,24 L860,24" marker-end="url(#ah7)"/>
|
||||
@@ -1005,7 +1006,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
</tbody></table></div>
|
||||
<h3 class="group">Models</h3>
|
||||
<div class="tablewrap"><table class="src-t"><colgroup><col style="width:4%"><col style="width:24%"><col style="width:72%"></colgroup><tbody>
|
||||
<tr id="src-25"><td>25</td><td><b>The model ladder, and no Haiku</b></td><td><span class="kind r">Benchmark</span> The rungs follow the Artificial Analysis Intelligence Index, except Fable 5.1 at max: the index scores it below Opus 5.5 at high, and the human put it on top for the hardest tasks.<br><span class="kind m">Own measurement</span> A chunk on Haiku 5.5 ran 2 h 17 min and wrote 3.1M tokens. The chunk before it, on Sonnet 5.5, took 29 min and 111k. Chunks on Haiku took 7.8 extra review rounds, against 2.8 on Sonnet. So Haiku left the ladder.</td></tr>
|
||||
<tr id="src-25"><td>25</td><td><b>The model ladder, and no Haiku</b></td><td><span class="kind r">Benchmark</span> The rungs follow the Artificial Analysis Intelligence Index: a rung stays only where no cheaper one scores as high. Fable 5.2 at max scores highest, so it is on top.<br><span class="kind m">Own measurement</span> A chunk on Haiku 5.5 ran 2 h 17 min and wrote 3.1M tokens. The chunk before it, on Sonnet 5.5, took 29 min and 111k. Chunks on Haiku took 7.8 extra review rounds, against 2.8 on Sonnet. So Haiku left the ladder.</td></tr>
|
||||
<tr id="src-26"><td>26</td><td><b>Judge and reviewers never run below the project's model</b></td><td><span class="kind m">Own measurement</span> 39 judge verdicts were replayed. The reference judge reproduced 31; on those, Opus 5.5 at medium missed none, and Sonnet 5.5 mislabelled 3 interrupts. A replay of review rounds: Sonnet 5.5 missed 44–55% of what the project's model found.</td></tr>
|
||||
</tbody></table></div>
|
||||
<h3 class="group">The machine</h3>
|
||||
@@ -1028,7 +1029,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<dt>Pass</dt><dd>One kind of review, such as Stability. It runs in rounds.</dd>
|
||||
<dt>Round</dt><dd>One review by one reviewer. At most two per pass per chunk, five in the release review.</dd>
|
||||
<dt>Gate</dt><dd>The checks a PR must pass to merge: tests, CI, plan format.</dd>
|
||||
<dt>Rung</dt><dd>A model at an effort level, such as "Opus 5.5 at high".</dd>
|
||||
<dt>Rung</dt><dd>A model at an effort level, such as "Opus 5.5 at xhigh".</dd>
|
||||
<dt>Release review</dt><dd>A release's last chunk: whole-project review before you cut it.</dd>
|
||||
<dt>Worktree</dt><dd>A separate git checkout per chunk, so your main checkout stays untouched.</dd>
|
||||
</dl>
|
||||
|
||||
+1
-1
@@ -39,7 +39,7 @@
|
||||
"includeCoAuthoredBy": false,
|
||||
"mcpServers": {},
|
||||
"modelSettings": {
|
||||
"claude-fable-5-1": {
|
||||
"claude-fable-5-2": {
|
||||
"effortLevel": "high"
|
||||
}
|
||||
},
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 292 | 0.8.0 | | owner | **Rebuild the routing ladder for Fable 5.2, placing each of its efforts by the Artificial Analysis Intelligence Index beside Opus 5.5's and Sonnet 5.5's, and replace Fable 5.1 wherever it is named.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 4, 1.4 by 3 | 2.4 |
|
||||
| 320 | 0.8.0 | | | **Bench a worker below the project's rung with the advisor tool against the same worker alone and the project's rung, and file the switch if it wins.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3, 1.1 by 2 | 2.0 |
|
||||
| 315 | 0.8.0 | | | **Start each reviewer under `agents/` without `CLAUDE.md`, and hand it the vocabulary, the prose rules and the project's decision titles.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 6 | 2.0 |
|
||||
| 312 | 0.8.0 | | | **Research the tooling agents read, search and run commands with, and file an item for each tool that cuts tokens a replay shows keep quality.** | 1 | 2 | 1 | 2 | 3 | 1.4 by 3, 1.5 by 2 | 2.0 |
|
||||
@@ -73,16 +72,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 292. Rebuild the routing ladder for Fable 5.2, placing each of its efforts by the Artificial Analysis Intelligence Index beside Opus 5.5's and Sonnet 5.5's, and replace Fable 5.1 wherever it is named.
|
||||
|
||||
Fable 5.2 (`claude-fable-5-2`) has been released. The ladder in `bin/model_routing/ladder.py` tops out at `Rung("claude-fable-5-1", "max")`, and the decision `routing: one ladder of (model, effort) rungs` holds "while no newer model joins the catalog", so it is due for a rebuild.
|
||||
|
||||
Read the index again (https://artificialanalysis.ai/leaderboards/models), at each effort it scores, with its cost per task. Rebuild the ladder by the decision's cut: a rung stays only where no cheaper rung scores as high. Fable 5.2 may earn more than one rung, and a cheaper effort of it may push an Opus 5.5 rung off. Where the order changes which rung a project's model starts on, or how far the L and E + I steps reach, say so in the decision. The rungs are the human's choice (`CLAUDE.md` → "Working loop", "Ask, don't guess"), so propose the new ladder with the index's figures and get the human's approval before changing code.
|
||||
|
||||
Then replace `claude-fable-5-1` where it is named: `ladder.py`, `bin/routing_figures/prices.py` (Fable 5.2's prices), `settings.json` → `modelSettings`, the tests in `bin/model_routing/`, `bin/loop_claude/`, `bin/judge_replay/` and `bin/rule_hook/`, `docs/decisions.md` and `presentation/index.html`. Keep Fable 5.1 in `prices.py` while archived chunks that ran on it are still priced from there.
|
||||
|
||||
The index, read 2026-11-11, as score and cost per task, Haiku 5.5 left out since it is off the ladder: Sonnet 5.5 at low 26 $0.51, medium 31 $0.78, high 42 $1.64, xhigh 50 $4.09, max 55 $10.02; Opus 5.5 at low 35 $0.98, medium 48 $2.45, high 52 $3.46, xhigh 57 $6.05, max 59 $10.04; Fable 5.2 at low 54 $3.10, medium 56 $3.80, high 62 $9.08, xhigh 62 $14.83, max 65 $26.54. The cut, from Sonnet 5.5 at high up, keeps Sonnet 5.5 at high, Opus 5.5 at medium, Fable 5.2 at low, Fable 5.2 at medium, Opus 5.5 at xhigh, Fable 5.2 at high and Fable 5.2 at max. It drops Opus 5.5 at high, under Fable 5.2 at low; Opus 5.5 at max, under Fable 5.2 at high; and Fable 5.2 at xhigh, which ties Fable 5.2 at high at a higher cost. Proposed to lilleman on 2026-11-11, awaiting approval.
|
||||
|
||||
### 320. Bench a worker below the project's rung with the advisor tool against the same worker alone and the project's rung, and file the switch if it wins.
|
||||
|
||||
Anthropic reports Sonnet 4.6 with an Opus 4.6 advisor scored 2.7 points more on SWE-bench Multilingual at 11.9% less cost than Sonnet alone (the vendor's own run; https://claude.com/blog/the-advisor-strategy). The docs say `/advisor` works with `-p`, but not outright that the advisor is called during a `-p` run, so probe that first. Each advisor call hands the advisor the whole conversation, so count its cost. Bench on `workflow-bench` with archived L 1 and L 2 items: Sonnet with the advisor, Sonnet alone, Opus at medium. lilleman approved on 2026-11-11.
|
||||
|
||||
Reference in New Issue
Block a user