Merge remote-tracking branch 'origin/main' into item-255
# Conflicts: # CHANGELOG.md
This commit is contained in:
@@ -24,7 +24,8 @@ Decisions live in `docs/decisions.md`:
|
||||
- reading cost: a rise is the last 5 chunks' median above the 5 before them, and files or refreshes one plan item
|
||||
- reading cost: larv-review measures each finished chunk from loop-claude's archive and appends the figure to `docs/reading-cost.md` in the next chunk's PR
|
||||
- review: `/security-review` is the security pass of record, except where the full scan runs on the same change and replaces it
|
||||
- review: a fix for a 🔴 or 🟡 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in `larv-review` §3's re-entry table the fix meets; from round 2 a pass reviews only the answers to its last round's findings and any other pass's fix that re-entered it; a fix for a 🟢, a ❓ or a prose finding reruns none
|
||||
- review: a comment is prose in any file; a branch changing only Markdown no agent follows runs only the prose pass and, for README or docs an outside reader lands on, the product-owner pass; plan rows a chunk files from a finding, re-scores or moves get no review
|
||||
- review: a fix for a 🔴 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in `larv-review` §3's re-entry table the fix meets; from round 2 a pass reviews only the answers to its last round's findings and any other pass's fix that re-entered it; a fix for a 🟡, a 🟢, a ❓ or a prose finding reruns none
|
||||
- review: a pass gets at most two rounds per chunk, with no third and no ask for one; the fixes answering round 2 merge once the gate passes
|
||||
- review: a pass skips only on a precedent: a decision, or a plan item crediting the pass, that the repo held before the chunk began
|
||||
- review: a project decision may opt every chunk out of the architect, the product-owner pass or the comprehension check, and a project opting out of the comprehension check voids code health
|
||||
@@ -38,7 +39,7 @@ Decisions live in `docs/decisions.md`:
|
||||
- review: the premises chunk's whole review is the human's approval
|
||||
- review: the principles audit runs on every chunk that changes code, config or tests, reads the project whole with one auditor checking the goals and every principles file that applies, and runs on no other chunk
|
||||
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
|
||||
- routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass; the seats and the security pass always on the project's
|
||||
- routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass or the branch changes only Markdown no agent follows, which reviews on the project's rung; the seats and the security pass always on the project's
|
||||
- routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and only where the E + I steps no review up and the worker runs no higher than the project's rung, then even below the worker's; no pass has such a rung
|
||||
- routing: a worker delegates a task only where it needs just the task's conclusion, and each subagent runs on the rung its task's L picks from the project's rung, never below the project's rung when it files or scores todo.md items
|
||||
- routing: a worker hands a task one rung up on each escalation trigger, up to two rungs above the rung its item picks unless the human agrees, and opens the subagent's prompt with the trigger so the routing figures count it
|
||||
|
||||
@@ -2,6 +2,13 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Haiku 5.5 is off the routing ladder, which now runs from Sonnet 5.5 at high to Fable 5.1 at max. A Sonnet 5.5 project's L 1 and L 2 workers now run on Sonnet 5.5 at high, where they ran on Haiku 5.5 at max: such a chunk took more than two hours and 3.1 million output tokens on a README change. A review replay can now lower a pass only to Sonnet 5.5 at high. `subagent-rung`, `review-replay` and `judge-replay` treat Haiku 5.5 as a model off the ladder.
|
||||
- larv-review runs fewer reviews on changes that hold no code:
|
||||
- A fix for a 🟡 no longer reruns the pass that found it. Only a 🔴 fix does, with each pass whose re-entry row the fix meets.
|
||||
- A comment counts as prose in every file, config included, so a comment in a workflow or compose file fires only the prose pass. Markdown an agent follows counts as config: `CLAUDE.md`, `AGENTS.md`, and files under `agents/`, `principles/`, `rules/` or `skills/`.
|
||||
- A branch that changes only other Markdown, such as `README.md`, `docs/` and `todo.md`, runs the prose pass, and the product-owner pass where an outside reader lands on the change. Its reviewers run on the project's rung, whatever its items' E + I. loop-claude's prompt says so, and rule-hook's `reviewer-rung` allows it.
|
||||
- A `todo.md` row or detail that a chunk files from a finding, re-scores or moves between releases is no longer reviewed. The chunk that ships the item reads it.
|
||||
|
||||
- New tool `loop-inbox` lists the interrupts of every loop-claude on the machine, oldest first, in one terminal. Each entry shows the loop's checkout, how long it has waited and the question's first lines. Pick one to answer the question as on the loop's terminal, except that Enter alone goes back; `/talk` opens claude in loop-inbox's terminal. Of a pasted answer, only the first line is sent, and the rest is dropped. The loop takes the first answer, from its terminal or loop-inbox, so it works as before with no loop-inbox running. A loop that ends drops off the list, even one killed outright. A loop whose claude is already open, such as for a release ready to cut or the routing calls, is listed but answered in that claude. A loop asking for its project's model, as on a project's first run, asks on its own terminal only. Below the questions, loop-inbox lists every running loop, with the ID and tagline of each `todo.md` item its chunk takes. Link it onto `PATH` as README → Install shows.
|
||||
- Ctrl-C in a claude that `loop-claude` opens on its terminal, for `/talk`, a release ready to cut or the routing calls, now reaches only that claude. Before, it also stopped `loop-claude`, which closed that claude a quarter second later.
|
||||
- `reading-cost` now lists the files a chunk read in the worktree `chunk-preflight` made, `<checkout>-item-<ID>`. It listed none, since no transcript command names that worktree.
|
||||
|
||||
@@ -175,9 +175,9 @@ class Main(unittest.TestCase):
|
||||
self.assertIn("usage limit: You've hit your limit", err)
|
||||
for argv in ([], ["off-ladder"], ["claude-opus-5-5", "x"]):
|
||||
self.assertEqual(self.run_main(argv, unused)[0], 1)
|
||||
code, _, err = self.run_main(["claude-haiku-5-5"], unused)
|
||||
code, _, err = self.run_main(["claude-fable-5-1"], unused)
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("no archived verdicts of projects on claude-haiku-5-5", err)
|
||||
self.assertIn("no archived verdicts of projects on claude-fable-5-1", err)
|
||||
(self.home / ".claude" / "loop-claude-models.json").write_text("[")
|
||||
self.assertEqual(self.run_main(["claude-opus-5-5"], unused)[0], 1)
|
||||
|
||||
|
||||
@@ -58,9 +58,10 @@ def review_launches(route: Route, project: Rung | None) -> list[str]:
|
||||
- the rung of each pass a replay lowered;
|
||||
- where the security pass runs on another rung than the worker's, that a full scan runs only from its subagent;
|
||||
- where the worker runs below the project's rung, where to launch a subagent that files or scores todo.md items;
|
||||
- the reviewers' rung each higher E + I picks.
|
||||
- the reviewers' rung each higher E + I picks;
|
||||
- where the reviewers run above the project's rung, that a branch changing only Markdown no agent follows reviews on it, as rule-hook's reviewer-rung allows.
|
||||
|
||||
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project, routed by item 3 of E + I 5, they open: `Launch each reviewer with the Agent tool set to model opus, effort high, the rung item 3's E + I of 5 picks.` and `Launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.` Where a replay lowered the prose pass to Haiku 5.5 at low, and the E + I steps no review up, a sentence follows: `Launch each prose-reviewer instead with the Agent tool set to model haiku, effort low, the rung the replay of its pass chose.`
|
||||
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project, routed by item 3 of E + I 5, they open: `Launch each reviewer with the Agent tool set to model opus, effort high, the rung item 3's E + I of 5 picks.` and `Launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.` Where a replay lowered the prose pass to Sonnet 5.5 at high, and the E + I steps no review up, a sentence follows: `Launch each prose-reviewer instead with the Agent tool set to model sonnet, effort high, the rung the replay of its pass chose.`
|
||||
"""
|
||||
if project is None:
|
||||
return []
|
||||
@@ -68,7 +69,8 @@ def review_launches(route: Route, project: Rung | None) -> list[str]:
|
||||
lowered = [f"Launch each {REVIEWERS[kind]} instead with the Agent tool set to {rung.agent_args()}, the rung the replay of its pass chose." for kind, rung in choice.lowered.items()]
|
||||
scan = ["Run a full scan only from such a subagent, since the scan's agents run on the model that starts them."] if choice.security != route.rung else []
|
||||
planning = [f"Your rung is below the project's: launch each subagent that files or scores todo.md items on the rung `~/.claude/bin/subagent-rung --plan {project.model} <L>` prints."] if position(route.rung) < position(project) else []
|
||||
return [*shared_launches(choice, route, project), *lowered, *scan, *planning, *taken_in_launches(choice)]
|
||||
markdown = [f"Where your branch changes only Markdown, none of it `CLAUDE.md`, `AGENTS.md` or under `agents/`, `principles/`, `rules/` or `skills/`, launch {EACH_REVIEWER} on the project's rung instead, with the Agent tool set to {project.agent_args()}."] if position(choice.reviewers) > position(project) else []
|
||||
return [*shared_launches(choice, route, project), *lowered, *scan, *planning, *taken_in_launches(choice), *markdown]
|
||||
|
||||
|
||||
def shared_launches(choice: ReviewChoice, route: Route, project: Rung) -> list[str]:
|
||||
@@ -127,7 +129,7 @@ def escalation(route: Route, project: Rung | None) -> list[str]:
|
||||
def reruns(route: Route, project: Rung | None) -> list[str]:
|
||||
"""The sentences naming the rung one above a review's, on which it reruns when it asks: shared_reruns' for a reviewer and a seat, then one for each pass a replay lowered. None off the ladder.
|
||||
|
||||
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project whose prose pass a replay lowered to Haiku 5.5 at low, routed by an item of E + I 4, they read: `A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`.` and `Rerun each prose-reviewer that asks with the Agent tool set to `model haiku, effort medium` instead.`
|
||||
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project whose prose pass a replay lowered to Sonnet 5.5 at high, routed by an item of E + I 4, they read: `A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`.` and `Rerun each prose-reviewer that asks with the Agent tool set to `model opus, effort medium` instead.`
|
||||
"""
|
||||
if project is None or route.rung not in LADDER:
|
||||
return []
|
||||
|
||||
@@ -1388,6 +1388,7 @@ class TypedAnswers(unittest.TestCase):
|
||||
class WorkerArgs(unittest.TestCase):
|
||||
SCAN = " Run a full scan only from such a subagent, since the scan's agents run on the model that starts them."
|
||||
SEATS = " Launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung."
|
||||
MARKDOWN = " Where your branch changes only Markdown, none of it `CLAUDE.md`, `AGENTS.md` or under `agents/`, `principles/`, `rules/` or `skills/`, launch each reviewer on the project's rung instead, with the Agent tool set to model opus, effort medium."
|
||||
# What the preflight found, as the fresh prompt quotes it.
|
||||
FACTS = "- A preflight fact."
|
||||
|
||||
@@ -1425,7 +1426,7 @@ class WorkerArgs(unittest.TestCase):
|
||||
def test_launches_reviewers_on_the_rung_the_items_e_plus_i_picks_above_the_workers(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
level = prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 6), opus, self.FACTS)
|
||||
self.assertIn("whether or not it is item 3. Launch each reviewer with the Agent tool set to model opus, effort xhigh, the rung item 3's E + I of 6 picks. Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8. When an escalation trigger", level)
|
||||
self.assertIn(f"whether or not it is item 3. Launch each reviewer with the Agent tool set to model opus, effort xhigh, the rung item 3's E + I of 6 picks. Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort max` from 7 and `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", level)
|
||||
below = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 5), opus, self.FACTS)
|
||||
self.assertIn(f"whether or not it is item 3. Launch each reviewer with the Agent tool set to model opus, effort high, the rung item 3's E + I of 5 picks.{self.SEATS}{self.SCAN} Your rung is below the project's:", below)
|
||||
self.assertNotIn("Launch each reviewer", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 6), opus))
|
||||
@@ -1441,17 +1442,17 @@ class WorkerArgs(unittest.TestCase):
|
||||
|
||||
def test_launches_reviewers_on_the_rung_the_highest_e_plus_i_taken_in_picks(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
self.assertIn(f"loop-claude runs you on claude-opus-5-5 at medium effort. Launch each reviewer with the Agent tool set to model opus, effort max, the rung picked by an item of E + I 7 taken into the chunk. Where an item you take into the chunk has an E + I of 8 or more, launch each reviewer with the Agent tool set to `model fable, effort max` from 8. When an escalation trigger", prompts.resumed("", Route(opus, 3, 3, 5, escape_and_impact_taken_in=7), opus))
|
||||
self.assertIn(f"loop-claude runs you on claude-opus-5-5 at medium effort. Launch each reviewer with the Agent tool set to model opus, effort max, the rung picked by an item of E + I 7 taken into the chunk. Where an item you take into the chunk has an E + I of 8 or more, launch each reviewer with the Agent tool set to `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", prompts.resumed("", Route(opus, 3, 3, 5, escape_and_impact_taken_in=7), opus))
|
||||
self.assertIn(" Launch each reviewer with the Agent tool set to model opus, effort high, the rung picked by an item of E + I 5 taken into the chunk.", prompts.resumed("", Route(opus, escape_and_impact_taken_in=5), opus))
|
||||
|
||||
def test_keeps_seats_and_the_security_pass_on_the_projects_rung_while_the_worker_runs_above_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
above = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-opus-5-5", "high"), 3, 4, 4), opus, self.FACTS)
|
||||
stepped_up = prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus)
|
||||
self.assertIn(f"{prompts.ANSWERS_RECORDED}. loop-claude runs you on claude-opus-5-5 at high effort.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8. When an escalation trigger", stepped_up)
|
||||
self.assertIn(f"{prompts.ANSWERS_RECORDED}. loop-claude runs you on claude-opus-5-5 at high effort.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", stepped_up)
|
||||
self.assertNotIn("the project's", prompts.resumed("", Route(opus, 3, 3, 4), opus))
|
||||
self.assertNotIn("runs you on", prompts.resumed("", Route(Rung("m")), None))
|
||||
self.assertIn(f"whether or not it is item 3.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8. When an escalation trigger", above)
|
||||
self.assertIn(f"whether or not it is item 3.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 6, `model opus, effort max` from 7 and `model fable, effort max` from 8.{self.MARKDOWN} When an escalation trigger", above)
|
||||
|
||||
def test_names_the_rungs_an_escalation_trigger_lets_the_worker_hand_a_task_to_at_most_two_above_its_items(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
@@ -1473,17 +1474,24 @@ class WorkerArgs(unittest.TestCase):
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus))
|
||||
self.assertNotIn("Where the hard part", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 3, 4), opus))
|
||||
|
||||
def test_lets_a_markdown_only_branch_review_on_the_projects_rung_where_the_reviewers_run_above_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
markdown = " Where your branch changes only Markdown, none of it `CLAUDE.md`, `AGENTS.md` or under `agents/`, `principles/`, `rules/` or `skills/`, launch each reviewer on the project's rung instead, with the Agent tool set to model opus, effort medium."
|
||||
self.assertIn(markdown, prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 7), opus, self.FACTS))
|
||||
self.assertIn(markdown, prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 4, 4), opus))
|
||||
self.assertNotIn("only Markdown", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus, self.FACTS))
|
||||
|
||||
def test_launches_each_pass_a_replay_lowered_on_its_replays_rung_and_reruns_it_one_above(self):
|
||||
opus, haiku = Rung("claude-opus-5-5", "medium"), Rung("claude-haiku-5-5", "low")
|
||||
with unittest.mock.patch.dict(ladder.REPLAYED, {("prose", "claude-opus-5-5"): haiku}):
|
||||
below = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 4), opus, self.FACTS)
|
||||
on_it = prompts.resumed("", Route(haiku, 3, 1, 4), opus)
|
||||
launched = " Launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung. Launch each prose-reviewer instead with the Agent tool set to model haiku, effort low, the rung the replay of its pass chose."
|
||||
rerun = " A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`. Rerun each prose-reviewer that asks with the Agent tool set to `model haiku, effort medium` instead."
|
||||
opus, sonnet = Rung("claude-opus-5-5", "medium"), Rung("claude-sonnet-5-5", "high")
|
||||
with unittest.mock.patch.dict(ladder.REPLAYED, {("prose", "claude-opus-5-5"): sonnet}):
|
||||
below = prompts.fresh("Do it.", "## Goals", Route(sonnet, 3, 2, 4), opus, self.FACTS)
|
||||
on_it = prompts.resumed("", Route(sonnet, 3, 1, 4), opus)
|
||||
launched = " Launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung. Launch each prose-reviewer instead with the Agent tool set to model sonnet, effort high, the rung the replay of its pass chose."
|
||||
rerun = " A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`. Rerun each prose-reviewer that asks with the Agent tool set to `model opus, effort medium` instead."
|
||||
self.assertIn(f"whether or not it is item 3.{launched}", below)
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model opus, effort high` from 5,", below)
|
||||
self.assertIn(f"{rerun} After your first read", below)
|
||||
self.assertIn(f"loop-claude runs you on claude-haiku-5-5 at low effort.{launched}", on_it)
|
||||
self.assertIn(f"loop-claude runs you on claude-sonnet-5-5 at high effort.{launched}", on_it)
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 5 or more,", on_it)
|
||||
self.assertTrue(on_it.endswith(rerun))
|
||||
|
||||
|
||||
@@ -49,11 +49,6 @@ class Rung:
|
||||
|
||||
# Least competent first.
|
||||
LADDER = (
|
||||
Rung("claude-haiku-5-5", "low"),
|
||||
Rung("claude-haiku-5-5", "medium"),
|
||||
Rung("claude-haiku-5-5", "high"),
|
||||
Rung("claude-haiku-5-5", "xhigh"),
|
||||
Rung("claude-haiku-5-5", "max"),
|
||||
Rung("claude-sonnet-5-5", "high"),
|
||||
Rung("claude-opus-5-5", "medium"),
|
||||
Rung("claude-opus-5-5", "high"),
|
||||
|
||||
@@ -14,10 +14,11 @@ OPUS = Rung("claude-opus-5-5", "medium")
|
||||
|
||||
|
||||
class Ladder(unittest.TestCase):
|
||||
def test_runs_from_haiku_at_low_effort_to_fable_at_max_with_opus_at_medium_between(self):
|
||||
self.assertEqual(LADDER[0], Rung("claude-haiku-5-5", "low"))
|
||||
def test_runs_from_sonnet_at_high_effort_to_fable_at_max_with_opus_at_medium_between(self):
|
||||
self.assertEqual(LADDER[0], Rung("claude-sonnet-5-5", "high"))
|
||||
self.assertEqual(LADDER[1], OPUS)
|
||||
self.assertEqual(LADDER[-1], Rung("claude-fable-5-1", "max"))
|
||||
self.assertEqual(len(LADDER), 11)
|
||||
self.assertEqual(len(LADDER), 6)
|
||||
self.assertEqual(len(set(LADDER)), len(LADDER))
|
||||
|
||||
def test_puts_a_project_on_its_models_cheapest_rung_and_none_off_the_ladder(self):
|
||||
@@ -53,7 +54,7 @@ class Ladder(unittest.TestCase):
|
||||
self.assertEqual(planning_rung(OPUS, 4), Rung("claude-opus-5-5", "high"))
|
||||
|
||||
def test_names_each_rung_by_a_family_the_agent_tool_takes(self):
|
||||
self.assertEqual({rung.agent_args().split(",")[0] for rung in LADDER}, {"model haiku", "model sonnet", "model opus", "model fable"})
|
||||
self.assertEqual({rung.agent_args().split(",")[0] for rung in LADDER}, {"model sonnet", "model opus", "model fable"})
|
||||
|
||||
def test_ranks_a_rung_off_the_ladder_below_every_rung_on_it(self):
|
||||
self.assertEqual(highest(OPUS, LADDER[-1], LADDER[0]), LADDER[-1])
|
||||
@@ -96,9 +97,9 @@ class Ladder(unittest.TestCase):
|
||||
self.assertEqual(Rung.from_agent("opus", "low"), Rung("claude-opus-5-5", "low"))
|
||||
|
||||
def test_puts_reviewers_on_the_highest_of_the_projects_rung_the_workers_and_the_one_e_plus_i_picks(self):
|
||||
haiku, fable = LADDER[4], LADDER[-1]
|
||||
self.assertEqual(review_rungs(OPUS, haiku, 4)["stability"], OPUS)
|
||||
self.assertEqual(review_rungs(OPUS, haiku, 5)["stability"], Rung("claude-opus-5-5", "high"))
|
||||
sonnet, fable = LADDER[0], LADDER[-1]
|
||||
self.assertEqual(review_rungs(OPUS, sonnet, 4)["stability"], OPUS)
|
||||
self.assertEqual(review_rungs(OPUS, sonnet, 5)["stability"], Rung("claude-opus-5-5", "high"))
|
||||
self.assertEqual(review_rungs(OPUS, fable, 5)["stability"], fable)
|
||||
self.assertEqual(review_rungs(OPUS, Rung("claude-opus-5-5", "high"), 7)["stability"], Rung("claude-opus-5-5", "max"))
|
||||
|
||||
@@ -106,14 +107,13 @@ class Ladder(unittest.TestCase):
|
||||
self.assertEqual(review_rungs(OPUS, LADDER[-1], 8), {**dict.fromkeys(REVIEWERS, LADDER[-1]), "seat": OPUS, "security": OPUS})
|
||||
|
||||
def test_runs_a_replayed_pass_on_its_replays_rung_below_the_workers_where_the_e_plus_i_steps_no_review_up_and_the_worker_runs_no_higher_than_the_project(self):
|
||||
sonnet, haiku, opus_high = Rung("claude-sonnet-5-5", "high"), LADDER[0], Rung("claude-opus-5-5", "high")
|
||||
with unittest.mock.patch.dict(ladder.REPLAYED, {("prose", "claude-opus-5-5"): haiku, ("stability", "claude-sonnet-5-5"): haiku}):
|
||||
self.assertEqual(review_rungs(OPUS, LADDER[4], 4), {**dict.fromkeys(REVIEWERS, OPUS), "prose": haiku, "seat": OPUS, "security": OPUS})
|
||||
self.assertEqual(review_rungs(OPUS, OPUS, 4)["prose"], haiku)
|
||||
sonnet, opus_high = Rung("claude-sonnet-5-5", "high"), Rung("claude-opus-5-5", "high")
|
||||
with unittest.mock.patch.dict(ladder.REPLAYED, {("prose", "claude-opus-5-5"): sonnet}):
|
||||
self.assertEqual(review_rungs(OPUS, sonnet, 4), {**dict.fromkeys(REVIEWERS, OPUS), "prose": sonnet, "seat": OPUS, "security": OPUS})
|
||||
self.assertEqual(review_rungs(OPUS, OPUS, 4)["prose"], sonnet)
|
||||
self.assertEqual(review_rungs(OPUS, opus_high, 4)["prose"], opus_high)
|
||||
self.assertEqual(review_rungs(OPUS, sonnet, 5)["prose"], opus_high)
|
||||
self.assertEqual(review_rungs(OPUS, LADDER[-1], 5)["prose"], LADDER[-1])
|
||||
self.assertEqual((review_rungs(sonnet, sonnet, 4)["prose"], review_rungs(sonnet, sonnet, 4)["stability"]), (sonnet, haiku))
|
||||
|
||||
def test_holds_each_replayed_rung_below_its_models_project_rung_on_the_ladder(self):
|
||||
for (kind, model), rung in ladder.REPLAYED.items():
|
||||
|
||||
@@ -7,15 +7,15 @@ from model_routing.ladder import Rung
|
||||
|
||||
class Environment(unittest.TestCase):
|
||||
PROJECT = Rung("claude-opus-5-5", "medium")
|
||||
WORKER = Rung("claude-haiku-5-5", "max")
|
||||
WORKER = Rung("claude-sonnet-5-5", "high")
|
||||
|
||||
def test_the_project_and_the_workers_rungs_and_the_e_plus_i_reviews_take_are_named_in_the_environment(self):
|
||||
self.assertEqual(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5)), {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5"})
|
||||
self.assertEqual(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5)), {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-sonnet-5-5 at high effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5"})
|
||||
self.assertEqual(environment(self.PROJECT, Route(self.WORKER))["LOOP_CLAUDE_ESCAPE_AND_IMPACT"], "0")
|
||||
self.assertEqual(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5, escape_and_impact_taken_in=6))["LOOP_CLAUDE_ESCAPE_AND_IMPACT"], "6")
|
||||
|
||||
def test_a_project_off_the_ladder_is_handed_an_empty_rung_so_none_is_inherited(self):
|
||||
self.assertEqual(environment(None, Route(self.WORKER)), {"LOOP_CLAUDE_PROJECT_RUNG": "", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "0"})
|
||||
self.assertEqual(environment(None, Route(self.WORKER)), {"LOOP_CLAUDE_PROJECT_RUNG": "", "LOOP_CLAUDE_WORKER_RUNG": "claude-sonnet-5-5 at high effort", "LOOP_CLAUDE_ESCAPE_AND_IMPACT": "0"})
|
||||
|
||||
def test_handed_reads_back_what_environment_named(self):
|
||||
self.assertEqual(handed(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5))), (self.PROJECT, self.WORKER, 5))
|
||||
|
||||
@@ -222,9 +222,9 @@ class Main(unittest.TestCase):
|
||||
return code, out.getvalue(), err.getvalue()
|
||||
|
||||
def test_replays_the_models_rounds_on_the_commit_reviewed_from_the_project_rung_down(self):
|
||||
code, out, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0] if rung.model == "claude-haiku-5-5" else [0, 1]})
|
||||
code, out, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0] if rung == MIDDLE else [0, 1]})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE, MIDDLE, CHEAP])
|
||||
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE, MIDDLE])
|
||||
_, reviewed, branch, base, prompt, cwd = self.reviews[0]
|
||||
# The default branch stands where it stood when the chunk began, so the prompt's diff against it shows the branch's change.
|
||||
self.assertEqual((reviewed, branch, base), ("1\n", "x\n", "0\n"))
|
||||
@@ -232,7 +232,7 @@ class Main(unittest.TestCase):
|
||||
self.assertFalse(pathlib.Path(cwd).exists())
|
||||
self.assertIn("The reference reviewer, claude-opus-5-5 at medium effort, found 2 of them.", out)
|
||||
self.assertIn(f" {self.checkout} 2026-10-14T10:00:00Z: 🟡 slow", out)
|
||||
self.assertTrue(out.endswith("finds: claude-sonnet-5-5 at high effort\n"))
|
||||
self.assertTrue(out.endswith("finds: none below the project's rung\n"), out)
|
||||
|
||||
def test_replays_the_prose_pass_with_the_prose_reviewer(self):
|
||||
code, out, err = self.run_main(["prose", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
@@ -243,8 +243,8 @@ class Main(unittest.TestCase):
|
||||
def test_tries_each_rung_down_to_the_ladders_lowest_below_the_lowest_a_worker_routes_to(self):
|
||||
code, out, err = self.run_main(["prose", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE, MIDDLE, CHEAP, *(Rung("claude-haiku-5-5", effort) for effort in ("xhigh", "high", "medium", "low"))])
|
||||
self.assertTrue(out.endswith("finds: claude-haiku-5-5 at low effort\n"), out)
|
||||
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE, MIDDLE])
|
||||
self.assertTrue(out.endswith("finds: claude-sonnet-5-5 at high effort\n"), out)
|
||||
|
||||
def review_folder_with_round_1(self):
|
||||
"""The chunk's review folder, holding round 1's report from before the replayed round began at 10:00, that round's own and a later pass's reports, written after, and an earlier attempt's report, from before the chunk began at 09:15."""
|
||||
@@ -404,9 +404,9 @@ class Main(unittest.TestCase):
|
||||
for argv in ([], ["claude-opus-5-5"], ["full scan", "claude-opus-5-5"], ["stability", "off-ladder"], ["stability", "claude-opus-5-5", "x"]):
|
||||
with self.subTest(argv=argv):
|
||||
self.assertEqual(self.run_main(argv, unused)[0], 1)
|
||||
code, _, err = self.run_main(["stability", "claude-haiku-5-5"], unused)
|
||||
code, _, err = self.run_main(["stability", "claude-sonnet-5-5"], unused)
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("no archived stability round of projects on claude-haiku-5-5", err)
|
||||
self.assertIn("no archived stability round of projects on claude-sonnet-5-5", err)
|
||||
|
||||
def test_prints_the_help(self):
|
||||
code, out, _ = self.run_main(["-h"], unused)
|
||||
|
||||
@@ -1,8 +1,12 @@
|
||||
"""Refuse an Agent call that launches a review below the rung it needs, worked out from the rungs and the E + I loop-claude hands its worker: docs/decisions.md → `routing:`."""
|
||||
import pathlib
|
||||
import typing
|
||||
|
||||
from rule_hook.rules import Breach
|
||||
from rule_hook.call import Call, text
|
||||
from failure import Failure
|
||||
from rule_hook.git import git
|
||||
import git_repo
|
||||
from model_routing.worker_rungs import handed
|
||||
from model_routing.ladder import position, review_rungs, ReviewKind, Rung
|
||||
from review_passes import REVIEWERS
|
||||
@@ -14,6 +18,9 @@ SEAT = "You are one seat of a comprehension panel."
|
||||
# The opening of the security pass's prompt: skills/larv-review/SKILL.md → "Security".
|
||||
SECURITY_OPENER = "You are the security pass for this chunk."
|
||||
SUBJECTS: dict[ReviewKind, str] = {"seat": "a comprehension seat", "security": "the security pass's subagent"}
|
||||
# Markdown an agent follows, which counts as config: Claude Code's memory files, and the folders that hold Claude Code's skills, agents and rules and this repo's principles.
|
||||
INSTRUCTION_FILES = {"CLAUDE.md", "AGENTS.md"}
|
||||
INSTRUCTION_FOLDERS = {"agents", "principles", "rules", "skills"}
|
||||
|
||||
|
||||
def reviewer_breaches(call: Call, environ: typing.Mapping[str, str]) -> list[Breach]:
|
||||
@@ -23,6 +30,9 @@ def reviewer_breaches(call: Call, environ: typing.Mapping[str, str]) -> list[Bre
|
||||
if kind is None or project is None or worker is None:
|
||||
return []
|
||||
required = review_rungs(project, worker, escape_and_impact)[kind]
|
||||
# A branch that changes only Markdown no agent follows holds no code or config for a reviewer on a higher rung to catch a defect in: skills/larv-review/SKILL.md → "Markdown only".
|
||||
if kind in REVIEWERS and position(required) > position(project) and markdown_only(call.cwd):
|
||||
required = project
|
||||
named = named_rung(call.given, worker)
|
||||
if position(named) >= position(required):
|
||||
return []
|
||||
@@ -48,3 +58,21 @@ def named_rung(given: dict[str, typing.Any], worker: Rung) -> Rung:
|
||||
if not model:
|
||||
return Rung(worker.model, effort or worker.effort)
|
||||
return Rung.from_agent(model, effort)
|
||||
|
||||
|
||||
def markdown_only(directory: str) -> bool:
|
||||
"""Whether every file the branch in directory changes, against the default branch as last fetched, is Markdown that no agent follows, counting uncommitted and untracked files; False where it changes none or git cannot say."""
|
||||
default = git_repo.default_ref(directory, guarded=True)
|
||||
base = default if isinstance(default, Failure) else git(directory, "merge-base", default, "HEAD")
|
||||
changed = base if isinstance(base, Failure) else git(directory, "diff", "--name-only", base)
|
||||
untracked = git(directory, "ls-files", "--others", "--exclude-standard")
|
||||
if isinstance(changed, Failure) or isinstance(untracked, Failure):
|
||||
return False
|
||||
paths = [*changed.splitlines(), *untracked.splitlines()]
|
||||
return bool(paths) and all(prose(path) for path in paths)
|
||||
|
||||
|
||||
def prose(path: str) -> bool:
|
||||
"""Whether path names Markdown that no agent follows."""
|
||||
parts = pathlib.PurePosixPath(path).parts
|
||||
return path.endswith(".md") and parts[-1] not in INSTRUCTION_FILES and not INSTRUCTION_FOLDERS & set(parts[:-1])
|
||||
|
||||
@@ -219,11 +219,40 @@ class ReviewerRung(unittest.TestCase):
|
||||
def refused(self, given: dict, environ: dict[str, str] = ENVIRON) -> list[str]:
|
||||
return rules(reviewer_breaches(Call("Agent", given, "/r"), environ))
|
||||
|
||||
def test_a_branch_changing_only_markdown_reviews_on_the_projects_rung_whatever_the_e_plus_i(self):
|
||||
repo = tempfile.mkdtemp()
|
||||
self.addCleanup(subprocess.run, ["rm", "-rf", repo])
|
||||
git(repo, "init", "-q", "-b", "main")
|
||||
git(repo, "commit", "-q", "--allow-empty", "-m", "base")
|
||||
git(repo, "checkout", "-q", "-b", "chunk")
|
||||
pathlib.Path(repo, "docs").mkdir()
|
||||
pathlib.Path(repo, "README.md").write_text("Read me.\n")
|
||||
pathlib.Path(repo, "docs", "decisions.md").write_text("Decided.\n")
|
||||
git(repo, "add", ".")
|
||||
git(repo, "commit", "-q", "-m", "prose")
|
||||
high = self.ENVIRON | {"LOOP_CLAUDE_ESCAPE_AND_IMPACT": "7"}
|
||||
on_project = {"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}
|
||||
self.assertEqual(rules(reviewer_breaches(Call("Agent", on_project, repo), high)), [])
|
||||
self.assertEqual(rules(reviewer_breaches(Call("Agent", {**on_project, "model": "sonnet", "effort": "high"}, repo), high)), ["reviewer-rung"])
|
||||
self.assertEqual(rules(reviewer_breaches(Call("Agent", on_project, repo), self.ABOVE)), [], "a worker that climbed doesn't lift a Markdown-only review either")
|
||||
for instructions in ("CLAUDE.md", "AGENTS.md", "skills/review/SKILL.md", "agents/reviewer.md", "rules/todo.md", "principles/tests.md"):
|
||||
with self.subTest(instructions=instructions):
|
||||
path = pathlib.Path(repo, instructions)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text("Do this.\n")
|
||||
self.assertEqual(rules(reviewer_breaches(Call("Agent", on_project, repo), high)), ["reviewer-rung"], "Markdown an agent follows is config")
|
||||
path.unlink()
|
||||
pathlib.Path(repo, "gate.yml").write_text("on: push\n")
|
||||
self.assertEqual(rules(reviewer_breaches(Call("Agent", on_project, repo), high)), ["reviewer-rung"], "an uncommitted change to another file counts")
|
||||
git(repo, "add", ".")
|
||||
git(repo, "commit", "-q", "-m", "config")
|
||||
self.assertEqual(rules(reviewer_breaches(Call("Agent", on_project, repo), high)), ["reviewer-rung"])
|
||||
|
||||
def test_a_reviewer_below_the_project_rung_is_refused_naming_the_arguments_to_pass(self):
|
||||
breaches = reviewer_breaches(Call("Agent", {"subagent_type": "stability-reviewer", "model": "haiku", "effort": "max", "prompt": "Review."}, "/r"), self.ENVIRON)
|
||||
breaches = reviewer_breaches(Call("Agent", {"subagent_type": "stability-reviewer", "model": "sonnet", "effort": "high", "prompt": "Review."}, "/r"), self.ENVIRON)
|
||||
self.assertEqual(rules(breaches), ["reviewer-rung"])
|
||||
self.assertIn("stability-reviewer", breaches[0].detail)
|
||||
self.assertIn("claude-haiku-5-5 at max effort", breaches[0].detail)
|
||||
self.assertIn("claude-sonnet-5-5 at high effort", breaches[0].detail)
|
||||
self.assertIn("model opus, effort medium", breaches[0].detail)
|
||||
|
||||
def test_a_reviewer_on_the_project_rung_or_a_rerun_above_it_passes(self):
|
||||
@@ -260,10 +289,10 @@ class ReviewerRung(unittest.TestCase):
|
||||
self.assertEqual(self.refused({"prompt": "You are one seat of a comprehension panel.", "model": "sonnet", "effort": "high"}, without), ["reviewer-rung"])
|
||||
|
||||
def test_a_replayed_passs_reviewer_is_held_to_its_replays_rung_below_the_workers_unless_the_e_plus_i_steps_reviews_up_or_the_worker_climbed(self):
|
||||
lowered = {"subagent_type": "prose-reviewer", "model": "haiku", "effort": "medium"}
|
||||
with unittest.mock.patch.dict(ladder.REPLAYED, {("prose", "claude-opus-5-5"): Rung("claude-haiku-5-5", "medium")}):
|
||||
lowered = {"subagent_type": "prose-reviewer", "model": "sonnet", "effort": "high"}
|
||||
with unittest.mock.patch.dict(ladder.REPLAYED, {("prose", "claude-opus-5-5"): Rung("claude-sonnet-5-5", "high")}):
|
||||
self.assertEqual(self.refused(lowered), [])
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "haiku", "effort": "low"}), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "sonnet", "effort": "medium"}), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({**lowered, "subagent_type": "stability-reviewer"}), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused(lowered, self.ENVIRON | {"LOOP_CLAUDE_ESCAPE_AND_IMPACT": "5"}), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused(lowered, self.ABOVE), ["reviewer-rung"])
|
||||
|
||||
@@ -64,7 +64,7 @@ class FirstReply(unittest.TestCase):
|
||||
|
||||
class DownFrom(unittest.TestCase):
|
||||
def test_runs_from_the_projects_rung_down_to_the_floor_highest_first(self):
|
||||
self.assertEqual(down_from(REFERENCE, CHEAP), [REFERENCE, MIDDLE, CHEAP])
|
||||
self.assertEqual(down_from(Rung("claude-opus-5-5", "high"), MIDDLE), [Rung("claude-opus-5-5", "high"), REFERENCE, MIDDLE])
|
||||
self.assertEqual(down_from(REFERENCE, MIDDLE), [REFERENCE, MIDDLE])
|
||||
|
||||
|
||||
|
||||
@@ -18,7 +18,8 @@ class SubagentRung(unittest.TestCase):
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 3), "model opus, effort medium")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 2), "model sonnet, effort high")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 4), "model opus, effort high")
|
||||
self.assertEqual(main.agent_args("claude-haiku-5-5", 1), "model haiku, effort low")
|
||||
self.assertEqual(main.agent_args("claude-sonnet-5-5", 1), "model sonnet, effort high")
|
||||
self.assertIsNone(main.agent_args("claude-haiku-5-5", 1))
|
||||
|
||||
def test_a_planning_task_never_runs_below_the_projects_rung(self):
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 1, planning=True), "model opus, effort medium")
|
||||
|
||||
+13
-3
@@ -162,12 +162,18 @@ Valid while no published standard sets either number, the reading-cost window st
|
||||
|
||||
2026-10-08, lilleman. Serves goal 1.4: run beside the full scan, `/security-review` started seconds apart from it each round and returned a near-identical report. Valid while the full scan covers what `/security-review` reports.
|
||||
|
||||
## review: a fix for a 🔴 or 🟡 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in `larv-review` §3's re-entry table the fix meets; from round 2 a pass reviews only the answers to its last round's findings and any other pass's fix that re-entered it; a fix for a 🟢, a ❓ or a prose finding reruns none
|
||||
## review: a fix for a 🔴 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in `larv-review` §3's re-entry table the fix meets; from round 2 a pass reviews only the answers to its last round's findings and any other pass's fix that re-entered it; a fix for a 🟡, a 🟢, a ❓ or a prose finding reruns none
|
||||
|
||||
2026-10-08, lilleman. Serves goal 1.4. An audit of the four costliest archived chunks found nearly every finding that mattered in a pass's first round; when every fix, a nit's included, re-entered review, later rounds mostly re-checked the last fix, reworded what an earlier round wrote, or contradicted another round on the same line. A round 2 caught real bugs only where a fix changed behaviour, so stability still reruns on every fix that does, and security on every fix that opens an untrusted-input boundary or touches authn/authz, secrets or crypto. Valid while round-2 reviews of nit and prose fixes keep changing nothing a user would notice.
|
||||
|
||||
2026-10-13, lilleman through todo item 97. Serves goals 1.4 and 1.5. From round 2 a pass reviews only how the chunk answered its previous round, each fix and each dismissal with its reason, and reports only one that is plainly wrong or worse than the change it answered. A fix from another pass that re-enters it is still reviewed as a first round would review it, limited to its lines. Reviewing all that changed had a reviewer find new gaps in each round's fixes, so a pass ran to its cap. Valid while round-2 findings outside a round's answers keep changing nothing a user would notice.
|
||||
|
||||
2026-10-29, lilleman. Serves goals 1.4 and 1.5. A 🟡 fix no longer reruns its pass. One project's chunk on 2026-10-29, a README paragraph and plan rows, took 10 review rounds in 45 minutes, and three of its four second rounds answered only 🟡 findings, such as the wording of a question's options. Round 2 found 12 🔴 in about 200 runs since 2026-10-21, mostly in round 1's fixes of 🔴 findings, so a 🔴 fix still reruns its pass. A 🟡 fix that breaks something merges unreviewed by its pass; another pass whose row it meets for a 🔴 fix in the same round still reviews it. Valid while 🔴 findings in later chunks, fixing code a 🟡 fix wrote, stay rare.
|
||||
|
||||
## review: a comment is prose in any file; a branch changing only Markdown no agent follows runs only the prose pass and, for README or docs an outside reader lands on, the product-owner pass; plan rows a chunk files from a finding, re-scores or moves get no review
|
||||
|
||||
2026-10-29, lilleman. Serves goals 1.4 and 1.5. One project's chunk on 2026-10-29 changed a README paragraph, `todo.md` and three lines of `gate.yml`, and the principles, security and stability passes all ran on it, each twice: 6 of the chunk's 10 rounds. A change to config is one that changes what runs, so a comment in it fires only the prose pass, as a comment in code already did. A branch that changes only Markdown no agent follows changes no code, config or tests, so no pass that reads those runs on it. A plan row a chunk files from a finding, re-scores or moves is read by the chunk that ships it, so reviewing it at filing reviewed it twice: that chunk spent a prose round of 4 minutes on its plan change. Markdown an agent follows, such as a skill, is config, since a change there changes what agents do. Valid while defects in prose-only chunks stay rare in later chunks' fixes.
|
||||
|
||||
## review: a pass gets at most two rounds per chunk, with no third and no ask for one; the fixes answering round 2 merge once the gate passes
|
||||
|
||||
2026-10-28, lilleman through todo item 246. Serves goals 1 and 1.4. One counter per pass per chunk, so no fix bounces between two passes without end. lilleman measured every repo's round titles since 2026-10-21. Round 1 ran 357 times and found 59 🔴. Round 2 ran about 200 times and found 12 🔴, mostly in round 1's fixes. Rounds 3 and later ran about 110 times and found no 🔴, at about $125 of review at list prices, and asks for a fourth or fifth round were 24 of the 59 interrupts. The cap had stood at three because a lower cap would ask the human more often; once nothing asks, that reason is gone. A rerun one rung up belongs to the round it reruns, so it doesn't count toward the cap: `review: a reviewer or comprehension seat that rules the work below standard`.
|
||||
@@ -262,7 +268,7 @@ The run has no time cap. Limiting it to the changed lines already bounds it by t
|
||||
|
||||
Valid while a mutation run on a chunk's changed lines stays cheap beside the review itself.
|
||||
|
||||
## routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass; the seats and the security pass always on the project's
|
||||
## routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one picked by the highest E + I among the chunk's items, unless a replay lowered their pass or the branch changes only Markdown no agent follows, which reviews on the project's rung; the seats and the security pass always on the project's
|
||||
|
||||
2026-10-14, lilleman. Serves goals 1.1 and 1.4.
|
||||
|
||||
@@ -274,6 +280,8 @@ On 2026-10-16 lilleman routed the reviewers on their item's E + I as well, on ei
|
||||
|
||||
Valid until the trial on real chunks shows whether a worker below the project's rung holds quality.
|
||||
|
||||
2026-10-29, lilleman. Serves goals 1.4 and 1.5. A branch that changes only Markdown no agent follows reviews on the project's rung, whatever its items' E + I. E + I scores an item before its chunk starts, by the change the item asks for, so one project's item, filed to build a control on a CI runner, kept its E + I of 7 when the chunk found no control the repo could build and shipped a README paragraph: five reviewers read prose on Opus 5.5 at xhigh. rule-hook's `reviewer-rung` reads the branch's files, committed or not, against the default branch as last fetched. Markdown an agent follows, `CLAUDE.md`, `AGENTS.md` and files under `agents/`, `principles/`, `rules/` or `skills/`, counts as config. Valid while a README or docs defect costs a reader no more than review on the project's rung lets through.
|
||||
|
||||
## routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows finding every 🔴 and 🟡 the project's rung finds, and only where the E + I steps no review up and the worker runs no higher than the project's rung, then even below the worker's; no pass has such a rung
|
||||
|
||||
2026-10-15, lilleman proposed running the stability reviewer on the worker's rung, and the agent proposed the replay that gates it. On 2026-10-16 lilleman asked for the prose pass to run on the worker's rung or lower, and, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. Serves goal 1.4 without giving up goal 1.1: a cheaper reviewer saves money only where it misses nothing.
|
||||
@@ -350,7 +358,7 @@ Valid while the archived verdicts show what a judge should return, and until a n
|
||||
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude and by the worker through `bin/subagent-rung`. The judge's rung is set by `routing: on a model judge-replay covered`. `bin/model_routing/ladder.py` holds the rungs and the steps.
|
||||
|
||||
The rungs, least competent first: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models, read 2026-10-14), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). That cut drops Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index falls among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scores it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price.
|
||||
The rungs, least competent first: Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The 2026-10-29 note below took Haiku 5.5 off. They first were: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models, read 2026-10-14), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). That cut drops Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index falls among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scores it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price.
|
||||
|
||||
A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. A model off the ladder routes nothing: claude runs it at the effort its settings give. A task's L steps its worker from the project's rung: 1 and 2 one down, 3 none, 4 one up. Its E + I steps its reviewers up from the project's rung: 4 or less none, 5 one up, 6 two up, 7 three up, 8 four up. A step past either end stops there. A worker errs more the more it changes and the newer that is, which L scores; review must look harder the further a defect would escape and the more it would break, which E + I scores (goals 1.1 and 1.4). Size raises L, so a large change runs its worker higher. The steps spread the values this repo's plan held when rescored on 2026-10-16, L 1 to 4 and E + I 2 to 7, so a worker on most items runs at or below the project's rung, and a review climbs only where a defect would escape far or break much. lilleman approved the first steps on 2026-10-16 as a guess: `score-outcomes` reports how well L, E and I each order finished chunks by review rounds and 🔴 findings, and `todo.md` item 170 re-tunes the steps once 50 chunks have shipped under the steps that hold today. A change to the ladder or the steps is lilleman's.
|
||||
|
||||
@@ -358,6 +366,8 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
|
||||
|
||||
2026-10-23, lilleman: 15 chunks had shipped under today's steps. Over them, L's rank correlation with review rounds was +0.53. They held only 3 🔴 findings, too few to test E and I. 50 chunks hold about 10 at that rate, and a re-tune on fewer chunks would fit noise.
|
||||
|
||||
2026-10-29, lilleman: Haiku 5.5 is off the ladder, so no worker, subagent or review runs on it, and a Sonnet 5.5 project's L 1 and L 2 workers stay on Sonnet 5.5 at high. One project's chunk on 2026-10-29, a README paragraph and plan rows, ran on Haiku 5.5 at max for 2 h 17 min and wrote 3.1 million output tokens; the chunk before it, item 7, a merge gate with tests, took 29 minutes and 111k tokens on Sonnet 5.5 at high. The routing figures had shown the same before: chunks started on Haiku 5.5 at max took 7.8 review rounds past each pass's first against 2.8 on Sonnet 5.5 at high (goals 1.4 and 1.5). A review replay can now lower a pass only to Sonnet 5.5 at high, and on a Sonnet 5.5 project to no rung.
|
||||
|
||||
Valid while the index ranks these models, and while no newer model joins the catalog.
|
||||
|
||||
## routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, or when the judge reports a taken-in item that picks a higher rung or an escalation trigger the worker cites
|
||||
|
||||
@@ -41,9 +41,13 @@ A pass is one kind of review, a row of the table, named by its first column. Its
|
||||
| Overrides | override-reviewer | `~/.claude/bin/rule-hook overrides`, run in the chunk's worktree, prints a line no round of this pass has ruled on |
|
||||
| Stability | stability-reviewer | changes code, config or tests |
|
||||
| Product-owner | product-owner-reviewer | changes an external surface: API, CLI, UI, error or deprecation text a consumer reads, README and docs an outside reader lands on |
|
||||
| Prose | prose-reviewer | changes prose a reader reads, such as docs, comments, help or error text, or the plan; one round |
|
||||
| Prose | prose-reviewer | changes prose a reader reads, such as docs, comments, help or error text, or the plan's text for an item the chunk ships, splits or answers; one round |
|
||||
|
||||
A comment is prose, not code.
|
||||
A comment is prose, not code, in any file. A change to config is one that changes what runs, so a comment in a workflow or compose file fires only the prose pass. Markdown an agent follows is config: `CLAUDE.md`, `AGENTS.md`, and files under `agents/`, `principles/`, `rules/` or `skills/`.
|
||||
|
||||
**Plan rows.** A `todo.md` row or detail that the chunk files from a finding, re-scores or moves between releases gets no review: the chunk that ships the item reads it.
|
||||
|
||||
**Markdown only.** A branch that changes only Markdown no agent follows, such as `README.md`, `docs/` and `todo.md`, changes no code, config or tests. It runs the prose pass, and the product-owner pass where it changes README or docs an outside reader lands on; no other pass. Its reviewers run on the project's rung, whatever its items' E + I; rule-hook's `reviewer-rung` allows that.
|
||||
|
||||
**Waves.** Launch a round's passes in up to three waves. Launch each wave's reviewers in one message: Agent calls in one message run together, so a wave lasts as long as its slowest reviewer. Start a wave only once every pass of the waves before it is done ("Round cap" → "Done"): line-level polish on code the architect wants restructured, or on a call the security pass will rewrite, is wasted work. Skip a wave none of whose passes runs.
|
||||
|
||||
@@ -77,18 +81,18 @@ Give a reviewer without Bash the branch diff and `git ls-files`'s output in its
|
||||
- **Done.** A pass is done when its latest round leaves no 🔴 or 🟡 standing: none was reported, or each was shown factually wrong, declined, filed, or named under the bar per "Findings", or settled in the PR summary per "Comprehension check" → "Places", or, for the prose pass and a pass at its cap, fixed. Resolve that round's 🟢s and ❓s per "Findings" too.
|
||||
- **Cap.** A pass takes at most two rounds per chunk; don't ask the user for a third. Answer its second round's findings per "Findings": those fixes merge once the gate passes, unreviewed by that pass. rule-hook's `round-cap` refuses a round commit that would take a pass past two, and no override waives it. A review the auto-reviewer posts after its second isn't a round: fix none of it and rerun no pass for it, but file a security finding it raises per `~/.claude/CLAUDE.md` → "Working loop", and any other 🔴 in `todo.md`.
|
||||
|
||||
**Re-entry.** A fix is a change no pass has seen. A fix for a 🟢, a ❓ or a prose finding reruns no pass, whatever it changes. A round's fixes for 🔴 and 🟡 findings re-enter together: run every pass that a row below names for any of them, a done pass included, in the waves of "Waves", then §4. A pass at its cap doesn't run again ("Round cap" → "Cap").
|
||||
**Re-entry.** A fix is a change no pass has seen. A fix for a 🟡, a 🟢, a ❓ or a prose finding reruns no pass, whatever it changes. A round's fixes for 🔴 findings re-enter together: run every pass that a row below names for any of them, a done pass included, in the waves of "Waves", then §4. A pass at its cap doesn't run again ("Round cap" → "Cap").
|
||||
|
||||
| The fix | Reruns |
|
||||
|---------|--------|
|
||||
| answers a 🔴 or 🟡 of a pass other than prose | that pass |
|
||||
| answers a 🔴 of a pass other than prose | that pass |
|
||||
| carries an architectural decision | Architecture |
|
||||
| changes code, config or tests | Principles |
|
||||
| opens an untrusted-input boundary or touches authn/authz, secrets or crypto | Security, unless the full scan reruns on it |
|
||||
| made rule-hook log an override | Overrides |
|
||||
| changes behaviour: what the code or its tests do | Stability |
|
||||
|
||||
A fix meets every row it fits. Prose has no row, so a comment or doc edit inside a fix reruns nothing. Example: a fix for a stability 🟡 that changes code and a comment reruns Principles and Stability.
|
||||
A fix meets every row it fits. Prose has no row, so a comment or doc edit inside a fix reruns nothing. Example: a fix for a stability 🔴 that changes code and a comment reruns Principles and Stability.
|
||||
|
||||
**What a round reviews.** An agent under `~/.claude/agents/` reviews the branch in round 1. From round 2 it reviews only the answers to its last round's findings, and each fix from another pass that re-entered its pass. Prompt it with the path of its last round's report and each finding's answer: the fix's diff, or the reason it was dismissed. Give it each other pass's fix as `git diff <the commit before that fix> <the fix's commit>`, since the fixes may be unpushed, with the finding it answered. It reports only an answer that is plainly wrong or worse than the change it answered, so its pass converges in its second round. Other reviewers review the branch as their tool does.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user