Merge pull request 'Route each subagent of a chunk by its rung' (#264) from route-subagents into main
This commit was merged in pull request #264.
This commit is contained in:
@@ -53,6 +53,9 @@
|
||||
!bin/score_outcomes/*.py
|
||||
!bin/shell_command.py
|
||||
!bin/statusline.py
|
||||
!bin/subagent-rung
|
||||
!bin/subagent_rung/
|
||||
!bin/subagent_rung/*.py
|
||||
!bin/test_*.py
|
||||
!bin/todo_table.py
|
||||
!bin/transcript_archive/
|
||||
|
||||
@@ -33,7 +33,7 @@ Decisions live in `docs/decisions.md`:
|
||||
- worker: a chunk's worker writes the simplest change that solves its item, test first wherever a test can check the change, and reads a technical principle only where the item, a finding or its own record names it; larv-review's reviewers hold the change to the principles and to the standards in `skills/larv-review/standards/`
|
||||
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
|
||||
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
|
||||
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model, and a chunk's worker may run below that rung, launching its reviewers on that rung
|
||||
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model, and a chunk's worker may run below that rung, launching its reviewers on that rung; the worker routes each subagent it delegates to by the R + C of the subagent's task
|
||||
|
||||
## Working rules
|
||||
|
||||
|
||||
+2
-1
@@ -2,7 +2,8 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- `loop-claude` routes each chunk's worker on a ladder of model and effort rungs, from Haiku 5.5 at low effort to Fable 5.1 at max, when the project's model is on it. As a chunk starts, the R + C of `todo.md`'s top item in the next release picks the rung: 11–12 picks one rung above the project's rung, so Opus 5.5 at high for an Opus 5.5 project, and so on up to four rungs. An item of R + C 10 or less runs the worker at or below the project's model's cheapest rung, which for Opus 5.5 is medium effort, below the xhigh `settings.json` sets. While the worker runs below the project's model's cheapest rung, it launches every reviewer, comprehension seat and security pass, the full scan included, in a subagent on that rung. It steps up one rung after every second unfinished verdict on a chunk, and to a higher rung when it takes in an item of higher R + C. Each cycle's header, and each finished or unfinished verdict, stop and step-up in the archive, name the worker's rung; a rerun resumes an open chunk on the latest rung archived for it. `docs/decisions.md` → `routing:` gives the ladder, its source and the bands. Expect a chunk on an easy item to cost much less than under xhigh, and perhaps to take more review rounds, and one on an item of R + C 15 or more to cost more.
|
||||
- A session delegates a task to a subagent only when it needs just the task's conclusion, such as a search across many files, and keeps edits in its own thread. It scores the task's R and C as for a `todo.md` item, and launches the subagent on the model and effort that the new `bin/subagent-rung <model> <R+C>` picks from the project's model, such as `model sonnet, effort high`. `CLAUDE.md` states the rule. Expect searches to run on cheaper models, and less of each session's context to fill with search output.
|
||||
- `loop-claude` routes each chunk's worker on a ladder of model and effort rungs, from Haiku 5.5 at low effort to Fable 5.1 at max, when the project's model is on it. As a chunk starts, the R + C of `todo.md`'s top item in the next release picks the rung: 11–12 picks one rung above the project's rung, so Opus 5.5 at high for an Opus 5.5 project, and so on up to four rungs. An item of R + C 10 or less runs the worker at or below the project's model's cheapest rung, which for Opus 5.5 is medium effort, below the xhigh `settings.json` sets. While the worker runs below the project's model's cheapest rung, it launches every reviewer, comprehension seat and security pass, the full scan included, in a subagent on that rung. While it runs above that rung, it launches each comprehension seat and the security pass on that rung. It steps up one rung after every second unfinished verdict on a chunk, and to a higher rung when it takes in an item of higher R + C. Each cycle's header, and each finished or unfinished verdict, stop and step-up in the archive, name the worker's rung; a rerun resumes an open chunk on the latest rung archived for it. `docs/decisions.md` → `routing:` gives the ladder, its source and the bands. Expect a chunk on an easy item to cost much less than under xhigh, and perhaps to take more review rounds, and one on an item of R + C 15 or more to cost more.
|
||||
- A session reads `todo.md` up to `## Details`, and the details of only the items its chunk takes. It reads a `docs/decisions.md` entry only where the entry's title leaves open what it needs. Both rules are in `CLAUDE.md`, so they hold from a session's start. The decisions rule replaces one in `rules/decisions.md`, which loads only once a session has opened `docs/decisions.md` or `AGENTS.md`. A chunk's opening line now counts the unreleased changes instead of listing them. Expect fewer tokens read before a chunk's first commit.
|
||||
- larv-review's stability and product-owner reviewers report only a defect, a requirement of the plan or the goals that the change breaks, a breach of a technical principle, or for stability a hard place the change touches; the stability reviewer leaves the style of comments to the prose reviewer, which now also reads help and error text. Each architect finding names a concrete later change the PR makes harder, and one that names none is dropped. The prose pass runs only on a chunk that changes prose a reader reads, such as docs, comments, help text or `todo.md`, and skips a code-only chunk. Expect fewer findings and fewer review rounds.
|
||||
- Every `todo.md` item carries a C score, how hard the change is to get right, from 1 to 10, in a column after S. C stays out of the score. `rank_todo.py` fails on an items table without the column, and the rule hook then blocks committing that `todo.md`. Before the next commit that changes a project's `todo.md`, add the column and score every item's C, as `rank_todo.py --help` shows.
|
||||
|
||||
@@ -52,6 +52,7 @@ The rules only reviewers apply, for quality, comments, READMEs and prose, live i
|
||||
- Keep the project's main checkout (e.g. `~/code/log`, never a worktree) current: when it is behind `origin` — a merge landed, or a fetch shows it — run `git pull --ff-only` there instead of reporting the lag, never with `--autostash`: a dirty checkout is a finding to report, not to carry along.
|
||||
- Run a project's language runtime, package manager, and task runner through its container/tooling setup, never directly on the host unless a decision puts them there. Example: `docker compose run --rm app <tool> …`.
|
||||
- Tear down test containers after a run.
|
||||
- Delegate a task to a subagent only when you need just its conclusion: a search or read across many files, or work independent enough to run in parallel. Keep edits, and whatever you must check line by line, in your own thread: you would read a delegated diff again to check it. Score the task's R and C as for a `todo.md` item. Launch the subagent with the model and effort that `~/.claude/bin/subagent-rung <model> <R+C>` prints. `<model>` is the project's model id, such as `claude-opus-5-5`: the one loop-claude's prompt names as the project's, or else your own. A reviewer, a comprehension seat or the security pass takes the model and effort loop-claude's prompt gives it, or else none.
|
||||
- Give a subagent only the access its job needs, and state the limits in its prompt. One that only reads, such as a prose review, gets no shell and no edits. One that probes a build tags the image as throwaway and removes it afterwards, writes only to the session scratchpad, and removes no image or file it did not create.
|
||||
- Check `README.md` first for build, test, install, and codegen commands, and fix one that misbehaves in the same change. Refresh `README.md` whenever you change what it describes, keeping it true, not longer.
|
||||
- Where `~/.claude/bin/rule-hook` does not check it, such as an image named in test code, a Kubernetes manifest, a `docker run` line or a file a Bash command writes, pin a container image to its full patch version and a GitHub Action to a semver tag.
|
||||
|
||||
@@ -75,6 +75,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit and Write call, and `rule-hook --help` states the rules and the override.
|
||||
- `score-outcomes [R S C]` lists the finished items whose guessed R, S and C sit nearest the given scores, with what their chunks measured, and each score's median outcome; `score-outcomes --help` states the measures.
|
||||
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
|
||||
- `subagent-rung <model> <R+C>` prints the Agent tool's model and effort for a subagent's task, picked by its R + C from the project's model; `subagent-rung --help` states the rule.
|
||||
|
||||
Test on the host, with `uv` on `PATH` to fetch mypy and the Python version Install names (`docs/decisions.md` → `bin/`): `python3 -m unittest discover -s bin -p 'test_*.py'`
|
||||
|
||||
@@ -91,7 +92,7 @@ pytest_add_cli_args=-k
|
||||
|
||||
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
|
||||
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, and `score_outcomes` imports `reading_cost`'s record path. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude reads. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, and `score_outcomes` imports `reading_cost`'s record path. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool; `model_routing/` holds the routing ladder, which loop-claude and `subagent-rung` read. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
|
||||
|
||||
## Tracking a new file
|
||||
|
||||
|
||||
@@ -21,16 +21,23 @@ def fresh(task: str, premises: str, route: Route, anchor: Rung | None) -> str:
|
||||
|
||||
|
||||
def routed_by(route: Route, anchor: Rung | None) -> str:
|
||||
said = f"loop-claude runs you on {route.rung}, routed by item {route.item}'s R + C of {route.risk_and_complexity}. Name each todo.md item this chunk takes, with its R + C, whether or not it is item {route.item}."
|
||||
if anchor is None or highest(route.rung, anchor) == route.rung:
|
||||
return said
|
||||
# The Agent tool takes a family name, such as opus, and runs that family's latest model; each rung names its family's latest, so the two match.
|
||||
return f"{said} Your rung is below the project's, {anchor}: launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool's model `{anchor.model.split('-')[1]}` and effort `{anchor.effort}`, and run a full scan only from such a subagent, since the scan's agents run on the model that starts them."
|
||||
return f"loop-claude runs you on {route.rung}, routed by item {route.item}'s R + C of {route.risk_and_complexity}. Name each todo.md item this chunk takes, with its R + C, whether or not it is item {route.item}.{off_the_projects_rung(route, anchor)}"
|
||||
|
||||
|
||||
def resumed(unfinished_evidence: str) -> str:
|
||||
def off_the_projects_rung(route: Route, anchor: Rung | None) -> str:
|
||||
"""Which subagents to launch on the project's rung, anchor, while the worker runs off it; empty while it runs on it."""
|
||||
if anchor is None or route.rung == anchor:
|
||||
return ""
|
||||
# Below the project's rung, the reviewers stay on it to hold quality; above it, the seats score every chunk on one rung, and the security pass spends no more than the project chose.
|
||||
below = highest(route.rung, anchor) == anchor
|
||||
launched = "each reviewer, each comprehension seat and the security pass's subagent" if below else "each comprehension seat and the security pass's subagent"
|
||||
return f" Your rung is {'below' if below else 'above'} the project's, {anchor}: launch {launched} with the Agent tool set to {anchor.agent_args()}. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them."
|
||||
|
||||
|
||||
def resumed(unfinished_evidence: str, route: Route, anchor: Rung | None) -> str:
|
||||
"""Names the project's rung again, since a step-up between cycles may have taken the worker above it."""
|
||||
ruled = f" A judge ruled the chunk unfinished, quoting this passage of your transcript as the work remaining:\n\n{quoted(unfinished_evidence)}\n\n" if unfinished_evidence else " "
|
||||
return f"Continue the task; the previous run ended before the chunk was finished.{ruled}The chunk is finished once it is merged, or nothing remains that you could do on your own. Even then, it is finished only once {ANSWERS_RECORDED}."
|
||||
return f"Continue the task; the previous run ended before the chunk was finished.{ruled}The chunk is finished once it is merged, or nothing remains that you could do on your own. Even then, it is finished only once {ANSWERS_RECORDED}.{off_the_projects_rung(route, anchor)}"
|
||||
|
||||
|
||||
def answered(question: str, answer: str) -> str:
|
||||
|
||||
@@ -88,7 +88,7 @@ def worker_start(cycle: Cycle, answered: QuotedAnswer | None) -> list[str]:
|
||||
if answered is not None:
|
||||
return worker_args(cycle.session, cycle.route.rung, prompts.answered(answered.question, answered.answer), resume=True)
|
||||
if session_open(cycle.session):
|
||||
return worker_args(cycle.session, cycle.route.rung, prompts.resumed(cycle.unfinished_evidence), resume=True)
|
||||
return worker_args(cycle.session, cycle.route.rung, prompts.resumed(cycle.unfinished_evidence, cycle.route, cycle.anchor), resume=True)
|
||||
return worker_args(cycle.session, cycle.route.rung, prompts.fresh(cycle.task, cycle.premises, cycle.route, cycle.anchor), resume=False)
|
||||
|
||||
|
||||
|
||||
@@ -48,7 +48,10 @@ counts from the model's cheapest rung, and an R + C of 8 or less picks a rung be
|
||||
such an item, or with a C missing, the chunk runs on the model's cheapest rung. The worker's first
|
||||
prompt names the rung and the item, and asks the worker to name each item the chunk takes, with its
|
||||
R + C. When that rung is below the model's cheapest, the prompt also has the worker launch each
|
||||
reviewer, each comprehension seat and the security pass in subagents on the cheapest rung. The
|
||||
reviewer, each comprehension seat and the security pass in subagents on the cheapest rung. When it
|
||||
is above, the prompt has the worker launch only each comprehension seat and the security pass on
|
||||
the cheapest rung. The prompt after each unfinished verdict
|
||||
says this again, since a step-up may have taken the worker above the cheapest rung. The
|
||||
judge reports the highest R + C the worker states, and the next cycle steps up to the rung that
|
||||
R + C picks, never down. Every second unfinished verdict on a chunk steps it up one rung. Each
|
||||
finished or unfinished verdict, stop and step-up in the archive names the worker's rung, with the
|
||||
|
||||
@@ -403,7 +403,7 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual(run.code, 130)
|
||||
first = next(call for call in run.calls if call["role"] == "worker")
|
||||
self.assertEqual((argument(first, "--model"), argument(first, "--effort")), ("claude-sonnet-5-5", "high"))
|
||||
self.assertIn("Your rung is below the project's, claude-opus-5-5 at medium effort: launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool's model `opus` and effort `medium`", argument(first, "-p"))
|
||||
self.assertIn("Your rung is below the project's, claude-opus-5-5 at medium effort: launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium", argument(first, "-p"))
|
||||
|
||||
def test_a_rerun_resumes_an_open_chunk_on_the_rung_it_last_ran(self):
|
||||
checkout = Checkout(self)
|
||||
|
||||
@@ -540,16 +540,23 @@ class WorkerArgs(unittest.TestCase):
|
||||
def test_keeps_reviewers_on_the_projects_rung_while_the_worker_runs_below_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
below = prompts.fresh("Do it.", "## Goals", routing.Route(Rung("claude-sonnet-5-5", "high"), 3, 5), opus)
|
||||
self.assertIn("whether or not it is item 3. Your rung is below the project's, claude-opus-5-5 at medium effort: launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool's model `opus` and effort `medium`, and run a full scan only from such a subagent, since the scan's agents run on the model that starts them.\n\n", below)
|
||||
for route in (routing.Route(opus, 3, 9), routing.Route(Rung("claude-opus-5-5", "high"), 3, 11)):
|
||||
self.assertNotIn("below the project's", prompts.fresh("Do it.", "## Goals", route, opus))
|
||||
self.assertIn("whether or not it is item 3. Your rung is below the project's, claude-opus-5-5 at medium effort: launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them.\n\n", below)
|
||||
self.assertNotIn("the project's", prompts.fresh("Do it.", "## Goals", routing.Route(opus, 3, 9), opus))
|
||||
|
||||
def test_keeps_seats_and_the_security_pass_on_the_projects_rung_while_the_worker_runs_above_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
above = prompts.fresh("Do it.", "## Goals", routing.Route(Rung("claude-opus-5-5", "high"), 3, 11), opus)
|
||||
stepped_up = prompts.resumed("", routing.Route(Rung("claude-opus-5-5", "high"), 3, 9), opus)
|
||||
self.assertTrue(stepped_up.endswith(" Your rung is above the project's, claude-opus-5-5 at medium effort: launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them."), stepped_up)
|
||||
self.assertNotIn("the project's", prompts.resumed("", routing.Route(opus, 3, 9), opus))
|
||||
self.assertIn("whether or not it is item 3. Your rung is above the project's, claude-opus-5-5 at medium effort: launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them.\n\n", above)
|
||||
|
||||
def test_opens_a_fresh_session_by_name_and_resumes_an_open_one(self):
|
||||
opened = session.session_named("A proj loop-claude", pathlib.Path("/h/.claude/projects"))
|
||||
fresh = worker.worker_args(opened, Rung("m"), prompts.fresh("Do it.", "## Goals\n1. g\n\n## Audience\nme", routing.Route(Rung("m")), None), resume=False)
|
||||
self.assertEqual(fresh[fresh.index("--session-id") + 1], str(opened.id))
|
||||
self.assertEqual(fresh[fresh.index("--name") + 1], "A proj loop-claude")
|
||||
resumed = worker.worker_args(opened, Rung("m"), prompts.resumed(""), resume=True)
|
||||
resumed = worker.worker_args(opened, Rung("m"), prompts.resumed("", routing.Route(Rung("m")), None), resume=True)
|
||||
self.assertEqual(resumed[resumed.index("--resume") + 1], str(opened.id))
|
||||
self.assertNotIn("--session-id", resumed)
|
||||
self.assertTrue(resumed[resumed.index("-p") + 1].startswith("Continue the task; the previous run ended before the chunk was finished. The chunk is finished once"))
|
||||
|
||||
@@ -11,6 +11,10 @@ class Rung:
|
||||
def flags(self) -> list[str]:
|
||||
return ["--model", self.model, *(["--effort", self.effort] if self.effort else [])]
|
||||
|
||||
def agent_args(self) -> str:
|
||||
"""The rung in the Agent tool's words. The tool takes a family name, such as opus, and runs that family's latest model, which is the model each rung names."""
|
||||
return f"model {self.model.split('-')[1]}, effort {self.effort}"
|
||||
|
||||
# The archive records this wording, and loop_claude.routing.ROUTED reads it back to resume a chunk on its rung.
|
||||
def __str__(self) -> str:
|
||||
return f"{self.model} at {self.effort} effort" if self.effort else self.model
|
||||
|
||||
@@ -15,5 +15,5 @@ class Types(unittest.TestCase):
|
||||
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
|
||||
if shutil.which("uv") is None:
|
||||
self.fail("uv on PATH fetches mypy; see README.md's Test line")
|
||||
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "rule_hook", "-p", "score_outcomes", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
|
||||
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "claude_code", "-m", "comprehension_history", "-m", "git_repo", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-m", "round_titles", "-p", "reading_cost", "-p", "rule_hook", "-p", "score_outcomes", "-p", "subagent_rung", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
|
||||
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
|
||||
|
||||
Executable
+9
@@ -0,0 +1,9 @@
|
||||
#!/usr/bin/env python3
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
|
||||
from subagent_rung import main
|
||||
|
||||
sys.exit(main.main(sys.argv[1:]))
|
||||
@@ -0,0 +1,40 @@
|
||||
"""Print the Agent tool's model and effort for a subagent's task: docs/decisions.md → `routing:`.
|
||||
|
||||
usage: subagent-rung <model> <R+C>
|
||||
|
||||
<model> is the project's model, such as claude-opus-5-5, and <R+C> the task's R + C, a whole
|
||||
number from 2 to 20, scored as for a todo.md item. It prints the rung that R + C picks from the
|
||||
model's cheapest rung, in the Agent tool's words, such as `model sonnet, effort high`. For a model
|
||||
off the routing ladder, the tool prints nothing on stdout, says so on stderr and exits 0; launch the
|
||||
subagent with no model or effort.
|
||||
|
||||
Exit 1, saying why, on bad arguments.
|
||||
"""
|
||||
import re
|
||||
import sys
|
||||
|
||||
from model_routing.ladder import picked, project_rung
|
||||
|
||||
USAGE = "usage: subagent-rung <model> <R+C>, R+C a whole number from 2 to 20"
|
||||
|
||||
|
||||
def agent_args(model: str, risk_and_complexity: int) -> str | None:
|
||||
"""The Agent tool's model and effort for the rung risk_and_complexity picks from model's cheapest rung; None for a model off the ladder."""
|
||||
anchor = project_rung(model)
|
||||
return picked(anchor, risk_and_complexity).agent_args() if anchor else None
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
if argv[:1] in (["-h"], ["--help"]):
|
||||
print(__doc__.strip())
|
||||
return 0
|
||||
if len(argv) != 2 or not re.fullmatch(r"[2-9]|1[0-9]|20", argv[1]):
|
||||
print(f"subagent-rung: cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; {USAGE}", file=sys.stderr)
|
||||
return 1
|
||||
model, total = argv
|
||||
args = agent_args(model, int(total))
|
||||
if args is None:
|
||||
print(f"subagent-rung: {model} is not on the routing ladder; pass the Agent tool no model or effort", file=sys.stderr)
|
||||
else:
|
||||
print(args)
|
||||
return 0
|
||||
@@ -0,0 +1,45 @@
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from subagent_rung import main
|
||||
|
||||
TOOL = pathlib.Path(__file__).resolve().parents[1] / "subagent-rung"
|
||||
|
||||
|
||||
class SubagentRung(unittest.TestCase):
|
||||
def run_tool(self, *args: str) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, text=True)
|
||||
|
||||
def test_prints_the_agent_tools_model_and_effort_for_the_rung_an_r_plus_c_picks(self):
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 9), "model opus, effort medium")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 7), "model sonnet, effort high")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 20), "model fable, effort max")
|
||||
self.assertEqual(main.agent_args("claude-haiku-5-5", 2), "model haiku, effort low")
|
||||
|
||||
def test_picks_nothing_for_a_model_off_the_ladder(self):
|
||||
self.assertIsNone(main.agent_args("claude-opus-5", 9))
|
||||
|
||||
def test_refuses_an_r_plus_c_outside_2_to_20_or_a_wrong_count_of_arguments(self):
|
||||
for argv in (["claude-opus-5-5", "1"], ["claude-opus-5-5", "02"], ["claude-opus-5-5", "21"], ["claude-opus-5-5", "x"], ["claude-opus-5-5"], ["9", "claude-opus-5-5"]):
|
||||
result = self.run_tool(*argv)
|
||||
self.assertEqual((result.returncode, result.stdout), (1, ""), argv)
|
||||
self.assertIn("usage: subagent-rung <model> <R+C>", result.stderr)
|
||||
|
||||
def test_the_launcher_prints_the_rung_and_exits_0_on_and_off_the_ladder(self):
|
||||
on = self.run_tool("claude-opus-5-5", "12")
|
||||
self.assertEqual((on.returncode, on.stdout, on.stderr), (0, "model opus, effort high\n", ""))
|
||||
off = self.run_tool("claude-opus-5", "12")
|
||||
self.assertEqual((off.returncode, off.stdout), (0, ""))
|
||||
self.assertEqual(off.stderr, "subagent-rung: claude-opus-5 is not on the routing ladder; pass the Agent tool no model or effort\n")
|
||||
for bound, rung in (("2", "model haiku, effort xhigh\n"), ("20", "model fable, effort max\n")):
|
||||
self.assertEqual(self.run_tool("claude-opus-5-5", bound).stdout, rung)
|
||||
for flag in ("-h", "--help"):
|
||||
self.assertIn("usage: subagent-rung <model> <R+C>", self.run_tool(flag).stdout)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1638,3 +1638,24 @@ Inherited architect, decided by:
|
||||
- Shape: `bin/loop_claude/loop/main.py:180` `archive_step`: `archive` is a function and a Path, and `project_archive` a function and a parameter, a few lines apart
|
||||
- Locality, Shape: `bin/loop_claude/loop/reload.py:41` `decoded_as`: a recursive decoder with a sentinel, and the comment above `LoopRun` half-contradicts the one inside it
|
||||
- Shape: `bin/loop_claude/routing.py:59` `stepped`: does two jobs, raising to a bundled item's rung and counting towards a one-rung step, and its name covers only the second
|
||||
|
||||
## 2026-10-14T22:22:01Z, PR #264 at 17e1ebe, against 8f7b0c5
|
||||
|
||||
Ruling: same
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Mid A | same | worse | same | same | same |
|
||||
| Inherited architect | same | worse | same | same | same |
|
||||
|
||||
Mid A, decided by:
|
||||
|
||||
- Locality: `bin/loop_claude/cycle/prompts.py:27` `off_the_projects_rung`: the prompt text now depends on how the worker's rung compares with the project's through `highest`, and why a chunk with no item needs no note rests on `first_route` in another module
|
||||
- Locality: `bin/loop_claude/routing.py:54` `kept` and `:59` `stepped`: the two now use different floors, so confirming a route below the project's rung survives a reload means checking `routed` in `loop/main.py` against both
|
||||
- Locality: `bin/loop_claude/cycle/run.py:36` `Cycle.anchor`: a second rung travels beside `route` into every prompt call, each with its None case
|
||||
|
||||
Inherited architect, decided by:
|
||||
|
||||
- Locality: `CLAUDE.md:55` "Working with the codebase": which model a subagent gets now spans this prose rule, `prompts.py`'s `off_the_projects_rung` and `ladder.py`'s `Rung.agent_args`; that reviewers above the project's rung inherit the worker's rung is said in neither file
|
||||
- Locality: `bin/model_routing/ladder.py:16` `Rung.agent_args`: the family name comes from `self.model.split('-')[1]`, tying the ladder's ids to the Agent tool's vocabulary with nothing checking it
|
||||
- Locality: `bin/loop_claude/routing.py:54` `kept` and `:59` `stepped`: whether a rerun resumes now depends on the R + C the archived route line holds, and the order of the checks is more to hold
|
||||
|
||||
+5
-3
@@ -170,15 +170,17 @@ The run has no time cap. Limiting it to the changed lines already bounds it by t
|
||||
|
||||
Valid while a mutation run on a chunk's changed lines stays cheap beside the review itself.
|
||||
|
||||
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model, and a chunk's worker may run below that rung, launching its reviewers on that rung
|
||||
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model, and a chunk's worker may run below that rung, launching its reviewers on that rung; the worker routes each subagent it delegates to by the R + C of the subagent's task
|
||||
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude now, and by larv-review and the worker once items 90 and 92 land. `bin/model_routing/ladder.py` holds the rungs and the bands.
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude, by the worker through `bin/subagent-rung`, and by larv-review once item 90 lands. `bin/model_routing/ladder.py` holds the rungs and the bands.
|
||||
|
||||
The rungs, least competent first: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models, read 2026-10-14), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). That cut drops Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index falls among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scores it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price.
|
||||
|
||||
A chunk's worker runs on the rung its item picks, below the project's rung too. Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks spends on work nothing ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each reviewer, each comprehension seat and the security pass's subagent on the project's rung, through the Agent tool's model and effort. It runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. The cost of a cheaper worker shows as extra review rounds and unfinished verdicts, which `todo.md` item 94 is to measure, interrupting lilleman when a rung takes more of either than the rung above. A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. lilleman chose medium knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
|
||||
|
||||
A model off the ladder routes nothing: claude runs it at the effort its settings give. R + C steps a task from the project's rung: 2–4 three rungs down, 5–6 two down, 7–8 one down, 9–10 none, 11–12 one up, 13–14 two up, 15–16 three up, 17 and above four up. A step past either end stops there. The bands spread the R + C that `todo.md`'s items held on 2026-10-14, 2 to 12, across the rungs, so most tasks step down. S never moves a rung, so a huge task with low R and C runs cheap. While the worker runs above the project's rung, the comprehension seats and the security pass inherit its rung; `todo.md` item 138 keeps them on the project's rung. `score-outcomes` had too few finished chunks to ground the bands, and a change to the ladder or the bands is lilleman's.
|
||||
A model off the ladder routes nothing: claude runs it at the effort its settings give. R + C steps a task from the project's rung: 2–4 three rungs down, 5–6 two down, 7–8 one down, 9–10 none, 11–12 one up, 13–14 two up, 15–16 three up, 17 and above four up. A step past either end stops there. The bands spread the R + C that `todo.md`'s items held on 2026-10-14, 2 to 12, across the rungs, so most tasks step down. S never moves a rung, so a huge task with low R and C runs cheap. While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers inherit the worker's higher rung until item 90 routes them. `score-outcomes` had too few finished chunks to ground the bands, and a change to the ladder or the bands is lilleman's.
|
||||
|
||||
The worker delegates a task only when it needs just the task's conclusion: a search or read across many files, or work independent enough to run in parallel. Edits, and whatever it must check line by line, stay in its own thread, since it would read a delegated diff again to check it. Measured over archived worker transcripts, subagent reports cost a median 8% of the worker's peak context, and `sed -n` and `grep` output made up 41% of its Bash results, so delegated searches save the most context. It scores each delegated task's R and C as for an item, and `bin/subagent-rung` turns them into the Agent tool's model and effort, counted from the project's rung. The rule names both, because the Agent tool sets a subagent's effort only when instructions ask. `CLAUDE.md` states the rule, since it holds in every session.
|
||||
|
||||
The worker changes rung only as the chunk starts, after every second unfinished verdict, or when the judge reports a taken-in item whose R + C picks a higher rung, because a model switch inside a session discards its prompt cache. loop-claude routes on the first row of the next release in `todo.md`'s table that waits on no question. It reads neither an order the plan states in prose nor the task, so a chunk may start on another item's rung: a higher one costs money, and a lower one holds until the judge reports the item the worker took in, after the first cycle. `todo.md` item 136 ends this, routing on the item the worker says it takes.
|
||||
|
||||
|
||||
@@ -282,3 +282,20 @@ Read most (opens and searches):
|
||||
- `bin/larv_review/plan.py` 1
|
||||
- `bin/loop_claude/cycle/judge.py` 1
|
||||
- `bin/loop_claude/cycle/prompts.py` 1
|
||||
|
||||
## Reading cost, chunk c08a3c73e956, finished 2026-10-14T22:04:41Z
|
||||
|
||||
93587 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `todo.md` 7
|
||||
- `CHANGELOG.md` 4
|
||||
- `README.md` 4
|
||||
- `bin/loop_claude/cycle/run.py` 3
|
||||
- `docs/decisions.md` 3
|
||||
- `bin/loop_claude/test_loop_claude.py` 2
|
||||
- `bin/transcript_archive/__init__.py` 2
|
||||
- `skills/larv-review/SKILL.md` 2
|
||||
- `AGENTS.md` 1
|
||||
- `bin/loop_claude/cycle/prompts.py` 1
|
||||
|
||||
@@ -28,9 +28,7 @@ Release 0.2.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 94 | 0.1.0 | decision | **Every 20th cycle, have loop-claude measure token spend and routing outcomes, file them here, and interrupt the human only when they call for a routing change.** | 3 | 5 | 6 | 6 | 7 | 1.1, 1.4 | 21.6 |
|
||||
| 90 | 0.1.0 | | **Route each reviewer by the chunk's R + C, once a replay of past chunks shows the routed reviewer finds the 🔴 and 🟡 findings on record.** | 4 | 5 | 8 | 6 | 7 | 1.4 | 20.2 |
|
||||
| 92 | 0.1.0 | | **Have the worker route each subagent by its task's R + C, and delegate a task only when the worker needs just its conclusion.** | 2 | 2 | 5 | 5 | 6 | 1.4 | 19.0 |
|
||||
| 89 | 0.1.0 | | **Run the judge on the rung its chunk's R + C picks, once a replay of archived transcripts shows it returns the verdicts on record.** | 3 | 4 | 7 | 5 | 6 | 1.4 | 17.5 |
|
||||
| 138 | 0.1.0 | decision | **Keep the comprehension seats and the security pass on the project's rung while a chunk's worker runs above it.** | 2 | 2 | 3 | 4 | 5 | 1.1 | 15.4 |
|
||||
| 136 | 0.1.0 | decision | **Route a chunk on the item its worker says it takes, in place of the next release's first row.** | 2 | 3 | 4 | 4 | 5 | 1.4 | 14.8 |
|
||||
| 105 | 0.2.0 | decision | **Compare review rounds per pass and reading cost for the five chunks before and after the `worker:` decision moved the principles out of the worker's context, and file an item if either rose.** | 1 | 2 | 3 | 5 | 6 | 1.4 | 20.0 |
|
||||
| 112 | 0.2.0 | | **Count an item as shipped only where its chunk built it, so an item a chunk dropped from the plan gets no outcome.** | 2 | 3 | 5 | 5 | 6 | 1.1 | 19.1 |
|
||||
@@ -89,26 +87,16 @@ Release 0.2.0 takes items 125, 114 and 124 first, whatever their scores: Shape i
|
||||
|
||||
### 94. Every 20th cycle, have loop-claude measure token spend and routing outcomes, file them here, and interrupt the human only when they call for a routing change.
|
||||
|
||||
Count the cycles across all projects in loop-claude's archive, which holds every project's transcripts on the machine. Measure in code, as `bin/reading-cost` does, so the check itself costs no model run (goal 1.4): tokens by project, subagent type, rung and tool, and per routed chunk its review rounds and unfinished verdicts. A loop on another project cannot commit here, so it writes the figures to the archive, and this repo's next chunk files or refreshes one plan item from them. The ladder's thresholds and the delegation rule are decisions, so a change to them is the human's. Interrupt when chunks on a rung took more review rounds or unfinished verdicts than chunks on the rung above, or when subagent reports fill more of the worker's peak context than the searches they replaced would have; never twice for the same figures (goal 1.2).
|
||||
|
||||
Needs 92.
|
||||
Count the cycles across all projects in loop-claude's archive, which holds every project's transcripts on the machine. Measure in code, as `bin/reading-cost` does, so the check itself costs no model run (goal 1.4): tokens by project, subagent type, rung and tool, and per routed chunk its review rounds and unfinished verdicts, and the rungs its worker delegated subagents on, so the figures show a cheap subagent whose conclusions cost review rounds. A loop on another project cannot commit here, so it writes the figures to the archive, and this repo's next chunk files or refreshes one plan item from them. The ladder's thresholds and the delegation rule are decisions, so a change to them is the human's. Interrupt when chunks on a rung took more review rounds or unfinished verdicts than chunks on the rung above, or when subagent reports fill more of the worker's peak context than the searches they replaced would have; never twice for the same figures (goal 1.2).
|
||||
|
||||
### 90. Route each reviewer by the chunk's R + C, once a replay of past chunks shows the routed reviewer finds the 🔴 and 🟡 findings on record.
|
||||
|
||||
larv-review passes the rung through the Agent tool's `model` and `effort`. A reviewer never steps down when the worker has stepped down on the same chunk, so a cheap worker always meets a reviewer on the project model or above.
|
||||
|
||||
### 92. Have the worker route each subagent by its task's R + C, and delegate a task only when the worker needs just its conclusion.
|
||||
|
||||
State in `CLAUDE.md` when the worker delegates: a search or read that spans many files whose conclusion is all it needs, and work independent enough to run in parallel. Edits and anything the worker must verify line by line stay in the main thread, since a delegated diff is read again to check it. The worker scores R and C for each delegated task as it would for an item, and passes the rung's model and effort. In item 73's measurement, reports cost a median 8% of the worker's peak context, and `sed -n` and `grep` output made up 41% of its Bash results, so delegating searches saves the most context. The Agent tool sets a subagent's effort only when instructions ask, so the rule names both model and effort.
|
||||
|
||||
### 89. Run the judge on the rung its chunk's R + C picks, once a replay of archived transcripts shows it returns the verdicts on record.
|
||||
|
||||
loop-claude starts the judge itself (`bin/loop_claude/cycle/judge.py`), so it routes without any skill change. Replay the judge on archived cycles at the candidate rung, and switch only when its interrupts and verdicts match those on record.
|
||||
|
||||
### 138. Keep the comprehension seats and the security pass on the project's rung while a chunk's worker runs above it.
|
||||
|
||||
`bin/loop_claude/cycle/prompts.py`'s `routed_by` names the project's rung only while the worker runs below it, so from Opus 5.5 at high up to Fable 5.1 at max the seats and the `/security-review` subagent inherit the worker's rung, and the seats' scores vary with each item's R + C. The other reviewers may stay on the worker's higher rung; item 90 routes them.
|
||||
|
||||
### 136. Route a chunk on the item its worker says it takes, in place of the next release's first row.
|
||||
|
||||
loop-claude routes on the first row of the next release, while the worker takes the item the plan's prose orders first, or a prerequisite, as the chunk that shipped item 88, a prerequisite, took item 88 in place of the first row. A first row of higher R + C pays for a dearer rung all chunk; one of lower R + C holds until the judge reports the item taken in. The judge reports the item the worker takes only after the first cycle, so routing the first cycle right needs the item before the worker starts.
|
||||
|
||||
Reference in New Issue
Block a user