Keep the shipped routing ladder beside the module that reads it, so a copy of bin/ carries it
This commit is contained in:
+1
-1
@@ -10,7 +10,6 @@
|
||||
!CLAUDE.md
|
||||
!README.md
|
||||
!ci.sh
|
||||
!routing-ladder.json
|
||||
!settings.json
|
||||
!todo.md
|
||||
|
||||
@@ -73,6 +72,7 @@
|
||||
!bin/metric_bench/*.py
|
||||
!bin/model_routing/
|
||||
!bin/model_routing/*.py
|
||||
!bin/model_routing/routing-ladder.json
|
||||
!bin/mutate-changes
|
||||
!bin/mutate_changes/
|
||||
!bin/mutate_changes/*.py
|
||||
|
||||
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- The routing ladder is now a file, `routing-ladder.json` at this repo's root, and a project may run on a ladder of its own: a `routing-ladder.json` in its config's folder, `~/.claude-projects/<key>/`, replaces the shipped ladder whole for that project. loop-claude, `subagent-rung` and rule-hook read the project's ladder; the routing figures, `judge-replay` and `review-replay` read the shipped one. `subagent-rung` now reads the config of the checkout it runs in, and exits 1 where that config or the ladder holds a fault; loop-claude stops on a faulty ladder as it does on a faulty config. `loop-claude --help` shows the file's shape, and README → Install → "Set up a project" says when to write one.
|
||||
- The routing ladder is now a file, `bin/model_routing/routing-ladder.json`, and a project may run on a ladder of its own: a `routing-ladder.json` in its config's folder, `~/.claude-projects/<key>/`, replaces the shipped ladder whole for that project. loop-claude, `subagent-rung` and rule-hook read the project's ladder; the routing figures, `judge-replay` and `review-replay` read the shipped one. `subagent-rung` now reads the config of the checkout it runs in, and exits 1 where that config or the ladder holds a fault; loop-claude stops on a faulty ladder as it does on a faulty config. `loop-claude --help` shows the file's shape, and README → Install → "Set up a project" says when to write one.
|
||||
|
||||
## 0.10.0
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Claude Code configuration
|
||||
|
||||
My personal `~/.claude`, version controlled: `CLAUDE.md` and `rules/` for instructions, the words under `CLAUDE.md` → "Vocabulary"; `.claude/rules/` for the rules that load in this repo alone; `agents/` and `skills/`; `principles/` and `skills/larv-review/standards/` for what the reviewer agents hold projects to; `docs/` for decisions and the comprehension history; `settings.json` for the harness and `.claude/settings.json` for the harness inside this repo; `routing-ladder.json` for the models and efforts loop-claude routes work to; `CHANGELOG.md` and `todo.md`; the tools in `bin/`; `workflow-bench/`, the scripts and prompts of the 2026-09-30 bench of larv-review and of the 2026-11-12 bench of the advisor tool; `reviewer-merge-trials/`, the 2026-11-09 trials of merging reviewers, with their method, results and scripts; `judge-field-trials/`, the 2026-11-11 trials of which facts code could read in place of loop-claude's judge, with their results and scripts; `stability-skip-count/`, the 2026-11-12 count of archived stability rounds, with its script, from which larv-review took when to skip a stability round 2; and `presentation/`, a page that explains this setup to newcomers, with the script that prints it to PDF.
|
||||
My personal `~/.claude`, version controlled: `CLAUDE.md` and `rules/` for instructions, the words under `CLAUDE.md` → "Vocabulary"; `.claude/rules/` for the rules that load in this repo alone; `agents/` and `skills/`; `principles/` and `skills/larv-review/standards/` for what the reviewer agents hold projects to; `docs/` for decisions and the comprehension history; `settings.json` for the harness and `.claude/settings.json` for the harness inside this repo; `CHANGELOG.md` and `todo.md`; the tools in `bin/`, with the routing ladder they ship, `bin/model_routing/routing-ladder.json`; `workflow-bench/`, the scripts and prompts of the 2026-09-30 bench of larv-review and of the 2026-11-12 bench of the advisor tool; `reviewer-merge-trials/`, the 2026-11-09 trials of merging reviewers, with their method, results and scripts; `judge-field-trials/`, the 2026-11-11 trials of which facts code could read in place of loop-claude's judge, with their results and scripts; `stability-skip-count/`, the 2026-11-12 count of archived stability rounds, with its script, from which larv-review took when to skip a stability round 2; and `presentation/`, a page that explains this setup to newcomers, with the script that prints it to PDF.
|
||||
|
||||
## Goals
|
||||
|
||||
@@ -106,7 +106,7 @@ cd ~/code/acme-billing && ~/.claude/bin/project-config
|
||||
|
||||
The last line prints each setting's path once the config is complete.
|
||||
|
||||
A project runs on the routing ladder this setup ships, `routing-ladder.json` at this repo's root: the rungs, each a model at an effort, that loop-claude routes workers, subagents and reviews to, least competent first. To run one project on other rungs, as when an outage or a client's account leaves it fewer models, copy that file to `~/.claude-projects/<key>/routing-ladder.json` and cut or change its rungs. The project's file replaces the shipped ladder whole. loop-claude reads it as each cycle starts, and `subagent-rung` and rule-hook each time they run. The loop never commits it, so commit it in the folder's repo yourself only where it should last. Run the project on a model its ladder holds, as with `LOOP_CLAUDE_WORKER_MODEL` and `LOOP_CLAUDE_JUDGE_MODEL`: loop-claude doesn't route a model off the ladder, and runs it at the effort its settings give. `loop-claude --help` shows the file's shape.
|
||||
A project runs on the routing ladder this setup ships, `bin/model_routing/routing-ladder.json`: the rungs, each a model at an effort, that loop-claude routes workers, subagents and reviews to, least competent first. To run one project on other rungs, as when an outage or a client's account leaves it fewer models, copy that file to `~/.claude-projects/<key>/routing-ladder.json` and cut or change its rungs. The project's file replaces the shipped ladder whole. loop-claude reads it as each cycle starts, and `subagent-rung` and rule-hook each time they run. The loop never commits it, so commit it in the folder's repo yourself only where it should last. Run the project on a model its ladder holds, as with `LOOP_CLAUDE_WORKER_MODEL` and `LOOP_CLAUDE_JUDGE_MODEL`: loop-claude doesn't route a model off the ladder, and runs it at the effort its settings give. `loop-claude --help` shows the file's shape.
|
||||
|
||||
## Update
|
||||
|
||||
@@ -172,7 +172,7 @@ Install puts only `loop-claude` and `loop-inbox` on `PATH`; call the others by t
|
||||
- `loop_state/` holds the files loop-claude and loop-inbox share: the question files (`waiting/`), the files a running loop lists itself in (`running_loops.py`, `session_lock.py`), their paths, and `hand_over.py`, which opens claude for `/talk`. A change to any of them changes both tools.
|
||||
- `machine_load/` takes the machine's heavy slots, its shared lock files, for `run_heavy`, and reads the memory share that loop-claude waits for. It also gives `mutate_changes` a slot's CPU count.
|
||||
- `mattermost_dm/` sends the human a Mattermost DM, for `mattermost-dm` and for loop-claude through `loop_effects`.
|
||||
- `model_routing/` holds the routing ladder's steps (`ladder.py`) and reads a ladder from its file, the shipped `routing-ladder.json` or a project's own (`ladder_file.py`), for loop-claude, `rule_hook`, both replay tools and `subagent-rung`. It finds a project's config folder through `project_config`. It also holds the list prices, which `routing_figures` and `transcript_archive` read.
|
||||
- `model_routing/` holds the routing ladder's steps (`ladder.py`) and reads a ladder from its file, the shipped `routing-ladder.json` beside it or a project's own (`ladder_file.py`), for loop-claude, `rule_hook`, both replay tools and `subagent-rung`. It finds a project's config folder through `project_config`. It also holds the list prices, which `routing_figures` and `transcript_archive` read.
|
||||
- `rank_todo.py`, `settings_filter.py` and `statusline.py` are tools of one file each, tested beside them.
|
||||
- `release_plan/` reads `todo.md`'s items table and release versions, says which release is next and whether it is ready to cut (`release_standing.py`), and ranks the plan (`ranking.py`, with its score formula in `formula.py` and its blocker order in `blockers.py`). `goals_file.py` reads the goals file's sections, goals and goal weights, for the ranking and for `loop_claude`'s premises, and `citations.py` holds `todo.md` and `docs/decisions.md` to each other. `plan_file.py` reads, ranks, checks and writes a `todo.md`, for `larv_review`, `rank_todo.py`, `rule_hook` and `todo_item`.
|
||||
- `repo_checks/` tests the repo itself.
|
||||
|
||||
@@ -76,14 +76,15 @@ medium, high, xhigh or max:
|
||||
{"model": "claude-fable-5-2", "effort": "low"}
|
||||
]
|
||||
|
||||
This setup ships one, routing-ladder.json at its root. A project that keeps a routing-ladder.json in
|
||||
its config's folder, ~/.claude-projects/<key>/, runs on that ladder instead, whole: write there
|
||||
every rung the project may run on. loop-claude reads the project's ladder as each cycle starts, and
|
||||
never commits the file. subagent-rung and rule-hook read the same ladder; the routing figures,
|
||||
judge-replay and review-replay, which span every project, read the shipped one.
|
||||
This setup ships one, bin/model_routing/routing-ladder.json. A project that keeps a
|
||||
routing-ladder.json in its config's folder, ~/.claude-projects/<key>/, runs on that ladder
|
||||
instead, whole: write there every rung the project may run on. loop-claude reads the project's
|
||||
ladder as each cycle starts, and never commits the file. subagent-rung and rule-hook read the same
|
||||
ladder; the routing figures, judge-replay and review-replay, which span every project, read the
|
||||
shipped one.
|
||||
|
||||
Where the project's model is on its routing ladder, the worker and the
|
||||
session /talk opens run on the rung that the chunk's todo.md item picks, and the judge of some
|
||||
Where the project's model is on its routing ladder, the worker and the session /talk opens run on
|
||||
the rung that the chunk's todo.md item picks, and the judge of some
|
||||
models on another; a chunk no item routes runs on the project's model. The worker's prompts name the
|
||||
rungs it launches its reviews on. The worker's rung steps up as a chunk stays unfinished, when the
|
||||
worker names on its `Chunk items:` line an item that picks a higher rung, or when the judge finds
|
||||
|
||||
@@ -952,8 +952,6 @@ class Reload(unittest.TestCase):
|
||||
checkout = Checkout(self)
|
||||
copy = checkout.temp / "bin"
|
||||
shutil.copytree(BIN, copy, ignore=shutil.ignore_patterns("__pycache__", "test_*"))
|
||||
# The copied tools read the ladder their setup ships from the folder above their bin/.
|
||||
shutil.copy(BIN.parent / "routing-ladder.json", copy.parent)
|
||||
checkout.script = copy / "loop-claude"
|
||||
checkout.env["FAKE_CLAUDE_EDIT"] = str(copy)
|
||||
run = checkout.run([
|
||||
|
||||
@@ -10,8 +10,8 @@ from project_config.settings import Unnamed
|
||||
|
||||
# The file's name in both places. Three readers, seated as the human who writes an override in a hurry, picked a bare list of objects over a list of strings, a list of pairs and a list under a key. Two of them picked one name for both files: an override is then a copy of the shipped file with rungs cut.
|
||||
NAME = "routing-ladder.json"
|
||||
# The ladder this setup ships, at the root of the checkout the running tool is in.
|
||||
SHIPPED = pathlib.Path(__file__).resolve().parents[2] / NAME
|
||||
# The ladder this setup ships, beside this module, so a copy of bin/ carries it: the tests of a reload and mutate-changes each run the tools from such a copy.
|
||||
SHIPPED = pathlib.Path(__file__).resolve().with_name(NAME)
|
||||
# A model as Rung.family reads it: the Agent tool's family name follows the first hyphen.
|
||||
MODEL = re.compile(r"[^\s-]+(-[^\s-]+)+")
|
||||
SHAPE = f'a JSON list of rungs, least competent first, each as {{"model": "claude-opus-5-5", "effort": "medium"}}, with an effort of {", ".join(EFFORTS[:-1])} or {EFFORTS[-1]}'
|
||||
|
||||
@@ -61,7 +61,7 @@ RULES: dict[RuleName, Rule] = {
|
||||
),
|
||||
# A rung is not a judgment call, and the refusal names the arguments to pass, so no marker waives it.
|
||||
"reviewer-rung": Rule(
|
||||
"a reviewer, a comprehension seat or the security pass runs on the rung loop-claude names for it: for a seat or the security pass the project's rung, and for a reviewer the highest of the project's rung, the worker's and the one the chunk's item's E + I picks, or the project's rung where the branch changes only Markdown no agent follows, so a worker below the project's rung cannot hand a review to its own cheaper model",
|
||||
"a reviewer, a comprehension seat or the security pass runs on the rung loop-claude names for it: for a seat or the security pass the project's rung, and for a reviewer the highest of the project's rung, the worker's and the one the chunk's item's E + I picks, or the project's rung where the branch changes only Markdown no agent follows, so a worker below the project's rung cannot hand a review to its own cheaper model. The rungs rank as on the project's routing ladder, and the review is refused while that ladder cannot be read",
|
||||
"an Agent call, in a claude loop-claude started, whose subagent type is an agent under agents/, or whose prompt opens with seat-prompts.md's opening sentence or the one larv-review's Security section sets; and a `security-review` or `claude-security` Skill call in such a claude, which runs on its caller's rung, unless the caller is a subagent whose prompt opens with that sentence",
|
||||
overridable=False,
|
||||
),
|
||||
|
||||
@@ -5,13 +5,15 @@ usage: subagent-rung [--plan] <model> <L>
|
||||
<model> is the project's model, such as claude-opus-5-5, and <L> the task's L, the chance it holds
|
||||
a defect, a level from 1 to 4 scored as for a todo.md item. It prints the rung that L picks from
|
||||
the model's cheapest rung, the project's rung, in the Agent tool's words, such as `model sonnet,
|
||||
effort high`. It reads the routing ladder of the project whose checkout it runs in:
|
||||
routing-ladder.json in the folder of the project's config, ~/.claude-projects/<key>/, or else
|
||||
the one this setup ships. loop-claude --help says how to write one. --plan, for a task that files or scores todo.md items, prints the project's rung where
|
||||
effort high`. --plan, for a task that files or scores todo.md items, prints the project's rung where
|
||||
that L picks a lower one, as ~/.claude/rules/todo-items.md lets only the project's rung or above
|
||||
plan. For a model off the routing ladder, the tool doesn't print anything on stdout, says so on stderr and
|
||||
exits 0; launch the subagent with no model or effort.
|
||||
|
||||
It reads the routing ladder of the project whose checkout it runs in: routing-ladder.json in the
|
||||
folder of the project's config, ~/.claude-projects/<key>/, or else the one this setup ships, as in
|
||||
a checkout no config names. loop-claude --help shows the file's shape.
|
||||
|
||||
Exit 1, saying why, on bad arguments, or where the project's config or its routing ladder is faulty
|
||||
or cannot be read.
|
||||
"""
|
||||
|
||||
+3
-3
@@ -514,9 +514,9 @@ On 2026-11-12, `bin/judge-replay claude-opus-5-5` replayed 60 verdicts, 15 of ea
|
||||
|
||||
Valid while the archived verdicts show what a judge should return, and until a new model joins the ladder: replay then.
|
||||
|
||||
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index, shipped as `routing-ladder.json`, which a project replaces with a file of that name in its config's folder; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
|
||||
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index, shipped as `bin/model_routing/routing-ladder.json`, which a project replaces with a file of that name in its config's folder; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
|
||||
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude and by the worker through `bin/subagent-rung`. The judge's rung is set by `routing: on a model judge-replay covered`. `routing-ladder.json` holds the rungs and `bin/model_routing/ladder.py` the steps; the 2026-11-15 note below says when a project runs on a ladder of its own.
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude and by the worker through `bin/subagent-rung`. The judge's rung is set by `routing: on a model judge-replay covered`. `bin/model_routing/routing-ladder.json` holds the rungs and `ladder.py` beside it the steps; the 2026-11-15 note below says when a project runs on a ladder of its own.
|
||||
|
||||
The rungs, least competent first: Sonnet 5.5 at high; Opus 5.5 at medium; Fable 5.2 at low and medium; Opus 5.5 at xhigh; Fable 5.2 at high and max. The 2026-11-11 note below put them so. Before it they were Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max, after the 2026-10-29 note took Haiku 5.5 off. They first were: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). The first cut, from the index read 2026-10-14, dropped Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index fell among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scored it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price. Fable 5.2 at max tops the index outright, so that exception has ended.
|
||||
|
||||
@@ -534,7 +534,7 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
|
||||
|
||||
2026-11-11, lilleman approved the rungs for Fable 5.2 from the index read that day. Its score and cost per task, for each effort from low to max: Sonnet 5.5 26 $0.51, 31 $0.78, 42 $1.64, 50 $4.09, 55 $10.02; Opus 5.5 35 $0.98, 48 $2.45, 52 $3.46, 57 $6.05, 59 $10.04; Fable 5.2 54 $3.10, 56 $3.80, 62 $9.08, 62 $14.83, 65 $26.54. The cut drops Opus 5.5 at high, since Fable 5.2 at low scores more for less; Opus 5.5 at max, since Fable 5.2 at high scores more for less; and Fable 5.2 at xhigh, which ties Fable 5.2 at high at a higher cost. Sonnet 5.5 at high stays the bottom rung, as the 2026-10-29 note chose. With Haiku 5.5 off, the cut alone would also keep Sonnet 5.5 at low and medium and Opus 5.5 at low, below it. On an Opus 5.5 project, an L 4 worker, an E + I 5 review and a worker's first escalation now run on Fable 5.2 at low, where they ran on Opus 5.5 at high; E + I 8 runs on Fable 5.2 at high, where it ran on Fable 5.1 at max. Three steps land on a rung the index scores lower than before: E + I 6 on Fable 5.2 at medium, 56 against Opus 5.5 at xhigh's 57; E + I 7 on Opus 5.5 at xhigh, 57 against Opus 5.5 at max's 59; and an L 3 worker's escalation stops, without lilleman's yes, at Fable 5.2 at medium, where it stopped at Opus 5.5 at xhigh. On a Sonnet 5.5 project, E + I 6 runs on Fable 5.2 at low, 54 against Opus 5.5 at high's 52; E + I 7 on Fable 5.2 at medium, 56 against Opus 5.5 at xhigh's 57; and E + I 8 on Opus 5.5 at xhigh, 57 against Opus 5.5 at max's 59. Fable 5.2 at max, on top, is reached only one rung up from Fable 5.2 at high, as by a reviewer's rerun. A Fable 5.2 project's rung is Fable 5.2 at low, so its L 1 and L 2 workers run on Opus 5.5 at medium.
|
||||
|
||||
2026-11-15, lilleman asked for the ladder as a file that a project can replace, on a day when an outage left only Fable 5.2 answering: the rungs were a constant in `bin/model_routing/ladder.py`, so no project could leave Sonnet 5.5 and Opus 5.5 without a release. The shipped ladder is `routing-ladder.json` at the repo's root, a JSON list of rungs, least competent first. A project that keeps a `routing-ladder.json` in its config's folder, `~/.claude-projects/<key>/`, runs on that ladder whole, and the steps count on it as on the shipped one. Every project still runs the same routing rules, and its config's folder supplies what differs, as goal 1.3 asks. lilleman chose a file per project over one per machine, so a single project can differ. The project's file is lilleman's to write, as any change to the ladder is, and the loop never commits it. loop-claude reads the project's ladder as each cycle starts; `subagent-rung` and rule-hook's `reviewer-rung` read it each time they run, from the checkout they run in. A ladder file that holds a fault stops each of them, since work routed on the shipped ladder in its place could run on a model the project cannot reach. The routing figures, `judge-replay` and `review-replay` span every project, so they read the shipped ladder: a rung only a project's own ladder holds is left out of the figures' calls, and no replay qualifies it. Three readers, seated as lilleman writing a project's file in a hurry, picked the file's shape, a bare list of `{"model", "effort"}` objects, over a list of strings, a list of pairs and a list under a key, and two of the three picked one name for both files.
|
||||
2026-11-15, lilleman asked for the ladder as a file that a project can replace, on a day when an outage left only Fable 5.2 answering: the rungs were a constant in `bin/model_routing/ladder.py`, so no project could leave Sonnet 5.5 and Opus 5.5 without a release. The shipped ladder is `bin/model_routing/routing-ladder.json`, a JSON list of rungs, least competent first, beside the code that reads it, so a copy of `bin/` still runs. A project that keeps a `routing-ladder.json` in its config's folder, `~/.claude-projects/<key>/`, runs on that ladder whole, and the steps count on it as on the shipped one. Every project still runs the same routing rules, and its config's folder supplies what differs, as goal 1.3 asks. lilleman chose a file per project over one per machine, so a single project can differ. The project's file is lilleman's to write, as any change to the ladder is, and the loop never commits it. loop-claude reads the project's ladder as each cycle starts; `subagent-rung` and rule-hook's `reviewer-rung` read it each time they run, from the checkout they run in. A ladder file that holds a fault stops each of them, since work routed on the shipped ladder in its place could run on a model the project cannot reach. The routing figures, `judge-replay` and `review-replay` span every project, so they read the shipped ladder: a rung only a project's own ladder holds is left out of the figures' calls, and no replay qualifies it. Three readers, seated as lilleman writing a project's file in a hurry, picked the file's shape, a bare list of `{"model", "effort"}` objects, over a list of strings, a list of pairs and a list under a key, and two of the three picked one name for both files.
|
||||
|
||||
Valid while the index ranks these models, and while no newer model joins the catalog.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user