Answer prose round 1: 0 🔴, 9 🟡 (0 misread); say what each docstring's code does, give the outage step in plain words and keep each choice in one place
CI / tests (pull_request) Successful in 2m31s
CI / tests (pull_request) Successful in 2m31s
This commit is contained in:
@@ -106,7 +106,7 @@ cd ~/code/acme-billing && ~/.claude/bin/project-config
|
||||
|
||||
The last line prints each setting's path once the config is complete.
|
||||
|
||||
A project runs on the routing ladder this setup ships, `bin/model_routing/routing-ladder.json`: the rungs, each a model at an effort, that loop-claude routes workers, subagents and reviews to, least competent first. To run one project on other rungs, as when an outage or a client's account leaves it fewer models, copy that file to `~/.claude-projects/<key>/routing-ladder.json` and cut or change its rungs. The project's file replaces the shipped ladder whole. loop-claude reads it as each cycle starts, and `subagent-rung` and rule-hook each time they run. The loop never commits it, so commit it in the folder's repo yourself only where it should last. Run the project on a model its ladder holds, as with `LOOP_CLAUDE_WORKER_MODEL` and `LOOP_CLAUDE_JUDGE_MODEL`: loop-claude doesn't route a model off the ladder, and runs it at the effort its settings give. `loop-claude --help` shows the file's shape.
|
||||
A project runs on the routing ladder this setup ships, `bin/model_routing/routing-ladder.json`: the rungs, least competent first, each a model at an effort, that loop-claude routes workers, subagents and reviews to. To run one project on other rungs, as when an outage or a client's account leaves it fewer models, copy that file to `~/.claude-projects/<key>/routing-ladder.json` and cut or change its rungs. Keep one model for each family, such as opus: Claude Code launches a subagent by its family's name. The project's file replaces the shipped ladder whole. loop-claude reads it as each cycle starts, and `subagent-rung` and rule-hook each time they run. The loop never commits it, so commit it in the folder's repo yourself only where it should last. Set the project's models to ones its ladder holds, for example with `LOOP_CLAUDE_WORKER_MODEL` and `LOOP_CLAUDE_JUDGE_MODEL`. A model off the ladder is not routed: every worker runs on that model, at the effort Claude Code's `settings.json` gives. Each cycle of a project on a ladder of its own says so on stderr, naming the file, and says where the worker's model is off it. `loop-claude --help` shows the file's shape.
|
||||
|
||||
## Update
|
||||
|
||||
@@ -172,7 +172,7 @@ Install puts only `loop-claude` and `loop-inbox` on `PATH`; call the others by t
|
||||
- `loop_state/` holds the files loop-claude and loop-inbox share: the question files (`waiting/`), the files a running loop lists itself in (`running_loops.py`, `session_lock.py`), their paths, and `hand_over.py`, which opens claude for `/talk`. A change to any of them changes both tools.
|
||||
- `machine_load/` takes the machine's heavy slots, its shared lock files, for `run_heavy`, and reads the memory share that loop-claude waits for. It also gives `mutate_changes` a slot's CPU count.
|
||||
- `mattermost_dm/` sends the human a Mattermost DM, for `mattermost-dm` and for loop-claude through `loop_effects`.
|
||||
- `model_routing/` holds the routing ladder's steps (`ladder.py`) and reads a ladder from its file, the shipped `routing-ladder.json` beside it or a project's own (`ladder_file.py`), for loop-claude, `rule_hook`, both replay tools and `subagent-rung`. It finds a project's config folder through `project_config`. It also holds the list prices, which `routing_figures` and `transcript_archive` read.
|
||||
- `model_routing/` holds the shipped routing ladder (`routing-ladder.json`) and its steps (`ladder.py`), and reads a ladder from that file or a project's own (`ladder_file.py`), for loop-claude, `rule_hook`, both replay tools and `subagent-rung`. It finds a project's config folder through `project_config`. It also holds the list prices, which `routing_figures` and `transcript_archive` read.
|
||||
- `rank_todo.py`, `settings_filter.py` and `statusline.py` are tools of one file each, tested beside them.
|
||||
- `release_plan/` reads `todo.md`'s items table and release versions, says which release is next and whether it is ready to cut (`release_standing.py`), and ranks the plan (`ranking.py`, with its score formula in `formula.py` and its blocker order in `blockers.py`). `goals_file.py` reads the goals file's sections, goals and goal weights, for the ranking and for `loop_claude`'s premises, and `citations.py` holds `todo.md` and `docs/decisions.md` to each other. `plan_file.py` reads, ranks, checks and writes a `todo.md`, for `larv_review`, `rank_todo.py`, `rule_hook` and `todo_item`.
|
||||
- `repo_checks/` tests the repo itself.
|
||||
|
||||
@@ -18,12 +18,12 @@ def environment(project: Rung | None, route: Route) -> dict[str, str]:
|
||||
|
||||
|
||||
def hands_rungs(environ: typing.Mapping[str, str]) -> bool:
|
||||
"""Whether the environment names a project's and a worker's rung, as loop-claude's does for a project on its ladder. A reader asks before it reads the project's ladder, which handed needs."""
|
||||
"""Whether the environment names a project's and a worker's rung, as loop-claude sets them for a project on its ladder. Where it names neither, no review launch is checked, so the caller skips reading the project's ladder."""
|
||||
return bool(environ.get(PROJECT_RUNG) and environ.get(WORKER_RUNG))
|
||||
|
||||
|
||||
def handed(environ: typing.Mapping[str, str], ladder: Ladder) -> tuple[Rung | None, Rung | None, int]:
|
||||
"""The project's and the worker's rungs, each None when unset or not on ladder, the project's ladder, and the reviews' E + I, 0 when unset or not a whole number. An older loop-claude hands no E + I, and 0 steps no review up."""
|
||||
"""The project's and the worker's rungs, each None when unset or not on ladder, and the reviews' E + I, 0 when unset or not a whole number. ladder is the project's. An older loop-claude hands no E + I, and 0 steps no review up."""
|
||||
escape_and_impact = environ.get(ESCAPE_AND_IMPACT, "")
|
||||
return on_the_ladder(environ.get(PROJECT_RUNG), ladder), on_the_ladder(environ.get(WORKER_RUNG), ladder), int(escape_and_impact) if escape_and_impact.isdecimal() else 0
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ Place = str | pathlib.Path
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Unnamed(Failure):
|
||||
"""No config names a checkout here: none names the one asked for, or git finds no checkout. A reader of the routing ladder takes the shipped one there, as in a third party's clone; a config that is faulty or cannot be read is a plain Failure."""
|
||||
"""No config names a checkout here: none names the one asked for, or git fails there, as outside a checkout. A config that is faulty or cannot be read is a plain Failure."""
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
|
||||
@@ -74,7 +74,7 @@ class Call:
|
||||
|
||||
|
||||
def calls(ladder: Ladder, figures: Figures) -> list[Call]:
|
||||
"""Each routing change the figures call for, comparing the rungs on ladder, which is the shipped one: the figures span every project, and a project's own ladder ranks only its own chunks."""
|
||||
"""Each routing change the figures call for, comparing the rungs on ladder."""
|
||||
compared = sorted((rung for rung, chunks in figures.routed.items() if len(chunks) >= MEASURED_AT_LEAST and ladder.position(rung) >= 0), key=ladder.position)
|
||||
found = []
|
||||
for lower, upper in zip(compared, compared[1:]):
|
||||
|
||||
+3
-3
@@ -514,9 +514,9 @@ On 2026-11-12, `bin/judge-replay claude-opus-5-5` replayed 60 verdicts, 15 of ea
|
||||
|
||||
Valid while the archived verdicts show what a judge should return, and until a new model joins the ladder: replay then.
|
||||
|
||||
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index, shipped as `bin/model_routing/routing-ladder.json`, which a project replaces with a file of that name in its config's folder; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
|
||||
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index, shipped as `bin/model_routing/routing-ladder.json`, which a project may replace with a file of that name in its config's folder; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
|
||||
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude and by the worker through `bin/subagent-rung`. The judge's rung is set by `routing: on a model judge-replay covered`. `bin/model_routing/routing-ladder.json` holds the rungs and `ladder.py` beside it the steps; the 2026-11-15 note below says when a project runs on a ladder of its own.
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one shipped ladder and one set of steps for every project, read by loop-claude and by the worker through `bin/subagent-rung`; a project runs on other rungs only through a ladder file of its own (2026-11-15 note). The judge's rung is set by `routing: on a model judge-replay covered`. `bin/model_routing/routing-ladder.json` holds the rungs and `ladder.py` beside it the steps.
|
||||
|
||||
The rungs, least competent first: Sonnet 5.5 at high; Opus 5.5 at medium; Fable 5.2 at low and medium; Opus 5.5 at xhigh; Fable 5.2 at high and max. The 2026-11-11 note below put them so. Before it they were Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max, after the 2026-10-29 note took Haiku 5.5 off. They first were: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). The first cut, from the index read 2026-10-14, dropped Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index fell among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scored it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price. Fable 5.2 at max tops the index outright, so that exception has ended.
|
||||
|
||||
@@ -534,7 +534,7 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
|
||||
|
||||
2026-11-11, lilleman approved the rungs for Fable 5.2 from the index read that day. Its score and cost per task, for each effort from low to max: Sonnet 5.5 26 $0.51, 31 $0.78, 42 $1.64, 50 $4.09, 55 $10.02; Opus 5.5 35 $0.98, 48 $2.45, 52 $3.46, 57 $6.05, 59 $10.04; Fable 5.2 54 $3.10, 56 $3.80, 62 $9.08, 62 $14.83, 65 $26.54. The cut drops Opus 5.5 at high, since Fable 5.2 at low scores more for less; Opus 5.5 at max, since Fable 5.2 at high scores more for less; and Fable 5.2 at xhigh, which ties Fable 5.2 at high at a higher cost. Sonnet 5.5 at high stays the bottom rung, as the 2026-10-29 note chose. With Haiku 5.5 off, the cut alone would also keep Sonnet 5.5 at low and medium and Opus 5.5 at low, below it. On an Opus 5.5 project, an L 4 worker, an E + I 5 review and a worker's first escalation now run on Fable 5.2 at low, where they ran on Opus 5.5 at high; E + I 8 runs on Fable 5.2 at high, where it ran on Fable 5.1 at max. Three steps land on a rung the index scores lower than before: E + I 6 on Fable 5.2 at medium, 56 against Opus 5.5 at xhigh's 57; E + I 7 on Opus 5.5 at xhigh, 57 against Opus 5.5 at max's 59; and an L 3 worker's escalation stops, without lilleman's yes, at Fable 5.2 at medium, where it stopped at Opus 5.5 at xhigh. On a Sonnet 5.5 project, E + I 6 runs on Fable 5.2 at low, 54 against Opus 5.5 at high's 52; E + I 7 on Fable 5.2 at medium, 56 against Opus 5.5 at xhigh's 57; and E + I 8 on Opus 5.5 at xhigh, 57 against Opus 5.5 at max's 59. Fable 5.2 at max, on top, is reached only one rung up from Fable 5.2 at high, as by a reviewer's rerun. A Fable 5.2 project's rung is Fable 5.2 at low, so its L 1 and L 2 workers run on Opus 5.5 at medium.
|
||||
|
||||
2026-11-15, lilleman asked for the ladder as a file that a project can replace, on a day when an outage left only Fable 5.2 answering: the rungs were a constant in `bin/model_routing/ladder.py`, so no project could leave Sonnet 5.5 and Opus 5.5 without a release. The shipped ladder is `bin/model_routing/routing-ladder.json`, a JSON list of rungs, least competent first, beside the code that reads it, so a copy of `bin/` still runs. A project that keeps a `routing-ladder.json` in its config's folder, `~/.claude-projects/<key>/`, runs on that ladder whole, and the steps count on it as on the shipped one. Every project still runs the same routing rules, and its config's folder supplies what differs, as goal 1.3 asks. lilleman chose a file per project over one per machine, so a single project can differ. The project's file is lilleman's to write, as any change to the ladder is, and the loop never commits it. loop-claude reads the project's ladder as each cycle starts; `subagent-rung` and rule-hook's `reviewer-rung` read it each time they run, from the checkout they run in. A ladder file that holds a fault stops each of them, since work routed on the shipped ladder in its place could run on a model the project cannot reach. The routing figures, `judge-replay` and `review-replay` span every project, so they read the shipped ladder: a rung only a project's own ladder holds is left out of the figures' calls, and no replay qualifies it. Three readers, seated as lilleman writing a project's file in a hurry, picked the file's shape, a bare list of `{"model", "effort"}` objects, over a list of strings, a list of pairs and a list under a key, and two of the three picked one name for both files.
|
||||
2026-11-15, lilleman asked for the ladder as a file that a project can replace, on a day when an outage left only Fable 5.2 answering: the rungs were a constant in `bin/model_routing/ladder.py`, so no project could leave Sonnet 5.5 and Opus 5.5 without a release. The shipped ladder is `bin/model_routing/routing-ladder.json`, a JSON list of rungs, least competent first. A project that keeps a `routing-ladder.json` in its config's folder, `~/.claude-projects/<key>/`, runs on that ladder whole, and the steps count on it as on the shipped one. Every project still runs the same routing rules and steps, so the workflow of goal 1.3 holds; the ladder joins what a project's config folder may supply, which that goal's list does not name yet. lilleman chose a file per project over one per machine, so a single project can differ. A ladder holds one model of each family, since the Agent tool launches a subagent or a review by its family's name. The project's file is lilleman's to write, as any change to the ladder is, and the loop never commits it. loop-claude reads the project's ladder as each cycle starts; `subagent-rung` and rule-hook's `reviewer-rung` read it each time they run, from the checkout they run in. A ladder file that holds a fault stops each of them, since work routed on the shipped ladder in its place could run on a model the project cannot reach. The routing figures, `judge-replay` and `review-replay` span every project, so they read the shipped ladder: a rung only a project's own ladder holds is left out of the figures' calls, and no replay qualifies it. lilleman shipped the change as 0.10.1, a patch cut from the tag v0.10.0, on the day of the outage: it adds a file format and an exit 1 to `subagent-rung`, which the technical principles' semantic versioning gives a minor, and lilleman asked for the patch "even if that breaks semver", so that the release in use could run again without taking the unreleased work on `main`.
|
||||
|
||||
Valid while the index ranks these models, and while no newer model joins the catalog.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user