Merge pull request 'Score risk as L + E + I, and route a worker on its item's L and its reviewers on its E + I' (#296) from item-146 into main
This commit was merged in pull request #296.
This commit is contained in:
@@ -38,19 +38,19 @@ Decisions live in `docs/decisions.md`:
|
||||
- tests: each test earns its keep as the technical principles' test bullet says, and the worker takes a test's expected values from its item, never from the code
|
||||
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
|
||||
- review: the stability pass mutation-tests the lines a chunk changes, in place of measuring test coverage
|
||||
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model
|
||||
- routing: a chunk's worker runs on the rung its item picks, below the project's rung too; its reviewers run on the higher of the worker's rung and the project's, the seats and the security pass always on the project's
|
||||
- routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
|
||||
- routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one its item's E + I picks, the seats and the security pass always on the project's
|
||||
- routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, or when the judge reports a taken-in item that picks a higher rung or an escalation trigger the worker cites
|
||||
- routing: loop-claude measures review rounds and unfinished verdicts per rung every 20 cycles, and calls for a routing change when, of two rungs with 5 or more chunks each, the lower took more of either than the one above, or when 5 or more subagents of one type had reports longer than the tool results they read
|
||||
- routing: a worker delegates a task only where it needs just the task's conclusion, and each subagent runs on the rung its task's R + C picks from the project's rung, never below the project's rung when it files or scores todo.md items
|
||||
- routing: a worker delegates a task only where it needs just the task's conclusion, and each subagent runs on the rung its task's L picks from the project's rung, never below the project's rung when it files or scores todo.md items
|
||||
- routing: a worker hands a task one rung up on each escalation trigger, up to two rungs above the rung its item picks unless the human agrees, and opens the subagent's prompt with the trigger so the routing figures count it
|
||||
- routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, resumes one rung up next cycle, up to two rungs above the rung its item picks
|
||||
- routing: on a model judge-replay covered, the judge runs on the rung its chunk's item picks, never below the project's rung
|
||||
- routing: the stability reviewer stays on the project's rung while the worker runs below it, until `stability-replay` shows a lower rung finding every 🔴 and 🟡 the project's rung finds
|
||||
- routing: on a model judge-replay covered, the judge runs on the rung its chunk's item's E + I picks for a review, never below the project's rung
|
||||
- routing: the stability reviewer runs no lower than the project's rung while the worker runs below it, until `stability-replay` shows a lower rung finding every 🔴 and 🟡 the project's rung finds
|
||||
|
||||
## Working rules
|
||||
|
||||
- `routing-calls-raised.md` sits at the root of loop-claude's archive (`$LOOP_CLAUDE_TRANSCRIPTS`, default `~/.claude/loop-claude-transcripts`). Each of its lines is a call or a `Reset`, with a stamp. A call is new when it is stamped after the latest `Reset` and after the latest stamp that `todo.md` or `docs/decisions.md` → `routing:` cites. When there are new calls, file one `todo.md` item to settle them, or refresh the item already filed. Its detail quotes those calls with their stamps, and the routed-chunks table of `routing-figures.md` beside it, never the project names its other tables hold. A change to the ladder, its bands or the delegation rule is the human's. The chunk that settles the calls cites the latest stamp it settled in `docs/decisions.md` → `routing:`, whatever the human decided, so they are not filed again. Once a chunk that changes the ladder, its bands or the delegation rule has merged, append `- <UTC time> Reset` to `routing-calls-raised.md`: the figures then count only chunks finished after it, and its calls may be raised again.
|
||||
- `routing-calls-raised.md` sits at the root of loop-claude's archive (`$LOOP_CLAUDE_TRANSCRIPTS`, default `~/.claude/loop-claude-transcripts`). Each of its lines is a call or a `Reset`, with a stamp. A call is new when it is stamped after the latest `Reset` and after the latest stamp that `todo.md` or `docs/decisions.md` → `routing:` cites. When there are new calls, file one `todo.md` item to settle them, or refresh the item already filed. Its detail quotes those calls with their stamps, and the routed-chunks table of `routing-figures.md` beside it, never the project names its other tables hold. A change to the ladder, its steps or the delegation rule is the human's. The chunk that settles the calls cites the latest stamp it settled in `docs/decisions.md` → `routing:`, whatever the human decided, so they are not filed again. Once a chunk that changes the ladder, its steps or the delegation rule has merged, append `- <UTC time> Reset` to `routing-calls-raised.md`: the figures then count only chunks finished after it, and its calls may be raised again.
|
||||
- When `~/.claude/settings.json` has uncommitted changes, commit them on a branch of their own and merge them through a PR: Claude Code writes that file from any session. Read their diff first. When it holds a credential, such as a token in `env` or `mcpServers`, ask the human to move it out of the file, and commit only once it is gone. A token in `env` moves to the shell profile; a server in `mcpServers` moves to `~/.claude.json` with `claude mcp add --scope user`.
|
||||
|
||||
## Scoring run
|
||||
|
||||
+4
-2
@@ -3,12 +3,14 @@
|
||||
## Unreleased
|
||||
|
||||
- Every goal in `README.md` → Goals is marked `**Maintain:**`, a state to hold, or `**Achieve:**`, a direction to pursue. Each achieve goal has a signal, a counter-signal and a one-line reason under it. A change that breaks a maintain goal is redesigned or dropped, whatever it scores: `rules/todo-items.md` says so, and larv-review's goal alignment step no longer merges such a PR. `rules/goals-and-audience.md` states the form for every project's README. Expect larv-review to stop a PR on a maintain goal where it asked you before.
|
||||
- Reshape every project's `todo.md` to the new shape below: until it is reshaped, that is the project's only work, as it is while README.md lacks Goals and Audience, and `rank_todo.py` says so. First move every machine to this release: an older release rejects the new formula, so it fails on every reshaped `todo.md`. Then mark the README's goals `**Maintain:**` or `**Achieve:**`, and set a bar on the new scale below the score of every item to keep, since the old bar would drop most items. Until then, an old-shape `todo.md` still ranks by its own formula and weights.
|
||||
- Reshape every project's `todo.md` to the new shape below: until it is reshaped, that is the project's only work, as it is while README.md lacks Goals and Audience, and `rank_todo.py` says so. First move every machine to this release: an older release rejects the new formula, so it fails on every reshaped `todo.md`. Then mark the README's goals `**Maintain:**` or `**Achieve:**`, score each item's L, E and I, and set a bar on the new scale below the score of every item to keep, since the old bar would drop most items. Until then, an old-shape `todo.md` still ranks by its own formula and weights.
|
||||
- `todo.md` ranks items by value per cost: this setup's own plan uses `Score = 100 * V / (2**S + 8)`, where 8 is a chunk's fixed cost, about one size-3 change, and `rank_todo.py --help` shows it. A Moves column replaces A, G and Goals, naming each achieve goal an item moves and how far, such as `1.1 by 8, 1.2 by 5`. V sums each named goal's weight times its move. `rank_todo.py` weighs each achieve goal by its position in README.md, the first heaviest, so the goal weights table goes. An item names the most specific goal it moves, never a goal beside its own sub-goal. `rules/todo-items.md` and `rank_todo.py --help` show the new shape.
|
||||
- L, E and I replace C, and set R. L is the chance the change holds a defect, from 1 to 4; E the chance a defect escapes tests and review, from 1 to 3; I the damage it would do, from 1 to 5, counting every project and session that runs the change. `rules/todo-items.md` anchors each level in words, and `rank_todo.py` writes R as L + E + I − 2, at least 7 when I is 5 and at least 5 when I is 4. The table's score columns are `| L | E | I | R | S |`. `score-outcomes` takes `L E I S`, and reports how well L, E and I each order finished chunks by review rounds and 🔴 findings.
|
||||
- `loop-claude` routes a chunk's worker on its item's L, from two rungs below the project's rung at L 1 to one above it at L 4. It routes the reviewers on the item's E + I, one rung above the project's for each point past 4, never below the worker's rung. It runs the judge of an Opus 5.5 project on the rung E + I picks, never below the project's, and keeps it there when the worker steps up. The comprehension seats and the security pass stay on the project's rung. rule-hook holds each reviewer launch to the reviewers' rung, which `loop-claude` hands the worker in `LOOP_CLAUDE_REVIEWER_RUNG`. A delegated task's subagent takes the rung its L picks: `subagent-rung` takes L in place of R + C. Until a project reshapes its `todo.md`, every chunk there runs on the project's rung. Expect reviews of items whose defects would escape far or break much to cost more, and escalation from an item of L 4 to reach Fable 5.1 only with your yes.
|
||||
- An item that restores a maintain goal the project breaks today is exempt as `defect`, and its Moves may name that maintain goal, weighed as the heaviest achieve goal. A defect ranks by its score like any item. Only two kinds rank ahead of the rest of their release: questions first, since an answer may change other items, then security items.
|
||||
- The stability reviewer applies `principles/technical-principles.md` as written, in place of its own copies of it. Its test-data check now asks for UUID v7, where it asked for randomised UUIDs.
|
||||
- larv-review's principles audit is a pass of its own, and runs on every chunk that changes code, config or tests, not only on one with an architectural decision. A chunk of prose or planning alone skips it.
|
||||
- `rule-hook` refuses an Agent call that launches a reviewer, a comprehension seat or the security pass on a rung below the one `loop-claude` names for it, and takes no override: pass the model and effort the refusal names. A reviewer is held to the higher of the worker's rung and the project's; a seat or the security pass to the project's.
|
||||
- `rule-hook` refuses an Agent call that launches a reviewer, a comprehension seat or the security pass on a rung below the one `loop-claude` names for it, and takes no override: pass the model and effort the refusal names.
|
||||
|
||||
## 0.1.0
|
||||
|
||||
|
||||
@@ -52,13 +52,13 @@ The rules only reviewers apply, for quality, comments, READMEs and prose, live i
|
||||
- Keep the project's main checkout (e.g. `~/code/log`, never a worktree) current: when it is behind `origin` — a merge landed, or a fetch shows it — run `git pull --ff-only` there instead of reporting the lag, never with `--autostash`: a dirty checkout is a finding to report, not to carry along.
|
||||
- Run a project's language runtime, package manager, and task runner through its container/tooling setup, never directly on the host unless a decision puts them there. Example: `docker compose run --rm app <tool> …`.
|
||||
- Tear down test containers after a run.
|
||||
- Delegate a task to a subagent only when you need just its conclusion: a search or read across many files, or work independent enough to run in parallel. Keep edits, and whatever you must check line by line, in your own thread: you would read a delegated diff again to check it. Score the task's R and C as for a `todo.md` item. `<model>` below is the project's model id, such as `claude-opus-5-5`: the one loop-claude's prompt names as the project's, or else your own. Launch the subagent with the model and effort of the first case that fits:
|
||||
- Delegate a task to a subagent only when you need just its conclusion: a search or read across many files, or work independent enough to run in parallel. Keep edits, and whatever you must check line by line, in your own thread: you would read a delegated diff again to check it. Score the task's L as for a `todo.md` item; a task that changes no file, such as a search, is L 1. `<model>` below is the project's model id, such as `claude-opus-5-5`: the one loop-claude's prompt names as the project's, or else your own. Launch the subagent with the model and effort of the first case that fits:
|
||||
1. A reviewer, a comprehension seat or the security pass: those loop-claude's prompt gives it. Given none, pass no model or effort, so it runs on your model.
|
||||
2. An escalated task, below: the rung loop-claude's prompt names for escalation, or case 3's rung where the subagent files or scores `todo.md` items and that rung is higher.
|
||||
3. A subagent that files or scores `todo.md` items: what `~/.claude/bin/subagent-rung --plan <model> <R+C>` prints, which is never below the project's rung.
|
||||
4. Any other: what `~/.claude/bin/subagent-rung <model> <R+C>` prints.
|
||||
3. A subagent that files or scores `todo.md` items: what `~/.claude/bin/subagent-rung --plan <model> <L>` prints, which is never below the project's rung.
|
||||
4. Any other: what `~/.claude/bin/subagent-rung <model> <L>` prints.
|
||||
- Escalate a task that proves too hard for your rung. Four escalation triggers say so:
|
||||
1. A re-score of the item, made on the project's rung or above, picks a rung above yours.
|
||||
1. A re-score of the item's L, made on the project's rung or above, picks a rung above yours.
|
||||
2. The same failing test or finding survives two fix attempts.
|
||||
3. A pass reports a 🔴 in round 2 or later.
|
||||
4. A review rules the work below standard: systems-architect's BLOCK, a finding against larv-review's `standards/quality.md`, or a vote `record-ruling` counts as worse, in the report that stands after any relaunch one rung up (`larv-review` §3 → "Rung up").
|
||||
|
||||
@@ -89,10 +89,10 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format.
|
||||
- `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure.
|
||||
- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit, Write and Agent call, and `rule-hook --help` states the rules and the override.
|
||||
- `score-outcomes [R S C]` lists the finished items whose guessed R, S and C sit nearest the given scores, with what their chunks measured, and each score's median outcome; `score-outcomes --help` states the measures.
|
||||
- `score-outcomes [L E I S]` lists the finished items whose guessed scores sit nearest the given ones, with what their chunks measured, the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings; `score-outcomes --help` states the measures.
|
||||
- `stability-replay <model>` replays archived stability reviews of the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again; `stability-replay --help` states the replay. Run it only in a sandbox, like loop-claude: each replayed reviewer has Bash and skips every permission prompt, and an archived prompt may name paths on the host.
|
||||
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
|
||||
- `subagent-rung [--plan] <model> <R+C>` prints the Agent tool's model and effort for a subagent's task, picked by its R + C from the project's model, and with `--plan` never below the project's rung; `subagent-rung --help` states the rule.
|
||||
- `subagent-rung [--plan] <model> <L>` prints the Agent tool's model and effort for a subagent's task, picked by its L from the project's model, and with `--plan` never below the project's rung; `subagent-rung --help` states the rule.
|
||||
|
||||
Test on the host, with `uv` on `PATH` to fetch mypy and the Python version Install names (`docs/decisions.md` → `bin/`): `python3 -m unittest discover -s bin -p 'test_*.py'`
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ It first replays every verdict on the reference judge: the model at xhigh effort
|
||||
judge ran on before loop-claude routed it. A verdict the reference judge returns another kind on
|
||||
may rest on records or judge rules that have changed since; either way no rung is held to it. It
|
||||
then replays the rest on each candidate rung, highest first: from the model's cheapest rung down
|
||||
to the rung the lowest R + C picks. It stops at the first rung that misses one. It prints each
|
||||
to the lowest rung a worker routes to. It stops at the first rung that misses one. It prints each
|
||||
verdict the reference judge did not reproduce, each verdict a rung missed, and the lowest rung
|
||||
that, like each rung above it, missed none. A rung it names is the evidence to add the model to
|
||||
JUDGE_REPLAYED in bin/loop_claude/routing.py, which routes that model's judge, and to record the
|
||||
|
||||
@@ -3,16 +3,17 @@ import json
|
||||
import re
|
||||
import typing
|
||||
|
||||
from loop_claude.cycle.prompts import ANSWER_DESTINATIONS, ANSWERS_RECORDED, either
|
||||
from loop_claude.cycle.prompts import ANSWER_DESTINATIONS, ANSWERS_RECORDED
|
||||
from claude_code.reply import claude_reply, Limited
|
||||
from loop_claude.effects import Effects
|
||||
from failure import Failure
|
||||
from loop_claude.project.records import HUMAN_RULE_NAMES
|
||||
from todo_table import LEVELS
|
||||
from markdown_text import listed
|
||||
from loop_claude.cycle.retry import retried
|
||||
from model_routing.ladder import Rung
|
||||
from loop_claude.stop import Stop
|
||||
from claude_code.entries import structured_output
|
||||
from todo_table import TOP_SCORE
|
||||
|
||||
# At 3 to 4 characters a token, 100,000 to 133,000 tokens, which leaves a 200,000-token context room for the reply; the largest session rendered here was 364,000 characters on 2026-10-03.
|
||||
JUDGE_PROMPT_CHARS = 400000
|
||||
@@ -45,7 +46,7 @@ JUDGE_RULES = (
|
||||
f"its own. Even then, it is finished only once {ANSWERS_RECORDED}. An answer counts as recorded "
|
||||
f"where README.md or the records above hold it, {', '.join(HUMAN_RULE_NAMES)} included. It also counts where "
|
||||
f"the transcript shows the agent writing it into {ANSWER_DESTINATIONS}; writing it into "
|
||||
f"{either(HUMAN_RULE_NAMES)} does not count. finished=false when work remains that a further run can "
|
||||
f"{listed(HUMAN_RULE_NAMES, 'or')} does not count. finished=false when work remains that a further run can "
|
||||
"continue: an open PR, a review not yet run, CI not yet reported, an answer not yet recorded."
|
||||
)
|
||||
|
||||
@@ -54,7 +55,7 @@ JUDGE_RULES = (
|
||||
class Reply:
|
||||
answer: str = dataclasses.field(metadata={"description": "With interrupt false, the passage of README.md or the records that answers the agent's question, quoted verbatim, without elisions. Empty otherwise."})
|
||||
# Routes the next cycle's worker. Counting the item the chunk was routed on is harmless: loop-claude never steps the worker down.
|
||||
bundled: int = dataclasses.field(metadata={"description": "The highest R + C the agent states for a todo.md item it took into this chunk; 0 when it states none."})
|
||||
bundled: int = dataclasses.field(metadata={"description": "The highest L the agent states for a todo.md item it took into this chunk; 0 when it states none."})
|
||||
# Routes the next cycle's worker one rung up: docs/decisions.md → `routing: a worker whose run ends asking for a higher rung`.
|
||||
escalation: str = dataclasses.field(metadata={"description": "Empty unless the chunk is unfinished and the agent's run ends asking loop-claude for a higher rung, citing an escalation trigger. Then: a verbatim quote, without elisions, of an earlier transcript passage showing the trigger fired after any earlier such request, such as a review round's commit title or the agent's report of the finding or failing test. Never the request itself."})
|
||||
evidence: str = dataclasses.field(metadata={"description": "A verbatim quote of the one transcript passage the verdict rests on, without elisions: what triggers the interrupt; for a quoted answer, the agent's question; else what shows the chunk finished or the work remaining."})
|
||||
@@ -65,7 +66,7 @@ class Reply:
|
||||
|
||||
REPLY_TYPES = typing.get_type_hints(Reply)
|
||||
JSON_TYPES = {bool: "boolean", int: "integer", str: "string"}
|
||||
HIGHEST_R_PLUS_C = 2 * TOP_SCORE
|
||||
HIGHEST_L = LEVELS["L"]
|
||||
JUDGE_SCHEMA = {
|
||||
"properties": {field.name: {**field.metadata, "type": JSON_TYPES[REPLY_TYPES[field.name]]} for field in dataclasses.fields(Reply)},
|
||||
"required": list(REPLY_TYPES),
|
||||
@@ -95,7 +96,7 @@ class Finished:
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Unfinished:
|
||||
evidence: str
|
||||
# The bundled R + C the judge saw, 0 for none.
|
||||
# The bundled L the judge saw, 0 for none.
|
||||
bundled: int = 0
|
||||
# The judge's quote showing that the worker's cited escalation trigger fired; empty for none.
|
||||
escalation: str = ""
|
||||
@@ -121,9 +122,9 @@ def verdict_from(output: object) -> Verdict | Failure:
|
||||
return QuotedAnswer(output["evidence"], output["question"], output["answer"])
|
||||
if output["finished"]:
|
||||
return Finished(output["evidence"])
|
||||
if not 0 <= output["bundled"] <= HIGHEST_R_PLUS_C:
|
||||
return Failure(f"the judge's bundled R + C {output['bundled']} is not from 0 to {HIGHEST_R_PLUS_C}")
|
||||
return Unfinished(output["evidence"], output["bundled"], output["escalation"])
|
||||
# A bundled L off its scale counts as none: holding the rung is safe, and three failures would stop an unattended loop.
|
||||
bundled = output["bundled"] if 0 <= output["bundled"] <= HIGHEST_L else 0
|
||||
return Unfinished(output["evidence"], bundled, output["escalation"])
|
||||
|
||||
|
||||
def judge(root: str, task: str, premises: str, records: str, rendered_transcript: str, rung: Rung, effects: Effects) -> Verdict | Stop:
|
||||
|
||||
@@ -1,19 +1,16 @@
|
||||
from model_routing.ladder import climbed, LADDER, review_rung, Rung
|
||||
from model_routing.ladder import climbed, LADDER, position, review_rungs, ReviewKind, Rung
|
||||
from model_routing.route import ESCALATION_MARKER, Route
|
||||
from loop_claude.routing import escalation_rungs
|
||||
from loop_claude.project.records import HUMAN_RULE_NAMES, RECORD_FILES
|
||||
from markdown_text import quoted
|
||||
|
||||
def either(names: tuple[str, ...]) -> str:
|
||||
"""The names as `a, b or c`."""
|
||||
return f"{', '.join(names[:-1])} or {names[-1]}"
|
||||
|
||||
from markdown_text import listed, quoted
|
||||
|
||||
# Where the judge may quote an answer from, named as the human and the worker see them; of README.md the judge sees only the premises.
|
||||
QUOTE_SOURCES = either(("README.md's Goals and Audience", *RECORD_FILES, *HUMAN_RULE_NAMES))
|
||||
QUOTE_SOURCES = listed(("README.md's Goals and Audience", *RECORD_FILES, *HUMAN_RULE_NAMES), "or")
|
||||
# Where the worker may write an answer the human typed, so it counts as recorded.
|
||||
ANSWER_DESTINATIONS = "README.md's Goals or Audience, " + ", ".join(RECORD_FILES) + ", todo.md or a comment"
|
||||
ANSWERS_RECORDED = f"every answer the human typed in it that a later chunk could need is recorded in {ANSWER_DESTINATIONS} at the one place it governs"
|
||||
# Each kind of review, as review_launches names those it launches.
|
||||
REVIEWS: tuple[tuple[str, ReviewKind], ...] = (("each reviewer", "reviewer"), ("each comprehension seat", "seat"), ("the security pass's subagent", "security"))
|
||||
|
||||
|
||||
def fresh(task: str, premises: str, route: Route, anchor: Rung | None) -> str:
|
||||
@@ -23,18 +20,30 @@ def fresh(task: str, premises: str, route: Route, anchor: Rung | None) -> str:
|
||||
|
||||
|
||||
def routed_by(route: Route, anchor: Rung | None) -> str:
|
||||
return f"loop-claude runs you on {route.rung}, routed by item {route.item}'s R + C of {route.risk_and_complexity}. Name each todo.md item this chunk takes, with its R + C, whether or not it is item {route.item}.{off_the_projects_rung(route, anchor)}{escalation(route, anchor)}{reruns(route, anchor)}"
|
||||
return f"loop-claude runs you on {route.rung}, routed by item {route.item}'s L of {route.likelihood}. Name each todo.md item this chunk takes, with its L and E + I, whether or not it is item {route.item}.{review_launches(route, anchor)}{escalation(route, anchor)}{reruns(route, anchor)}"
|
||||
|
||||
|
||||
def off_the_projects_rung(route: Route, anchor: Rung | None) -> str:
|
||||
"""Which subagents to launch on the project's rung, anchor, and below it which on the --plan rung; empty while the worker runs on anchor, as every chunk no item routed does until it steps up."""
|
||||
if anchor is None or route.rung == anchor:
|
||||
def review_launches(route: Route, anchor: Rung | None) -> str:
|
||||
"""Which reviews to launch on a rung other than the worker's, and on which; below the project's rung, anchor, also where to launch a subagent that files or scores todo.md items. Empty off the ladder, and where every review runs on the worker's rung.
|
||||
|
||||
For a worker on Sonnet 5.5 at high, on an Opus 5.5 project, routed by item 3 of E + I 5, it opens: ` Launch each reviewer with the Agent tool set to model opus, effort high, the rung item 3's E + I of 5 picks. Launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.`
|
||||
"""
|
||||
if anchor is None:
|
||||
return ""
|
||||
# Below the project's rung, the reviewers stay on it to hold quality; above it, the seats score every chunk on one rung, and the security pass spends no more than the project chose.
|
||||
below = review_rung("reviewer", anchor, route.rung) == anchor
|
||||
launched = "each reviewer, each comprehension seat and the security pass's subagent" if below else "each comprehension seat and the security pass's subagent"
|
||||
planning = f" Launch each subagent that files or scores todo.md items on the rung `~/.claude/bin/subagent-rung --plan {anchor.model} <R+C>` prints." if below else ""
|
||||
return f" Your rung is {'below' if below else 'above'} the project's, {anchor}: launch {launched} with the Agent tool set to {anchor.agent_args()}. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them.{planning}"
|
||||
rungs = review_rungs(anchor, route.rung, route.escape_and_impact)
|
||||
launched: dict[Rung, list[str]] = {}
|
||||
for reviews, kind in REVIEWS:
|
||||
if rungs[kind] != route.rung:
|
||||
launched.setdefault(rungs[kind], []).append(reviews)
|
||||
launches = "".join(f" Launch {listed(reviews, 'and')} with the Agent tool set to {rung.agent_args()}, {rung_named(rung, route, anchor)}." for rung, reviews in launched.items())
|
||||
scan = " Run a full scan only from such a subagent, since the scan's agents run on the model that starts them." if rungs["security"] != route.rung else ""
|
||||
planning = f" Your rung is below the project's: launch each subagent that files or scores todo.md items on the rung `~/.claude/bin/subagent-rung --plan {anchor.model} <L>` prints." if position(route.rung) < position(anchor) else ""
|
||||
return f"{launches}{scan}{planning}"
|
||||
|
||||
|
||||
def rung_named(rung: Rung, route: Route, anchor: Rung) -> str:
|
||||
"""Why a review runs on rung: it is the project's, anchor, or the one the route's item's E + I picks."""
|
||||
return "the project's rung" if rung == anchor else f"the rung item {route.item}'s E + I of {route.escape_and_impact} picks"
|
||||
|
||||
|
||||
def escalation(route: Route, anchor: Rung | None) -> str:
|
||||
@@ -55,13 +64,13 @@ def reruns(route: Route, anchor: Rung | None) -> str:
|
||||
"""The prompt's sentence naming the rung one above a reviewer's and a seat's, on which each reruns when it asks; empty off the ladder, and leaving out one already on the top rung."""
|
||||
if anchor is None or route.rung not in LADDER:
|
||||
return ""
|
||||
# The rungs off_the_projects_rung launches them on: the seats always on anchor, the reviewers on anchor only while the worker runs below it.
|
||||
launched = (("reviewer", review_rung("reviewer", anchor, route.rung)), ("comprehension seat", review_rung("seat", anchor, route.rung)))
|
||||
rungs = review_rungs(anchor, route.rung, route.escape_and_impact)
|
||||
launched = (("reviewer", rungs["reviewer"]), ("comprehension seat", rungs["seat"]))
|
||||
rerun = {name: climbed(rung, 1) for name, rung in launched if rung != LADDER[-1]}
|
||||
if not rerun:
|
||||
return ""
|
||||
if len(set(rerun.values())) == 1:
|
||||
return f" A {' or '.join(rerun)} that asks for a rerun one rung up reruns with the Agent tool set to `{next(iter(rerun.values())).agent_args()}`."
|
||||
return f" A {listed(list(rerun), 'or')} that asks for a rerun one rung up reruns with the Agent tool set to `{next(iter(rerun.values())).agent_args()}`."
|
||||
return f" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `{rerun['reviewer'].agent_args()}`, and a comprehension seat with `{rerun['comprehension seat'].agent_args()}`."
|
||||
|
||||
|
||||
@@ -69,7 +78,7 @@ def resumed(unfinished_evidence: str, route: Route, anchor: Rung | None) -> str:
|
||||
"""Names the worker's rung, so it learns whether its request for a higher rung went through, and the project's again, since a step-up between cycles may have taken the worker above it."""
|
||||
on = f" loop-claude runs you on {route.rung}." if anchor else ""
|
||||
ruled = f" A judge ruled the chunk unfinished, quoting this passage of your transcript as the work remaining:\n\n{quoted(unfinished_evidence)}\n\n" if unfinished_evidence else " "
|
||||
return f"Continue the task; the previous run ended before the chunk was finished.{ruled}The chunk is finished once it is merged, or nothing remains that you could do on your own. Even then, it is finished only once {ANSWERS_RECORDED}.{on}{off_the_projects_rung(route, anchor)}{escalation(route, anchor)}{reruns(route, anchor)}"
|
||||
return f"Continue the task; the previous run ended before the chunk was finished.{ruled}The chunk is finished once it is merged, or nothing remains that you could do on your own. Even then, it is finished only once {ANSWERS_RECORDED}.{on}{review_launches(route, anchor)}{escalation(route, anchor)}{reruns(route, anchor)}"
|
||||
|
||||
|
||||
def answered(question: str, answer: str) -> str:
|
||||
|
||||
@@ -93,7 +93,7 @@ def work(cycle: Cycle, answered: QuotedAnswer | None) -> WorkerRan | Stop:
|
||||
|
||||
def worker_environment(cycle: Cycle) -> dict[str, str]:
|
||||
"""The rungs the worker's claude is handed, which rule-hook holds its review launches to."""
|
||||
return worker_rungs.environment(cycle.anchor, cycle.route.rung)
|
||||
return worker_rungs.environment(cycle.anchor, cycle.route)
|
||||
|
||||
|
||||
def worker_start(cycle: Cycle, answered: QuotedAnswer | None) -> list[str]:
|
||||
|
||||
@@ -42,37 +42,44 @@ run, is asked for one. LOOP_CLAUDE_JUDGE_MODEL overrides it for the judge, LOOP_
|
||||
for the worker and the interrupt; with both set, nothing is asked.
|
||||
|
||||
Where that model is on the routing ladder (bin/model_routing/ladder.py), the worker and the
|
||||
interrupt run on one of the ladder's rungs: a model at an effort. As a chunk starts, the R + C of
|
||||
todo.md's top item in the next release that does not wait picks the rung: an item waits while it is
|
||||
marked question, or while an item that blocks it is still in the table. The R + C
|
||||
counts from the model's cheapest rung, and an R + C of 8 or less picks a rung below it; without
|
||||
such an item, or with a C missing, the chunk runs on the model's cheapest rung. The worker's first
|
||||
prompt names the rung and the item, and asks the worker to name each item the chunk takes, with its
|
||||
R + C. When that rung is below the model's cheapest, the prompt also has the worker launch each
|
||||
reviewer, each comprehension seat and the security pass in subagents on the cheapest rung. When it
|
||||
is above, the prompt has the worker launch only each comprehension seat and the security pass on
|
||||
the cheapest rung. The prompt after each unfinished verdict
|
||||
says this again, since a step-up may have taken the worker above the cheapest rung. A reviewer or
|
||||
interrupt run on one of the ladder's rungs: a model at an effort. As a chunk starts, todo.md's top
|
||||
item in the next release that does not wait routes it: an item waits while it is marked question,
|
||||
or while an item that blocks it is still in the table. The item's L picks the worker's rung,
|
||||
stepping from the model's cheapest rung by WORKER_STEPS in bin/model_routing/ladder.py, below it
|
||||
for a low L; without such an item, or with its L, E or I unscored, the chunk runs on the model's
|
||||
cheapest rung. The
|
||||
item's E + I picks the reviewers' rung, stepping up from the cheapest rung by REVIEW_STEPS there,
|
||||
never below the worker's rung. The worker's first prompt names the rung and the item, and asks the worker to
|
||||
name each item the chunk takes, with its L and E + I. The prompt names the rung of each review that
|
||||
runs on a rung other than the worker's: the reviewers' rung for the reviewers, and the cheapest rung
|
||||
for each comprehension seat and the security pass. When the worker runs below the cheapest rung,
|
||||
the prompt has it launch each subagent that files or scores todo.md items on the rung
|
||||
subagent-rung --plan prints. The prompt after
|
||||
each unfinished verdict says this again, since a step-up may have moved the worker. A reviewer or
|
||||
seat that rules the work below standard may ask to rerun one rung up; each prompt names that rung.
|
||||
loop-claude hands the worker's claude its own rung and the project's, in LOOP_CLAUDE_WORKER_RUNG
|
||||
and LOOP_CLAUDE_PROJECT_RUNG, and rule-hook refuses a review launched below the rung named for it.
|
||||
The judge reports the highest R + C the worker states, and the next
|
||||
cycle steps up to the rung that R + C picks, never down. Every second unfinished verdict on a chunk steps it up one rung.
|
||||
So does an unfinished verdict on a worker that ended its run asking for a higher rung, once the judge quotes
|
||||
the transcript showing that the escalation trigger it named fired
|
||||
(~/.claude/CLAUDE.md → "Working with the codebase" lists them). That step stops two rungs above the rung the chunk's item picks, and the
|
||||
worker's prompts offer it only while the worker is below that limit. Each
|
||||
finished or unfinished verdict, stop and step-up in the archive names the worker's rung, with the
|
||||
item and R + C that routed the chunk; a rerun resumes an open chunk on the latest rung named there,
|
||||
and counts its unfinished verdicts afresh. A rung no longer on the ladder, or below the one its
|
||||
item's R + C picks, is dropped, and the chunk is routed afresh from todo.md. A todo.md or an
|
||||
archive that cannot be read or parsed is said on stderr, and the chunk is routed without it.
|
||||
loop-claude hands the worker's claude its own rung, the project's and the reviewers', in
|
||||
LOOP_CLAUDE_WORKER_RUNG, LOOP_CLAUDE_PROJECT_RUNG and LOOP_CLAUDE_REVIEWER_RUNG, and rule-hook
|
||||
refuses a review launched below the rung named for it. The judge reports the highest L the worker
|
||||
states, and the next cycle steps up to the rung that L picks, never down; an L off its scale counts
|
||||
as none. Every second unfinished
|
||||
verdict on a chunk steps it up one rung. So does an unfinished verdict on a worker that ended its
|
||||
run asking for a higher rung, once the judge quotes the transcript showing that the escalation
|
||||
trigger it named fired (~/.claude/CLAUDE.md → "Working with the codebase" lists them). That step
|
||||
stops two rungs above the rung the chunk's item picks, and the worker's prompts offer it only while
|
||||
the worker is below that limit. Each finished or unfinished verdict, stop and step-up in the
|
||||
archive names the worker's rung, with the item, L and E + I that routed the chunk; a rerun resumes
|
||||
an open chunk on the latest rung named there, and counts its unfinished verdicts afresh. A rung no
|
||||
longer on the ladder, or below the one its item's L picks, is dropped, and the chunk is routed
|
||||
afresh from todo.md. So is a chunk whose archived line names an R + C, as loop-claude wrote before
|
||||
it routed on L. A todo.md or an archive that cannot be read or parsed is said on stderr, and the chunk is routed
|
||||
without it.
|
||||
|
||||
The judge of a model listed in JUDGE_REPLAYED (bin/loop_claude/routing.py) runs on the rung the
|
||||
routed item's R + C picks from that model, never below the model's cheapest rung, and on the
|
||||
cheapest rung when no item routed the chunk. Its rung stays put when the worker steps up or takes in
|
||||
another item. A model is listed once judge-replay found its cheapest rung returning every verdict
|
||||
the reference judge reproduced. Any other model's judge runs at the effort its settings give.
|
||||
routed item's E + I picks for a review from that model, never below the model's cheapest rung, and
|
||||
on the cheapest rung when no item routed the chunk. Its rung stays put when the worker steps up or
|
||||
takes in another item. A model is listed once judge-replay found its cheapest rung returning every
|
||||
verdict the reference judge reproduced. Any other model's judge runs at the effort its settings
|
||||
give.
|
||||
|
||||
Counting the cycles a verdict closed across all projects in the archive, the first loop to close
|
||||
one 20 or more cycles after the last measurement measures every finished chunk there, with no model
|
||||
@@ -110,9 +117,8 @@ Exit 1, saying why, when:
|
||||
- the worker or the judge fails three times running, retried a minute apart within the cycle. A
|
||||
worker fails when claude cannot start, exits non-zero, is killed, or ends without a result event;
|
||||
a judge when claude cannot start, exits non-zero or prints no json reply, or its reply misses or
|
||||
mistypes a field, interrupts or quotes an answer without a question, gives an unfinished verdict
|
||||
an R + C outside 0 to 20, or quotes a passage the transcript does not hold; it holds an elided
|
||||
passage when each fragment appears in order.
|
||||
mistypes a field, interrupts or quotes an answer without a question, or quotes a passage the
|
||||
transcript does not hold; it holds an elided passage when each fragment appears in order.
|
||||
A usage limit is not a failure, and restarts the count of three: the worker or the judge is
|
||||
retried 2 minutes after the limit resets, or every 10 minutes when the reset time is unknown or
|
||||
more than 2 minutes past. The judge's reply gives its reset time only as text, so the judge is
|
||||
@@ -203,7 +209,7 @@ def run_loop(argv: list[str], env: typing.Mapping[str, str], paths: Paths, launc
|
||||
|
||||
def routed(loop_run: LoopRun, anchor: Rung | None, archived: typing.Callable[[], Route | None], item: typing.Callable[[], Item | None]) -> LoopRun:
|
||||
"""The loop run with its chunk's route, or None when the project's model is off the ladder. A chunk keeps its route, or on a rerun the one its archive last names,
|
||||
while that is at or above the rung its item's R + C picks from anchor, the project's rung; else the plan item routes it afresh."""
|
||||
while that is at or above the rung its item's L picks from anchor, the project's rung; else the plan item routes it afresh."""
|
||||
if anchor is None:
|
||||
return dataclasses.replace(loop_run, route=None)
|
||||
route = loop_run.route or archived()
|
||||
|
||||
@@ -32,24 +32,36 @@ def decode_handed(handed: str, kind: type[T]) -> T | None:
|
||||
except ValueError:
|
||||
return None
|
||||
decoded = decoded_as(fields, kind)
|
||||
return None if decoded is REJECTED else typing.cast(T, decoded)
|
||||
return None if decoded in (REJECTED, RETIRED) else typing.cast(T, decoded)
|
||||
|
||||
|
||||
REJECTED = object()
|
||||
# A dataclass handed with a field its RETIRED names: where None may stand in for it, it decodes as None.
|
||||
RETIRED = object()
|
||||
|
||||
|
||||
def decoded_as(value: object, kind: typing.Any) -> typing.Any:
|
||||
"""value as kind, or REJECTED: a dataclass from an object holding its fields, a union's first member that fits, else a value of exactly kind."""
|
||||
"""value as kind, REJECTED or RETIRED: a dataclass from an object holding its fields, a union's first member that fits, else a value of exactly kind."""
|
||||
if isinstance(kind, types.UnionType):
|
||||
return next((found for option in typing.get_args(kind) if (found := decoded_as(value, option)) is not REJECTED), REJECTED)
|
||||
found = [decoded_as(value, option) for option in typing.get_args(kind)]
|
||||
fits = [each for each in found if each is not REJECTED and each is not RETIRED]
|
||||
if fits:
|
||||
return fits[0]
|
||||
return None if RETIRED in found and type(None) in typing.get_args(kind) else REJECTED
|
||||
if isinstance(kind, type) and dataclasses.is_dataclass(kind):
|
||||
shape = typing.get_type_hints(kind)
|
||||
if isinstance(value, dict) and value.keys() & getattr(kind, "RETIRED", frozenset()):
|
||||
return RETIRED
|
||||
hints = typing.get_type_hints(kind)
|
||||
shape = {field.name: hints[field.name] for field in dataclasses.fields(kind)}
|
||||
# A field with a default may be missing, so a field added since the running script started does not stop the loop.
|
||||
required = {field.name for field in dataclasses.fields(kind) if field.default is dataclasses.MISSING and field.default_factory is dataclasses.MISSING}
|
||||
if not isinstance(value, dict) or not required <= value.keys() <= shape.keys():
|
||||
return REJECTED
|
||||
fields = {name: decoded_as(field, shape[name]) for name, field in value.items()}
|
||||
return REJECTED if any(field is REJECTED for field in fields.values()) else kind(**fields)
|
||||
# A dataclass that checks its fields, as Route does, does so in a static method named fits.
|
||||
if any(field in (REJECTED, RETIRED) for field in fields.values()) or not getattr(kind, "fits", lambda _: True)(fields):
|
||||
return REJECTED
|
||||
return kind(**fields)
|
||||
return value if type(value) is kind else REJECTED
|
||||
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@ def raise_calls(root: pathlib.Path, new: list[str], rung: Rung, paths: Paths, ef
|
||||
if not effects.on_terminal():
|
||||
print("Not on a terminal: the calls are raised again at the next measurement.", file=sys.stderr)
|
||||
return None
|
||||
prompt = f"Put these calls for a routing change from loop-claude's figures to the human, and change nothing until they answer:\n{listed}The figures are in {root / archive.FIGURES}. The ladder, its bands and the delegation rule are docs/decisions.md → `routing:`, and a change to them is the human's. Record what they decide as ~/.claude/CLAUDE.md says; the next chunk of this repo files the plan item for these calls."
|
||||
prompt = f"Put these calls for a routing change from loop-claude's figures to the human, and change nothing until they answer:\n{listed}The figures are in {root / archive.FIGURES}. The ladder, its steps and the delegation rule are docs/decisions.md → `routing:`, and a change to them is the human's. Record what they decide as ~/.claude/CLAUDE.md says; the next chunk of this repo files the plan item for these calls."
|
||||
if unstarted := hand_over(rung, [prompt], str(paths.claude_dir), paths, {}, effects):
|
||||
return Stop(1, f"could not start claude for the routing calls: {unstarted.reason}")
|
||||
# Raised once the human has seen them, answered or not: an unanswered call reaches the plan through AGENTS.md, so it is not asked twice.
|
||||
|
||||
+12
-12
@@ -1,7 +1,7 @@
|
||||
"""Which ladder rung a chunk's worker and judge run on: docs/decisions.md → `routing:`."""
|
||||
import dataclasses
|
||||
|
||||
from model_routing.ladder import climbed, highest, LADDER, picked, position, project_rung, Rung, stepped_up
|
||||
from model_routing.ladder import climbed, highest, LADDER, position, project_rung, review_floor, Rung, stepped_up, worker_rung
|
||||
from failure import Failure
|
||||
from todo_table import Item, items, waiting
|
||||
from model_routing.route import Route
|
||||
@@ -14,21 +14,21 @@ JUDGE_REPLAYED = ("claude-opus-5-5",)
|
||||
|
||||
|
||||
def routed_item(todo: str) -> Item | Failure | None:
|
||||
"""The next release's first row that does not wait, as todo_table.waiting says, in the table's order; None when there is none, or it has no C."""
|
||||
"""The next release's first row that does not wait, as todo_table.waiting says, in the table's order; None when there is none, or it lacks L, E or I."""
|
||||
found = items(todo)
|
||||
if isinstance(found, Failure) or not found:
|
||||
return found if isinstance(found, Failure) else None
|
||||
next_release = min(item.release for item in found)
|
||||
waits = waiting(found)
|
||||
takes = next((item for item in found if item.release == next_release and item.id not in waits), None)
|
||||
return takes if takes and takes.risk_and_complexity else None
|
||||
return takes if takes and takes.likelihood and takes.escape_and_impact else None
|
||||
|
||||
|
||||
def first_route(anchor: Rung, item: Item | None) -> Route:
|
||||
"""The route a chunk starts on: the rung its item's R + C picks from anchor, the project's rung."""
|
||||
"""The route a chunk starts on: the rung its item's L picks from anchor, the project's rung."""
|
||||
if item is None:
|
||||
return Route(anchor)
|
||||
return Route(picked(anchor, item.risk_and_complexity), item.id, item.risk_and_complexity)
|
||||
return Route(worker_rung(anchor, item.likelihood), item.id, item.likelihood, item.escape_and_impact)
|
||||
|
||||
|
||||
def kept(route: Route, anchor: Rung) -> bool:
|
||||
@@ -42,8 +42,8 @@ def kept_rung(anchor: Rung, route: Route) -> Rung:
|
||||
|
||||
|
||||
def item_rung(anchor: Rung, route: Route) -> Rung:
|
||||
"""The rung the route's item's R + C picks from anchor, the project's rung, or anchor itself when no item routed it. The worker may run on a rung below anchor."""
|
||||
return picked(anchor, route.risk_and_complexity) if route.item else anchor
|
||||
"""The rung the route's item's L picks from anchor, the project's rung, or anchor itself when no item routed it. The worker may run on a rung below anchor."""
|
||||
return worker_rung(anchor, route.likelihood) if route.item else anchor
|
||||
|
||||
|
||||
def escalation_rungs(anchor: Rung, route: Route) -> tuple[Rung, ...]:
|
||||
@@ -62,7 +62,7 @@ def escalation_cap(anchor: Rung, route: Route) -> Rung:
|
||||
|
||||
|
||||
def stepped(anchor: Rung, route: Route, bundled: int, escalated: bool) -> Route:
|
||||
"""The route after one more unfinished verdict, never on a lower rung.
|
||||
"""The route after one more unfinished verdict. Its rules, in order: the rung never falls below kept_rung; a bundled item lifts it to the rung that item's L picks; every second unfinished verdict steps it up one rung; so does an escalation the judge found, up to escalation_cap, two rungs above item_rung; and both steps on one verdict make one step.
|
||||
|
||||
bundled and escalated are the verdict's fields, not the verdict, since loop_claude.cycle.judge imports this module through prompts.
|
||||
"""
|
||||
@@ -77,12 +77,12 @@ def stepped(anchor: Rung, route: Route, bundled: int, escalated: bool) -> Route:
|
||||
|
||||
|
||||
def taken_in(anchor: Rung, route: Route, bundled: int) -> Rung:
|
||||
"""kept_rung, raised to the rung bundled picks from anchor; bundled is the highest R + C the judge saw the worker take into the chunk, 0 for none."""
|
||||
return highest(kept_rung(anchor, route), picked(anchor, bundled)) if bundled else kept_rung(anchor, route)
|
||||
"""kept_rung, raised to the rung bundled picks from anchor; bundled is the highest L the judge saw the worker take into the chunk, 0 for none."""
|
||||
return highest(kept_rung(anchor, route), worker_rung(anchor, bundled)) if bundled else kept_rung(anchor, route)
|
||||
|
||||
|
||||
def judge_rung(judge_model: str, route: Route) -> Rung:
|
||||
"""The rung the route's item picks, counted from the judge model's cheapest rung and never below it, whatever rung the worker stepped to.
|
||||
"""The rung the route's item's E + I picks for a review, counted from the judge model's cheapest rung and never below it, whatever rung the worker stepped to.
|
||||
|
||||
An item the worker takes in beyond the routed one lifts the worker's rung, through taken_in, but not the judge's, until todo.md item 153 lifts the judge too.
|
||||
|
||||
@@ -91,4 +91,4 @@ def judge_rung(judge_model: str, route: Route) -> Rung:
|
||||
anchor = project_rung(judge_model)
|
||||
if anchor is None or judge_model not in JUDGE_REPLAYED:
|
||||
return Rung(judge_model)
|
||||
return highest(anchor, item_rung(anchor, route))
|
||||
return review_floor(anchor, route.escape_and_impact)
|
||||
|
||||
@@ -115,8 +115,10 @@ if os.path.basename(sys.argv[0]) == "loop-claude":
|
||||
"""
|
||||
|
||||
PREMISES = "# Project\n\n## Goals\n\n1. Work.\n\n## Audience\n\nMe.\n"
|
||||
# Item 4's R + C of 11 steps one rung up from the project's.
|
||||
TODO = "## Items\n\n| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |\n|---|---|---|---|---|---|---|---|---|---|---|\n| 4 | 0.1.0 | | **Do it.** | 5 | 2 | 6 | 8 | 9 | 1 | 25 |\n"
|
||||
# Item 4's L of 4 steps one rung up from the project's.
|
||||
TODO = "## Items\n\n| ID | Release | Exempt | Item | L | E | I | R | S | Moves | Score |\n|---|---|---|---|---|---|---|---|---|---|---|\n| 4 | 0.1.0 | | **Do it.** | 4 | 2 | 3 | 7 | 2 | 1 by 9 | 25 |\n"
|
||||
# Item 4's L of 2 steps one rung down.
|
||||
CHEAP = TODO.replace("| 4 | 2 | 3 | 7 |", "| 2 | 1 | 2 | 3 |")
|
||||
|
||||
|
||||
def worker(**step):
|
||||
@@ -369,31 +371,31 @@ class Cycles(unittest.TestCase):
|
||||
self.assertIn('"text": "next chunk"', copies["<UTC> Stopped: Ctrl-C\n\nWorker: worker-model\n"])
|
||||
self.assertEqual(len(list(checkout.archive().glob(f"proj/*/session/{session}.orphaned-1-ab.jsonl"))), 1)
|
||||
|
||||
def test_routes_the_worker_by_its_items_r_plus_c_and_steps_it_up_within_the_chunk(self):
|
||||
def test_routes_the_worker_by_its_items_l_and_steps_it_up_within_the_chunk(self):
|
||||
checkout = Checkout(self)
|
||||
checkout.env["LOOP_CLAUDE_WORKER_MODEL"] = "claude-opus-5-5"
|
||||
(checkout.root / "todo.md").write_text(TODO)
|
||||
(checkout.root / "todo.md").write_text(CHEAP)
|
||||
run = checkout.run([
|
||||
worker(say="one"), judge(evidence="one"),
|
||||
worker(say="two"), judge(evidence="two"),
|
||||
worker(say="three"), judge(evidence="three", bundled=17),
|
||||
worker(say="three"), judge(evidence="three", bundled=4),
|
||||
worker(say="four"), judge(finished=True, evidence="four"),
|
||||
worker(say="next chunk"),
|
||||
])
|
||||
self.assertEqual(run.code, 130)
|
||||
workers = [call for call in run.calls if call["role"] == "worker"]
|
||||
self.assertEqual([(argument(call, "--model"), argument(call, "--effort")) for call in workers], [
|
||||
("claude-sonnet-5-5", "high"),
|
||||
("claude-sonnet-5-5", "high"),
|
||||
("claude-opus-5-5", "medium"),
|
||||
("claude-opus-5-5", "high"),
|
||||
("claude-opus-5-5", "high"),
|
||||
("claude-opus-5-5", "xhigh"),
|
||||
("claude-fable-5-1", "max"),
|
||||
("claude-opus-5-5", "high"),
|
||||
("claude-sonnet-5-5", "high"),
|
||||
])
|
||||
self.assertEqual({(argument(call, "--model"), argument(call, "--effort")) for call in run.calls if call["role"] == "judge"}, {("judge-model", None)})
|
||||
self.assertIn("loop-claude runs you on claude-opus-5-5 at high effort, routed by item 4's R + C of 11. Name each todo.md item this chunk takes, with its R + C, whether or not it is item 4.", argument(workers[0], "-p"))
|
||||
self.assertIn("loop-claude runs you on claude-sonnet-5-5 at high effort, routed by item 4's L of 2. Name each todo.md item this chunk takes, with its L and E + I, whether or not it is item 4.", argument(workers[0], "-p"))
|
||||
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> three\n\nJudge: judge-model\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's R + C of 11\n\n<UTC> Stepped up\n\nWorker: claude-fable-5-1 at max effort, routed by item 4's R + C of 11\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: finished\n\nEvidence:\n> four\n\nJudge: judge-model\nWorker: claude-fable-5-1 at max effort, routed by item 4's R + C of 11\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> three\n\nJudge: judge-model\nWorker: claude-opus-5-5 at medium effort, routed by item 4's L of 2 and E + I of 3\n\n<UTC> Stepped up\n\nWorker: claude-opus-5-5 at high effort, routed by item 4's L of 2 and E + I of 3\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: finished\n\nEvidence:\n> four\n\nJudge: judge-model\nWorker: claude-opus-5-5 at high effort, routed by item 4's L of 2 and E + I of 3\n", outcomes)
|
||||
|
||||
def test_steps_the_worker_up_a_rung_when_the_judge_finds_in_the_transcript_the_escalation_trigger_it_cites(self):
|
||||
checkout = Checkout(self)
|
||||
@@ -414,13 +416,13 @@ class Cycles(unittest.TestCase):
|
||||
("claude-opus-5-5", "high"),
|
||||
])
|
||||
outcomes = sorted(verdict.removeprefix(f"Task: {TASK}\n\n") for verdict in checkout.archived().values())
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at high effort, routed by item 4's R + C of 11\n\n<UTC> Stepped up\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's R + C of 11\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's R + C of 11\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nEscalation:\n> Stability round 2: a 🔴.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at high effort, routed by item 4's L of 4 and E + I of 5\n\n<UTC> Stepped up\n\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5\n", outcomes)
|
||||
self.assertIn("<UTC> Verdict: unfinished\n\nEvidence:\n> I ask for a higher rung.\n\nJudge: judge-model\nWorker: claude-opus-5-5 at xhigh effort, routed by item 4's L of 4 and E + I of 5\n", outcomes)
|
||||
|
||||
def test_runs_a_cheap_items_worker_below_the_projects_rung_with_its_reviewers_and_judge_on_it(self):
|
||||
checkout = Checkout(self)
|
||||
checkout.env["LOOP_CLAUDE_WORKER_MODEL"] = checkout.env["LOOP_CLAUDE_JUDGE_MODEL"] = "claude-opus-5-5"
|
||||
(checkout.root / "todo.md").write_text(TODO.replace("| 5 | 2 | 6 |", "| 2 | 2 | 5 |"))
|
||||
(checkout.root / "todo.md").write_text(CHEAP)
|
||||
run = checkout.run([worker(say="one"), judge(finished=True, evidence="one"), worker(say="next chunk")])
|
||||
self.assertEqual(run.code, 130)
|
||||
first = next(call for call in run.calls if call["role"] == "worker")
|
||||
@@ -428,7 +430,7 @@ class Cycles(unittest.TestCase):
|
||||
ruling = next(call for call in run.calls if call["role"] == "judge")
|
||||
self.assertEqual((argument(ruling, "--model"), argument(ruling, "--effort")), ("claude-opus-5-5", "medium"))
|
||||
self.assertIn("Judge: claude-opus-5-5 at medium effort\nWorker: claude-sonnet-5-5 at high effort", "".join(checkout.archived().values()))
|
||||
self.assertIn("Your rung is below the project's, claude-opus-5-5 at medium effort: launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium", argument(first, "-p"))
|
||||
self.assertIn("Launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.", argument(first, "-p"))
|
||||
|
||||
def test_a_rerun_resumes_an_open_chunk_on_the_rung_it_last_ran(self):
|
||||
checkout = Checkout(self)
|
||||
|
||||
@@ -196,11 +196,11 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: finished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n"])
|
||||
self.assertIsNone(session.transcript_file(self.session))
|
||||
|
||||
def test_a_worker_is_handed_the_project_and_its_own_rung_in_its_environment(self):
|
||||
def test_a_worker_is_handed_the_projects_rung_its_own_and_its_reviewers_in_its_environment(self):
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("finished")])
|
||||
cycle = dataclasses.replace(self.cycle(fake), anchor=Rung("claude-opus-5-5", "medium"), route=Route(Rung("claude-haiku-5-5", "max"), 163, 5))
|
||||
cycle = dataclasses.replace(self.cycle(fake), anchor=Rung("claude-opus-5-5", "medium"), route=Route(Rung("claude-haiku-5-5", "max"), 163, 1, 5))
|
||||
self.assertEqual(run.run_cycle(cycle), Finished("working"))
|
||||
self.assertEqual(fake.worker_environments, [{"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort"}])
|
||||
self.assertEqual(fake.worker_environments, [{"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": "claude-opus-5-5 at high effort"}])
|
||||
|
||||
def test_the_claude_an_interrupt_resumes_is_handed_the_same_rungs_as_the_worker(self):
|
||||
def answer(args):
|
||||
@@ -208,7 +208,7 @@ class Cycles(unittest.TestCase):
|
||||
handle.write(json.dumps(human("the second")) + "\n")
|
||||
return 0
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("interrupt"), judged("finished")], on_terminal_claude=[answer], git=[ran()])
|
||||
cycle = dataclasses.replace(self.cycle(fake), anchor=Rung("claude-opus-5-5", "medium"), route=Route(Rung("claude-haiku-5-5", "max"), 163, 5))
|
||||
cycle = dataclasses.replace(self.cycle(fake), anchor=Rung("claude-opus-5-5", "medium"), route=Route(Rung("claude-haiku-5-5", "max"), 163, 1, 5))
|
||||
self.assertEqual(run.run_cycle(cycle), Finished("working"))
|
||||
self.assertEqual(fake.terminal_environments, fake.worker_environments)
|
||||
self.assertEqual(fake.terminal_environments[0]["LOOP_CLAUDE_WORKER_RUNG"], "claude-haiku-5-5 at max effort")
|
||||
@@ -523,11 +523,11 @@ TODO = """# todo
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |
|
||||
| ID | Release | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 7 | 0.2.0 | question | **Ask first.** | 9 | 1 | 9 | 9 | 9 | 1 | 30 |
|
||||
| 3 | 0.2.0 | | **Do the next thing.** | 5 | 3 | 6 | 8 | 9 | 1 | 25 |
|
||||
| 1 | 0.3.0 | | **Do a later thing.** | 1 | 1 | 1 | 9 | 9 | 1 | 30 |
|
||||
| 7 | 0.2.0 | question | **Ask first.** | 4 | 3 | 5 | 10 | 1 | 1 by 9 | 30 |
|
||||
| 3 | 0.2.0 | | **Do the next thing.** | 4 | 2 | 3 | 7 | 3 | 1 by 9 | 25 |
|
||||
| 1 | 0.3.0 | | **Do a later thing.** | 1 | 1 | 1 | 1 | 1 | 1 by 9 | 30 |
|
||||
"""
|
||||
|
||||
|
||||
@@ -535,90 +535,110 @@ class Routing(unittest.TestCase):
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
HIGH = Rung("claude-opus-5-5", "high")
|
||||
XHIGH = Rung("claude-opus-5-5", "xhigh")
|
||||
MAX = Rung("claude-opus-5-5", "max")
|
||||
SONNET = Rung("claude-sonnet-5-5", "high")
|
||||
|
||||
def test_routes_on_the_next_releases_top_item_that_waits_on_no_question(self):
|
||||
self.assertEqual(routing.routed_item(TODO).id, 3)
|
||||
self.assertIsNone(routing.routed_item(TODO.replace("| | **Do", "| question | **Do")))
|
||||
self.assertIsNone(routing.routed_item("# todo\n"))
|
||||
self.assertIsNone(routing.routed_item(TODO[:TODO.index("| 7 |")]))
|
||||
without_c = TODO.replace(" C |", "").replace("|---|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|---|").replace("| 9 | 1 | 9 |", "| 9 | 1 |").replace("| 5 | 3 | 6 |", "| 5 | 3 |").replace("| 1 | 1 | 1 |", "| 1 | 1 |")
|
||||
self.assertIsNone(routing.routed_item(without_c))
|
||||
self.assertEqual(routing.routed_item(TODO.replace("| 5 | 3 |", "| x | 3 |")), Failure("item 3's R 'x' is not a whole number from 1 to 10"))
|
||||
old_shape = "## Items\n\n| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |\n|---|---|---|---|---|---|---|---|---|---|---|\n| 3 | 0.2.0 | | **Do the next thing.** | 5 | 3 | 6 | 8 | 9 | 1 | 25 |\n"
|
||||
self.assertIsNone(routing.routed_item(old_shape))
|
||||
self.assertEqual(routing.routed_item(TODO.replace("| 4 | 2 | 3 |", "| x | 2 | 3 |")), Failure("item 3's L 'x' is not a whole number from 1 to 4"))
|
||||
|
||||
def test_routes_past_an_item_a_waiting_item_blocks(self):
|
||||
todo = "## Items\n\n| ID | Release | Blocked by | Exempt | Item | R | S | C | A | G | Goals | Score |\n|---|---|---|---|---|---|---|---|---|---|---|---|\n| 7 | 0.2.0 | | question | **Ask first.** | 9 | 1 | 9 | 9 | 9 | 1 | 30 |\n| 8 | 0.2.0 | 7 | | **Use the answer.** | 1 | 1 | 1 | 9 | 9 | 1 | 30 |\n| 9 | 0.2.0 | 8 | | **Build on it.** | 1 | 1 | 1 | 9 | 9 | 1 | 30 |\n| 3 | 0.2.0 | | | **Do the next thing.** | 5 | 3 | 6 | 8 | 9 | 1 | 25 |\n"
|
||||
todo = "## Items\n\n| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |\n|---|---|---|---|---|---|---|---|---|---|---|---|\n| 7 | 0.2.0 | | question | **Ask first.** | 4 | 3 | 5 | 10 | 1 | 1 by 9 | 30 |\n| 8 | 0.2.0 | 7 | | **Use the answer.** | 1 | 1 | 1 | 1 | 1 | 1 by 9 | 30 |\n| 9 | 0.2.0 | 8 | | **Build on it.** | 1 | 1 | 1 | 1 | 1 | 1 by 9 | 30 |\n| 3 | 0.2.0 | | | **Do the next thing.** | 4 | 2 | 3 | 7 | 3 | 1 by 9 | 25 |\n"
|
||||
self.assertEqual(routing.routed_item(todo).id, 3)
|
||||
self.assertEqual(routing.routed_item(todo.replace("| question |", "| |")).id, 7)
|
||||
unranked = todo.replace("| 7 | 0.2.0 | | question | **Ask first.** | 9 | 1 | 9 | 9 | 9 | 1 | 30 |\n", "") + "| 7 | 0.2.0 | | question | **Ask first.** | 9 | 1 | 9 | 9 | 9 | 1 | 30 |\n"
|
||||
self.assertEqual(routing.routed_item(unranked).id, 3)
|
||||
first = "| 7 | 0.2.0 | | question | **Ask first.** | 4 | 3 | 5 | 10 | 1 | 1 by 9 | 30 |\n"
|
||||
self.assertEqual(routing.routed_item(todo.replace(first, "") + first).id, 3)
|
||||
|
||||
def test_starts_a_chunk_on_the_rung_its_items_r_plus_c_picks_below_the_projects_too(self):
|
||||
self.assertEqual(routing.first_route(self.OPUS, routing.routed_item(TODO)), Route(self.HIGH, 3, 11))
|
||||
cheap = routing.routed_item(TODO.replace("| 5 | 3 | 6 |", "| 1 | 3 | 1 |"))
|
||||
self.assertEqual(routing.first_route(self.OPUS, cheap), Route(Rung("claude-haiku-5-5", "xhigh"), 3, 2))
|
||||
def test_starts_a_chunk_on_the_rung_its_items_l_picks_below_the_projects_too(self):
|
||||
self.assertEqual(routing.first_route(self.OPUS, routing.routed_item(TODO)), Route(self.HIGH, 3, 4, 5))
|
||||
cheap = routing.routed_item(TODO.replace("| 4 | 2 | 3 |", "| 1 | 2 | 3 |"))
|
||||
self.assertEqual(routing.first_route(self.OPUS, cheap), Route(Rung("claude-haiku-5-5", "max"), 3, 1, 5))
|
||||
self.assertEqual(routing.first_route(self.OPUS, None), Route(self.OPUS))
|
||||
|
||||
def test_steps_up_a_rung_after_every_second_unfinished_verdict(self):
|
||||
once = routing.stepped(self.OPUS, Route(self.OPUS, 3, 9), 0, False)
|
||||
self.assertEqual(once, Route(self.OPUS, 3, 9, 1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, once, 0, False), Route(self.HIGH, 3, 9, 0))
|
||||
once = routing.stepped(self.OPUS, Route(self.OPUS, 3, 3, 4), 0, False)
|
||||
self.assertEqual(once, Route(self.OPUS, 3, 3, 4, unfinished=1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, once, 0, False), Route(self.HIGH, 3, 3, 4))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(LADDER[-1], unfinished=1), 0, False).rung, LADDER[-1])
|
||||
|
||||
def test_steps_up_to_the_rung_a_bundled_items_r_plus_c_picks_and_never_down(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS), 13, False).rung, self.XHIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 11, False).rung, self.XHIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS, unfinished=1), 13, False).rung, Rung("claude-opus-5-5", "max"))
|
||||
def test_steps_up_to_the_rung_a_bundled_items_l_picks_and_never_down(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS), 4, False).rung, self.HIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 4, False).rung, self.XHIGH)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS, unfinished=1), 4, False).rung, self.XHIGH)
|
||||
|
||||
def test_steps_up_from_a_rung_below_the_projects_one_rung_at_a_time(self):
|
||||
low = Route(Rung("claude-haiku-5-5", "xhigh"), 3, 2, 1)
|
||||
self.assertEqual(routing.stepped(self.OPUS, low, 0, False).rung, Rung("claude-haiku-5-5", "max"))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 2), 0, False).rung, low.rung)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 2), 7, False).rung, Rung("claude-sonnet-5-5", "high"))
|
||||
low = Route(Rung("claude-haiku-5-5", "max"), 3, 1, 3, unfinished=1)
|
||||
self.assertEqual(routing.stepped(self.OPUS, low, 0, False).rung, self.SONNET)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 1, 3), 0, False).rung, low.rung)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(low.rung, 3, 1, 3), 2, False).rung, self.SONNET)
|
||||
|
||||
def test_steps_up_a_rung_on_an_escalation_the_judge_found_up_to_two_above_the_items_rung(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.HIGH, 3, 11), 0, True), Route(self.XHIGH, 3, 11, 0))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH, 3, 11), 0, True), Route(Rung("claude-opus-5-5", "max"), 3, 11, 0))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "max"), 3, 11), 0, True), Route(Rung("claude-opus-5-5", "max"), 3, 11, 1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.HIGH, 3, 11, 1), 0, True), Route(self.XHIGH, 3, 11, 0))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.HIGH, 3, 4, 5), 0, True), Route(self.XHIGH, 3, 4, 5))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH, 3, 4, 5), 0, True), Route(self.MAX, 3, 4, 5))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.MAX, 3, 4, 5), 0, True), Route(self.MAX, 3, 4, 5, unfinished=1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.HIGH, 3, 4, 5, unfinished=1), 0, True), Route(self.XHIGH, 3, 4, 5))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.XHIGH), 0, True), Route(self.XHIGH, unfinished=1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.OPUS, 3, 9), 13, True), Route(self.XHIGH, 3, 9, 1))
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(self.SONNET, 3, 2, 4), 4, True), Route(self.HIGH, 3, 2, 4, unfinished=1))
|
||||
|
||||
def test_a_rung_an_edit_took_off_the_ladder_counts_as_the_projects(self):
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 0, False).rung, self.OPUS)
|
||||
self.assertEqual(routing.stepped(self.OPUS, Route(Rung("claude-opus-5-5", "low")), 2, False).rung, self.OPUS)
|
||||
|
||||
def test_runs_the_judge_on_the_rung_its_chunks_item_picks_never_below_the_projects(self):
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.HIGH, 3, 11)), self.HIGH)
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.XHIGH, 3, 11)), self.HIGH)
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(Rung("claude-haiku-5-5", "xhigh"), 3, 2)), self.OPUS)
|
||||
def test_runs_the_judge_on_the_rung_its_chunks_items_e_plus_i_picks_never_below_the_projects(self):
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.OPUS, 3, 3, 5)), self.HIGH)
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.XHIGH, 3, 4, 5)), self.HIGH)
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.SONNET, 3, 2, 7)), self.MAX)
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(Rung("claude-haiku-5-5", "max"), 3, 1, 3)), self.OPUS)
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5", Route(self.HIGH)), self.OPUS)
|
||||
self.assertEqual(routing.judge_rung("judge-model", Route(self.HIGH, 3, 11)), Rung("judge-model"))
|
||||
self.assertEqual(routing.judge_rung("claude-sonnet-5-5", Route(self.HIGH, 3, 11)), Rung("claude-sonnet-5-5"))
|
||||
self.assertEqual(routing.judge_rung("judge-model", Route(self.HIGH, 3, 4, 5)), Rung("judge-model"))
|
||||
self.assertEqual(routing.judge_rung("claude-sonnet-5-5", Route(self.HIGH, 3, 4, 5)), Rung("claude-sonnet-5-5"))
|
||||
|
||||
def test_refuses_a_route_whose_item_has_levels_off_their_scales_or_levels_without_an_item(self):
|
||||
for item, likelihood, escape_and_impact in ((3, 0, 5), (3, 5, 5), (3, 4, 1), (3, 4, 9), (0, 4, 5), (0, 4, 0), (0, 0, 5)):
|
||||
with self.subTest(item=item, likelihood=likelihood, escape_and_impact=escape_and_impact):
|
||||
with self.assertRaises(ValueError):
|
||||
Route(self.HIGH, item, likelihood, escape_and_impact)
|
||||
|
||||
def test_takes_a_route_whose_item_holds_each_levels_lowest_value(self):
|
||||
self.assertEqual(Route(self.HIGH, 3, 1, 2).escape_and_impact, 2)
|
||||
|
||||
def test_routes_nothing_on_an_item_without_e_or_i(self):
|
||||
without_i = TODO.replace("| E | I | R |", "| E | R |").replace("|---|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|---|").replace("| 4 | 3 | 5 | 10 |", "| 4 | 3 | 10 |").replace("| 4 | 2 | 3 | 7 |", "| 4 | 2 | 7 |").replace("| 1 | 1 | 1 | 1 |", "| 1 | 1 | 1 |")
|
||||
self.assertIsNone(routing.routed_item(without_i))
|
||||
|
||||
def test_archives_the_route_in_a_line_it_reads_back(self):
|
||||
route = Route(self.HIGH, 3, 11)
|
||||
self.assertEqual(str(route), "claude-opus-5-5 at high effort, routed by item 3's R + C of 11")
|
||||
route = Route(self.HIGH, 3, 4, 5)
|
||||
self.assertEqual(str(route), "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 5")
|
||||
self.assertEqual(route_from(str(route)), route)
|
||||
self.assertEqual(route_from(str(Route(self.HIGH))), Route(self.HIGH))
|
||||
self.assertEqual(str(Route(Rung("worker-model"))), "worker-model")
|
||||
self.assertIsNone(route_from("worker-model"))
|
||||
|
||||
def test_reads_no_route_from_a_line_whose_levels_are_off_their_scales_or_an_r_plus_c_route_wrote(self):
|
||||
for line in ("claude-opus-5-5 at high effort, routed by item 3's L of 5 and E + I of 5", "claude-opus-5-5 at high effort, routed by item 3's L of 0 and E + I of 5", "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 9", "claude-opus-5-5 at high effort, routed by item 3's L of 4 and E + I of 1", "claude-opus-5-5 at high effort, routed by item 3's R + C of 11"):
|
||||
with self.subTest(line=line):
|
||||
self.assertIsNone(route_from(line))
|
||||
|
||||
def test_keeps_a_chunks_route_or_its_archived_one_at_or_above_the_rung_its_item_picks_and_else_routes_afresh(self):
|
||||
def run_with(route=None):
|
||||
return main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=route)
|
||||
item = routing.routed_item(TODO)
|
||||
archived = Route(self.XHIGH, 3, 11)
|
||||
archived = Route(self.XHIGH, 3, 4, 5)
|
||||
def never():
|
||||
raise AssertionError("not asked")
|
||||
self.assertIsNone(main.routed(run_with(archived), None, never, never).route)
|
||||
self.assertEqual(main.routed(run_with(archived), self.OPUS, never, never).route, archived)
|
||||
self.assertEqual(main.routed(run_with(), self.OPUS, lambda: archived, never).route, archived)
|
||||
self.assertEqual(main.routed(run_with(), self.OPUS, lambda: None, lambda: item).route, Route(self.HIGH, 3, 11))
|
||||
below = Route(Rung("claude-sonnet-5-5", "high"))
|
||||
self.assertEqual(main.routed(run_with(below), self.OPUS, never, lambda: None).route, Route(self.OPUS))
|
||||
cheap = Route(Rung("claude-sonnet-5-5", "high"), 4, 6)
|
||||
self.assertEqual(main.routed(run_with(), self.OPUS, lambda: None, lambda: item).route, Route(self.HIGH, 3, 4, 5))
|
||||
self.assertEqual(main.routed(run_with(Route(self.SONNET)), self.OPUS, never, lambda: None).route, Route(self.OPUS))
|
||||
cheap = Route(self.SONNET, 4, 2, 3)
|
||||
self.assertEqual(main.routed(run_with(cheap), self.OPUS, never, never).route, cheap)
|
||||
self.assertEqual(main.routed(run_with(Route(Rung("claude-haiku-5-5", "max"), 4, 7)), self.OPUS, never, lambda: None).route, Route(self.OPUS))
|
||||
self.assertEqual(main.routed(run_with(Route(Rung("claude-haiku-5-5", "max"), 4, 2, 3)), self.OPUS, never, lambda: None).route, Route(self.OPUS))
|
||||
|
||||
|
||||
class RouteSources(unittest.TestCase):
|
||||
@@ -628,7 +648,7 @@ class RouteSources(unittest.TestCase):
|
||||
todo = pathlib.Path(root, "todo.md")
|
||||
todo.write_text(TODO)
|
||||
self.assertEqual(main.plan_item(root).id, 3)
|
||||
for text, said in ((TODO.replace("| 5 | 3 |", "| x | 3 |"), "routing on the project's rung: item 3's R 'x' is not a whole number from 1 to 10\n"), (b"\xff", "routing on the project's rung: could not read todo.md: ")):
|
||||
for text, said in ((TODO.replace("| 4 | 2 | 3 |", "| x | 2 | 3 |"), "routing on the project's rung: item 3's L 'x' is not a whole number from 1 to 4\n"), (b"\xff", "routing on the project's rung: could not read todo.md: ")):
|
||||
with self.subTest(said=said):
|
||||
todo.write_bytes(text if isinstance(text, bytes) else text.encode())
|
||||
with contextlib.redirect_stderr(io.StringIO()) as stderr:
|
||||
@@ -644,7 +664,7 @@ class RouteSources(unittest.TestCase):
|
||||
transcript.parent.mkdir(parents=True)
|
||||
transcript.write_text(json.dumps({"type": "queue-operation", "timestamp": "1"}) + "\n")
|
||||
self.assertIsNone(main.archived_route(archive, opened))
|
||||
route = Route(Rung("claude-opus-5-5", "high"), 3, 11)
|
||||
route = Route(Rung("claude-opus-5-5", "high"), 3, 4, 5)
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(route), "working", "j", ""), NOW))
|
||||
self.assertEqual(main.archived_route(archive, opened), route)
|
||||
@@ -664,7 +684,7 @@ class Steps(unittest.TestCase):
|
||||
def test_a_finished_verdict_clears_the_route_and_the_evidence(self):
|
||||
run_with = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="old", route=Route(self.OPUS))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Finished("x")), dataclasses.replace(run_with, route=None, unfinished_evidence=""))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Unfinished("new", 13)), dataclasses.replace(run_with, route=Route(Rung("claude-opus-5-5", "xhigh"), unfinished=1), unfinished_evidence="new"))
|
||||
self.assertEqual(main.after_verdict(run_with, self.OPUS, Unfinished("new", 4)), dataclasses.replace(run_with, route=Route(Rung("claude-opus-5-5", "high"), unfinished=1), unfinished_evidence="new"))
|
||||
|
||||
def test_archives_a_step_up_as_it_happens_so_a_rerun_resumes_on_it(self):
|
||||
with tempfile.TemporaryDirectory() as home:
|
||||
@@ -673,12 +693,12 @@ class Steps(unittest.TestCase):
|
||||
transcript = pathlib.Path(home, "projects", "-fake", f"{opened.id}.jsonl")
|
||||
transcript.parent.mkdir(parents=True)
|
||||
transcript.write_text(json.dumps({"type": "queue-operation", "timestamp": "1"}) + "\n")
|
||||
before = Route(self.OPUS, 3, 9)
|
||||
before = Route(self.OPUS, 3, 3, 4)
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(str(before), "working", "j", ""), NOW))
|
||||
after = main.after_verdict(main.LoopRun(cycle_number=1, root="/r", task="Do it.", unfinished_evidence="", route=before), self.OPUS, Unfinished("working", 13))
|
||||
after = main.after_verdict(main.LoopRun(cycle_number=1, root="/r", task="Do it.", unfinished_evidence="", route=before), self.OPUS, Unfinished("working", 4))
|
||||
self.assertIsNone(main.archive_step(archive, opened, after, before, FakeEffects().effects()))
|
||||
self.assertEqual(main.archived_route(archive, opened), Route(Rung("claude-opus-5-5", "xhigh"), 3, 9))
|
||||
self.assertEqual(main.archived_route(archive, opened), Route(Rung("claude-opus-5-5", "high"), 3, 3, 4))
|
||||
self.assertIsNone(main.archive_step(archive, opened, after, after.route, FakeEffects().effects()))
|
||||
self.assertIsNone(main.archive_step(archive, opened, after, None, FakeEffects().effects()))
|
||||
blocked = pathlib.Path(home, "blocked")
|
||||
@@ -688,59 +708,70 @@ class Steps(unittest.TestCase):
|
||||
|
||||
|
||||
class WorkerArgs(unittest.TestCase):
|
||||
SCAN = " Run a full scan only from such a subagent, since the scan's agents run on the model that starts them."
|
||||
SEATS = " Launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung."
|
||||
|
||||
def test_names_the_route_in_a_fresh_prompt_only_where_an_item_routed_it(self):
|
||||
routed = prompts.fresh("Do it.", "## Goals", Route(Rung("m", "high"), 3, 11), None)
|
||||
self.assertIn("Continue any open draft PR or worktree for the chunk before creating a new one.\n\nloop-claude runs you on m at high effort, routed by item 3's R + C of 11. Name each todo.md item this chunk takes, with its R + C, whether or not it is item 3.\n\nThe chunk's", routed)
|
||||
routed = prompts.fresh("Do it.", "## Goals", Route(Rung("m", "high"), 3, 4, 5), None)
|
||||
self.assertIn("Continue any open draft PR or worktree for the chunk before creating a new one.\n\nloop-claude runs you on m at high effort, routed by item 3's L of 4. Name each todo.md item this chunk takes, with its L and E + I, whether or not it is item 3.\n\nThe chunk's", routed)
|
||||
unrouted = prompts.fresh("Do it.", "## Goals", Route(Rung("m")), None)
|
||||
self.assertIn("before creating a new one.\n\nThe chunk's", unrouted)
|
||||
self.assertNotIn("loop-claude runs you on", unrouted)
|
||||
|
||||
def test_keeps_reviewers_on_the_projects_rung_while_the_worker_runs_below_it(self):
|
||||
def test_keeps_every_review_on_the_projects_rung_while_the_worker_runs_below_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
below = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 5), opus)
|
||||
self.assertIn("whether or not it is item 3. Your rung is below the project's, claude-opus-5-5 at medium effort: launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them. Launch each subagent that files or scores todo.md items on the rung `~/.claude/bin/subagent-rung --plan claude-opus-5-5 <R+C>` prints. When an escalation trigger", below)
|
||||
self.assertNotIn("the project's", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 9), opus))
|
||||
below = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 4), opus)
|
||||
self.assertIn(f"whether or not it is item 3. Launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.{self.SCAN} Your rung is below the project's: launch each subagent that files or scores todo.md items on the rung `~/.claude/bin/subagent-rung --plan claude-opus-5-5 <L>` prints. When an escalation trigger", below)
|
||||
self.assertNotIn("Launch each", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus))
|
||||
|
||||
def test_launches_reviewers_on_the_rung_the_items_e_plus_i_picks_above_the_workers(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
level = prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 6), opus)
|
||||
self.assertIn("whether or not it is item 3. Launch each reviewer with the Agent tool set to model opus, effort xhigh, the rung item 3's E + I of 6 picks. When an escalation trigger", level)
|
||||
below = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 5), opus)
|
||||
self.assertIn(f"whether or not it is item 3. Launch each reviewer with the Agent tool set to model opus, effort high, the rung item 3's E + I of 5 picks.{self.SEATS}{self.SCAN} Your rung is below the project's:", below)
|
||||
self.assertNotIn("Launch each reviewer", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 6), opus))
|
||||
|
||||
def test_keeps_seats_and_the_security_pass_on_the_projects_rung_while_the_worker_runs_above_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
above = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-opus-5-5", "high"), 3, 11), opus)
|
||||
stepped_up = prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 9), opus)
|
||||
self.assertIn(" Your rung is above the project's, claude-opus-5-5 at medium effort: launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them. When an escalation trigger", stepped_up)
|
||||
self.assertNotIn("the project's", prompts.resumed("", Route(opus, 3, 9), opus))
|
||||
self.assertIn(f"{prompts.ANSWERS_RECORDED}. loop-claude runs you on claude-opus-5-5 at high effort. Your rung is above", stepped_up)
|
||||
above = prompts.fresh("Do it.", "## Goals", Route(Rung("claude-opus-5-5", "high"), 3, 4, 4), opus)
|
||||
stepped_up = prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus)
|
||||
self.assertIn(f"{prompts.ANSWERS_RECORDED}. loop-claude runs you on claude-opus-5-5 at high effort.{self.SEATS}{self.SCAN} When an escalation trigger", stepped_up)
|
||||
self.assertNotIn("the project's", prompts.resumed("", Route(opus, 3, 3, 4), opus))
|
||||
self.assertNotIn("runs you on", prompts.resumed("", Route(Rung("m")), None))
|
||||
self.assertIn("whether or not it is item 3. Your rung is above the project's, claude-opus-5-5 at medium effort: launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium. Run a full scan only from such a subagent, since the scan's agents run on the model that starts them. When an escalation trigger", above)
|
||||
self.assertIn(f"whether or not it is item 3.{self.SEATS}{self.SCAN} When an escalation trigger", above)
|
||||
|
||||
def test_names_the_rungs_an_escalation_trigger_lets_the_worker_hand_a_task_to_at_most_two_above_its_items(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
fresh = prompts.fresh("Do it.", "## Goals", Route(opus, 3, 9), opus)
|
||||
fresh = prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus)
|
||||
self.assertIn("whether or not it is item 3. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", fresh)
|
||||
self.assertIn(" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort xhigh`, one rung above yours; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 9), opus))
|
||||
self.assertIn(" An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `model opus, effort max`, one rung above yours, only once the human agrees; loop-claude will not resume you higher for a trigger either, so ask the human for that yes. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 9), opus))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("claude-fable-5-1", "max"), 3, 20), opus))
|
||||
self.assertIn("hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort max`, one rung above yours;", prompts.resumed("", Route(Rung("claude-opus-5-5", "max"), 3, 13), opus))
|
||||
self.assertIn(" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort xhigh`, one rung above yours; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus))
|
||||
self.assertIn(" An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `model opus, effort max`, one rung above yours, only once the human agrees; loop-claude will not resume you higher for a trigger either, so ask the human for that yes. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 3, 4), opus))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("claude-fable-5-1", "max"), 3, 4, 8), opus))
|
||||
self.assertIn("hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort max`, one rung above yours;", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 5), opus))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("m")), None))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("claude-opus-5-5", "low"), 3, 9), opus))
|
||||
self.assertNotIn("escalation", prompts.resumed("", Route(Rung("claude-opus-5-5", "low"), 3, 3, 4), opus))
|
||||
self.assertIn("before creating a new one.\n\nWhen an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort high`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number. Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last such request: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`. A reviewer", prompts.fresh("Do it.", "## Goals", Route(opus), opus))
|
||||
|
||||
def test_offers_the_worker_a_higher_rung_for_the_rest_of_the_chunk_up_to_the_escalation_cap(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
offer = " Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last such request: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model opus, effort xhigh`."
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 9), opus))
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 9), opus))
|
||||
self.assertNotIn("Where the hard part", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 9), opus))
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus))
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer}", prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 3, 4), opus))
|
||||
self.assertNotIn("Where the hard part", prompts.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 3, 4), opus))
|
||||
|
||||
def test_names_the_rung_a_reviewer_or_seat_reruns_on_one_above_its_own(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
both = " A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`."
|
||||
self.assertTrue(prompts.resumed("", Route(opus, 3, 9), opus).endswith(both))
|
||||
self.assertTrue(prompts.resumed("", Route(Rung("claude-sonnet-5-5", "high"), 3, 5), opus).endswith(both))
|
||||
self.assertTrue(prompts.resumed("", Route(opus, 3, 3, 4), opus).endswith(both))
|
||||
self.assertTrue(prompts.resumed("", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 4), opus).endswith(both))
|
||||
self.assertTrue(prompts.fresh("Do it.", "## Goals", Route(opus), opus).split("\n\nThe chunk's")[0].endswith(both))
|
||||
self.assertTrue(prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 11), opus).endswith(" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort xhigh`, and a comprehension seat with `model opus, effort high`."))
|
||||
self.assertTrue(prompts.resumed("", Route(Rung("claude-fable-5-1", "max"), 3, 20), opus).endswith(" A comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`."))
|
||||
self.assertTrue(prompts.resumed("", Route(Rung("claude-opus-5-5", "high"), 3, 4, 4), opus).endswith(" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort xhigh`, and a comprehension seat with `model opus, effort high`."))
|
||||
self.assertTrue(prompts.resumed("", Route(opus, 3, 3, 7), opus).endswith(" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort max`, and a comprehension seat with `model opus, effort high`."))
|
||||
self.assertTrue(prompts.resumed("", Route(Rung("claude-fable-5-1", "max"), 3, 4, 8), opus).endswith(" A comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model opus, effort high`."))
|
||||
self.assertNotIn("rerun", prompts.resumed("", Route(Rung("m")), None))
|
||||
self.assertNotIn("rerun", prompts.fresh("Do it.", "## Goals", Route(Rung("m", "high"), 3, 11), None))
|
||||
self.assertIn(f"{both}\n\nThe chunk's", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 9), opus))
|
||||
self.assertNotIn("rerun", prompts.fresh("Do it.", "## Goals", Route(Rung("m", "high"), 3, 4, 5), None))
|
||||
self.assertIn(f"{both}\n\nThe chunk's", prompts.fresh("Do it.", "## Goals", Route(opus, 3, 3, 4), opus))
|
||||
fable = Rung("claude-fable-5-1", "max")
|
||||
self.assertNotIn("rerun", prompts.resumed("", Route(fable), fable))
|
||||
|
||||
@@ -840,13 +871,13 @@ class Verdict(unittest.TestCase):
|
||||
self.assertEqual(verdict_from({"bundled": 0, "escalation": "", "answer": "", "interrupt": True, "evidence": "x", "finished": False, "question": " "}), Failure("the judge interrupted without a question"))
|
||||
self.assertEqual(verdict_from(None), Failure("the judge returned no reply"))
|
||||
|
||||
def test_takes_a_bundled_r_plus_c_from_0_to_20_on_an_unfinished_verdict(self):
|
||||
def test_takes_a_bundled_l_on_its_scale_and_counts_any_other_as_none(self):
|
||||
reply = {"answer": "", "escalation": "", "interrupt": False, "evidence": "e", "finished": False, "question": ""}
|
||||
self.assertEqual(verdict_from({**reply, "bundled": 20}), Unfinished("e", 20))
|
||||
self.assertEqual(verdict_from({**reply, "bundled": 4}), Unfinished("e", 4))
|
||||
self.assertEqual(verdict_from({**reply, "bundled": True}), Failure("the judge's reply has no bundled of type integer"))
|
||||
for bundled in (21, -1):
|
||||
for bundled in (5, 13, -1):
|
||||
with self.subTest(bundled=bundled):
|
||||
self.assertEqual(verdict_from({**reply, "bundled": bundled}), Failure(f"the judge's bundled R + C {bundled} is not from 0 to 20"))
|
||||
self.assertEqual(verdict_from({**reply, "bundled": bundled}), Unfinished("e", 0))
|
||||
|
||||
|
||||
def test_takes_an_escalation_quote_on_an_unfinished_verdict(self):
|
||||
@@ -871,7 +902,7 @@ class LoopRuns(unittest.TestCase):
|
||||
self.assertIsNone(reload.decode_handed('{"count": "1", "name": "n"}', Stringly))
|
||||
|
||||
def test_decodes_a_route_inside_the_state_or_none(self):
|
||||
state = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=Route(Rung("claude-opus-5-5", "high"), 3, 11, 1))
|
||||
state = main.LoopRun(cycle_number=1, root="/r", task="do it", unfinished_evidence="", route=Route(Rung("claude-opus-5-5", "high"), 3, 4, 5, 1))
|
||||
self.assertEqual(reload.decode_handed(json.dumps(dataclasses.asdict(state)), main.LoopRun), state)
|
||||
unrouted = dataclasses.replace(state, route=None)
|
||||
self.assertEqual(reload.decode_handed(json.dumps(dataclasses.asdict(unrouted)), main.LoopRun), unrouted)
|
||||
@@ -880,6 +911,13 @@ class LoopRuns(unittest.TestCase):
|
||||
self.assertIsNone(reload.decode_handed(json.dumps(handed), main.LoopRun))
|
||||
del handed["route"]["rung"]
|
||||
self.assertIsNone(reload.decode_handed(json.dumps(handed), main.LoopRun))
|
||||
off_scale = dataclasses.asdict(state)
|
||||
off_scale["route"]["likelihood"] = 9
|
||||
self.assertIsNone(reload.decode_handed(json.dumps(off_scale), main.LoopRun))
|
||||
|
||||
def test_drops_a_route_an_older_loop_claude_handed_under_r_plus_c_so_the_chunk_routes_afresh(self):
|
||||
older = {"cycle_number": 3, "root": "/r", "task": "do it", "unfinished_evidence": "", "route": {"rung": {"model": "claude-opus-5-5", "effort": "high"}, "item": 3, "risk_and_complexity": 11, "unfinished": 1}}
|
||||
self.assertEqual(reload.decode_handed(json.dumps(older), main.LoopRun), main.LoopRun(3, "/r", "do it", ""))
|
||||
|
||||
def test_decodes_a_state_from_before_a_field_with_a_default_joined_it(self):
|
||||
self.assertEqual(reload.decode_handed(json.dumps({"cycle_number": 3, "root": "/r", "task": "do it", "unfinished_evidence": ""}), main.LoopRun), main.LoopRun(3, "/r", "do it", ""))
|
||||
@@ -961,7 +999,6 @@ class JudgeFailures(unittest.TestCase):
|
||||
("verdict_from", 'f"the judge\'s reply has no {name}', field),
|
||||
("verdict_from", "'the judge interrupted without a question'", unasked),
|
||||
("verdict_from", "'the judge quoted an answer without a question'", unasked),
|
||||
("verdict_from", 'f"the judge\'s bundled R + C', "gives an unfinished verdict an R + C outside 0 to 20"),
|
||||
("run_judge", "f'{verdict.reason}; it replied:", field),
|
||||
("run_judge", "f'the judge quoted {verdict.evidence!r}, which is not in the transcript", "quotes a passage the transcript does not hold"),
|
||||
("claude_reply", "f'could not start claude:", started),
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import re
|
||||
import typing
|
||||
|
||||
FENCE = re.compile(r"^ {0,3}(`{3,}|~{3,})", re.M)
|
||||
|
||||
@@ -30,3 +31,8 @@ def sections(text: str) -> dict[str, list[str]]:
|
||||
# Blockquoted, so a passage holding blank lines still ends visibly where the text after it starts.
|
||||
def quoted(passage: str) -> str:
|
||||
return "\n".join(f"> {line}" for line in passage.splitlines())
|
||||
|
||||
|
||||
def listed(names: typing.Sequence[str], conjunction: str) -> str:
|
||||
"""The names as `a, b and c`, with conjunction, such as `and` or `or`, before the last."""
|
||||
return names[0] if len(names) == 1 else f"{', '.join(names[:-1])} {conjunction} {names[-1]}"
|
||||
|
||||
+30
-13
@@ -1,4 +1,4 @@
|
||||
"""The routing ladder, and the rung an R + C picks from the project's rung: docs/decisions.md → `routing:`."""
|
||||
"""The routing ladder, and the rungs a task's L and E + I pick from the project's rung: docs/decisions.md → `routing:`."""
|
||||
import dataclasses
|
||||
import re
|
||||
import typing
|
||||
@@ -59,9 +59,10 @@ LADDER = (
|
||||
Rung("claude-opus-5-5", "max"),
|
||||
Rung("claude-fable-5-1", "max"),
|
||||
)
|
||||
# (highest R + C in the band, rungs it steps from the project's rung); a higher R + C steps ABOVE_THE_BANDS.
|
||||
BANDS = ((4, -3), (6, -2), (8, -1), (10, 0), (12, 1), (14, 2), (16, 3))
|
||||
ABOVE_THE_BANDS = 4
|
||||
# The rungs a task's L steps its worker from the project's rung.
|
||||
WORKER_STEPS = {1: -2, 2: -1, 3: 0, 4: 1}
|
||||
# The rungs a task's E + I steps a review up from the project's rung; an E + I under 5 steps none.
|
||||
REVIEW_STEPS = {5: 1, 6: 2, 7: 3, 8: 4}
|
||||
|
||||
|
||||
def project_rung(model: str) -> Rung | None:
|
||||
@@ -69,14 +70,24 @@ def project_rung(model: str) -> Rung | None:
|
||||
return next((rung for rung in LADDER if rung.model == model), None)
|
||||
|
||||
|
||||
def picked(anchor: Rung, risk_and_complexity: int) -> Rung:
|
||||
"""The rung a task's R + C picks from anchor, the project's rung, stopping at either end of the ladder."""
|
||||
return climbed(anchor, next((step for highest, step in BANDS if risk_and_complexity <= highest), ABOVE_THE_BANDS))
|
||||
def worker_rung(project: Rung, likelihood: int) -> Rung:
|
||||
"""The rung a task's L picks for its worker, counted from the project's rung, stopping at either end of the ladder."""
|
||||
return climbed(project, WORKER_STEPS[likelihood])
|
||||
|
||||
|
||||
def floored(anchor: Rung, risk_and_complexity: int) -> Rung:
|
||||
"""The rung a task's R + C picks from anchor, the project's rung, never below anchor."""
|
||||
return highest(anchor, picked(anchor, risk_and_complexity))
|
||||
def lowest_worker_rung(project: Rung) -> Rung:
|
||||
"""The lowest rung any L routes a worker to, counted from the project's rung."""
|
||||
return climbed(project, min(WORKER_STEPS.values()))
|
||||
|
||||
|
||||
def planning_rung(project: Rung, likelihood: int) -> Rung:
|
||||
"""The rung of a task that files or scores todo.md items: the one its L picks, never below the project's rung."""
|
||||
return highest(project, worker_rung(project, likelihood))
|
||||
|
||||
|
||||
def review_floor(project: Rung, escape_and_impact: int) -> Rung:
|
||||
"""The rung a task's E + I picks for a review, counted up from the project's rung. An E + I of 0, for no task, picks the project's rung."""
|
||||
return climbed(project, REVIEW_STEPS.get(escape_and_impact, 0))
|
||||
|
||||
|
||||
def stepped_up(rung: Rung) -> Rung:
|
||||
@@ -84,9 +95,15 @@ def stepped_up(rung: Rung) -> Rung:
|
||||
return climbed(rung, 1)
|
||||
|
||||
|
||||
def review_rung(kind: ReviewKind, project: Rung, worker: Rung) -> Rung:
|
||||
"""The rung a review runs on: a reviewer on the higher of the worker's rung and the project's; a seat or the security pass on the project's."""
|
||||
return highest(project, worker) if kind == "reviewer" else project
|
||||
def review_rungs(project: Rung, worker: Rung, escape_and_impact: int) -> dict[ReviewKind, Rung]:
|
||||
"""The rung each kind of review in a chunk runs on, where its worker runs on worker and its item has that E + I. The reviewers run on the higher of the worker's rung and review_floor's."""
|
||||
reviewers = highest(worker, review_floor(project, escape_and_impact))
|
||||
return {kind: review_rung(kind, project, reviewers) for kind in typing.get_args(ReviewKind)}
|
||||
|
||||
|
||||
def review_rung(kind: ReviewKind, project: Rung, reviewers: Rung) -> Rung:
|
||||
"""The rung a review of kind runs on: a reviewer on the reviewers' rung; a seat or the security pass on the project's, so the seats score every chunk on one rung, and the security pass spends no more than the project chose."""
|
||||
return reviewers if kind == "reviewer" else project
|
||||
|
||||
|
||||
def highest(*rungs: Rung) -> Rung:
|
||||
|
||||
@@ -1,34 +1,77 @@
|
||||
"""A chunk's route: its worker's rung and the plan item that picked it, in the words the archive records each verdict's worker in."""
|
||||
import dataclasses
|
||||
import re
|
||||
import typing
|
||||
|
||||
from todo_table import LEVELS
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
# How a worker opens the prompt of a subagent it escalates a task to, `Escalation trigger 2: …`: docs/decisions.md → `routing: a worker hands a task one rung up`.
|
||||
ESCALATION_MARKER = "Escalation trigger"
|
||||
ROUTED = re.compile(r"(?P<rung>.+?)(?:, routed by item (?P<item>[0-9]+)'s R \+ C of (?P<sum>[0-9]+))?")
|
||||
# An archived worker line: the rung, then the item and the levels that routed it, if any. route_from reads only LEVELS_ROUTED's levels, so a line an older loop-claude wrote, naming an R + C, routes its chunk afresh; archived_worker still reads its rung for the routing figures.
|
||||
ROUTED = re.compile(r"(?P<rung>.+?)(?:, routed by item (?P<item>[0-9]+)'s (?P<levels>.+))?")
|
||||
LEVELS_ROUTED = re.compile(r"L of (?P<likelihood>[0-9]+) and E \+ I of (?P<escape_and_impact>[0-9]+)")
|
||||
|
||||
|
||||
class ArchivedWorker(typing.NamedTuple):
|
||||
"""What an archived worker line names: the worker's rung, the item that routed it, 0 for none, and the text naming that item's levels, which an older loop-claude wrote as an R + C."""
|
||||
rung: Rung
|
||||
item: int
|
||||
levels: str
|
||||
|
||||
|
||||
def on_their_scales(likelihood: int, escape_and_impact: int) -> bool:
|
||||
"""Whether L and E + I are levels on the scales todo_table.LEVELS gives; E and I each start at 1."""
|
||||
return 1 <= likelihood <= LEVELS["L"] and 2 <= escape_and_impact <= LEVELS["E"] + LEVELS["I"]
|
||||
|
||||
|
||||
def levels_fit(item: int, likelihood: int, escape_and_impact: int) -> bool:
|
||||
"""Whether a route may hold these: an item with levels on their scales, or no item and no levels."""
|
||||
return on_their_scales(likelihood, escape_and_impact) if item else not (likelihood or escape_and_impact)
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Route:
|
||||
"""A chunk's worker rung and the plan item whose R + C picked it, 0 for none.
|
||||
"""A chunk's worker rung and the plan item that picked it, 0 for none. The item's L picked the worker's rung; its E + I picks the reviewers'. A route holds only levels that levels_fit, so no rung is picked from a level off its scale.
|
||||
|
||||
unfinished counts the unfinished verdicts towards the next one-rung step; a reload hands it over with the rung.
|
||||
"""
|
||||
# The fields a loop-claude from before L and E + I replaced R + C hands on reload; the reloaded script drops such a route, and the chunk routes afresh.
|
||||
RETIRED: typing.ClassVar[frozenset[str]] = frozenset({"risk_and_complexity"})
|
||||
rung: Rung
|
||||
item: int = 0
|
||||
risk_and_complexity: int = 0
|
||||
likelihood: int = 0
|
||||
escape_and_impact: int = 0
|
||||
unfinished: int = 0
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Raises on a programming error only, so nothing catches it: route_from and the reload decoder check levels_fit before they build a route.
|
||||
if not levels_fit(self.item, self.likelihood, self.escape_and_impact):
|
||||
raise ValueError(f"a route by item {self.item} cannot hold an L of {self.likelihood} and an E + I of {self.escape_and_impact}")
|
||||
|
||||
@staticmethod
|
||||
def fits(fields: typing.Mapping[str, typing.Any]) -> bool:
|
||||
"""Whether a route may be built from these fields, as the reload decoder asks of a handed one."""
|
||||
return levels_fit(fields.get("item", 0), fields.get("likelihood", 0), fields.get("escape_and_impact", 0))
|
||||
|
||||
def __str__(self) -> str:
|
||||
"""What the archive records as each verdict's worker; ROUTED reads it back."""
|
||||
return f"{self.rung}, routed by item {self.item}'s R + C of {self.risk_and_complexity}" if self.item else str(self.rung)
|
||||
return f"{self.rung}, routed by item {self.item}'s L of {self.likelihood} and E + I of {self.escape_and_impact}" if self.item else str(self.rung)
|
||||
|
||||
|
||||
def archived_worker(worker: str) -> ArchivedWorker | None:
|
||||
"""What an archived worker line names; None for a model run off the ladder."""
|
||||
found = ROUTED.fullmatch(worker)
|
||||
rung = Rung.parse(found["rung"]) if found else None
|
||||
return ArchivedWorker(rung, int(found["item"] or 0), found["levels"] or "") if found and rung else None
|
||||
|
||||
|
||||
def route_from(worker: str) -> Route | None:
|
||||
"""The route an archived worker line names; None for a model run off the ladder."""
|
||||
found = ROUTED.fullmatch(worker)
|
||||
if not found:
|
||||
"""The route an archived worker line names; None for a model run off the ladder, or a route by anything but levels on their scales."""
|
||||
archived = archived_worker(worker)
|
||||
if archived is None or not archived.item:
|
||||
return Route(archived.rung) if archived else None
|
||||
levels = LEVELS_ROUTED.fullmatch(archived.levels)
|
||||
if not levels or not levels_fit(archived.item, int(levels["likelihood"]), int(levels["escape_and_impact"])):
|
||||
return None
|
||||
rung = Rung.parse(found["rung"])
|
||||
return Route(rung, int(found["item"] or 0), int(found["sum"] or 0)) if rung else None
|
||||
return Route(archived.rung, archived.item, int(levels["likelihood"]), int(levels["escape_and_impact"]))
|
||||
|
||||
@@ -4,7 +4,8 @@ import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from model_routing.ladder import floored, highest, LADDER, picked, project_rung, review_rung, Rung, stepped_up
|
||||
from model_routing.ladder import highest, LADDER, lowest_worker_rung, planning_rung, project_rung, review_floor, review_rung, review_rungs, Rung, stepped_up, worker_rung, WORKER_STEPS
|
||||
from todo_table import LEVELS
|
||||
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
|
||||
@@ -22,20 +23,31 @@ class Ladder(unittest.TestCase):
|
||||
self.assertIsNone(project_rung("claude-opus-5"))
|
||||
self.assertIsNone(project_rung("opus"))
|
||||
|
||||
def test_steps_each_band_of_r_plus_c_from_the_projects_rung(self):
|
||||
def test_steps_a_workers_rung_by_its_l_from_the_projects_rung(self):
|
||||
position = LADDER.index(OPUS)
|
||||
self.assertEqual([LADDER.index(picked(OPUS, total)) - position for total in range(2, 21)], [-3, -3, -3, -2, -2, -1, -1, 0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 4, 4])
|
||||
self.assertEqual([LADDER.index(worker_rung(OPUS, likelihood)) - position for likelihood in range(1, 5)], [-2, -1, 0, 1])
|
||||
|
||||
def test_steps_a_review_up_by_its_e_plus_i_from_the_projects_rung_and_never_down(self):
|
||||
position = LADDER.index(OPUS)
|
||||
self.assertEqual([LADDER.index(review_floor(OPUS, total)) - position for total in range(2, 9)], [0, 0, 0, 1, 2, 3, 4])
|
||||
self.assertEqual(review_floor(OPUS, 0), OPUS)
|
||||
|
||||
def test_a_step_past_either_end_stops_there(self):
|
||||
self.assertEqual(picked(OPUS, 4), Rung("claude-haiku-5-5", "xhigh"))
|
||||
self.assertEqual(picked(OPUS, 17), Rung("claude-fable-5-1", "max"))
|
||||
self.assertEqual(picked(LADDER[1], 2), LADDER[0])
|
||||
self.assertEqual(worker_rung(OPUS, 1), Rung("claude-haiku-5-5", "max"))
|
||||
self.assertEqual(worker_rung(LADDER[1], 1), LADDER[0])
|
||||
self.assertEqual(review_floor(Rung("claude-opus-5-5", "xhigh"), 8), LADDER[-1])
|
||||
self.assertEqual(stepped_up(LADDER[-1]), LADDER[-1])
|
||||
self.assertEqual(stepped_up(OPUS), Rung("claude-opus-5-5", "high"))
|
||||
|
||||
def test_floors_the_picked_rung_at_the_projects_rung(self):
|
||||
self.assertEqual(floored(OPUS, 2), OPUS)
|
||||
self.assertEqual(floored(OPUS, 12), Rung("claude-opus-5-5", "high"))
|
||||
def test_steps_a_worker_for_every_level_on_ls_scale(self):
|
||||
self.assertEqual(set(WORKER_STEPS), set(range(1, LEVELS["L"] + 1)))
|
||||
|
||||
def test_names_the_lowest_rung_any_l_routes_a_worker_to(self):
|
||||
self.assertEqual(lowest_worker_rung(OPUS), Rung("claude-haiku-5-5", "max"))
|
||||
|
||||
def test_plans_on_the_rung_l_picks_but_never_below_the_projects(self):
|
||||
self.assertEqual(planning_rung(OPUS, 1), OPUS)
|
||||
self.assertEqual(planning_rung(OPUS, 4), Rung("claude-opus-5-5", "high"))
|
||||
|
||||
def test_names_each_rung_by_a_family_the_agent_tool_takes(self):
|
||||
self.assertEqual({rung.agent_args().split(",")[0] for rung in LADDER}, {"model haiku", "model sonnet", "model opus", "model fable"})
|
||||
@@ -80,15 +92,22 @@ class Ladder(unittest.TestCase):
|
||||
self.assertNotIn(Rung.from_agent(model, effort), LADDER)
|
||||
self.assertEqual(Rung.from_agent("opus", "low"), Rung("claude-opus-5-5", "low"))
|
||||
|
||||
def test_holds_a_reviewer_to_the_higher_of_the_workers_rung_and_the_projects_and_anything_else_to_the_projects(self):
|
||||
def test_puts_reviewers_on_the_highest_of_the_projects_rung_the_workers_and_the_one_e_plus_i_picks(self):
|
||||
haiku, fable = LADDER[4], LADDER[-1]
|
||||
self.assertEqual(review_rung("reviewer", OPUS, haiku), OPUS)
|
||||
self.assertEqual(review_rungs(OPUS, haiku, 4)["reviewer"], OPUS)
|
||||
self.assertEqual(review_rungs(OPUS, haiku, 5)["reviewer"], Rung("claude-opus-5-5", "high"))
|
||||
self.assertEqual(review_rungs(OPUS, fable, 5)["reviewer"], fable)
|
||||
self.assertEqual(review_rungs(OPUS, Rung("claude-opus-5-5", "high"), 7)["reviewer"], Rung("claude-opus-5-5", "max"))
|
||||
|
||||
def test_keeps_the_seats_and_the_security_pass_on_the_projects_rung_whatever_the_worker_or_e_plus_i(self):
|
||||
self.assertEqual(review_rungs(OPUS, LADDER[-1], 8), {"reviewer": LADDER[-1], "seat": OPUS, "security": OPUS})
|
||||
|
||||
def test_holds_a_reviewer_to_the_reviewers_rung_and_anything_else_to_the_projects(self):
|
||||
fable = LADDER[-1]
|
||||
self.assertEqual(review_rung("reviewer", OPUS, fable), fable)
|
||||
for kind in ("seat", "security"):
|
||||
with self.subTest(kind=kind):
|
||||
self.assertEqual(review_rung(kind, OPUS, fable), OPUS)
|
||||
self.assertEqual(review_rung(kind, OPUS, haiku), OPUS)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import unittest
|
||||
|
||||
from model_routing.worker_rungs import environment, handed
|
||||
from model_routing.route import Route
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
|
||||
@@ -8,17 +9,19 @@ class Environment(unittest.TestCase):
|
||||
PROJECT = Rung("claude-opus-5-5", "medium")
|
||||
WORKER = Rung("claude-haiku-5-5", "max")
|
||||
|
||||
def test_the_project_and_the_worker_rung_are_named_in_the_environment(self):
|
||||
self.assertEqual(environment(self.PROJECT, self.WORKER), {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort"})
|
||||
def test_the_project_the_worker_and_the_reviewers_rungs_are_named_in_the_environment(self):
|
||||
self.assertEqual(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5)), {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": "claude-opus-5-5 at high effort"})
|
||||
self.assertEqual(environment(self.PROJECT, Route(self.WORKER))["LOOP_CLAUDE_REVIEWER_RUNG"], "claude-opus-5-5 at medium effort")
|
||||
self.assertEqual(environment(self.PROJECT, Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 2))["LOOP_CLAUDE_REVIEWER_RUNG"], "claude-opus-5-5 at xhigh effort")
|
||||
|
||||
def test_a_project_off_the_ladder_is_handed_an_empty_rung_so_none_is_inherited(self):
|
||||
self.assertEqual(environment(None, self.WORKER), {"LOOP_CLAUDE_PROJECT_RUNG": "", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort"})
|
||||
def test_a_project_off_the_ladder_is_handed_empty_rungs_so_none_is_inherited(self):
|
||||
self.assertEqual(environment(None, Route(self.WORKER)), {"LOOP_CLAUDE_PROJECT_RUNG": "", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": ""})
|
||||
|
||||
def test_handed_reads_back_what_environment_named(self):
|
||||
self.assertEqual(handed(environment(self.PROJECT, self.WORKER)), (self.PROJECT, self.WORKER))
|
||||
self.assertEqual(handed(environment(None, self.WORKER)), (None, self.WORKER))
|
||||
self.assertEqual(handed(environment(self.PROJECT, Route(self.WORKER, 3, 1, 5))), (self.PROJECT, self.WORKER, Rung("claude-opus-5-5", "high")))
|
||||
self.assertEqual(handed(environment(None, Route(self.WORKER))), (None, self.WORKER, None))
|
||||
|
||||
def test_a_missing_or_unknown_rung_reads_as_none(self):
|
||||
self.assertEqual(handed({}), (None, None))
|
||||
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 bogus"}), (None, None))
|
||||
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at bogus effort"}), (None, None))
|
||||
self.assertEqual(handed({}), (None, None, None))
|
||||
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 bogus"}), (None, None, None))
|
||||
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at bogus effort"}), (None, None, None))
|
||||
|
||||
@@ -3,20 +3,23 @@
|
||||
loop-claude sets these variables for its worker and for the claude an interrupt resumes; a human's shell does not, and rule-hook then checks no review launch."""
|
||||
import typing
|
||||
|
||||
from model_routing.ladder import LADDER, Rung
|
||||
from model_routing.ladder import LADDER, review_rungs, Rung
|
||||
from model_routing.route import Route
|
||||
|
||||
PROJECT_RUNG = "LOOP_CLAUDE_PROJECT_RUNG"
|
||||
WORKER_RUNG = "LOOP_CLAUDE_WORKER_RUNG"
|
||||
REVIEWER_RUNG = "LOOP_CLAUDE_REVIEWER_RUNG"
|
||||
|
||||
|
||||
def environment(project: Rung | None, worker: Rung) -> dict[str, str]:
|
||||
"""The variables a worker's claude starts with. A project off the ladder gets an empty rung, so none is inherited from the environment."""
|
||||
return {PROJECT_RUNG: str(project) if project else "", WORKER_RUNG: str(worker)}
|
||||
def environment(project: Rung | None, route: Route) -> dict[str, str]:
|
||||
"""The variables a worker's claude on route starts with. A project off the ladder gets empty project and reviewer rungs, so none is inherited from the environment."""
|
||||
reviewers = review_rungs(project, route.rung, route.escape_and_impact)["reviewer"] if project else None
|
||||
return {PROJECT_RUNG: str(project) if project else "", WORKER_RUNG: str(route.rung), REVIEWER_RUNG: str(reviewers) if reviewers else ""}
|
||||
|
||||
|
||||
def handed(environ: typing.Mapping[str, str]) -> tuple[Rung | None, Rung | None]:
|
||||
"""The project's and the worker's rungs, each None when unset or not on the ladder."""
|
||||
return on_the_ladder(environ.get(PROJECT_RUNG)), on_the_ladder(environ.get(WORKER_RUNG))
|
||||
def handed(environ: typing.Mapping[str, str]) -> tuple[Rung | None, Rung | None, Rung | None]:
|
||||
"""The project's, the worker's and the reviewers' rungs, each None when unset or not on the ladder."""
|
||||
return on_the_ladder(environ.get(PROJECT_RUNG)), on_the_ladder(environ.get(WORKER_RUNG)), on_the_ladder(environ.get(REVIEWER_RUNG))
|
||||
|
||||
|
||||
def on_the_ladder(value: str | None) -> Rung | None:
|
||||
|
||||
+50
-36
@@ -7,8 +7,9 @@ Reads todo.md (default: ./todo.md), the README.md beside it, and docs/decisions.
|
||||
one. Rewrites each item's Score from the formula and sorts the items by release; within a release,
|
||||
question rows come first, then security rows, then the rest, each group by score from highest, then
|
||||
by ID. Then it moves each blocker, an item another's Blocked by cell names, that ranks below an item
|
||||
it blocks to just above the first such item. Orders the details to match. Adds the Blocked by column
|
||||
to a table without it. --check writes nothing and fails when the file is not ranked as it would be.
|
||||
it blocks to just above the first such item. Orders the details to match. Writes each item's R, and
|
||||
adds the Blocked by and R columns to a table without them. --check writes nothing and fails when the
|
||||
file is not ranked as it would be.
|
||||
|
||||
A valid todo.md; ## Details and each item's detail are optional:
|
||||
|
||||
@@ -24,9 +25,9 @@ A valid todo.md; ## Details and each item's detail are optional:
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | R | S | C | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| 1 | 0.1.0 | | | **Make the first thing work.** | 2 | 3 | 4 | 1 by 9 | 0 |
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 1 | 0.1.0 | | | **Make the first thing work.** | 2 | 2 | 1 | 3 | 3 | 1 by 9 | 0 |
|
||||
|
||||
## Details
|
||||
|
||||
@@ -34,7 +35,7 @@ A valid todo.md; ## Details and each item's detail are optional:
|
||||
|
||||
What the tagline cannot say.
|
||||
|
||||
- The formula holds numbers, + - * / ** and parentheses over R, S and V.
|
||||
- The formula holds numbers, + - * / ** and parentheses over L, E, I, R, S and V.
|
||||
- V sums, over the goals in an item's Moves cell, each goal's weight times how far the item moves it.
|
||||
README.md marks each goal **Maintain:** or **Achieve:**. The one ranked k of n achieve goals, in
|
||||
reading order, a sub-goal such as 1.1 included, weighs (1/k) / (1 + 1/2 + ... + 1/n). Only an item
|
||||
@@ -42,8 +43,10 @@ A valid todo.md; ## Details and each item's detail are optional:
|
||||
goal.
|
||||
- The bar is judged on the Score as shown, to one decimal.
|
||||
- A new item takes Next ID, and Next ID then goes up by one; every ID stays below it.
|
||||
- An item's R, S and C are whole numbers from 1 to 10. C, how hard the change is to get right, stays
|
||||
out of the formula.
|
||||
- An item's L, E and I are levels that ~/.claude/rules/todo-items.md anchors: L from 1 to 4, E from 1
|
||||
to 3 and I from 1 to 5.
|
||||
rank_todo.py writes its R, L + E + I - 2, at least 7 when I is 5 and at least 5 when I is 4. Its S
|
||||
is a whole number from 1 to 10.
|
||||
- An item's Moves cell is empty, or names each goal it moves and how far, a whole number from 1 to
|
||||
10, such as `1.1 by 8, 1.2 by 5`: the most specific goal, and a parent only for a move none of its
|
||||
sub-goals covers, so never a goal beside its own sub-goal. An empty cell scores V 0.
|
||||
@@ -56,11 +59,11 @@ A valid todo.md; ## Details and each item's detail are optional:
|
||||
loop. An ID at or above Next ID fails; one below it that the table no longer holds was finished
|
||||
or deleted, and is dropped.
|
||||
- ## Items holds the table and nothing else: an order goes in the scores or the Blocked by column.
|
||||
- The table holds the columns above. A table without Blocked by gains it. One without C fails, and so
|
||||
does one with a column this version does not know, since writing it would drop that column.
|
||||
- A table in the old shape, with A, G and Goals columns in place of Moves, still ranks, by a formula
|
||||
over R, S, A, G and W and a | Goal | W | table that weighs every README goal, W being the heaviest
|
||||
cited goal's weight; ranking it says to reshape it.
|
||||
- The table holds the columns above. One missing L, E or I fails, and so does one with a column this
|
||||
version does not know, since writing it would drop that column.
|
||||
- A table in the old shape, with C, A, G and Goals columns in place of L, E, I and Moves, still
|
||||
ranks, by a formula over R, S, A, G and W and a | Goal | W | table that weighs every README goal, W
|
||||
being the heaviest cited goal's weight; ranking it says to reshape it.
|
||||
|
||||
Exit 1, saying why, on bad arguments, a file that cannot be read or written, a todo.md, README.md or
|
||||
docs/decisions.md that breaks the rules above, an item that is not exempt scoring under the bar, or
|
||||
@@ -74,9 +77,9 @@ import re
|
||||
import sys
|
||||
import typing
|
||||
|
||||
from todo_table import BLOCKED_BY, cells, HEADER_START, Item, items_table, MOVES, OLD_SHAPE, parse_row, Shape, shape_of, Table, table_rows
|
||||
from todo_table import BLOCKED_BY, cells, HEADER_START, Item, items_table, MOVES, OLD_SHAPE, parse_row, risk, Shape, SHAPE, shape_of, Table, table_rows
|
||||
from failure import Failure
|
||||
from markdown_text import FENCE, sections
|
||||
from markdown_text import FENCE, listed, sections
|
||||
from file_writes import replace_file
|
||||
|
||||
LARGEST_FORMULA = 200
|
||||
@@ -87,7 +90,7 @@ WEIGHTS_HEADER = "| Goal | W |"
|
||||
FIRST = {"question": 0, "security": 1}
|
||||
# How README.md marks a goal; None where it marks it neither way.
|
||||
Mark = typing.Literal["Maintain", "Achieve"] | None
|
||||
RESHAPE = f"todo.md's items table is in the old shape, scoring A and G against goal weights todo.md states. Until todo.md has the shape rank_todo.py --help shows, reshaping it is the project's only work, as ~/.claude/CLAUDE.md → 'Working loop' says: mark README.md's goals **Maintain:** or **Achieve:**, name each item's {MOVES}, and set the bar below the score of every item to keep"
|
||||
RESHAPE = f"todo.md's items table is in the old shape, scoring A and G against goal weights todo.md states. Until todo.md has the shape rank_todo.py --help shows, reshaping it is the project's only work, as ~/.claude/CLAUDE.md → 'Working loop' says: mark README.md's goals **Maintain:** or **Achieve:**, name each item's {MOVES}, score its L, E and I in place of C, and set the bar below the score of every item to keep"
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -208,11 +211,15 @@ def rank(text: str, readme: str, decisions: str) -> Ranked | Failure:
|
||||
return Ranked(ranked, under_bar, notes(items, scoring))
|
||||
|
||||
|
||||
# The columns ranking fills itself, so a table without one gains it.
|
||||
WRITTEN = (BLOCKED_BY, "R", "Score")
|
||||
|
||||
|
||||
def rewrite_table(lines: list[str], table: Table, shape: Shape, scored: list[tuple[Item, float]]) -> str:
|
||||
"""lines with the table's rows in scored's order, under a header naming every column of shape, so a table from before BLOCKED_BY joined gains the column."""
|
||||
columns = shape.columns
|
||||
"""lines with the table's rows in scored's order, under a header naming, in shape's order, each column the table holds and each of WRITTEN."""
|
||||
columns = tuple(column for column in shape.columns if column in table.columns or column in WRITTEN)
|
||||
header = ["| " + " | ".join(columns) + " |", "|" + "---|" * len(columns)]
|
||||
rows = [render_item(item, value, shape) for item, value in scored]
|
||||
rows = [render_item(item, value, shape, columns) for item, value in scored]
|
||||
return "\n".join(lines[:table.header] + header + rows + lines[table.start + len(table.rows):])
|
||||
|
||||
|
||||
@@ -288,25 +295,31 @@ def read_table(lines: list[str]) -> Table | Failure:
|
||||
return Failure(f"## Items holds no table whose header opens {HEADER_START}; rank_todo.py --help shows a valid one")
|
||||
if isinstance(table, Failure):
|
||||
return table
|
||||
# A table without C fails, so no chunk starts on an unscored item.
|
||||
if "C" not in table.columns:
|
||||
return Failure("todo.md's items table has no C column; add one after S, and score each item's C from 1 to 10 per rank_todo.py --help")
|
||||
return unknown_columns(table) or outside_items(lines, table) or table
|
||||
shape = shape_of(table.columns)
|
||||
if missing := [column for column in shape.risk_columns if column not in table.columns]:
|
||||
old = older_columns(table, shape)
|
||||
dropped = f", and drop {listed(old, 'and')}" if old else ""
|
||||
columns = f"{listed(missing, 'and')} column; add it" if len(missing) == 1 else f"{listed(missing, 'and')} columns; add them"
|
||||
return Failure(f"todo.md's items table has no {columns}, scoring each item's {listed(shape.risk_columns, 'and')} as rank_todo.py --help shows{dropped}")
|
||||
return unknown_columns(table, shape) or outside_items(lines, table) or table
|
||||
|
||||
|
||||
def unknown_columns(table: Table) -> Failure | None:
|
||||
def unknown_columns(table: Table, shape: Shape) -> Failure | None:
|
||||
"""Why ranking, which writes only the columns of the table's shape, would drop one of the table's columns; None when it would drop none."""
|
||||
known = shape_of(table.columns).columns
|
||||
unknown = [column for column in table.columns if column not in known]
|
||||
if old := [column for column in unknown if column in OLD_SHAPE.columns]:
|
||||
listed = old[0] if len(old) == 1 else f"{', '.join(old[:-1])} and {old[-1]}"
|
||||
kind = f"a column of the old shape that {MOVES} replaced; drop it" if len(old) == 1 else f"columns of the old shape that {MOVES} replaced; drop them"
|
||||
return Failure(f"todo.md's items table has {listed}, {kind}")
|
||||
unknown = [column for column in table.columns if column not in shape.columns]
|
||||
if old := older_columns(table, shape):
|
||||
kind = "a column of an older shape; drop it" if len(old) == 1 else "columns of an older shape; drop them"
|
||||
return Failure(f"todo.md's items table has {listed(old, 'and')}, {kind}, as rank_todo.py --help shows")
|
||||
if unknown:
|
||||
return Failure(f"todo.md's items table has the column {', '.join(unknown)}, which this rank_todo.py does not know; the columns it knows are {', '.join(known)}. Rename it, or move this machine to the release that added it")
|
||||
return Failure(f"todo.md's items table has the column {', '.join(unknown)}, which this rank_todo.py does not know; the columns it knows are {', '.join(shape.columns)}. Rename it, or move this machine to the release that added it")
|
||||
return None
|
||||
|
||||
|
||||
def older_columns(table: Table, shape: Shape) -> list[str]:
|
||||
"""The table's columns that only OLD_SHAPE knows, where the table is in SHAPE."""
|
||||
return [column for column in table.columns if column not in SHAPE.columns and column in OLD_SHAPE.columns] if shape is SHAPE else []
|
||||
|
||||
|
||||
def read_items(table: Table, scoring: Scoring) -> list[Item] | Failure:
|
||||
items = []
|
||||
for row in table.rows:
|
||||
@@ -554,13 +567,14 @@ def parse_item(row: str, scoring: Scoring, table: Table) -> Item | Failure:
|
||||
return Failure(f"item {item.id} cites goal {goal}, which README.md's ## Goals does not number")
|
||||
if scoring.goals[goal] == "Maintain" and item.exempt != "defect":
|
||||
return Failure(f"item {item.id} cites goal {goal}, a maintain goal, which only a defect that restores it may name; name the achieve goals the item moves")
|
||||
return item
|
||||
# R is written here, as the Score is, so --check finds a stale one, and a reader keeps the R the table stored.
|
||||
return dataclasses.replace(item, scores=item.scores | {"R": risk(item.scores["L"], item.scores["E"], item.scores["I"])})
|
||||
|
||||
|
||||
def render_item(item: Item, value: float, shape: Shape) -> str:
|
||||
"""The item's row, each cell under its column of shape."""
|
||||
written = {"ID": str(item.id), "Release": release_name(item), BLOCKED_BY: ", ".join(map(str, item.blocked_by)), "Exempt": item.exempt, "Item": item.tagline, **{name: str(item.scores[name]) for name in shape.score_columns}, shape.goals_column: shape.write_goals(item.moves), "Score": f"{value:.1f}"}
|
||||
return "| " + " | ".join(written[column] for column in shape.columns) + " |"
|
||||
def render_item(item: Item, value: float, shape: Shape, columns: tuple[str, ...]) -> str:
|
||||
"""The item's row, each cell under its column."""
|
||||
written = {"ID": str(item.id), "Release": release_name(item), BLOCKED_BY: ", ".join(map(str, item.blocked_by)), "Exempt": item.exempt, "Item": item.tagline, **{name: str(score) for name, score in item.scores.items()}, shape.goals_column: shape.write_goals(item.moves), "Score": f"{value:.1f}"}
|
||||
return "| " + " | ".join(written[column] for column in columns) + " |"
|
||||
|
||||
|
||||
def rank_details(text: str, items: list[Item]) -> str | Failure:
|
||||
|
||||
@@ -5,9 +5,9 @@ import re
|
||||
import statistics
|
||||
import typing
|
||||
|
||||
from model_routing.route import archived_worker, ESCALATION_MARKER
|
||||
from claude_code.entries import assistant_parts, parts, tool_call, usage_count
|
||||
from git_command import commits
|
||||
from model_routing.route import ESCALATION_MARKER, route_from
|
||||
from model_routing.ladder import position, Rung
|
||||
from round_titles import round_counts
|
||||
from shell_command import simple_commands
|
||||
@@ -91,7 +91,7 @@ def measured(chunk: Chunk, home: str) -> ChunkFigures:
|
||||
reports[kind] = reports.get(kind, Reports(0, 0, 0)) + Reports(1, len(reported.get(subagent.launched_by, "")), sum(map(len, results(subagent.entries).values())))
|
||||
for _, tools, spent in spend(subagent.entries):
|
||||
add(tokens, chunk.project, kind, rung, tools, spent)
|
||||
start = route_from(cycles[0][1]) if cycles else None
|
||||
start = archived_worker(cycles[0][1]) if cycles else None
|
||||
return ChunkFigures(
|
||||
tokens=tokens,
|
||||
start=start.rung if start and start.item else None,
|
||||
@@ -133,8 +133,8 @@ def rung_at(cycles: list[tuple[str, str]], timestamp: str) -> Rung | str:
|
||||
"""The rung of the cycle running at timestamp: the worker named by the first outcome stamped at or after it, or else by the last outcome. A worker off the ladder is returned as its text."""
|
||||
second = timestamp[:19] + "Z"
|
||||
found = next((rung for stamp, rung in cycles if stamp >= second), cycles[-1][1] if cycles else "unrouted")
|
||||
route = route_from(found)
|
||||
return route.rung if route else found
|
||||
archived = archived_worker(found)
|
||||
return archived.rung if archived else found
|
||||
|
||||
|
||||
def subagent_rung(entries: list[dict], arguments: dict, worker: Rung | str) -> Rung:
|
||||
|
||||
@@ -18,7 +18,7 @@ from transcript_archive import outcomes, write
|
||||
|
||||
LOW = Rung("claude-opus-5-5", "medium")
|
||||
HIGH = Rung("claude-opus-5-5", "high")
|
||||
LOW_ROUTE = f"{LOW}, routed by item 7's R + C of 9"
|
||||
LOW_ROUTE = f"{LOW}, routed by item 7's L of 3 and E + I of 4"
|
||||
|
||||
|
||||
def message(id_, at, *parts, model="claude-opus-5-5", tokens=(1, 10, 100)):
|
||||
@@ -84,6 +84,13 @@ class Measured(unittest.TestCase):
|
||||
worker = [commit("m1", "Answer stability round 1: 1 🔴, 0 🟡"), commit("m2", "Answer stability round 2: 0 🔴, 0 🟡"), commit("m3", "Answer stability round 2: 0 🔴, 0 🟡"), commit("m4", "Answer prose round 1: 0 🔴, 1 🟡")]
|
||||
self.assertEqual(measured(Chunk("log", [], worker, []), "/h").rounds_past_first, 1)
|
||||
|
||||
def test_reads_the_rung_of_a_route_an_older_loop_claude_archived_by_r_plus_c(self):
|
||||
older = f"{LOW}, routed by item 7's R + C of 9"
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.finished(older, "m", "j"))
|
||||
found = measured(Chunk("log", [kept], [message("m1", "2026-10-14T09:00:00Z")], []), "/h")
|
||||
self.assertEqual(found.start, LOW)
|
||||
self.assertEqual(found.tokens["rung"], {str(LOW): 111})
|
||||
|
||||
def test_starts_no_route_for_a_chunk_no_item_routed(self):
|
||||
kept = stamped("2026-10-14T10:00:00Z", outcomes.finished(str(LOW), "m", "j"))
|
||||
found = measured(Chunk("log", [kept], [message("m1", "2026-10-14T09:00:00Z")], []), "/h")
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
"""Refuse an Agent call that launches a review below the rung it needs, worked out from the two rungs loop-claude hands its worker: docs/decisions.md → `routing:`."""
|
||||
"""Refuse an Agent call that launches a review below the rung it needs, worked out from the rungs loop-claude hands its worker: docs/decisions.md → `routing:`."""
|
||||
import pathlib
|
||||
import typing
|
||||
|
||||
from rule_hook.rules import Breach
|
||||
from rule_hook.call import Call, text
|
||||
from model_routing.worker_rungs import handed
|
||||
from model_routing.ladder import position, review_rung, ReviewKind, Rung
|
||||
from model_routing.ladder import highest, position, review_rung, ReviewKind, Rung
|
||||
|
||||
# A reviewer is an agent under agents/, as CLAUDE.md → "Vocabulary" defines it.
|
||||
REVIEWERS = frozenset(path.stem for path in (pathlib.Path(__file__).resolve().parents[2] / "agents").glob("*.md"))
|
||||
@@ -18,11 +18,12 @@ SUBJECTS: dict[ReviewKind, str] = {"seat": "a comprehension seat", "security": "
|
||||
|
||||
def reviewer_breaches(call: Call, environ: typing.Mapping[str, str]) -> list[Breach]:
|
||||
"""The breach of an Agent call that launches a review below its rung; none where loop-claude handed no rungs."""
|
||||
project, worker = handed(environ)
|
||||
project, worker, reviewers = handed(environ)
|
||||
kind = review_kind(call.given)
|
||||
if kind is None or project is None or worker is None:
|
||||
return []
|
||||
required = review_rung(kind, project, worker)
|
||||
# A worker an older loop-claude started was handed no reviewers' rung; its reviewers are held to the rung that loop-claude gave them.
|
||||
required = review_rung(kind, project, reviewers or highest(project, worker))
|
||||
named = named_rung(call.given, worker)
|
||||
if position(named) >= position(required):
|
||||
return []
|
||||
|
||||
@@ -41,7 +41,7 @@ RULES: dict[RuleName, Rule] = {
|
||||
),
|
||||
# A rung is not a judgment call, and the refusal names the arguments to pass, so no marker waives it.
|
||||
"reviewer-rung": Rule(
|
||||
"a reviewer, a comprehension seat or the security pass's subagent runs on the rung loop-claude names for it: the project's rung, or for a reviewer the worker's rung where that is higher, so a worker below the project's rung cannot hand a review to its own cheaper model",
|
||||
"a reviewer, a comprehension seat or the security pass's subagent runs on the rung loop-claude names for it: for a seat or the security pass the project's rung, and for a reviewer the highest of the project's rung, the worker's and the one the chunk's item's E + I picks, so a worker below the project's rung cannot hand a review to its own cheaper model",
|
||||
"an Agent call, in a claude loop-claude started, whose subagent type is an agent under agents/, or whose prompt opens with seat-prompts.md's opening sentence or the one larv-review's Security section sets",
|
||||
overridable=False,
|
||||
),
|
||||
|
||||
@@ -166,10 +166,10 @@ class OverrideLog(unittest.TestCase):
|
||||
|
||||
|
||||
class ReviewerRung(unittest.TestCase):
|
||||
# A worker routed to Haiku 5.5 at max, on a project whose rung is Opus 5.5 at medium.
|
||||
ENVIRON = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort"}
|
||||
# The same project, with a worker routed above it to Fable 5.1 at max.
|
||||
ABOVE = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-fable-5-1 at max effort"}
|
||||
# A worker routed to Haiku 5.5 at max, on a project whose rung is Opus 5.5 at medium, with its reviewers on the project's rung.
|
||||
ENVIRON = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": "claude-opus-5-5 at medium effort"}
|
||||
# The same project, with a worker and its reviewers routed above it to Fable 5.1 at max.
|
||||
ABOVE = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 at medium effort", "LOOP_CLAUDE_WORKER_RUNG": "claude-fable-5-1 at max effort", "LOOP_CLAUDE_REVIEWER_RUNG": "claude-fable-5-1 at max effort"}
|
||||
|
||||
def refused(self, given: dict, environ: dict[str, str] = ENVIRON) -> list[str]:
|
||||
return rules(reviewer_breaches(Call("Agent", given, "/r"), environ))
|
||||
@@ -200,9 +200,19 @@ class ReviewerRung(unittest.TestCase):
|
||||
def test_an_effort_off_the_ladder_for_its_model_is_refused(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "opus", "effort": "low"}), ["reviewer-rung"])
|
||||
|
||||
def test_a_reviewer_is_held_to_the_workers_rung_where_that_is_higher(self):
|
||||
def test_a_reviewer_is_held_to_the_reviewers_rung_where_that_is_higher(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}, self.ABOVE), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "fable", "effort": "max"}, self.ABOVE), [])
|
||||
lifted = self.ENVIRON | {"LOOP_CLAUDE_REVIEWER_RUNG": "claude-opus-5-5 at high effort"}
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}, lifted), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"prompt": "You are one seat of a comprehension panel.", "model": "opus", "effort": "medium"}, lifted), [])
|
||||
|
||||
def test_holds_a_reviewer_to_the_higher_of_the_workers_and_the_projects_rung_where_loop_claude_handed_no_reviewers_rung(self):
|
||||
without = {name: rung for name, rung in self.ABOVE.items() if name != "LOOP_CLAUDE_REVIEWER_RUNG"}
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}, without), ["reviewer-rung"])
|
||||
below = {name: rung for name, rung in self.ENVIRON.items() if name != "LOOP_CLAUDE_REVIEWER_RUNG"}
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "sonnet", "effort": "high"}, below), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"prompt": "You are one seat of a comprehension panel.", "model": "sonnet", "effort": "high"}, without), ["reviewer-rung"])
|
||||
|
||||
def test_a_seat_or_a_security_launch_is_held_to_the_project_rung_alone(self):
|
||||
seat = {"prompt": "You are one seat of a comprehension panel. Tools: Read, Grep, Glob.", "model": "opus", "effort": "medium"}
|
||||
|
||||
@@ -3,14 +3,11 @@ import pathlib
|
||||
import sys
|
||||
|
||||
from failure import Failure
|
||||
from model_routing.ladder import LADDER, picked, position, project_rung, Rung
|
||||
from model_routing.ladder import LADDER, lowest_worker_rung, position, project_rung, Rung
|
||||
from claude_code.reply import Limited
|
||||
from loop_claude.project.model_choice import read_models_file
|
||||
from loop_claude.stop import Stop
|
||||
|
||||
LOWEST_R_PLUS_C = 2
|
||||
|
||||
|
||||
def project_rung_or_exit(tool: str, argv: list[str], help_text: str) -> Rung | int:
|
||||
"""The project's rung of the model argv names, or the exit code: 0 after printing help_text for -h or --help, 1 after saying why argv cannot be read."""
|
||||
if argv[:1] in (["-h"], ["--help"]):
|
||||
@@ -31,8 +28,8 @@ def projects_on(model: str, home: pathlib.Path) -> list[str] | Failure:
|
||||
|
||||
|
||||
def down_from(top: Rung) -> list[Rung]:
|
||||
"""top and each rung below it, down to the rung the lowest R + C picks from top, highest first."""
|
||||
return list(LADDER[position(picked(top, LOWEST_R_PLUS_C)):position(top) + 1])[::-1]
|
||||
"""top, the project's rung, and each rung below it, highest first, down to the lowest rung a worker routes to: no route uses a rung below it, so a replay there could change no routing."""
|
||||
return list(LADDER[position(lowest_worker_rung(top)):position(top) + 1])[::-1]
|
||||
|
||||
|
||||
def limited_exit(tool: str, limited: Limited) -> int:
|
||||
|
||||
@@ -63,8 +63,8 @@ class FirstReply(unittest.TestCase):
|
||||
|
||||
|
||||
class DownFrom(unittest.TestCase):
|
||||
def test_runs_from_the_projects_rung_down_to_the_rung_the_lowest_r_plus_c_picks(self):
|
||||
self.assertEqual(down_from(REFERENCE), [REFERENCE, MIDDLE, CHEAP, LOWER])
|
||||
def test_runs_from_the_projects_rung_down_to_the_lowest_rung_a_worker_routes_to(self):
|
||||
self.assertEqual(down_from(REFERENCE), [REFERENCE, MIDDLE, CHEAP])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+26
-14
@@ -1,6 +1,6 @@
|
||||
"""Show what finished chunks measured beside the scores their items were guessed at, to score a new item by.
|
||||
|
||||
usage: score-outcomes [R S C]
|
||||
usage: score-outcomes [L E I S]
|
||||
|
||||
Run it inside a project checkout or one of its worktrees. It fetches origin, when the checkout has
|
||||
that remote, and reads origin's default branch, or else the local main or master; a failed fetch
|
||||
@@ -8,8 +8,9 @@ reads origin as last fetched and says so on stderr.
|
||||
|
||||
Git history is the record. A chunk is a merge commit on the default branch's first-parent line
|
||||
whose change removed items from todo.md, and changed more than todo.md and the records below; it
|
||||
shipped those items. Each item's guess is its R, S and C in the todo.md the merge's first parent
|
||||
held; C only where that todo.md scored one. The chunk measures:
|
||||
shipped those items. Each item's guess is the scores it held in the todo.md of the merge's first
|
||||
parent: L, E, I, R and S. An item scored before L, E and I replaced C holds R and S, and C too where
|
||||
its table also had A, G and Goals. The chunk measures:
|
||||
|
||||
- lines changed: lines added plus lines deleted against the first parent, leaving out todo.md and
|
||||
the records tools append to, docs/comprehension-history.md and docs/reading-cost.md;
|
||||
@@ -22,14 +23,18 @@ held; C only where that todo.md scored one. The chunk measures:
|
||||
|
||||
A project that merges without merge commits, by fast-forward or squash, has no chunk to measure.
|
||||
|
||||
Given R, S and C, it lists the 5 finished items whose guesses sit closest, closest first: the
|
||||
least mean squared difference over the scores both hold, C only where the item had one, the newest
|
||||
first among equals. Each line gives what the item's chunk measured, which items shipped with it, and its tagline.
|
||||
Given L, E and I, on the levels ~/.claude/rules/todo-items.md anchors, and S, it works out R from
|
||||
L, E and I, as rank_todo.py does, and lists the 5 finished items whose guesses sit closest, closest
|
||||
first: the least mean squared difference over the scores both hold, so R and S alone for an item
|
||||
scored before L, E and I, the newest first among equals. Each line gives what the item's chunk
|
||||
measured, which items shipped with it, and its tagline.
|
||||
|
||||
Then, for each score, the median of the outcome it predicts at each value it was guessed at: R
|
||||
against 🔴 findings, S against lines changed, C against review rounds past each pass's first. It counts only chunks that
|
||||
shipped one item, since a chunk measures all its items at once, and reports a score only once 10
|
||||
such chunks hold it and its outcome.
|
||||
Then the median of the outcome R and S each predict at each value they were guessed at: R against
|
||||
🔴 findings, S against lines changed. And how well L, E and I each order chunks by review rounds
|
||||
past each pass's first and by 🔴 findings: Spearman's rank correlation, from -1 to +1, where +1
|
||||
orders them exactly; none where every chunk held the same level, or the same outcome. It counts only chunks that shipped one
|
||||
item, since a chunk measures all its items at once, and reports a score only once 10 such chunks
|
||||
hold it and its outcomes.
|
||||
|
||||
Exit 1, saying why, on bad arguments, when git cannot start, finds no checkout or reads a shallow
|
||||
clone, or the default branch cannot be read.
|
||||
@@ -39,16 +44,22 @@ import sys
|
||||
|
||||
from git_repo import default_branch, file_at, full_history, git
|
||||
from failure import Failure
|
||||
from todo_table import LEVELS, risk, TOP_SCORE
|
||||
from markdown_text import listed
|
||||
from score_outcomes.outcomes import nearest, Outcome, outcomes_of, render, report, shipped
|
||||
from utc_stamp import utc
|
||||
|
||||
# The scores score-outcomes takes, in order.
|
||||
GIVEN = ("L", "E", "I", "S")
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
if argv[:1] in (["-h"], ["--help"]):
|
||||
print(__doc__.strip())
|
||||
return 0
|
||||
if argv and (len(argv) != 3 or not all(re.fullmatch(r"[1-9]|10", arg) for arg in argv)):
|
||||
print(f"score-outcomes: cannot read the arguments {' '.join(argv)}; usage: score-outcomes [R S C], each a whole number from 1 to 10", file=sys.stderr)
|
||||
if argv and (len(argv) != len(GIVEN) or not all(re.fullmatch(r"[1-9][0-9]?", arg) and int(arg) <= LEVELS.get(name, TOP_SCORE) for name, arg in zip(GIVEN, argv))):
|
||||
scales = listed([f"{name} from 1 to {LEVELS.get(name, TOP_SCORE)}" for name in GIVEN], "and")
|
||||
print(f"score-outcomes: cannot read the arguments {' '.join(argv)}; usage: score-outcomes [L E I S], {scales}", file=sys.stderr)
|
||||
return 1
|
||||
found = default_branch(stale_ok=True)
|
||||
if isinstance(found, Failure):
|
||||
@@ -60,8 +71,9 @@ def main(argv: list[str]) -> int:
|
||||
if isinstance(outcomes, Failure):
|
||||
return failed(outcomes)
|
||||
if argv:
|
||||
given = dict(zip("RSC", map(int, argv)))
|
||||
print(f"Finished items nearest R {given['R']}, S {given['S']}, C {given['C']}, closest first:\n")
|
||||
given = dict(zip(GIVEN, map(int, argv)))
|
||||
given["R"] = risk(given["L"], given["E"], given["I"])
|
||||
print(f"Finished items nearest L {given['L']}, E {given['E']}, I {given['I']}, so R {given['R']}, and S {given['S']}, closest first:\n")
|
||||
print("\n".join(render(outcome) for outcome in nearest(outcomes, given)) or "- None: no merge commit on the default branch shipped an item from todo.md. A project that merges by squash or fast-forward has none.")
|
||||
print()
|
||||
print("Median outcome at each guessed score, among chunks that shipped one item:\n")
|
||||
|
||||
@@ -4,6 +4,7 @@ import typing
|
||||
|
||||
from comprehension_history import HISTORY
|
||||
from todo_table import Item, items
|
||||
from markdown_text import listed
|
||||
from reading_cost.figures import RECORD
|
||||
from round_titles import round_counts, round_findings
|
||||
|
||||
@@ -26,12 +27,14 @@ class Outcome:
|
||||
findings: tuple[int, int] | None
|
||||
|
||||
|
||||
# Each pass takes a first round whatever the change, so only the rounds after it measure how hard the change was to get right.
|
||||
MEASURES: dict[str, tuple[str, typing.Callable[[Outcome], int | None]]] = {
|
||||
"R": ("🔴 findings", lambda outcome: outcome.findings[0] if outcome.findings else None),
|
||||
"S": ("lines changed", lambda outcome: outcome.lines),
|
||||
"C": ("review rounds past each pass's first", lambda outcome: outcome.rounds_past_first),
|
||||
}
|
||||
# An outcome's name and how to read it from an Outcome, None where the chunk did not record it.
|
||||
Measure = tuple[str, typing.Callable[[Outcome], int | None]]
|
||||
# The scores whose median outcome the report gives at each guessed value.
|
||||
MEDIANS: dict[str, Measure] = {"R": ("🔴 findings", lambda outcome: outcome.findings[0] if outcome.findings else None), "S": ("lines changed", lambda outcome: outcome.lines)}
|
||||
# The levels R is built from; for each, the report gives how well it orders chunks by review rounds and by 🔴 findings.
|
||||
ORDERED = ("L", "E", "I")
|
||||
# The scores render shows, in this order; C only on an item scored before L, E and I replaced it.
|
||||
SHOWN = ("L", "E", "I", "R", "S", "C")
|
||||
|
||||
|
||||
def shipped(before: str, after: str) -> list[Item]:
|
||||
@@ -66,22 +69,50 @@ def nearest(outcomes: list[Outcome], given: dict[str, int], count: int = 5) -> l
|
||||
|
||||
|
||||
def report(outcomes: list[Outcome]) -> list[str]:
|
||||
"""For each score, the median outcome at each value it was guessed at, among chunks that shipped one item: a chunk measures all its items at once."""
|
||||
"""Among chunks that shipped one item, since a chunk measures all its items at once: for each score in MEDIANS, the median outcome at each value it was guessed at; for each level in ORDERED, how well it orders chunks by review rounds and by 🔴 findings."""
|
||||
single = [outcome for outcome in outcomes if not outcome.shipped_with]
|
||||
lines = []
|
||||
for name, (what, measure) in MEASURES.items():
|
||||
for name, (what, measure) in MEDIANS.items():
|
||||
held = [(outcome.item.scores[name], value) for outcome in single if name in outcome.item.scores and (value := measure(outcome)) is not None]
|
||||
if len(held) < ENOUGH:
|
||||
chunks = "chunk that shipped one item records" if len(held) == 1 else "chunks that shipped one item record"
|
||||
lines.append(f"- {name}, {what}: {len(held)} {chunks} {name} and its outcome; the report needs {ENOUGH}.")
|
||||
lines.append(f"- {name}, {what}: {too_few(len(held), name, 'its outcome')}")
|
||||
continue
|
||||
medians = [(guess, [value for at, value in held if at == guess]) for guess in sorted({guess for guess, _ in held})]
|
||||
lines.append(f"- {name}, {what}: " + ", ".join(f"{guess} → {statistics.median(found):g} ({plural(len(found), 'chunk')})" for guess, found in medians))
|
||||
for name in ORDERED:
|
||||
# Each pass takes a first round whatever the change, so only the rounds after it measure how hard the change was to get right.
|
||||
rows = [(outcome.item.scores[name], outcome.rounds_past_first, outcome.findings[0]) for outcome in single if name in outcome.item.scores and outcome.findings is not None]
|
||||
opening = f"- {name}, by review rounds past each pass's first and 🔴 findings: "
|
||||
if len(rows) < ENOUGH:
|
||||
lines.append(opening + too_few(len(rows), name, "both outcomes"))
|
||||
continue
|
||||
guesses, rounds, red = ([row[column] for row in rows] for column in range(3))
|
||||
found = [rank_correlation(guesses, rounds), rank_correlation(guesses, red)]
|
||||
undefined = ", since every chunk held one value" if None in found else ""
|
||||
lines.append(opening + f"rank correlation {listed(['none' if value is None else f'{value:+.2f}' for value in found], 'and')}{undefined} ({plural(len(rows), 'chunk')})")
|
||||
return lines
|
||||
|
||||
|
||||
def too_few(held: int, name: str, what: str) -> str:
|
||||
chunks = "chunk that shipped one item records" if held == 1 else "chunks that shipped one item record"
|
||||
return f"{held} {chunks} {name} and {what}; the report needs {ENOUGH}."
|
||||
|
||||
|
||||
def rank_correlation(guesses: list[int], outcomes: list[int]) -> float | None:
|
||||
"""Spearman's rank correlation, tied values sharing their mean rank; None where either side holds one value, which no order can rank."""
|
||||
if len(set(guesses)) < 2 or len(set(outcomes)) < 2:
|
||||
return None
|
||||
return statistics.correlation(ranks(guesses), ranks(outcomes))
|
||||
|
||||
|
||||
def ranks(values: list[int]) -> list[float]:
|
||||
"""Each value's rank from 1, tied values sharing their mean rank."""
|
||||
ordered = sorted(values)
|
||||
return [ordered.index(value) + (ordered.count(value) + 1) / 2 for value in values]
|
||||
|
||||
|
||||
def render(outcome: Outcome) -> str:
|
||||
scores = ", ".join(f"{name} {outcome.item.scores[name]}" for name in MEASURES if name in outcome.item.scores)
|
||||
scores = ", ".join(f"{name} {outcome.item.scores[name]}" for name in SHOWN if name in outcome.item.scores)
|
||||
findings = f"{outcome.findings[0]} 🔴, {outcome.findings[1]} 🟡" if outcome.findings else "findings unknown"
|
||||
others = outcome.shipped_with
|
||||
together = f", with item{'s' if len(others) > 1 else ''} {', '.join(map(str, others))}" if others else ""
|
||||
|
||||
@@ -9,7 +9,7 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from test_env import isolated_env
|
||||
from todo_table import Item
|
||||
from score_outcomes.outcomes import nearest, Outcome, outcomes_of, render, report, shipped
|
||||
from score_outcomes.outcomes import nearest, Outcome, outcomes_of, rank_correlation, render, report, shipped
|
||||
|
||||
TOOL = pathlib.Path(__file__).resolve().parents[1] / "score-outcomes"
|
||||
HEADER = "| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |\n|---|---|---|---|---|---|---|---|---|---|---|\n"
|
||||
@@ -26,8 +26,8 @@ def item(id_: int, scores: dict[str, int]) -> Item:
|
||||
return Item(id_, (0, 1, 0), "", f"**Item {id_}.**", scores, (("1", None),), ())
|
||||
|
||||
|
||||
def outcome(id_: int, r: int, s: int, c: int | None, lines: int = 10, rounds: int = 1, findings: tuple[int, int] | None = (0, 0), merged: str = "2026-10-13T10:00:00Z", shipped_with: tuple[int, ...] = ()) -> Outcome:
|
||||
return Outcome(item(id_, {"R": r, "S": s} | ({"C": c} if c else {})), "aaaaaaa", merged, shipped_with, lines, 1, rounds, findings)
|
||||
def outcome(id_: int, r: int, s: int, c: int | None, lines: int = 10, rounds: int = 1, findings: tuple[int, int] | None = (0, 0), merged: str = "2026-10-13T10:00:00Z", shipped_with: tuple[int, ...] = (), levels: dict[str, int] | None = None) -> Outcome:
|
||||
return Outcome(item(id_, {"R": r, "S": s} | ({"C": c} if c else {}) | (levels or {})), "aaaaaaa", merged, shipped_with, lines, 1, rounds, findings)
|
||||
|
||||
|
||||
class Shipped(unittest.TestCase):
|
||||
@@ -67,6 +67,11 @@ class Nearest(unittest.TestCase):
|
||||
def test_lists_five_by_default(self):
|
||||
self.assertEqual(len(nearest([outcome(id_, 3, 4, 5) for id_ in range(6)], {"R": 3, "S": 4, "C": 5})), 5)
|
||||
|
||||
def test_compares_an_item_scored_before_l_e_and_i_by_its_r_and_s(self):
|
||||
given = {"L": 2, "E": 2, "I": 3, "R": 5, "S": 4}
|
||||
found = [outcome(1, 8, 4, None, levels={"L": 4, "E": 3, "I": 3}), outcome(2, 5, 4, 9, merged="2026-10-12T10:00:00Z"), outcome(3, 5, 4, None, levels={"L": 2, "E": 2, "I": 3})]
|
||||
self.assertEqual([each.item.id for each in nearest(found, given)], [3, 2, 1])
|
||||
|
||||
def test_lists_the_newest_first_among_equals(self):
|
||||
found = [outcome(1, 3, 4, 5, merged="2026-10-12T10:00:00Z"), outcome(2, 3, 4, 5, merged="2026-10-13T09:00:00Z")]
|
||||
self.assertEqual([each.item.id for each in nearest(found, {"R": 3, "S": 4, "C": 5})], [2, 1])
|
||||
@@ -74,17 +79,40 @@ class Nearest(unittest.TestCase):
|
||||
|
||||
class Report(unittest.TestCase):
|
||||
def test_gives_each_guessed_values_median_outcome_once_ten_single_item_chunks_hold_it(self):
|
||||
found = [outcome(id_, 2, 1 + id_ % 2, 4, lines=10 * id_, rounds=id_) for id_ in range(1, 11)]
|
||||
found = [outcome(id_, 2, 1 + id_ % 2, 4, lines=10 * id_, rounds=id_, findings=(1, 3)) for id_ in range(1, 11)]
|
||||
lines = report(found + [outcome(11, 2, 2, None, lines=1000, findings=None, shipped_with=(12,))])
|
||||
self.assertIn("- R, 🔴 findings: 2 → 0 (10 chunks)", lines)
|
||||
self.assertIn("- R, 🔴 findings: 2 → 1 (10 chunks)", lines)
|
||||
self.assertIn("- S, lines changed: 1 → 60 (5 chunks), 2 → 50 (5 chunks)", lines)
|
||||
self.assertIn("- C, review rounds past each pass's first: 4 → 5.5 (10 chunks)", lines)
|
||||
self.assertFalse([line for line in lines if line.startswith("- C")])
|
||||
|
||||
def test_says_how_many_chunks_it_has_under_ten(self):
|
||||
self.assertIn("- C, review rounds past each pass's first: 1 chunk that shipped one item records C and its outcome; the report needs 10.", report([outcome(1, 2, 2, 4), outcome(2, 2, 2, None)]))
|
||||
self.assertIn("- S, lines changed: 1 chunk that shipped one item records S and its outcome; the report needs 10.", report([outcome(1, 2, 2, 4), outcome(2, 2, 2, None, shipped_with=(1,))]))
|
||||
too_few = report([outcome(1, 2, 2, None, levels={"L": 2, "E": 1, "I": 1}), outcome(2, 2, 2, None, levels={"L": 2}, findings=None)])
|
||||
for name in "LEI":
|
||||
with self.subTest(name=name):
|
||||
self.assertIn(f"- {name}, by review rounds past each pass's first and 🔴 findings: 1 chunk that shipped one item records {name} and both outcomes; the report needs 10.", too_few)
|
||||
|
||||
def test_ranks_ties_by_their_mean_rank(self):
|
||||
self.assertAlmostEqual(rank_correlation([1, 1, 2, 2, 2], [1, 2, 3, 4, 5]), 0.866, places=3)
|
||||
self.assertAlmostEqual(rank_correlation([1, 1, 2, 3], [1, 2, 3, 4]), 0.949, places=3)
|
||||
self.assertAlmostEqual(rank_correlation([1, 2, 3, 4], [1, 1, 2, 3]), 0.949, places=3)
|
||||
|
||||
def test_ranks_an_outcome_that_holds_two_values(self):
|
||||
self.assertAlmostEqual(rank_correlation([1, 2, 3, 4, 5], [1, 1, 2, 2, 2]), 0.866, places=3)
|
||||
|
||||
def test_gives_how_well_l_e_and_i_each_order_chunks_by_review_rounds_and_red_findings(self):
|
||||
levels = [1, 1, 2, 2, 2, 3, 3, 3, 4, 4]
|
||||
found = [outcome(id_, 2, 2, None, rounds=level, findings=(5 - level, 0), levels={"L": level, "E": 2, "I": level}) for id_, level in enumerate(levels, 1)]
|
||||
lines = report(found)
|
||||
self.assertIn("- L, by review rounds past each pass's first and 🔴 findings: rank correlation +1.00 and -1.00 (10 chunks)", lines)
|
||||
self.assertIn("- I, by review rounds past each pass's first and 🔴 findings: rank correlation +1.00 and -1.00 (10 chunks)", lines)
|
||||
self.assertIn("- E, by review rounds past each pass's first and 🔴 findings: rank correlation none and none, since every chunk held one value (10 chunks)", lines)
|
||||
|
||||
|
||||
class Render(unittest.TestCase):
|
||||
def test_shows_an_items_l_e_and_i_before_its_r_and_s(self):
|
||||
self.assertTrue(render(outcome(4, 3, 3, None, levels={"L": 2, "E": 2, "I": 1})).startswith("- Item 4, L 2, E 2, I 1, R 3, S 3:"))
|
||||
|
||||
def test_names_the_items_shipped_together_and_unknown_findings(self):
|
||||
self.assertEqual(render(outcome(4, 2, 3, None, lines=1, rounds=2, findings=None, shipped_with=(5, 6))), "- Item 4, R 2, S 3: 1 line in 1 file, 2 rounds past each pass's first, findings unknown; merged 2026-10-13T10:00:00Z in aaaaaaa, with items 5, 6. **Item 4.**")
|
||||
self.assertEqual(render(outcome(4, 2, 3, 4, findings=(1, 2))), "- Item 4, R 2, S 3, C 4: 10 lines in 1 file, 1 round past each pass's first, 1 🔴, 2 🟡; merged 2026-10-13T10:00:00Z in aaaaaaa. **Item 4.**")
|
||||
@@ -121,17 +149,23 @@ class ScoreOutcomesTool(unittest.TestCase):
|
||||
self.commit("Answer stability round 1: 1 🔴, 2 🟡; fix", {"a.py": "1\n2\n4\n"})
|
||||
self.git("switch", "-q", "main")
|
||||
self.git("merge", "-q", "--no-ff", "-m", "Merge chunk", "chunk")
|
||||
ran = self.run_tool("2", "3", "10")
|
||||
ran = self.run_tool("1", "2", "1", "3")
|
||||
self.assertEqual(ran.returncode, 0, ran.stderr)
|
||||
self.assertIn("Finished items nearest L 1, E 2, I 1, so R 2, and S 3, closest first:", ran.stdout)
|
||||
self.assertIn("- Item 4, R 2, S 3, C 10: 3 lines in 1 file, 0 rounds past each pass's first, 1 🔴, 2 🟡; merged ", ran.stdout)
|
||||
self.assertNotIn("Item 5", ran.stdout)
|
||||
|
||||
def test_refuses_a_score_outside_one_to_ten_or_a_missing_score(self):
|
||||
for args in (("2", "11", "5"), ("0", "3", "5"), ("2", "3")):
|
||||
def test_takes_each_score_at_the_top_of_its_scale(self):
|
||||
ran = self.run_tool("4", "3", "5", "10")
|
||||
self.assertEqual(ran.returncode, 0, ran.stderr)
|
||||
self.assertIn("Finished items nearest L 4, E 3, I 5, so R 10, and S 10, closest first:", ran.stdout)
|
||||
|
||||
def test_refuses_a_score_off_its_scale_or_a_missing_score(self):
|
||||
for args in (("5", "2", "3", "4"), ("2", "4", "3", "4"), ("2", "2", "6", "4"), ("2", "2", "3", "11"), ("0", "2", "3", "4"), ("2", "2", "3")):
|
||||
with self.subTest(args):
|
||||
ran = self.run_tool(*args)
|
||||
self.assertEqual(ran.returncode, 1)
|
||||
self.assertIn("usage: score-outcomes [R S C]", ran.stderr)
|
||||
self.assertIn("usage: score-outcomes [L E I S]", ran.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Replay archived stability rounds on the rungs below the project's, to find which rungs may run the stability reviewer: docs/decisions.md → `routing: the stability reviewer stays on the project's rung`.
|
||||
"""Replay archived stability rounds on the rungs below the project's, to find which rungs may run the stability reviewer: docs/decisions.md → `routing: the stability reviewer runs no lower than the project's rung`.
|
||||
|
||||
usage: stability-replay <model>
|
||||
|
||||
@@ -15,12 +15,12 @@ names it and why.
|
||||
It first replays every round on the project's rung, the reference reviewer. Findings on record
|
||||
that the reference reviewer does not raise again may rest on code or rules that have changed
|
||||
since; no rung is held to them. It then replays the rounds where the reference raised any, on each
|
||||
rung below, highest first, down to the rung the lowest R + C picks, and stops at the first rung
|
||||
rung below, highest first, down to the lowest rung a worker routes to, and stops at the first rung
|
||||
that misses one. For each replay, claude on the project's rung matches the report to the findings
|
||||
on record, and counts a finding only where the report raises it as 🔴 or 🟡. The tool prints each
|
||||
finding a rung missed and the lowest rung that, like each rung above it, missed none. That rung is
|
||||
the evidence for running the stability reviewer on the worker's rung whenever the worker runs at
|
||||
or above it; docs/decisions.md → `routing: the stability reviewer stays on the project's rung`
|
||||
or above it; docs/decisions.md → `routing: the stability reviewer runs no lower than the project's rung`
|
||||
records the run.
|
||||
|
||||
Like the loop, it tries claude three times running before it counts a reply as failed. A failed
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Which archived stability rounds to replay, the code each reviewed, and the report: docs/decisions.md → `routing: the stability reviewer stays on the project's rung`."""
|
||||
"""Which archived stability rounds to replay, the code each reviewed, and the report: docs/decisions.md → `routing: the stability reviewer runs no lower than the project's rung`."""
|
||||
import dataclasses
|
||||
import datetime
|
||||
import pathlib
|
||||
|
||||
+17
-16
@@ -1,31 +1,32 @@
|
||||
"""Print the Agent tool's model and effort for a subagent's task: docs/decisions.md → `routing:`.
|
||||
|
||||
usage: subagent-rung [--plan] <model> <R+C>
|
||||
usage: subagent-rung [--plan] <model> <L>
|
||||
|
||||
<model> is the project's model, such as claude-opus-5-5, and <R+C> the task's R + C, a whole
|
||||
number from 2 to 20, scored as for a todo.md item. It prints the rung that R + C picks from the
|
||||
model's cheapest rung, the project's rung, in the Agent tool's words, such as `model sonnet, effort
|
||||
high`. --plan, for a task that files or scores todo.md items, prints the project's rung where that
|
||||
R + C picks a lower one, as ~/.claude/rules/todo-items.md lets only the project's rung or above plan. For a model
|
||||
off the routing ladder, the tool prints nothing on stdout, says so on stderr and exits 0; launch the
|
||||
subagent with no model or effort.
|
||||
<model> is the project's model, such as claude-opus-5-5, and <L> the task's L, the chance it holds
|
||||
a defect, a level from 1 to 4 scored as for a todo.md item. It prints the rung that L picks from
|
||||
the model's cheapest rung, the project's rung, in the Agent tool's words, such as `model sonnet,
|
||||
effort high`. --plan, for a task that files or scores todo.md items, prints the project's rung where
|
||||
that L picks a lower one, as ~/.claude/rules/todo-items.md lets only the project's rung or above
|
||||
plan. For a model off the routing ladder, the tool prints nothing on stdout, says so on stderr and
|
||||
exits 0; launch the subagent with no model or effort.
|
||||
|
||||
Exit 1, saying why, on bad arguments.
|
||||
"""
|
||||
import re
|
||||
import sys
|
||||
|
||||
from model_routing.ladder import floored, picked, project_rung
|
||||
from todo_table import LEVELS
|
||||
from model_routing.ladder import planning_rung, project_rung, worker_rung
|
||||
|
||||
USAGE = "usage: subagent-rung [--plan] <model> <R+C>, R+C a whole number from 2 to 20"
|
||||
USAGE = f"usage: subagent-rung [--plan] <model> <L>, L a level from 1 to {LEVELS['L']}"
|
||||
|
||||
|
||||
def agent_args(model: str, risk_and_complexity: int, planning: bool = False) -> str | None:
|
||||
"""The Agent tool's model and effort for the rung risk_and_complexity picks from model's cheapest rung, never below it when planning; None for a model off the ladder."""
|
||||
def agent_args(model: str, likelihood: int, planning: bool = False) -> str | None:
|
||||
"""The Agent tool's model and effort for the rung likelihood picks from model's cheapest rung, never below it when planning; None for a model off the ladder."""
|
||||
anchor = project_rung(model)
|
||||
if anchor is None:
|
||||
return None
|
||||
return (floored if planning else picked)(anchor, risk_and_complexity).agent_args()
|
||||
return (planning_rung if planning else worker_rung)(anchor, likelihood).agent_args()
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
@@ -34,11 +35,11 @@ def main(argv: list[str]) -> int:
|
||||
return 0
|
||||
planning = argv[:1] == ["--plan"]
|
||||
operands = argv[1:] if planning else argv
|
||||
if len(operands) != 2 or not re.fullmatch(r"[2-9]|1[0-9]|20", operands[1]):
|
||||
if len(operands) != 2 or not re.fullmatch(r"[1-9][0-9]*", operands[1]) or int(operands[1]) > LEVELS["L"]:
|
||||
print(f"subagent-rung: cannot read the arguments{' ' + ' '.join(argv) if argv else ''}; {USAGE}", file=sys.stderr)
|
||||
return 1
|
||||
model, total = operands
|
||||
args = agent_args(model, int(total), planning)
|
||||
model, likelihood = operands
|
||||
args = agent_args(model, int(likelihood), planning)
|
||||
if args is None:
|
||||
print(f"subagent-rung: {model} is not on the routing ladder; pass the Agent tool no model or effort", file=sys.stderr)
|
||||
else:
|
||||
|
||||
@@ -14,37 +14,36 @@ class SubagentRung(unittest.TestCase):
|
||||
def run_tool(self, *args: str) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, text=True)
|
||||
|
||||
def test_prints_the_agent_tools_model_and_effort_for_the_rung_an_r_plus_c_picks(self):
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 9), "model opus, effort medium")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 7), "model sonnet, effort high")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 20), "model fable, effort max")
|
||||
self.assertEqual(main.agent_args("claude-haiku-5-5", 2), "model haiku, effort low")
|
||||
def test_prints_the_agent_tools_model_and_effort_for_the_rung_an_l_picks(self):
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 3), "model opus, effort medium")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 2), "model sonnet, effort high")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 4), "model opus, effort high")
|
||||
self.assertEqual(main.agent_args("claude-haiku-5-5", 1), "model haiku, effort low")
|
||||
|
||||
def test_a_planning_task_never_runs_below_the_projects_rung(self):
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 2, planning=True), "model opus, effort medium")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 12, planning=True), "model opus, effort high")
|
||||
self.assertEqual(self.run_tool("--plan", "claude-opus-5-5", "7").stdout, "model opus, effort medium\n")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 1, planning=True), "model opus, effort medium")
|
||||
self.assertEqual(main.agent_args("claude-opus-5-5", 4, planning=True), "model opus, effort high")
|
||||
self.assertEqual(self.run_tool("--plan", "claude-opus-5-5", "2").stdout, "model opus, effort medium\n")
|
||||
|
||||
def test_picks_nothing_for_a_model_off_the_ladder(self):
|
||||
self.assertIsNone(main.agent_args("claude-opus-5", 9))
|
||||
self.assertIsNone(main.agent_args("claude-opus-5", 3))
|
||||
|
||||
def test_refuses_an_r_plus_c_outside_2_to_20_or_a_wrong_count_of_arguments(self):
|
||||
for argv in (["claude-opus-5-5", "1"], ["claude-opus-5-5", "02"], ["claude-opus-5-5", "21"], ["claude-opus-5-5", "x"], ["claude-opus-5-5"], ["9", "claude-opus-5-5"], ["claude-opus-5-5", "9", "--plan"], ["--planning", "claude-opus-5-5", "9"]):
|
||||
def test_refuses_an_l_outside_1_to_4_or_a_wrong_count_of_arguments(self):
|
||||
for argv in (["claude-opus-5-5", "0"], ["claude-opus-5-5", "01"], ["claude-opus-5-5", "5"], ["claude-opus-5-5", "x"], ["claude-opus-5-5"], ["3", "claude-opus-5-5"], ["claude-opus-5-5", "3", "--plan"], ["--planning", "claude-opus-5-5", "3"]):
|
||||
result = self.run_tool(*argv)
|
||||
self.assertEqual((result.returncode, result.stdout), (1, ""), argv)
|
||||
self.assertIn("usage: subagent-rung [--plan] <model> <R+C>", result.stderr)
|
||||
self.assertIn("usage: subagent-rung [--plan] <model> <L>", result.stderr)
|
||||
|
||||
def test_the_launcher_prints_the_rung_and_exits_0_on_and_off_the_ladder(self):
|
||||
on = self.run_tool("claude-opus-5-5", "12")
|
||||
on = self.run_tool("claude-opus-5-5", "4")
|
||||
self.assertEqual((on.returncode, on.stdout, on.stderr), (0, "model opus, effort high\n", ""))
|
||||
off = self.run_tool("claude-opus-5", "12")
|
||||
off = self.run_tool("claude-opus-5", "4")
|
||||
self.assertEqual((off.returncode, off.stdout), (0, ""))
|
||||
self.assertEqual(off.stderr, "subagent-rung: claude-opus-5 is not on the routing ladder; pass the Agent tool no model or effort\n")
|
||||
for bound, rung in (("2", "model haiku, effort xhigh\n"), ("20", "model fable, effort max\n")):
|
||||
for bound, rung in (("1", "model haiku, effort max\n"), ("4", "model opus, effort high\n")):
|
||||
self.assertEqual(self.run_tool("claude-opus-5-5", bound).stdout, rung)
|
||||
for flag in ("-h", "--help"):
|
||||
self.assertIn("usage: subagent-rung [--plan] <model> <R+C>", self.run_tool(flag).stdout)
|
||||
|
||||
self.assertIn("usage: subagent-rung [--plan] <model> <L>", self.run_tool(flag).stdout)
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -21,5 +21,12 @@ class Sections(unittest.TestCase):
|
||||
self.assertEqual(markdown_text.quoted("merged\n\nPR #1"), "> merged\n> \n> PR #1")
|
||||
|
||||
|
||||
class Listed(unittest.TestCase):
|
||||
def test_joins_names_with_commas_and_the_conjunction_before_the_last(self):
|
||||
self.assertEqual(markdown_text.listed(["a"], "and"), "a")
|
||||
self.assertEqual(markdown_text.listed(["a", "b"], "and"), "a and b")
|
||||
self.assertEqual(markdown_text.listed(("a", "b", "c"), "or"), "a, b or c")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
+102
-49
@@ -29,13 +29,13 @@ TODO = """# todo
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | R | S | C | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| 1 | 0.2.0 | | | **Ship the later thing.** | 1 | 1 | 3 | 1 by 9 | 0 |
|
||||
| 2 | 0.1.0 | | | **Tighten the tests.** | 2 | 4 | 3 | 2.1 by 1 | 0 |
|
||||
| 3 | 0.1.0 | | | **Make it fast, `a \\| b`.** | 3 | 8 | 3 | 2.1 by 8, 1 by 6 | 0 |
|
||||
| 4 | 0.1.0 | | security | **Rotate the leaked key.** | 9 | 2 | 3 | 2.1 by 1 | 0 |
|
||||
| 5 | 0.1.0 | | decision | **Move the writes out.** | 5 | 5 | 3 | 2.1 by 1 | 0 |
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 1 | 0.2.0 | | | **Ship the later thing.** | 1 | 1 | 1 | 1 | 1 | 1 by 9 | 0 |
|
||||
| 2 | 0.1.0 | | | **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2.1 by 1 | 0 |
|
||||
| 3 | 0.1.0 | | | **Make it fast, `a \\| b`.** | 2 | 1 | 2 | 3 | 8 | 2.1 by 8, 1 by 6 | 0 |
|
||||
| 4 | 0.1.0 | | security | **Rotate the leaked key.** | 3 | 3 | 5 | 9 | 2 | 2.1 by 1 | 0 |
|
||||
| 5 | 0.1.0 | | decision | **Move the writes out.** | 1 | 1 | 4 | 5 | 5 | 2.1 by 1 | 0 |
|
||||
|
||||
## Details
|
||||
|
||||
@@ -100,11 +100,11 @@ class Rank(unittest.TestCase):
|
||||
|
||||
def test_scores_and_sorts_by_release_then_security_then_score_and_orders_the_details_to_match(self):
|
||||
ranked = self.ranked()
|
||||
self.assertIn("""| 4 | 0.1.0 | | security | **Rotate the leaked key.** | 9 | 2 | 3 | 2.1 by 1 | 8.3 |
|
||||
| 3 | 0.1.0 | | | **Make it fast, `a \\| b`.** | 3 | 8 | 3 | 2.1 by 8, 1 by 6 | 2.6 |
|
||||
| 2 | 0.1.0 | | | **Tighten the tests.** | 2 | 4 | 3 | 2.1 by 1 | 2.1 |
|
||||
| 5 | 0.1.0 | | decision | **Move the writes out.** | 5 | 5 | 3 | 2.1 by 1 | 1.0 |
|
||||
| 1 | 0.2.0 | | | **Ship the later thing.** | 1 | 1 | 3 | 1 by 9 | 300.0 |
|
||||
self.assertIn("""| 4 | 0.1.0 | | security | **Rotate the leaked key.** | 3 | 3 | 5 | 9 | 2 | 2.1 by 1 | 8.3 |
|
||||
| 3 | 0.1.0 | | | **Make it fast, `a \\| b`.** | 2 | 1 | 2 | 3 | 8 | 2.1 by 8, 1 by 6 | 2.6 |
|
||||
| 2 | 0.1.0 | | | **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2.1 by 1 | 2.1 |
|
||||
| 5 | 0.1.0 | | decision | **Move the writes out.** | 1 | 1 | 4 | 5 | 5 | 2.1 by 1 | 1.0 |
|
||||
| 1 | 0.2.0 | | | **Ship the later thing.** | 1 | 1 | 1 | 1 | 1 | 1 by 9 | 300.0 |
|
||||
""", ranked.text)
|
||||
self.assertTrue(ranked.text.endswith("## Details\n\n### 3. Make it fast, `a | b`.\n\nFast detail.\n\n### 1. Ship the later thing.\n\nLater detail.\n\n## Notes\n\nKept as written.\n"))
|
||||
self.assertEqual(ranked.under_bar, [])
|
||||
@@ -116,16 +116,16 @@ class Rank(unittest.TestCase):
|
||||
self.assertEqual({goal: round(weight, 4) for goal, weight in rank_todo.goal_weights(rank_todo.readme_goals(readme)).items()}, {"1": 0.5455, "1.1": 0.5455, "1.2": 0.2727, "2": 0.1818})
|
||||
todo = TODO.replace("2.1 by 8, 1 by 6", "1.2 by 2, 2 by 3").replace("| 2.1 by 1 |", "| 1 by 1 |")
|
||||
ranked = self.ranked(todo, readme)
|
||||
self.assertIn("| **Make it fast, `a \\| b`.** | 3 | 8 | 3 | 1.2 by 2, 2 by 3 | 0.4 |", ranked.text)
|
||||
self.assertIn("| **Make it fast, `a \\| b`.** | 2 | 1 | 2 | 3 | 8 | 1.2 by 2, 2 by 3 | 0.4 |", ranked.text)
|
||||
|
||||
def test_scores_an_item_that_moves_no_goal_zero(self):
|
||||
ranked = self.ranked(TODO.replace("| 2 | 4 | 3 | 2.1 by 1 |", "| 2 | 4 | 3 | |"))
|
||||
self.assertIn("| **Tighten the tests.** | 2 | 4 | 3 | | 0.0 |", ranked.text)
|
||||
ranked = self.ranked(TODO.replace("| 1 | 2 | 1 | 2 | 4 | 2.1 by 1 |", "| 1 | 2 | 1 | 2 | 4 | |"))
|
||||
self.assertIn("| **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | | 0.0 |", ranked.text)
|
||||
self.assertEqual(ranked.under_bar, ["item 2 scores 0.0, under the bar 2; delete it, or mark it with a reason the bar does not apply"])
|
||||
|
||||
def test_holds_each_item_a_decision_cites_and_says_how_to_settle_a_finished_one(self):
|
||||
self.assertIsInstance(self.ranked(decisions=DECISIONS + "Until `todo.md` item 4 lands.\n"), rank_todo.Ranked)
|
||||
gone = self.ranked(TODO.replace("| 5 | 0.1.0 | | decision | **Move the writes out.** | 5 | 5 | 3 | 2.1 by 1 | 0 |\n", ""))
|
||||
gone = self.ranked(TODO.replace("| 5 | 0.1.0 | | decision | **Move the writes out.** | 1 | 1 | 4 | 5 | 5 | 2.1 by 1 | 0 |\n", ""))
|
||||
self.assertEqual(gone.reason, "docs/decisions.md cites \"`todo.md` item 5\", which todo.md no longer holds; revise that decision to say what holds now")
|
||||
|
||||
def test_names_each_item_under_the_bar_that_is_not_exempt(self):
|
||||
@@ -156,13 +156,13 @@ class Rank(unittest.TestCase):
|
||||
self.assertEqual(self.ranked(todo, readme), Failure(reason))
|
||||
|
||||
def test_ranks_questions_first_then_security_whatever_they_score_and_a_defect_by_its_score(self):
|
||||
todo = TODO.replace("| 2 | 0.1.0 | | |", "| 2 | 0.1.0 | | defect |").replace("| 9 | 2 | 3 | 2.1 by 1 |", "| 9 | 8 | 3 | 2.1 by 1 |").replace("| 5 | 0.1.0 | | decision |", "| 5 | 0.1.0 | | question |")
|
||||
todo = TODO.replace("| 2 | 0.1.0 | | |", "| 2 | 0.1.0 | | defect |").replace("| 3 | 3 | 5 | 9 | 2 | 2.1 by 1 |", "| 3 | 3 | 5 | 9 | 8 | 2.1 by 1 |").replace("| 5 | 0.1.0 | | decision |", "| 5 | 0.1.0 | | question |")
|
||||
rows = [line.split(" | ")[0] for line in self.ranked(todo, decisions="").text.splitlines() if line.startswith("| ") and " | 0." in line]
|
||||
self.assertEqual(rows, ["| 5", "| 4", "| 3", "| 2", "| 1"])
|
||||
|
||||
def test_weighs_the_maintain_goal_a_defect_restores_as_the_heaviest_achieve_goal(self):
|
||||
ranked = self.ranked(TODO.replace("| 2 | 0.1.0 | | | **Tighten the tests.** | 2 | 4 | 3 | 2.1 by 1 |", "| 2 | 0.1.0 | | defect | **Tighten the tests.** | 2 | 4 | 3 | 2 by 3 |"))
|
||||
self.assertIn("| **Tighten the tests.** | 2 | 4 | 3 | 2 by 3 | 12.5 |", ranked.text)
|
||||
ranked = self.ranked(TODO.replace("| 2 | 0.1.0 | | | **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2.1 by 1 |", "| 2 | 0.1.0 | | defect | **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2 by 3 |"))
|
||||
self.assertIn("| **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2 by 3 | 12.5 |", ranked.text)
|
||||
|
||||
def test_refuses_a_goal_named_beside_its_own_sub_goal(self):
|
||||
readme = "## Goals\n\n1. **Achieve:** A.\n 1. **Maintain:** B.\n 2. **Achieve:** C.\n2. **Achieve:** D.\n"
|
||||
@@ -177,7 +177,7 @@ class Rank(unittest.TestCase):
|
||||
|
||||
def test_accepts_a_move_of_ten_and_a_variable_base(self):
|
||||
ranked = self.ranked(TODO.replace("| 1 by 9 |", "| 1 by 10 |").replace("100 * V / 2**S", "V * S ** 2"))
|
||||
self.assertIn("| **Ship the later thing.** | 1 | 1 | 3 | 1 by 10 | 6.7 |", ranked.text)
|
||||
self.assertIn("| **Ship the later thing.** | 1 | 1 | 1 | 1 | 1 | 1 by 10 | 6.7 |", ranked.text)
|
||||
|
||||
def test_asks_to_drop_a_goal_weights_table_from_a_table_with_moves(self):
|
||||
weighted = TODO.replace("## Items\n", "| Goal | W |\n|---|---|\n| 1 | 1.00 |\n\n## Items\n")
|
||||
@@ -190,21 +190,25 @@ class Rank(unittest.TestCase):
|
||||
self.assertIn("| 1 by 9 | 2.0 |", self.ranked(TODO.replace("100 * V / 2**S", "R + S / S")).text)
|
||||
self.assertEqual(self.ranked(TODO.replace("100 * V / 2**S", "1 ** (1 / (S - S))")), Failure("the formula gives item 1 no finite score"))
|
||||
self.assertEqual(self.ranked(TODO.replace("100 * V / 2**S", "(1 / (S - S)) ** 0")), Failure("the formula gives item 1 no finite score"))
|
||||
self.assertEqual(self.ranked(TODO.replace("100 * V / 2**S", "W")), Failure("the formula 'W' may hold only numbers, R, S, V, + - * / ** and parentheses"))
|
||||
self.assertEqual(self.ranked(TODO.replace("100 * V / 2**S", "W")), Failure("the formula 'W' may hold only numbers, E, I, L, R, S, V, + - * / ** and parentheses"))
|
||||
self.assertEqual(self.ranked(TODO.replace("2.1 by 8, 1 by 6", "2.1: 8, 1: 6")), Failure("item 3's Moves '2.1: 8, 1: 6' is not empty or goals such as `1.1 by 8, 1.2 by 5`, each moved by a whole number from 1 to 10"))
|
||||
|
||||
def test_refuses_a_file_off_the_format(self):
|
||||
row = "| 2 | 0.1.0 | | | **Tighten the tests.** | 2 | 4 | 3 | 2.1 by 1 | 0 |"
|
||||
row = "| 2 | 0.1.0 | | | **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2.1 by 1 | 0 |"
|
||||
broken = {
|
||||
row: row.replace("0.1.0", "0.1"),
|
||||
"| 2 | 4 | 3 | 2.1 by 1 |": "| 2 | 4 | 11 | 2.1 by 1 |",
|
||||
"Tighten the tests.** | 2 | 4 | 3 | 2.1 by 1": "Tighten the tests.** | 2 | 4 | 3 | 2.1 by 11",
|
||||
"Move the writes out.** | 5 | 5 | 3 | 2.1 by 1": "Move the writes out.** | 5 | 5 | 3 | 2.1 by 0",
|
||||
"| 1 | 2 | 1 | 2 | 4 | 2.1 by 1 |": "| 5 | 2 | 1 | 2 | 4 | 2.1 by 1 |",
|
||||
"Tighten the tests.** | 1 | 2 | 1 | 2 | 4 |": "Tighten the tests.** | 1 | 4 | 1 | 2 | 4 |",
|
||||
"Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2.1": "Tighten the tests.** | 1 | 2 | 6 | 2 | 4 | 2.1",
|
||||
"Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2.1 by": "Tighten the tests.** | 0 | 2 | 1 | 2 | 4 | 2.1 by",
|
||||
"| 2 | 4 | 2.1 by 1 | 0 |\n| 3": "| 2 | 11 | 2.1 by 1 | 0 |\n| 3",
|
||||
"Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2.1 by 1": "Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2.1 by 11",
|
||||
"Move the writes out.** | 1 | 1 | 4 | 5 | 5 | 2.1 by 1": "Move the writes out.** | 1 | 1 | 4 | 5 | 5 | 2.1 by 0",
|
||||
"2.1 by 8, 1 by 6": "2.1: 8, 1: 6",
|
||||
"2.1 by 8, 1 by 6 ": "2.1 by 8,1 by 6 ",
|
||||
"| 3 | 8 | 3 | 2.1 by 8, 1 by 6 |": "| 3 | 8 | 3 | 2.1, 1 |",
|
||||
row + "\n": row.replace("4 | 3", "4 | ²") + "\n",
|
||||
"| 2.1 by 1 | 0 |\n\n## Details": "| 2.1 by 1 | 0 |\n\n| 9 | 0.1.0 | | | **Stray.** | 1 | 1 | 3 | 1 by 1 | 0 |\n\n## Details",
|
||||
"| 2 | 1 | 2 | 3 | 8 | 2.1 by 8, 1 by 6 |": "| 2 | 1 | 2 | 3 | 8 | 2.1, 1 |",
|
||||
row + "\n": row.replace("2 | 4 |", "2 | ² |") + "\n",
|
||||
"| 2.1 by 1 | 0 |\n\n## Details": "| 2.1 by 1 | 0 |\n\n| 9 | 0.1.0 | | | **Stray.** | 1 | 1 | 1 | 1 | 1 | 1 by 1 | 0 |\n\n## Details",
|
||||
"| 4 | 0.1.0 | | security |": "| 4 | 0.1.0 | | later |",
|
||||
"| 5 | 0.1.0 | | decision |": "| 5 | 0.1.0 | | |",
|
||||
"`Next ID = 6`": "`Next ID = 5`",
|
||||
@@ -221,6 +225,55 @@ class Rank(unittest.TestCase):
|
||||
self.assertIsInstance(self.ranked(TODO.replace(old, new)), Failure)
|
||||
|
||||
|
||||
class Risk(unittest.TestCase):
|
||||
"""R is L + E + I - 2, which rank_todo.py writes, at least 7 when I is 5 and at least 5 when I is 4."""
|
||||
def ranked(self, todo):
|
||||
return rank_todo.rank(todo, README, DECISIONS)
|
||||
|
||||
def test_writes_r_from_l_e_and_i_over_a_stale_or_missing_r(self):
|
||||
ranked = self.ranked(TODO).text
|
||||
for levels, risk in (("| 1 | 1 | 1 |", 1), ("| 2 | 1 | 2 |", 3), ("| 1 | 2 | 1 |", 2)):
|
||||
with self.subTest(levels=levels):
|
||||
self.assertIn(f"{levels} {risk} |", ranked)
|
||||
stale = TODO.replace("| 2 | 1 | 2 | 3 | 8 |", "| 2 | 1 | 2 | 10 | 8 |")
|
||||
self.assertEqual(self.ranked(stale).text, ranked)
|
||||
without_r = TODO.replace("| I | R | S |", "| I | S |").replace("|---|---|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|---|---|")
|
||||
without_r = re.sub(r"(\*\* \| \d+ \| \d+ \| \d+) \| \d+ \|", r"\1 |", without_r)
|
||||
self.assertEqual(self.ranked(without_r).text, ranked)
|
||||
|
||||
def test_writes_r_into_a_blank_r_cell(self):
|
||||
blank = TODO.replace("| 2 | 1 | 2 | 3 | 8 |", "| 2 | 1 | 2 | | 8 |")
|
||||
self.assertEqual(self.ranked(blank).text, self.ranked(TODO).text)
|
||||
|
||||
def test_leaves_a_stale_r_to_ranking_so_a_reader_keeps_the_r_the_table_stored(self):
|
||||
stale = TODO.replace("| 2 | 1 | 2 | 3 | 8 |", "| 2 | 1 | 2 | 10 | 8 |")
|
||||
self.assertEqual(next(item for item in todo_table.items(stale) if item.id == 3).scores["R"], 10)
|
||||
self.assertNotEqual(self.ranked(stale).text, stale)
|
||||
|
||||
def test_holds_r_at_7_when_i_is_5_and_at_5_when_i_is_4(self):
|
||||
cases = {"| 1 | 1 | 5 |": 7, "| 1 | 1 | 4 |": 5, "| 2 | 2 | 4 |": 6, "| 4 | 3 | 5 |": 10, "| 3 | 3 | 3 |": 7}
|
||||
for levels, risk in cases.items():
|
||||
with self.subTest(levels=levels):
|
||||
ranked = self.ranked(TODO.replace("| 1 | 2 | 1 | 2 | 4 |", f"{levels} 1 | 4 |"))
|
||||
self.assertIn(f"**Tighten the tests.** {levels} {risk} | 4 |", ranked.text)
|
||||
|
||||
def test_routes_a_worker_on_l_and_its_reviewers_on_e_plus_i(self):
|
||||
item = next(item for item in todo_table.items(TODO) if item.id == 4)
|
||||
self.assertEqual((item.likelihood, item.escape_and_impact), (3, 8))
|
||||
|
||||
def test_names_the_one_level_column_a_table_lacks(self):
|
||||
without_i = TODO.replace("| E | I | R |", "| E | R |").replace("|---|---|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|---|---|")
|
||||
without_i = re.sub(r"(\*\* \| \d+ \| \d+) \| \d+ \|", r"\1 |", without_i)
|
||||
self.assertEqual(self.ranked(without_i), Failure("todo.md's items table has no I column; add it, scoring each item's L, E and I as rank_todo.py --help shows"))
|
||||
|
||||
def test_asks_a_table_from_before_l_e_and_i_for_them_yet_reads_its_rows(self):
|
||||
before = TODO.replace("| L | E | I | R | S | Moves |", "| R | S | C | Moves |").replace("|---|---|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|---|")
|
||||
before = re.sub(r"(\*\* \|) \d+ \| \d+ \| \d+ \| (\d+) \| (\d+) \|", r"\1 \2 | \3 | 3 |", before)
|
||||
self.assertEqual(self.ranked(before), Failure("todo.md's items table has no L, E and I columns; add them, scoring each item's L, E and I as rank_todo.py --help shows, and drop C"))
|
||||
items = todo_table.items(before)
|
||||
self.assertEqual([(item.scores, item.likelihood, item.escape_and_impact) for item in items][1], ({"R": 2, "S": 4}, 0, 0))
|
||||
|
||||
|
||||
class OldShape(unittest.TestCase):
|
||||
"""A table with A, G and Goals, from before the Moves column, ranks by the formula and goal weights it states until its project reshapes it."""
|
||||
def ranked(self, todo=OLD_SHAPE, readme=README):
|
||||
@@ -249,18 +302,18 @@ class OldShape(unittest.TestCase):
|
||||
def test_refuses_a_formula_over_v(self):
|
||||
self.assertEqual(self.ranked(OLD_SHAPE.replace("-R - S/4 + 2*A + 2*G*W", "V")), Failure("the formula 'V' may hold only numbers, A, G, R, S, W, + - * / ** and parentheses"))
|
||||
|
||||
def test_asks_for_a_c_column_in_a_table_without_one(self):
|
||||
def test_ranks_a_table_from_before_c_joined_and_says_to_reshape_it(self):
|
||||
legacy = OLD_SHAPE.split("## Items")[0] + "## Items\n\n| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |\n|---|---|---|---|---|---|---|---|---|---|\n| 2 | 0.1.0 | | **Tighten the tests.** | 2 | 4 | 5 | 5 | 2.1 | 0 |\n"
|
||||
self.assertEqual(rank_todo.rank(legacy, README, "").reason, "todo.md's items table has no C column; add one after S, and score each item's C from 1 to 10 per rank_todo.py --help")
|
||||
self.assertEqual(rank_todo.rank(legacy, README, "").notes, [rank_todo.RESHAPE])
|
||||
|
||||
def test_refuses_a_table_with_both_moves_and_goals(self):
|
||||
both = OLD_SHAPE.replace("| Goals | Score |\n|---|", "| Goals | Moves | Score |\n|---|---|").replace(" | 0 |\n", " | 1 by 1 | 0 |\n")
|
||||
self.assertEqual(self.ranked(both), Failure("todo.md's items table has A, G and Goals, columns of the old shape that Moves replaced; drop them"))
|
||||
self.assertEqual(self.ranked(both), Failure("todo.md's items table has no L, E and I columns; add them, scoring each item's L, E and I as rank_todo.py --help shows, and drop C, A, G and Goals"))
|
||||
one = TODO.replace("| Moves | Score |\n|---|", "| Moves | G | Score |\n|---|---|").replace(" | 0 |\n", " | 1 | 0 |\n")
|
||||
self.assertEqual(self.ranked(one), Failure("todo.md's items table has G, a column of the old shape that Moves replaced; drop it"))
|
||||
self.assertEqual(self.ranked(one), Failure("todo.md's items table has G, a column of an older shape; drop it, as rank_todo.py --help shows"))
|
||||
|
||||
def test_names_the_goals_column_a_table_misspells_by_its_shape(self):
|
||||
misspelt = TODO.replace("| C | Moves |", "| C | Move |")
|
||||
misspelt = TODO.replace("| S | Moves |", "| S | Move |")
|
||||
self.assertEqual(rank_todo.rank(misspelt, README, DECISIONS), Failure("todo.md's items table has no Moves column; rank_todo.py --help shows a valid one"))
|
||||
self.assertEqual(todo_table.items(misspelt), Failure("todo.md's items table has no Moves column; rank_todo.py --help shows a valid one"))
|
||||
self.assertEqual(self.ranked(OLD_SHAPE.replace("| G | Goals |", "| G | Goal |")), Failure("todo.md's items table has no Goals column; rank_todo.py --help shows a valid one"))
|
||||
@@ -272,11 +325,11 @@ class Blockers(unittest.TestCase):
|
||||
|
||||
def test_ranks_each_blocker_just_above_the_first_item_it_blocks(self):
|
||||
ranked = self.ranked(TODO.replace("| 3 | 0.1.0 | |", "| 3 | 0.1.0 | 5, 2 |"))
|
||||
self.assertIn("""| 4 | 0.1.0 | | security | **Rotate the leaked key.** | 9 | 2 | 3 | 2.1 by 1 | 8.3 |
|
||||
| 2 | 0.1.0 | | | **Tighten the tests.** | 2 | 4 | 3 | 2.1 by 1 | 2.1 |
|
||||
| 5 | 0.1.0 | | decision | **Move the writes out.** | 5 | 5 | 3 | 2.1 by 1 | 1.0 |
|
||||
| 3 | 0.1.0 | 2, 5 | | **Make it fast, `a \\| b`.** | 3 | 8 | 3 | 2.1 by 8, 1 by 6 | 2.6 |
|
||||
| 1 | 0.2.0 | | | **Ship the later thing.** | 1 | 1 | 3 | 1 by 9 | 300.0 |
|
||||
self.assertIn("""| 4 | 0.1.0 | | security | **Rotate the leaked key.** | 3 | 3 | 5 | 9 | 2 | 2.1 by 1 | 8.3 |
|
||||
| 2 | 0.1.0 | | | **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2.1 by 1 | 2.1 |
|
||||
| 5 | 0.1.0 | | decision | **Move the writes out.** | 1 | 1 | 4 | 5 | 5 | 2.1 by 1 | 1.0 |
|
||||
| 3 | 0.1.0 | 2, 5 | | **Make it fast, `a \\| b`.** | 2 | 1 | 2 | 3 | 8 | 2.1 by 8, 1 by 6 | 2.6 |
|
||||
| 1 | 0.2.0 | | | **Ship the later thing.** | 1 | 1 | 1 | 1 | 1 | 1 by 9 | 300.0 |
|
||||
""", ranked.text)
|
||||
self.assertEqual(self.ranked(ranked.text), ranked)
|
||||
|
||||
@@ -290,7 +343,7 @@ class Blockers(unittest.TestCase):
|
||||
self.assertEqual(rows, ["| 4", "| 2", "| 5", "| 3", "| 1"])
|
||||
|
||||
def test_drops_a_blocker_the_table_no_longer_holds_as_finished(self):
|
||||
finished = TODO.replace("| 4 | 0.1.0 | | security | **Rotate the leaked key.** | 9 | 2 | 3 | 2.1 by 1 | 0 |\n", "").replace("| 2 | 0.1.0 | |", "| 2 | 0.1.0 | 4 |")
|
||||
finished = TODO.replace("| 4 | 0.1.0 | | security | **Rotate the leaked key.** | 3 | 3 | 5 | 9 | 2 | 2.1 by 1 | 0 |\n", "").replace("| 2 | 0.1.0 | |", "| 2 | 0.1.0 | 4 |")
|
||||
ranked = self.ranked(finished)
|
||||
self.assertIn("| 2 | 0.1.0 | | | **Tighten the tests.**", ranked.text)
|
||||
self.assertEqual(ranked.notes, ["item 2: dropped 4 from its Blocked by; the table no longer holds it"])
|
||||
@@ -311,19 +364,19 @@ class Blockers(unittest.TestCase):
|
||||
self.assertEqual(self.ranked(TODO.replace("| 3 | 0.1.0 | |", "| 3 | 0.1.0 | 2 and 5 |")), Failure("item 3's Blocked by '2 and 5' is not empty or IDs separated by \", \""))
|
||||
|
||||
def test_adds_the_column_to_a_table_without_it(self):
|
||||
without = TODO.replace("| Blocked by ", "").replace("|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|")
|
||||
without = TODO.replace("| Blocked by ", "").replace("|---|---|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|---|---|")
|
||||
without = re.sub(r"\| (\d) \| (0\.[12]\.0) \| \| ", r"| \1 | \2 | ", without)
|
||||
self.assertEqual(self.ranked(without), self.ranked(TODO))
|
||||
unscored = TODO.replace(" | Score |", " |").replace("|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|").replace(" | 0 |\n", " |\n")
|
||||
unscored = TODO.replace(" | Score |", " |").replace("|---|---|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|---|---|").replace(" | 0 |\n", " |\n")
|
||||
self.assertEqual(self.ranked(unscored), self.ranked(TODO))
|
||||
|
||||
def test_refuses_a_column_it_does_not_know_rather_than_drop_it_and_asks_for_c_beside_blocked_by(self):
|
||||
def test_refuses_a_column_it_does_not_know_rather_than_drop_it_and_one_an_older_shape_held(self):
|
||||
extra = TODO.replace("| Moves | Score |\n|---|", "| Moves | Score | Later |\n|---|---|").replace(" | 0 |\n", " | 0 | x |\n")
|
||||
self.assertEqual(self.ranked(extra), Failure("todo.md's items table has the column Later, which this rank_todo.py does not know; the columns it knows are ID, Release, Blocked by, Exempt, Item, R, S, C, Moves, Score. Rename it, or move this machine to the release that added it"))
|
||||
self.assertEqual(self.ranked(extra), Failure("todo.md's items table has the column Later, which this rank_todo.py does not know; the columns it knows are ID, Release, Blocked by, Exempt, Item, L, E, I, R, S, Moves, Score. Rename it, or move this machine to the release that added it"))
|
||||
self.assertEqual([item.id for item in todo_table.items(extra)], [1, 2, 3, 4, 5])
|
||||
without_c = TODO.replace("| R | S | C |", "| R | S |").replace("|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|")
|
||||
without_c = re.sub(r"(\*\* \| \d+ \| \d+) \| \d+ \|", r"\1 |", without_c)
|
||||
self.assertEqual(self.ranked(without_c).reason, "todo.md's items table has no C column; add one after S, and score each item's C from 1 to 10 per rank_todo.py --help")
|
||||
with_c = TODO.replace("| S | Moves |", "| S | C | Moves |").replace("|---|---|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|---|---|---|---|")
|
||||
with_c = re.sub(r"(\*\* \| \d+ \| \d+ \| \d+ \| \d+ \| \d+) \|", r"\1 | 3 |", with_c)
|
||||
self.assertEqual(self.ranked(with_c), Failure("todo.md's items table has C, a column of an older shape; drop it, as rank_todo.py --help shows"))
|
||||
|
||||
def test_refuses_text_under_items_beside_the_table(self):
|
||||
last = TODO.split("\n## Details")[0] + "\n"
|
||||
@@ -340,14 +393,14 @@ class Blockers(unittest.TestCase):
|
||||
table = TODO[TODO.index("| ID | Release |"):TODO.index("\n\n## Details")] + "\n"
|
||||
above = table + "\n" + TODO.replace(table, "")
|
||||
self.assertEqual(self.ranked(above).reason, "the items table sits under no ## heading; move it under ## Items")
|
||||
header = "| ID | Release | Blocked by | Exempt | Item | R | S | C | Moves | Score |\n|---|---|---|---|---|---|---|---|---|---|\n"
|
||||
header = "| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |\n|---|---|---|---|---|---|---|---|---|---|---|---|\n"
|
||||
self.assertEqual(todo_table.items(TODO + "\n" + header.replace("| Score |", "| Later |")), Failure("todo.md holds 2 tables whose header opens | ID | Release |; keep one items table"))
|
||||
repeated = TODO.replace("| Moves | Score |\n|---|", "| Moves | Score | Exempt |\n|---|---|").replace(" | 0 |\n", " | 0 | x |\n")
|
||||
self.assertEqual(todo_table.items(repeated), Failure("todo.md's items table names the column Exempt twice; keep one"))
|
||||
|
||||
def test_reads_an_empty_table_and_fails_on_a_header_with_no_divider(self):
|
||||
header = "| ID | Release | Blocked by | Exempt | Item | R | S | C | Moves | Score |"
|
||||
self.assertEqual(todo_table.items(f"## Items\n\n{header}\n|---|---|---|---|---|---|---|---|---|---|\n\n## Details\n"), [])
|
||||
header = "| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |"
|
||||
self.assertEqual(todo_table.items(f"## Items\n\n{header}\n|---|---|---|---|---|---|---|---|---|---|---|---|\n\n## Details\n"), [])
|
||||
self.assertEqual(todo_table.items(f"## Items\n\n{header}"), Failure(f"the table under {header} has no |---| divider line"))
|
||||
|
||||
|
||||
|
||||
+36
-16
@@ -4,8 +4,11 @@ import typing
|
||||
|
||||
from failure import Failure
|
||||
|
||||
# Each score a Shape names, and each goal's move, runs from 1 to TOP_SCORE.
|
||||
# Each score a Shape names, and each goal's move, runs from 1 to TOP_SCORE, but for the levels L, E and I, which run from 1 to their LEVELS: ~/.claude/rules/todo-items.md anchors each level.
|
||||
TOP_SCORE = 10
|
||||
LEVELS = {"L": 4, "E": 3, "I": 5}
|
||||
# The least R each impact level I allows, since a severe enough consequence outranks how unlikely it is.
|
||||
RISK_FLOOR = {4: 5, 5: 7}
|
||||
EXEMPTIONS = ("", "decision", "defect", "owner", "principle", "question", "security")
|
||||
# The column naming the items that block an item: each must be done before the item starts.
|
||||
BLOCKED_BY = "Blocked by"
|
||||
@@ -22,6 +25,12 @@ class Shape:
|
||||
"""A shape of the items table: its columns, and how its goals cell reads and writes."""
|
||||
# The score columns, in the order the table holds them.
|
||||
score_columns: tuple[str, ...]
|
||||
# The columns a reader does without: a missing BLOCKED_BY reads as blocked by nothing, a missing score column as that score unset, and a missing Score column as not yet ranked.
|
||||
optional: tuple[str, ...]
|
||||
# The columns rank_todo.py needs to rank a row, so no chunk starts on an item whose risk is unscored.
|
||||
risk_columns: tuple[str, ...]
|
||||
# The scores rank_todo.py works out and writes, so a blank cell reads as no score yet.
|
||||
derived: tuple[str, ...]
|
||||
# The column naming the goals an item serves; read_goals reads it, given the item's ID to name in a failure, and write_goals writes it back.
|
||||
goals_column: str
|
||||
read_goals: typing.Callable[[str, str], Moves | Failure]
|
||||
@@ -29,13 +38,13 @@ class Shape:
|
||||
|
||||
@property
|
||||
def columns(self) -> tuple[str, ...]:
|
||||
"""Every column of the shape, in order. A reader takes cells by column name, so it reads a table from before BLOCKED_BY or C joined, or with a column a later release adds."""
|
||||
"""Every column of the shape, in order. A reader takes cells by column name, so it reads a table from before an optional column joined, or with a column a later release adds."""
|
||||
return ("ID", "Release", BLOCKED_BY, "Exempt", "Item", *self.score_columns, self.goals_column, "Score")
|
||||
|
||||
@property
|
||||
def needed(self) -> tuple[str, ...]:
|
||||
"""The columns without which no row can be read: a missing BLOCKED_BY reads as blocked by nothing, a missing C as no C, and a missing Score as no score yet."""
|
||||
return tuple(column for column in self.columns if column not in (BLOCKED_BY, "C", "Score"))
|
||||
"""The columns without which no row can be read."""
|
||||
return tuple(column for column in self.columns if column not in self.optional)
|
||||
|
||||
|
||||
def parse_moves(id_: str, cell: str) -> Moves | Failure:
|
||||
@@ -65,9 +74,10 @@ def write_cited(moves: Moves) -> str:
|
||||
return ", ".join(goal for goal, _ in moves)
|
||||
|
||||
|
||||
SHAPE = Shape(("R", "S", "C"), MOVES, parse_moves, write_moves)
|
||||
# The shape from before MOVES replaced A, G and Goals, which older todo.md files hold.
|
||||
OLD_SHAPE = Shape(("R", "S", "C", "A", "G"), "Goals", parse_cited, write_cited)
|
||||
# L, E and I are optional, since score_outcomes reads, out of git history, tables from before they replaced C; R too, since rank_todo.py writes it from them.
|
||||
SHAPE = Shape(("L", "E", "I", "R", "S"), (BLOCKED_BY, "L", "E", "I", "R", "Score"), ("L", "E", "I"), ("R",), MOVES, parse_moves, write_moves)
|
||||
# The shape from before MOVES replaced A, G and Goals, and L, E and I replaced C, which older todo.md files hold; one from before C joined lacks it.
|
||||
OLD_SHAPE = Shape(("R", "S", "C", "A", "G"), (BLOCKED_BY, "C", "Score"), (), (), "Goals", parse_cited, write_cited)
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -81,9 +91,19 @@ class Item:
|
||||
blocked_by: tuple[int, ...]
|
||||
|
||||
@property
|
||||
def risk_and_complexity(self) -> int:
|
||||
"""R + C, which routes the item's chunk to a rung; 0 without a C."""
|
||||
return self.scores["R"] + self.scores["C"] if "C" in self.scores else 0
|
||||
def likelihood(self) -> int:
|
||||
"""L, which routes the item's worker to a rung; 0 without one."""
|
||||
return self.scores.get("L", 0)
|
||||
|
||||
@property
|
||||
def escape_and_impact(self) -> int:
|
||||
"""E + I, which routes the item's reviewers to a rung; 0 without them."""
|
||||
return self.scores["E"] + self.scores["I"] if "E" in self.scores and "I" in self.scores else 0
|
||||
|
||||
|
||||
def risk(likelihood: int, escape: int, impact: int) -> int:
|
||||
"""R from its levels: L + E + I - 2, never under RISK_FLOOR's value for I."""
|
||||
return max(likelihood + escape + impact - 2, RISK_FLOOR.get(impact, 0))
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -101,8 +121,7 @@ class Table:
|
||||
|
||||
def shape_of(columns: tuple[str, ...]) -> Shape:
|
||||
"""OLD_SHAPE where the columns include A, G or Goals and not MOVES; else SHAPE, so a misspelt MOVES is reported missing."""
|
||||
old_only = set(OLD_SHAPE.columns) - set(SHAPE.columns)
|
||||
return OLD_SHAPE if MOVES not in columns and old_only.intersection(columns) else SHAPE
|
||||
return OLD_SHAPE if MOVES not in columns and {"A", "G", OLD_SHAPE.goals_column}.intersection(columns) else SHAPE
|
||||
|
||||
|
||||
def items(todo: str) -> list[Item] | None | Failure:
|
||||
@@ -171,7 +190,7 @@ def parse_row(row: str, table: Table) -> Item | Failure:
|
||||
if misformed := check_cells(row, named):
|
||||
return misformed
|
||||
shape = shape_of(table.columns)
|
||||
scores = parse_scores(named["ID"], named, shape.score_columns)
|
||||
scores = parse_scores(named["ID"], {name: cell for name, cell in named.items() if cell or name not in shape.derived}, shape.score_columns)
|
||||
if isinstance(scores, Failure):
|
||||
return scores
|
||||
blocked_by = parse_blockers(named["ID"], named.get(BLOCKED_BY, ""))
|
||||
@@ -199,13 +218,14 @@ def check_cells(row: str, named: dict[str, str]) -> Failure | None:
|
||||
|
||||
|
||||
def parse_scores(id_: str, named: dict[str, str], names: tuple[str, ...]) -> dict[str, int] | Failure:
|
||||
"""Each score in names that the row holds; only C may be missing, since the table holds every needed column."""
|
||||
"""Each score in names that the row holds; only an optional one may be missing, since the table holds every needed column."""
|
||||
scores: dict[str, int] = {}
|
||||
for name in names:
|
||||
if name not in named:
|
||||
continue
|
||||
if not re.fullmatch(r"[0-9]+", named[name]) or not 1 <= int(named[name]) <= TOP_SCORE:
|
||||
return Failure(f"item {id_}'s {name} {named[name]!r} is not a whole number from 1 to {TOP_SCORE}")
|
||||
top = LEVELS.get(name, TOP_SCORE)
|
||||
if not re.fullmatch(r"[0-9]+", named[name]) or not 1 <= int(named[name]) <= top:
|
||||
return Failure(f"item {id_}'s {name} {named[name]!r} is not a whole number from 1 to {top}")
|
||||
scores[name] = int(named[name])
|
||||
return scores
|
||||
|
||||
|
||||
@@ -1984,3 +1984,33 @@ Inherited architect, decided by:
|
||||
- Navigation: `docs/decisions.md:239` `routing: a chunk's worker runs on the rung its item picks…`: the routing entry is split into titled entries, so a symptom leads to a heading
|
||||
- Shape: `bin/model_routing/ladder.py:87` `review_rung`: one named rule that the worker's prompt and the hook share
|
||||
- Locality: `bin/rule_hook/reviewer_rung.py:13` `SEAT`: the hook recognises a launch by a sentence in two markdown files, so rewording either turns the check off without a failure
|
||||
|
||||
## 2026-10-16T16:33:58Z, PR #296 at 50b9daa, against d1b0fe3
|
||||
|
||||
Ruling: same
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Mid A | same | worse | better | same | same |
|
||||
| Inherited architect | same | worse | better | same | same |
|
||||
|
||||
Mid A, decided by:
|
||||
|
||||
- Locality: `bin/loop_claude/loop/reload.py:38` `decoded_as`: looks up a `RETIRED` attribute and a static `fits` by name on whatever class it decodes, beside a second sentinel, so a rule on one class steers a generic module and only a comment says so
|
||||
- Locality: `bin/model_routing/route.py:11` `route_from`: two parsers of one archived line, and `Route.__post_init__` trusts that callers ran `levels_fit` first
|
||||
- Locality: `bin/rule_hook/reviewer_rung.py:25` `reviewer_breaches`: the required rung comes from a third environment variable, with a fallback to the old formula for a worker an older loop-claude started
|
||||
- Locality: `bin/rank_todo.py:570` `parse_item`: a function named parse also computes R and writes it into the item's scores
|
||||
- Shape: `bin/model_routing/ladder.py:62` `worker_rung`: worker, planning and review rungs are named for whom they serve, and plain step dicts replace the scan over band pairs
|
||||
- Shape, Self-sufficiency: `bin/loop_claude/cycle/prompts.py:26` `review_launches`: one mapping gives each review's rung, and the docstring shows the sentence it renders
|
||||
- Shape: `bin/todo_table.py:23` `Shape`: grew from 4 fields to 7, and `score_outcomes/outcomes.py`'s `report` does two jobs
|
||||
|
||||
Inherited architect, decided by:
|
||||
|
||||
- Locality: `bin/loop_claude/loop/reload.py:43` `decoded_as`: a second sentinel and two class hooks found by name through `getattr`, so a dropped route at 3am depends on rules spread across two files and held by convention
|
||||
- Locality: `bin/model_routing/ladder.py:73` `worker_rung`: raises KeyError for an L off its scale, safe only through guards in `judge.py`'s `verdict_from` and `Route.__post_init__`
|
||||
- Locality: `bin/rule_hook/reviewer_rung.py:26` `reviewer_breaches`: the reviewers' rung is worked out in loop-claude and reaches the hook through a third variable, beside an inline fallback restating the old rule
|
||||
- Shape: `bin/loop_claude/cycle/prompts.py:26` `review_launches`: `review_rungs` is the one source the prompts, the worker's environment and the hook read, and the docstring shows the sentence produced
|
||||
- Shape: `bin/model_routing/ladder.py:63` `WORKER_STEPS`: two small dicts with intent-revealing names replace the band tuple, and `lowest_worker_rung` replaces a magic constant
|
||||
- Shape: `bin/todo_table.py:21` `Shape`: names `optional`, `risk_columns` and `derived` as fields where `C` was hard-coded, though its seven-argument constructors take counting
|
||||
- Shape: `bin/model_routing/route.py:16` `ArchivedWorker`: the file doubles, almost all of it for old R + C archive lines
|
||||
- Self-sufficiency: `bin/score_outcomes/outcomes.py:37` `SHOWN`: its comment says C shows on any item scored before L, E and I, though only a table with A, G and Goals keeps it
|
||||
|
||||
+25
-23
@@ -62,7 +62,7 @@
|
||||
|
||||
2026-10-15, lilleman asked for it through todo item 144, implemented by claude. Serves goal 1.1: a cheap reviewer that finds the work below standard can have a more capable one check that ruling, and work below standard gets a more capable rung to fix it.
|
||||
|
||||
The work is below standard where systems-architect returns BLOCK, a reviewer reports a finding against larv-review's `standards/quality.md`, or `record-ruling` counts a comprehension seat's vote in a paired ruling as worse. That reviewer or seat may ask, and the worker then relaunches it on the rung loop-claude's prompt names, one above the rung it ran on. `bin/loop_claude/cycle/prompts.py`'s `reruns` names it: the seats run on the project's rung, and the reviewers too while the worker runs below it. Off the ladder, or for a reviewer or seat already on the top rung, the prompt names none, and its report stands.
|
||||
The work is below standard where systems-architect returns BLOCK, a reviewer reports a finding against larv-review's `standards/quality.md`, or `record-ruling` counts a comprehension seat's vote in a paired ruling as worse. That reviewer or seat may ask, and the worker then relaunches it on the rung loop-claude's prompt names, one above the rung it ran on. `bin/loop_claude/cycle/prompts.py`'s `reruns` names it: the seats run on the project's rung, and the reviewers on the highest of the worker's rung and the one their item's E + I picks. Off the ladder, or for a reviewer or seat already on the top rung, the prompt names none, and its report stands.
|
||||
|
||||
A reviewer's relaunch gets the report that asked and confirms or dismisses each of its findings with a reason, so no finding goes unanswered; a seat's relaunch reads afresh, since a seat reports no findings. The relaunch's report stands in place of the one that asked, in the same round, so it adds no round, and a relaunch cannot ask again. So the round cap still bounds a pass, and one round launches a reviewer at most twice. The ruling that stands is the fourth escalation trigger in `CLAUDE.md`. lilleman ruled that no ruling raises the judge's rung, which `routing: on a model judge-replay covered` sets.
|
||||
|
||||
@@ -130,21 +130,21 @@ Valid while the ladder orders its rungs by competence.
|
||||
|
||||
2026-10-02, lilleman. Serves goal 1: findings were filed faster than chunks closed them, so the plan grew without telling what mattered. Each item is scored as `~/.claude/rules/todo-items.md` says; this repo's formula counts value highest and size least. An item under the bar is not kept; this repo's bar started at 10 and was set to 9 the same day, purging the 17 items under it. An Exempt cell keeps a row past the bar where a rule needs it kept. lilleman retired the two decisions whose plans the purge removed: the stats line in the transcript archive, and three benches before a workflow change merges. A finding under the bar is recorded nowhere, so a later review may raise it again, and the run rescores it. That costs one scoring; a ledger of dropped findings would regrow the list the bar caps. larv-review's "Round cap" → "Done" counts a finding named under the bar as settled, so one raised again holds no pass open. `rank_todo.py --check` is part of larv-review's gate, since only a check keeps the stored Score from going stale. The format is mandatory in every project, as the premises are (goal 1.3); the formula and bar are each project's own. A script ranks the table so a re-rank after each addition or edit is deterministic and cheap. A 🟡 against a technical principle is exempt as `principle`, because its audience value scores low and the bar would drop every one. A 🟢 against a principle is scored like any other finding. 2026-10-08, lilleman: the goal weights in `todo.md` depart from an even spread by position, which would weigh goal 4, leaking nothing, at 0.11. Valid while one human sets the bar and reads the ranking, and while the items marked `principle` stay few enough to rank.
|
||||
|
||||
2026-10-15, lilleman: the scores get a researched method, which `todo.md` items 146, 149, 151 and 152 build, after PR #283 shipped the ranking; the evidence is in `docs/scoring-and-goals-research.md`. R becomes L + E + I − 2: the chance the change holds a defect, set first by its size; the chance the defect escapes tests, review and monitoring; and the worst impact dimension, each about ×10 per level. I5, irreversible, holds R at 7 or more, and I4 at 5 or more, since a severe enough consequence outranks how unlikely it is. C goes from every table and tool: what it scored beyond size, spread and novelty, which L already holds, is complexity, which adds almost nothing once size is accounted for. A task's R alone then picks its rung. Items rank by V / C, value per cost, the cost being 2^S, and V sums each cited achieve goal's sum-reciprocal weight times how far the item moves it. A goes, and the Audience says whose value the goals serve. R stays out of the ranking and sets how carefully an item is done; an item that lowers risk scores it as value. Each release fills by priority up to a capacity it states, in place of the bar. Each README goal is `maintain`, which vetoes an item that breaks it, or `achieve`, with a signal, a counter-signal and a one-line reason, as this README's Goals now mark them; each project keeps settled decisions to test its goals against. Each part of this method holds once the item that builds it ships. 2026-10-16, lilleman: this replaces "A task's R alone then picks its rung". A task's worker routes on its L, and its reviewers on its E + I (goals 1.1 and 1.4). How much a change does, and how new that is, make a worker err. How far a defect escapes, and what it breaks, set how hard review must look. I's reach counts every project and session that runs the change, so a change to a tool every session runs scores I at level 3, every consumer affected.
|
||||
2026-10-15, lilleman: the scores get a researched method, which `todo.md` items 149, 151 and 152 build, after PR #283 shipped the ranking and PR #296 the risk scores; the evidence is in `docs/scoring-and-goals-research.md`. R becomes L + E + I − 2: the chance the change holds a defect, set first by its size; the chance the defect escapes tests, review and monitoring; and the worst impact dimension, each about ×10 per level. I5, irreversible, holds R at 7 or more, and I4 at 5 or more, since a severe enough consequence outranks how unlikely it is. C goes from every table and tool: what it scored beyond size, spread and novelty, which L already holds, is complexity, which adds almost nothing once size is accounted for. Items rank by value per cost, V / 2^S, and V sums each cited achieve goal's sum-reciprocal weight times how far the item moves it. A goes, and the Audience says whose value the goals serve. R stays out of the ranking and sets how carefully an item is done; an item that lowers risk scores it as value. Each release fills by priority up to a capacity it states, in place of the bar. Each README goal is `maintain`, which vetoes an item that breaks it, or `achieve`, with a signal, a counter-signal and a one-line reason, as this README's Goals now mark them; each project keeps settled decisions to test its goals against. Each part of this method holds once the item that builds it ships. 2026-10-16, lilleman: a task's L and E + I pick its rungs in place of R, as `routing: one ladder` records.
|
||||
|
||||
2026-10-16, PR #283: items rank by `Score = 100 * V / 2**S`. 2**S is the cost; the 100 lets a score read to one decimal. V sums, over the achieve goals an item's Moves cell names, each goal's weight times how far the item moves it, 1 to 10. The goal ranked k of n achieve goals in reading order weighs (1/k) / (1 + 1/2 + … + 1/n), the sum-reciprocal weight. A Moves column replaces A, G and Goals, and `rank_todo.py` derives the weights from README.md, so the goal weights table and the 2026-10-08 weights are gone. An item that cited only maintain goals now names the achieve goal it moves, or, as a defect restoring one, that maintain goal, as the entry below says. A panel of three readers seated as lilleman picked the spelling: two put a Moves column holding `1.1 by 8, 1.2 by 5` first, and one put `1.1: 8` first; two chose `2**S`, since a Python reader takes `^` for XOR; all three chose one decimal of `100 * V / 2**S` over three significant digits or a log. The bar moved to 0.1, lilleman's call answering the review of PR #283: on one-decimal scores it drops an item scoring under 0.05, in practice one that moves no achieve goal, and item 149 revisits it. A table in the old shape still ranks by the formula and weights it states, and ranking it says to reshape it, so no other project's commits stop; reshaping it is that project's only work until done, as `CLAUDE.md` → "Working loop" says. The C column stays until `todo.md` item 146 drops it. The old shape's ranking is valid until `todo.md` item 156.
|
||||
2026-10-16, PR #283: items rank by `Score = 100 * V / 2**S`. 2**S is the cost; the 100 lets a score read to one decimal. V sums, over the achieve goals an item's Moves cell names, each goal's weight times how far the item moves it, 1 to 10. The goal ranked k of n achieve goals in reading order weighs (1/k) / (1 + 1/2 + … + 1/n), the sum-reciprocal weight. A Moves column replaces A, G and Goals, and `rank_todo.py` derives the weights from README.md, so the goal weights table and the 2026-10-08 weights are gone. An item that cited only maintain goals now names the achieve goal it moves, or, as a defect restoring one, that maintain goal, as the entry below says. A panel of three readers seated as lilleman picked the spelling: two put a Moves column holding `1.1 by 8, 1.2 by 5` first, and one put `1.1: 8` first; two chose `2**S`, since a Python reader takes `^` for XOR; all three chose one decimal of `100 * V / 2**S` over three significant digits or a log. The bar moved to 0.1, lilleman's call answering the review of PR #283: on one-decimal scores it drops an item scoring under 0.05, in practice one that moves no achieve goal, and item 149 revisits it. A table in the old shape still ranks by the formula and weights it states, and ranking it says to reshape it, so no other project's commits stop; reshaping it is that project's only work until done, as `CLAUDE.md` → "Working loop" says. PR #296 dropped the C column, and put L, E and I right before R, which `rank_todo.py` writes from them: of three readers seated as lilleman, two ranked that order first, since a reader meets each level before the R it builds and R sits beside S, and the third ranked it second, after an order with S before the levels. The old shape's ranking is valid until `todo.md` item 156.
|
||||
|
||||
2026-10-16, lilleman, answering the review of PR #283. An item that restores a maintain goal the project breaks today is a defect: exempt as `defect`. Its Moves name that maintain goal and how much of it is broken, and the goal weighs as the heaviest achieve goal, since a broken maintain goal outranks any achieve goal. Every defect ranks by its score, so a trivial one waits behind valuable work. Only questions and then security findings rank first in their release: a question's answer may change other items, and a question waits for the human anyway, so ranking it first delays no chunk. An item names the most specific goal it moves, and a parent only for a move none of its sub-goals covers, so a sub-goal's move does not count twice; `rank_todo.py` refuses a goal named beside its own sub-goal. Each project reshapes its own `todo.md`; `CLAUDE.md` → "Working loop" makes the reshape the only work until done, as it makes writing missing premises. An item that makes this project easier to read moves goal 1.4 as well as 1.1, since reading cost is measured in tokens per chunk; a goal of its own for readability is left for later. Cost gains a fixed part, F in `Score = 100 * V / (2**S + F)`: every chunk carries a cost its size does not shrink, the reading before its first commit, about 130k tokens, and its review passes, so `2**S` alone overrated tiny items. F is 8, about one S 3 change, until `todo.md` item 158 measures it.
|
||||
|
||||
2026-10-16, lilleman, ruling on the plan in PR #283: release 0.2.0 takes plan items 146, 152, 73, 50, 65, 84, 153 and 67, the split of `routing: one ladder`, and the question of how many chunks `score-outcomes` and the reading-cost trend wait for, ahead of their scores, to put budget and efficiency in the other projects first. A standing goal cannot capture that priority for now, so it is a one-time ruling: the later releases, and every release after 0.2.0, fill from the score again.
|
||||
|
||||
2026-10-15, lilleman. Serves goal 1.4: loop-claude routes a chunk's first cycle on the item the chunk takes (`routing: a chunk's worker runs on the rung its item picks`). An item lists the items that must be done before it starts in a Blocked by column, and `rank_todo.py` ranks each blocker above every item it blocks. The table's order is `todo.md`'s only order: `rank_todo.py` fails on text beside the table under `## Items`, since a prose order there, "Release 0.3.0 takes items 125, 114 and 124 first", contradicted the table and loop-claude could not read it. A decision that wants an order sets the scores instead. An item waits while it is marked `question` or a blocker of it is still in the table, whatever the rows' order, so a release's first row that does not wait is the item a chunk takes. A blocker the table no longer holds was finished or deleted, and is dropped, so the cell shows what still blocks. A panel of three readers seated as lilleman picked the spelling: two ranked "Blocked by" first, as the cell lists only what still blocks, and all three chose bare IDs such as `12, 14` in a column right after Release, read before the tagline and kept apart from the scores. `rank_todo.py` adds the column to a table without it. The tools read cells by column name, so a later column leaves a reader on an older release working; `rank_todo.py`, which writes every column, fails on one it does not know rather than drop it.
|
||||
2026-10-15, lilleman. Serves goal 1.4: loop-claude routes a chunk's first cycle on the item the chunk takes (`routing: a chunk's worker runs on the rung its item's L picks`). An item lists the items that must be done before it starts in a Blocked by column, and `rank_todo.py` ranks each blocker above every item it blocks. The table's order is `todo.md`'s only order: `rank_todo.py` fails on text beside the table under `## Items`, since a prose order there, "Release 0.3.0 takes items 125, 114 and 124 first", contradicted the table and loop-claude could not read it. A decision that wants an order sets the scores instead. An item waits while it is marked `question` or a blocker of it is still in the table, whatever the rows' order, so a release's first row that does not wait is the item a chunk takes. A blocker the table no longer holds was finished or deleted, and is dropped, so the cell shows what still blocks. A panel of three readers seated as lilleman picked the spelling: two ranked "Blocked by" first, as the cell lists only what still blocks, and all three chose bare IDs such as `12, 14` in a column right after Release, read before the tagline and kept apart from the scores. `rank_todo.py` adds the column to a table without it. The tools read cells by column name, so a later column leaves a reader on an older release working; `rank_todo.py`, which writes every column, fails on one it does not know rather than drop it.
|
||||
|
||||
## todo.md: only a session or subagent on the project's rung or above scores or reshapes the plan; one below it drafts new items and changes to filed ones for a subagent at or above that rung to apply and score, and removes finished ones
|
||||
|
||||
2026-10-15, lilleman. Serves goal 1.1: a cheaper rung writes worse scores, and a wrong score routes a chunk onto the wrong rung, ranks the plan wrong and drops an item under the bar that should stay. Below the project's rung, a session or subagent may not score an item, move it between releases, split a milestone, delete an item under the bar, or rewrite a filed item. It drafts each new item, and each change to a filed item, such as a refresh a rule asks for. Before the next commit, one subagent on the project's rung or above applies and scores the drafts. One launch covers all of a chunk's drafts (goal 1.4). A security finding gets that subagent at once, since it is filed the moment it is found. Removing a finished item closes work and plans none, so any rung does it. Scoring a delegated task's R + C runs on any rung too: a launch on the project's rung to score each delegation would cost more than the cheaper task saves (goal 1.4).
|
||||
2026-10-15, lilleman. Serves goal 1.1: a cheaper rung writes worse scores, and a wrong score routes a chunk onto the wrong rung, ranks the plan wrong and drops an item under the bar that should stay. Below the project's rung, a session or subagent may not score an item, move it between releases, split a milestone, delete an item under the bar, or rewrite a filed item. It drafts each new item, and each change to a filed item, such as a refresh a rule asks for. Before the next commit, one subagent on the project's rung or above applies and scores the drafts. One launch covers all of a chunk's drafts (goal 1.4). A security finding gets that subagent at once, since it is filed the moment it is found. Removing a finished item closes work and plans none, so any rung does it. Scoring a delegated task's L runs on any rung too: a launch on the project's rung to score each delegation would cost more than the cheaper task saves (goal 1.4).
|
||||
|
||||
Three places carry the rule. `rules/todo-items.md` states it and loads with `todo.md`. `CLAUDE.md`'s delegation rule launches the scoring subagent on the rung `bin/subagent-rung --plan` prints: the one the task's R + C picks, never below the project's, so a hard scoring task still climbs. `bin/loop_claude/cycle/prompts.py`'s `off_the_projects_rung` names the same command to a worker below the project's rung, never a fixed rung, which would disagree with `--plan` on a scoring task of R + C 11 or more. Valid while a worker or subagent may run below the project's rung.
|
||||
Three places carry the rule. `rules/todo-items.md` states it and loads with `todo.md`. `CLAUDE.md`'s delegation rule launches the scoring subagent on the rung `bin/subagent-rung --plan` prints: the one the task's L picks, never below the project's, so a hard scoring task still climbs. `bin/loop_claude/cycle/prompts.py`'s `review_launches` names the same command to a worker below the project's rung, never a fixed rung, which would disagree with `--plan` on a scoring task of L 4. Valid while a worker or subagent may run below the project's rung.
|
||||
|
||||
## settings.json: Claude Code runs in auto mode with the dangerous-mode prompt skipped
|
||||
|
||||
@@ -172,7 +172,7 @@ Valid while no published standard sets either number, the reading-cost window st
|
||||
|
||||
## rule-hook: a PreToolUse hook enforces each rule a program can check, a `# rule-override <rule>: <reason>` marker overrides one, and larv-review's overrides pass rules on each override before merge
|
||||
|
||||
2026-10-12, lilleman. Serves goals 1.2 and 1.3. Anthropic's best-practices page calls `CLAUDE.md` advisory and hooks deterministic, so a rule that must hold every time moves into `bin/rule-hook`, and `CLAUDE.md` keeps it only where the hook cannot check it. A Claude Code hook ties the check to one vendor ("Prefer the vendor-neutral option", `principles/technical-principles.md`), which is the target itself here: this repo configures Claude Code. What the tie buys: the hook blocks before a write lands, Edit and Write calls included, in every project with no per-repo install; a git `commit-msg` hook sees only commits, and CI sees them only after the push. The hook checks each write, so the agent fixes the line it just wrote; checking the staged diff at commit time would also cover files a Bash command writes, at the cost of later feedback. The marker's spelling was picked by a panel of three fresh readers, two to one. Each override is logged in the checkout's git directory, which is never pushed (goal 4). The plan had loop-claude's judge to rule on overrides inside a loop; lilleman approved larv-review's overrides pass in every run, because it rules before merge and an outrageous override still reaches the human through its ask. Valid while a PreToolUse hook that exits 2 blocks the call and hands its stderr to the model, and while an agent commits through Claude Code's Bash tool.
|
||||
2026-10-12, lilleman. Serves goals 1.2 and 1.3. Anthropic's best-practices page calls `CLAUDE.md` advisory and hooks deterministic, so a rule that must hold every time moves into `bin/rule-hook`, and `CLAUDE.md` keeps it only where the hook cannot check it. A Claude Code hook ties the check to one vendor ("Prefer the vendor-neutral option", `principles/technical-principles.md`), which is the target itself here: this repo configures Claude Code. What the tie buys: the hook blocks before a write lands, Edit and Write calls included, in every project with no per-repo install; a git `commit-msg` hook sees only commits, and CI sees them only after the push. The hook checks each write, so the agent fixes the line it just wrote; checking the staged diff at commit time would also cover files a Bash command writes, at the cost of later feedback. The marker's spelling was picked by a panel of three fresh readers, two to one. Each override is logged in the checkout's git directory, which is never pushed (goal 4). The plan had loop-claude's judge to rule on overrides inside a loop; lilleman approved larv-review's overrides pass in every run, because it rules before merge and an outrageous override still reaches the human through its ask. 2026-10-16, lilleman, answering plan item 166: `reviewer-rung` is the one rule no marker overrides; the comment above it in `bin/rule_hook/rules.py` says why. Valid while a PreToolUse hook that exits 2 blocks the call and hands its stderr to the model, and while an agent commits through Claude Code's Bash tool.
|
||||
|
||||
## rules/: `CLAUDE.md` holds the worker's rules that no file path can scope, and each topic's rules for the worker live in a `rules/` file that Claude Code loads by its `paths:` frontmatter
|
||||
|
||||
@@ -226,23 +226,25 @@ The run has no time cap. Limiting it to the changed lines already bounds it by t
|
||||
|
||||
Valid while a mutation run on a chunk's changed lines stays cheap beside the review itself.
|
||||
|
||||
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's R + C picks a rung by fixed bands, counted from the cheapest rung of the project's model
|
||||
## routing: one ladder of (model, effort) rungs, ordered by the Artificial Analysis Intelligence Index with Fable 5.1 at max on top; a task's L picks its worker's rung and its E + I its reviewers', by fixed steps counted from the cheapest rung of the project's model
|
||||
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude and by the worker through `bin/subagent-rung`. The judge's rung is set by `routing: on a model judge-replay covered`. `bin/model_routing/ladder.py` holds the rungs and the bands.
|
||||
2026-10-14, lilleman. Serves goal 1.4, and goal 1.3: one ladder, the same in every project, read by loop-claude and by the worker through `bin/subagent-rung`. The judge's rung is set by `routing: on a model judge-replay covered`. `bin/model_routing/ladder.py` holds the rungs and the steps.
|
||||
|
||||
The rungs, least competent first: Haiku 5.5 at low, medium, high, xhigh and max; Sonnet 5.5 at high; Opus 5.5 at medium, high, xhigh and max; Fable 5.1 at max. The order and the cut come from the Artificial Analysis Intelligence Index (https://artificialanalysis.ai/leaderboards/models, read 2026-10-14), which scores each model at each effort and gives a cost per task. A rung stays only where no cheaper rung scores as high, so a step up always buys more score (goal 1.4). That cut drops Sonnet 5.5 at low, medium, xhigh and max, Opus 5.5 at low, and every Fable 5.1 rung, and every tie in the index falls among the dropped rungs. lilleman put Fable 5.1 at max back on top: the index scores it under Opus 5.5 at high, but Anthropic's model catalog names Fable the most capable model, and for the hardest tasks lilleman wants the highest quality at any price.
|
||||
|
||||
A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. A model off the ladder routes nothing: claude runs it at the effort its settings give. R + C steps a task from the project's rung: 2–4 three rungs down, 5–6 two down, 7–8 one down, 9–10 none, 11–12 one up, 13–14 two up, 15–16 three up, 17 and above four up. A step past either end stops there. The bands spread the R + C that `todo.md`'s items held on 2026-10-14, 2 to 12, across the rungs, so most tasks step down. S never moves a rung, so a huge task with low R and C runs cheap. `score-outcomes` had too few finished chunks to ground the bands, and a change to the ladder or the bands is lilleman's.
|
||||
A project's rung is its model's cheapest rung, which for Opus 5.5 is its default effort, medium. A model off the ladder routes nothing: claude runs it at the effort its settings give. A task's L steps its worker from the project's rung: 1 two rungs down, 2 one down, 3 none, 4 one up. Its E + I steps its reviewers up from the project's rung: 4 or less none, 5 one up, 6 two up, 7 three up, 8 four up. A step past either end stops there. A worker errs more the more it changes and the newer that is, which L scores; review must look harder the further a defect would escape and the more it would break, which E + I scores (goals 1.1 and 1.4). Size raises L, so a large change runs its worker higher. The steps spread the values this repo's plan held when rescored on 2026-10-16, L 1 to 4 and E + I 2 to 7, so a worker on most items runs at or below the project's rung, and a review climbs only where a defect would escape far or break much. lilleman approved these steps on 2026-10-16 as a first guess: `score-outcomes` reports how well L, E and I each order finished chunks by review rounds and 🔴 findings, and `todo.md` item 170 re-tunes the steps once 10 chunks have shipped under them. A change to the ladder or the steps is lilleman's.
|
||||
|
||||
Valid while the index ranks these models, and while no newer model joins the catalog.
|
||||
|
||||
## routing: a chunk's worker runs on the rung its item picks, below the project's rung too; its reviewers run on the higher of the worker's rung and the project's, the seats and the security pass always on the project's
|
||||
## routing: a chunk's worker runs on the rung its item's L picks, below the project's rung too; its reviewers run on the highest of the worker's rung, the project's and the one its item's E + I picks, the seats and the security pass always on the project's
|
||||
|
||||
2026-10-14, lilleman. Serves goals 1.1 and 1.4.
|
||||
|
||||
Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks would spend on work that never ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each reviewer, each comprehension seat and the security pass's subagent on the project's rung, through the Agent tool's model and effort. The worker runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. lilleman chose the project's rung, medium, knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
|
||||
Only chunks run below the project's rung can prove that a cheaper worker holds quality (goal 1.1). So on 2026-10-14 lilleman chose a trial on real chunks: a replay of past chunks would spend on work that never ships (goal 1.4). Review holds quality meanwhile. While the worker runs below the project's rung, loop-claude's prompt has it launch each comprehension seat and the security pass's subagent on the project's rung, and each reviewer there too unless its item's E + I picks a higher rung, through the Agent tool's model and effort. The worker runs a full scan only from such a subagent, because the `claude-security` plugin's researchers and verifiers inherit the model that starts them. A subagent that cannot run the scan leaves it unavailable, so larv-review runs `/security-review` in its place. lilleman chose the project's rung, medium, knowing `settings.json` runs Opus 5.5 at xhigh, so a routed worker runs below the effort an interactive session takes.
|
||||
|
||||
While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers inherit the worker's higher rung. On 2026-10-15 lilleman dropped routing them by R + C instead: a worker climbs because its chunk proved harder than its score said, which is when a cheaper reviewer is the bigger risk. A reviewer or seat that rules the work below standard may rerun one rung up: `review: a reviewer or comprehension seat that rules the work below standard`.
|
||||
While the worker runs above the project's rung, loop-claude's prompt has it launch the comprehension seats and the security pass's subagent on the project's rung too. So the seats score every chunk on one rung, and the security pass spends no more than the project chose. The prompt after each unfinished verdict repeats this, since a step-up may have taken the worker above the project's rung. The other reviewers never run below the worker's rung: a worker climbs because its chunk proved harder than its score said, which is when a cheaper reviewer is the bigger risk. A reviewer or seat that rules the work below standard may rerun one rung up: `review: a reviewer or comprehension seat that rules the work below standard`.
|
||||
|
||||
On 2026-10-16 lilleman routed the reviewers on their item's E + I as well, on either side of the project's rung, which may lift them above both the worker's rung and the project's: how far a defect would escape, and what it would break, set how hard review must look (goal 1.1). loop-claude names that rung in the worker's prompt, and hands it to the worker's claude beside the project's and the worker's, so rule-hook holds each reviewer launch to it. A low E + I never steps a reviewer below the project's rung: lilleman chose to lower a pass only where a replay shows a lower rung finding every 🔴 and 🟡 the project's rung finds, which `todo.md` item 159 measures for each pass.
|
||||
|
||||
Valid until the trial on real chunks shows whether a worker below the project's rung holds quality.
|
||||
|
||||
@@ -250,7 +252,7 @@ Valid until the trial on real chunks shows whether a worker below the project's
|
||||
|
||||
2026-10-14, lilleman. Serves goal 1.4.
|
||||
|
||||
Because a model switch inside a session discards its prompt cache, the worker changes rung only as the chunk starts, after every second unfinished verdict, when the judge reports a taken-in item whose R + C picks a higher rung, or when the judge finds the escalation trigger a worker cites in asking for a higher rung (`routing: a worker whose run ends asking for a higher rung`). loop-claude routes on the first row of the next release in `todo.md`'s table that does not wait: an item waits while it is marked `question` or a blocker of it is still in the table. Since `todo.md` holds no order but its table's (`todo.md: one table`), that row is the item the worker takes. An item the worker takes in beyond that row lifts the worker's rung once the judge reports it, after the first cycle.
|
||||
Because a model switch inside a session discards its prompt cache, the worker changes rung only as the chunk starts, after every second unfinished verdict, when the judge reports a taken-in item whose L picks a higher rung, or when the judge finds the escalation trigger a worker cites in asking for a higher rung (`routing: a worker whose run ends asking for a higher rung`). loop-claude routes on the first row of the next release in `todo.md`'s table that does not wait: an item waits while it is marked `question` or a blocker of it is still in the table. Since `todo.md` holds no order but its table's (`todo.md: one table`), that row is the item the worker takes. An item the worker takes in beyond that row lifts the worker's rung once the judge reports it, after the first cycle.
|
||||
|
||||
Valid while a model switch inside a session discards its prompt cache.
|
||||
|
||||
@@ -260,13 +262,13 @@ Valid while a model switch inside a session discards its prompt cache.
|
||||
|
||||
The cost of a cheaper worker shows as extra review rounds and unfinished verdicts. Every 20 cycles across the archive, loop-claude measures them in code, with no model run (goal 1.4), and interrupts lilleman with the calls not raised before, when chunks routed onto a rung took more of either than chunks routed onto the next rung up, each rung with 5 chunks or more. Five keeps one odd chunk from calling for a change (goal 1.2). It also calls when the reports of 5 or more subagents of one type were longer than the tool results they read, so a delegated search cost the worker more context than it saved. `bin/routing_figures/` measures and words the calls. A loop on another project cannot commit here, so it writes the figures and the calls raised at the archive's root, and this repo's next chunk files the plan item, as `AGENTS.md` says. Off a terminal, a call is said on stderr and the loop goes on: it is about this setup, not the running project's chunk. A reset line in the calls file restarts the figures after a routing change, so they show whether the change worked.
|
||||
|
||||
Valid while loop-claude's archive holds every finished chunk's review rounds and unfinished verdicts.
|
||||
Since 2026-10-16 the reviewers may run above the worker's rung, on the one their item's E + I picks, so chunks whose worker ran on a lower rung may take more rounds from stronger reviewers, and a call could blame the worker's rung for them; `todo.md` item 171 groups the rounds by the reviewers' rung too. Valid while loop-claude's archive holds every finished chunk's review rounds and unfinished verdicts.
|
||||
|
||||
## routing: a worker delegates a task only where it needs just the task's conclusion, and each subagent runs on the rung its task's R + C picks from the project's rung, never below the project's rung when it files or scores todo.md items
|
||||
## routing: a worker delegates a task only where it needs just the task's conclusion, and each subagent runs on the rung its task's L picks from the project's rung, never below the project's rung when it files or scores todo.md items
|
||||
|
||||
2026-10-15, lilleman. Serves goals 1.1 and 1.4.
|
||||
|
||||
The worker delegates a task only when it needs just the task's conclusion: a search or read across many files, or work independent enough to run in parallel. Edits, and whatever it must check line by line, stay in its own thread, since it would read a delegated diff again to check it. Measured over archived worker transcripts, subagent reports cost a median 8% of the worker's peak context, and `sed -n` and `grep` output made up 41% of its Bash results, so delegated searches save the most context. It scores each delegated task's R and C as for an item, and `bin/subagent-rung` turns them into the Agent tool's model and effort, counted from the project's rung. The rule names both, because the Agent tool sets a subagent's effort only when instructions ask. `CLAUDE.md` states the rule, since it holds in every session. A subagent that files or scores `todo.md` items takes `bin/subagent-rung --plan`, which never prints a rung below the project's, the floor `todo.md: only a session or subagent on the project's rung or above scores or reshapes the plan` sets. A task that proves too hard for the worker's rung goes one rung up instead: `routing: a worker hands a task one rung up`.
|
||||
The worker delegates a task only when it needs just the task's conclusion: a search or read across many files, or work independent enough to run in parallel. Edits, and whatever it must check line by line, stay in its own thread, since it would read a delegated diff again to check it. Measured over archived worker transcripts, subagent reports cost a median 8% of the worker's peak context, and `sed -n` and `grep` output made up 41% of its Bash results, so delegated searches save the most context. It scores each delegated task's L as for an item, and `bin/subagent-rung` turns it into the Agent tool's model and effort, counted from the project's rung. The rule names both, because the Agent tool sets a subagent's effort only when instructions ask. `CLAUDE.md` states the rule, since it holds in every session. A subagent that files or scores `todo.md` items takes `bin/subagent-rung --plan`, which never prints a rung below the project's, the floor `todo.md: only a session or subagent on the project's rung or above scores or reshapes the plan` sets. A task that proves too hard for the worker's rung goes one rung up instead: `routing: a worker hands a task one rung up`.
|
||||
|
||||
Valid while delegated searches save more context than their reports cost.
|
||||
|
||||
@@ -292,24 +294,24 @@ The archive quotes the judge's passage under `Escalation:` in the unfinished ver
|
||||
|
||||
Valid while the worker reports in its transcript the triggers that fired.
|
||||
|
||||
## routing: on a model judge-replay covered, the judge runs on the rung its chunk's item picks, never below the project's rung
|
||||
## routing: on a model judge-replay covered, the judge runs on the rung its chunk's item's E + I picks for a review, never below the project's rung
|
||||
|
||||
2026-10-15, lilleman's plan, settled by the replay below. Serves goals 1.2 and 1.4, under the ladder and bands of `routing: one ladder`.
|
||||
2026-10-15, lilleman's plan, settled by the replay below. Serves goals 1.2 and 1.4, under the ladder and steps of `routing: one ladder`.
|
||||
|
||||
On a model listed in `JUDGE_REPLAYED` in `bin/loop_claude/routing.py`, the judge runs on the rung its chunk's item picks, never below the project's rung, and keeps that rung when the worker steps up. Without an item, it runs on the project's rung. Each archived verdict names the judge's rung on a `Judge:` line.
|
||||
On a model listed in `JUDGE_REPLAYED` in `bin/loop_claude/routing.py`, the judge runs on the rung its chunk's item's E + I picks for a review, never below the project's rung, and keeps that rung when the worker steps up. Without an item, it runs on the project's rung. Each archived verdict names the judge's rung on a `Judge:` line.
|
||||
|
||||
On 2026-10-15, `bin/judge-replay claude-opus-5-5` replayed the latest 15 archived verdicts of each kind from the Opus 5.5 projects, 39 in all. The reference judge, Opus 5.5 at the xhigh effort `settings.json` gave every judge before, reproduced 31 of them. The other 8 may rest on records or judge rules that have changed since; either way no rung is held to them. On the 31, Haiku 5.5 missed 7 at xhigh and 8 at max. That run tried each judge once. 4 of Haiku's misses, 1 at xhigh and 3 at max, misquoted the transcript, which the loop and `judge-replay` now retry; without them it missed 6 and 5. Sonnet 5.5 at high returned another kind on 3 interrupts, and Opus 5.5 at medium missed none. A missed interrupt lets the worker settle a decision that is lilleman's (goal 1.2), so the judge never runs below the project's rung.
|
||||
|
||||
Only Opus 5.5 was replayed, so the judge of a project on another model keeps the effort its settings give until `judge-replay` covers that model (goal 1.1). Every routed chunk on record had an item of R + C 9, so the replay could not show that the project's rung suffices for harder items, and the judge climbs with the routed item as the worker does. For R + C 11 to 14 it still runs at or below xhigh (goal 1.4); from 15, `routing: one ladder` records that lilleman wants the highest quality at any price. An item the worker takes in beyond the routed one lifts the worker's rung but not the judge's, until `todo.md` item 153 lifts the judge too.
|
||||
Only Opus 5.5 was replayed, so the judge of a project on another model keeps the effort its settings give until `judge-replay` covers that model (goal 1.1). Every routed chunk on record had an item of R + C 9, so the replay could not show that the project's rung suffices for harder items, and the judge climbs with the routed item. Since 2026-10-16, lilleman's call, it climbs with the item's E + I, as a review does: the judge checks the worker's decisions as the reviewers check its code, and a decision the judge lets through reaches lilleman, as a defect that review lets through reaches the users (goal 1.2). It runs above the project's rung only on an E + I of 5 or more (goal 1.4). An item the worker takes in beyond the routed one lifts the worker's rung but not the judge's, until `todo.md` item 153 lifts the judge too.
|
||||
|
||||
Valid while the archived verdicts show what a judge should return, and until a new model joins the ladder: replay then.
|
||||
|
||||
## routing: the stability reviewer stays on the project's rung while the worker runs below it, until `stability-replay` shows a lower rung finding every 🔴 and 🟡 the project's rung finds
|
||||
## routing: the stability reviewer runs no lower than the project's rung while the worker runs below it, until `stability-replay` shows a lower rung finding every 🔴 and 🟡 the project's rung finds
|
||||
|
||||
2026-10-15, lilleman proposed running the stability reviewer on the worker's rung; the replay that gates it is the agent's proposal. Serves goals 1.1 and 1.4: the stability pass is the only one that mutation-tests the changed lines, so a defect it misses may merge, and a cheaper reviewer saves money only where it misses nothing.
|
||||
|
||||
`bin/stability-replay <model>` replays archived stability rounds on the code each reviewed, first on the project's rung, then on each rung below it, highest first, and stops at the first that misses a finding the project's rung found again. A rung qualifies only where it finds every such 🔴 and 🟡.
|
||||
|
||||
On 2026-10-15, `bin/stability-replay claude-opus-5-5` replayed the latest 10 rounds holding a 🔴 or 🟡, 40 findings in all, 8 of the rounds from this repo. Opus 5.5 at medium, the project's rung, found 29 of them again. Sonnet 5.5 at high missed 13 of those 29, 2 of them 🔴, so no rung below the project's rung qualifies, and Haiku 5.5, below Sonnet on the ladder, was not tried. So the stability reviewer runs where `routing: a chunk's worker runs on the rung its item picks` puts every reviewer: on the project's rung while the worker runs below it.
|
||||
On 2026-10-15, `bin/stability-replay claude-opus-5-5` replayed the latest 10 rounds holding a 🔴 or 🟡, 40 findings in all, 8 of the rounds from this repo. Opus 5.5 at medium, the project's rung, found 29 of them again. Sonnet 5.5 at high missed 13 of those 29, 2 of them 🔴, so no rung below the project's rung qualifies, and Haiku 5.5, below Sonnet on the ladder, was not tried. So the stability reviewer runs where `routing: a chunk's worker runs on the rung its item's L picks` puts every reviewer: never below the project's rung while the worker runs below it.
|
||||
|
||||
Valid until a new model joins the ladder, or the archive holds rounds a new reviewer prompt produced: replay then.
|
||||
|
||||
@@ -503,3 +503,20 @@ Read most (opens and searches):
|
||||
- `.gitignore` 1
|
||||
- `CLAUDE.md` 1
|
||||
- `principles/technical-principles.md` 1
|
||||
|
||||
## Reading cost, chunk 9e8e845a4e6d, finished 2026-10-16T05:20:56Z
|
||||
|
||||
319478 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `todo.md` 7
|
||||
- `bin/loop_claude/test_loop_claude.py` 6
|
||||
- `bin/rule_hook/test_rule_hook.py` 5
|
||||
- `bin/loop_claude/cycle/run.py` 4
|
||||
- `bin/loop_claude/effects.py` 4
|
||||
- `CHANGELOG.md` 3
|
||||
- `README.md` 3
|
||||
- `bin/rule_hook/rules.py` 3
|
||||
- `AGENTS.md` 2
|
||||
- `bin/loop_claude/cycle/worker.py` 2
|
||||
|
||||
+14
-7
@@ -7,15 +7,22 @@ paths:
|
||||
|
||||
- **`todo.md` is mandatory in every project, in the shape `rank_todo.py --help` shows**, which `~/.claude/bin/rank_todo.py` checks and ranks; its `--help` shows a valid file. `## Scoring` holds the formula and the bar; `## Items` is one table of every item; `## Details` holds each item's detail under its ID. Cite an item by its ID. The formula and bar are the project's own to tune.
|
||||
- **Order items only by score and the Blocked by column.** An item's Blocked by cell lists the IDs of the items that must be done before it starts, such as `12, 14`; `rank_todo.py --help` says how it ranks them. No text in `todo.md` outside the table orders items: where an item must go first, score it so, and say why in its detail.
|
||||
- **Plan only on the project's rung or above.** loop-claude's prompt tells you when you run below it. Below it, never score or re-score an item, move it between releases, split a milestone into items, delete an item under the bar, or rewrite a filed item's tagline or detail. Draft each new item, and each change to a filed item, instead. Before your next commit, launch one subagent to apply the drafts, score them and set their release. Launch it on the rung `CLAUDE.md`'s delegation rule gives a subagent that files or scores items. Keep the scores and releases it writes. For a security finding, launch it at once. Any rung may remove a finished item's row and detail, and score a delegated task's R + C. A subagent launched for another task leaves `todo.md` alone.
|
||||
- **Plan only on the project's rung or above.** loop-claude's prompt tells you when you run below it. Below it, never score or re-score an item, move it between releases, split a milestone into items, delete an item under the bar, or rewrite a filed item's tagline or detail. Draft each new item, and each change to a filed item, instead. Before your next commit, launch one subagent to apply the drafts, score them and set their release. Launch it on the rung `CLAUDE.md`'s delegation rule gives a subagent that files or scores items. Keep the scores and releases it writes. For a security finding, launch it at once. Any rung may remove a finished item's row and detail, and score a delegated task's L. A subagent launched for another task leaves `todo.md` alone.
|
||||
- **An item's release allows the bump that semver read strictly asks for.** An item that changes what exists needs a major, or before 1.0.0 a minor. One that only adds needs a minor, and a fix a patch.
|
||||
- **Score an item when it is filed or edited.** Each of R, S and C is a whole number from 1 to 10:
|
||||
- **R** — the risk that the change creates serious bugs or issues.
|
||||
- **S** — the size of the change.
|
||||
- **C** — how hard the change is to get right: novel design, subtle invariants, many interacting parts. A huge rename is low C; a ten-line lock fix can be high. C stays out of the score.
|
||||
- **Score an item when it is filed or edited.** S, the size of the change, is a whole number from 1 to 10. Score L, E and I by the levels below, each level about ten times as likely, or as damaging, as the one below it. `rank_todo.py` writes R, the risk that the change creates serious bugs or issues: L + E + I − 2, at least 7 when I is 5 and at least 5 when I is 4, since a severe enough consequence outranks how unlikely it is. A task's L picks its worker's rung, and its E + I its reviewers' rung.
|
||||
|
||||
| Level | L — chance the change holds a defect | E — chance the defect escapes | I — damage, worst of availability, data, secrets, reach and reversibility |
|
||||
|---|---|---|---|
|
||||
| 1 | S 1–2, and nothing new | a test or the gate fails on it | one person's tooling or something cosmetic; a revert fixes it in minutes |
|
||||
| 2 | S 3–5, in one component, a pattern the repo already has | only review or a manual check sees it | one feature or service degraded; a revert within the hour |
|
||||
| 3 | one of: S 6 or more, several components, a new tool or pattern, live config or state | nothing checks it before users hit it | a service down, or every consumer affected; hours to restore |
|
||||
| 4 | two or more of level 3's, or a rewrite of code with known defects | — | everything down, a backup restore, or a secret or PII exposed |
|
||||
| 5 | — | — | irreversible: data lost for good, or a one-way door |
|
||||
|
||||
- **I's reach counts every project and session that runs the change.** A change to a hook or a tool that every session runs is I level 3, every consumer affected, even where one person uses it.
|
||||
- **Name in Moves each achieve goal in `README.md` → `## Goals` the item moves, and how far,** from 1 to 10, judged by its value to the audience in `## Audience`: `1.1 by 8, 1.2 by 5`. Name the most specific goal it moves, and a parent goal only for a move none of its sub-goals covers, never beside its own sub-goal. Leave out every maintain goal, which vetoes, except on an item marked `defect` that restores it: name that goal, and how much of it is broken. `rank_todo.py --help` says how it weighs each goal.
|
||||
- **A table with A, G and Goals in place of Moves is in the old shape.** Reshaping it is the project's only work until done, as `CLAUDE.md` → "Working loop" says, and `rank_todo.py` says how. Until then, A is the item's value to the audience and G its value to the goals it cites.
|
||||
- **Check R, S and C against finished chunks.** Guess them, then run `~/.claude/bin/score-outcomes <R> <S> <C>` in the project. It lists the finished items guessed nearest, what their chunks measured, and the median outcome at each guessed value. Set each score to the value guessed for the finished items whose outcome matches what you expect of this one. Keep your guess when no listed item resembles the one you are scoring.
|
||||
- **A table with C, A, G and Goals in place of L, E, I and Moves is in the old shape.** Reshaping it is the project's only work until done, as `CLAUDE.md` → "Working loop" says, and `rank_todo.py` says how. Until then, A is the item's value to the audience and G its value to the goals it cites.
|
||||
- **Check L, E, I and S against finished chunks.** Guess them, then run `~/.claude/bin/score-outcomes <L> <E> <I> <S>` in the project. It lists the finished items guessed nearest, what their chunks measured, the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings. Set each score to the value guessed for the finished items whose outcome matches what you expect of this one. Keep your guess when no listed item resembles the one you are scoring.
|
||||
- **Read the places the comprehension seats found hard before filing an item.** `~/.claude/bin/larv-review places`, run in the project, prints them. Where the item's change touches one, its detail says to make that place easier to read.
|
||||
- **An item scoring under the bar is deleted, with every item that needs it, and a finding under the bar is not filed.** The chunk's summary names each, with its score.
|
||||
- **The Exempt cell keeps an item whatever its score:** `security` for a security finding, `defect` for a 🔴 finding or an item that restores a maintain goal the project breaks today, `question` for a question only the human can answer, which ranks first in its release, `decision` for an item `docs/decisions.md` cites, `owner` for an item the human told you to keep, and, where no other reason fits, `principle` for a 🟡 finding against a file under `~/.claude/principles/`, whose item's detail names that principle. Every item `docs/decisions.md` cites carries one of these reasons.
|
||||
|
||||
@@ -6,108 +6,85 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 169`
|
||||
`Next ID = 174`
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | R | S | C | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| 166 | 0.2.0 | | question | **Decide whether `docs/decisions.md` names rule-hook's refusal of a review launched below its rung, in `routing:` and in `rule-hook:`.** | 1 | 1 | 1 | 1.2 by 3 | 4.4 |
|
||||
| 146 | 0.2.0 | | decision | **Score R as L + E + I − 2 from anchored levels, with R at least 7 when I is 5 and at least 5 when I is 4; give L, E and I their own columns and have `rank_todo.py` compute R; drop C; and route a task's worker on its L and its reviewers on its E + I.** | 6 | 7 | 7 | 1.1 by 8, 1.2 by 8, 1.4 by 8 | 2.8 |
|
||||
| 152 | 0.2.0 | 146 | decision | **Anchor each score level in `rules/todo-items.md` with words, and each project's with two or three items it shipped, and have a scorer compare a new item against them.** | 3 | 4 | 4 | 1.1 by 7 | 6.4 |
|
||||
| 50 | 0.2.0 | | principle | **Grant the bench's runs only the tools a review needs, in place of skipping every permission.** | 2 | 2 | 3 | 1.1 by 3 | 5.5 |
|
||||
| 65 | 0.2.0 | | defect | **Archive each judge reply the loop rejects.** | 3 | 4 | 3 | 3 by 3 | 5.5 |
|
||||
| 153 | 0.2.0 | | decision | **Run the judge on the rung the highest R + C in its chunk picks, an item the worker takes in included.** | 3 | 4 | 5 | 1.2 by 5 | 3.0 |
|
||||
| 161 | 0.2.0 | | | **Have loop-claude run a preflight script before each fresh worker and put its findings in the worker's prompt, so the worker no longer spends tool calls finding the chunk's mechanical facts.** | 4 | 6 | 5 | 1.1 by 4, 1.4 by 6, 1.5 by 6 | 2.9 |
|
||||
| 84 | 0.2.0 | | | **Check, at each release's scoring run, that seats in a paired ruling agree more often than two scoring runs on near-identical code do, as the paired-ruling decision assumes.** | 2 | 4 | 5 | 1.4 by 5 | 2.3 |
|
||||
| 168 | 0.2.0 | | | **Have the human answer an interrupt on loop-claude's terminal and resume the worker headless with the answer, opening the interactive session only when they ask for it.** | 4 | 5 | 4 | 1.5 by 8 | 1.8 |
|
||||
| 159 | 0.2.0 | | | **Run the prose pass on the lowest rung that a replay of archived prose rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.** | 3 | 5 | 5 | 1.4 by 5 | 1.4 |
|
||||
| 73 | 0.2.0 | | | **Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.** | 4 | 5 | 5 | 1.4 by 4 | 1.1 |
|
||||
| 80 | 0.3.0 | | defect | **State in README → Install that every machine taking this checkout must be a sandbox, as `settings.json`'s auto mode assumes.** | 1 | 1 | 1 | 2 by 4 | 17.5 |
|
||||
| 21 | 0.3.0 | | defect | **Give `settings.json`'s model one writer, and sort its keys.** | 3 | 3 | 4 | 2 by 5 | 13.7 |
|
||||
| 29 | 0.3.0 | | | **Make the contributor a standing member of every project's audience, unless its `## Audience` says maintainability does not matter.** | 3 | 2 | 3 | 1.1 by 5, 1.4 by 5 | 13.7 |
|
||||
| 119 | 0.3.0 | | | **Retitle the comprehension-floor decision so it says a chunk lifting a project towards the floor works first on its lowest-scoring dimension, and that the floor blocks nothing.** | 1 | 1 | 1 | 1.1 by 4, 1.4 by 4 | 13.1 |
|
||||
| 127 | 0.3.0 | | | **Define seat, place, paired ruling, scoring run and the judge's records in `CLAUDE.md` → "Vocabulary".** | 1 | 1 | 2 | 1.1 by 4, 1.4 by 4 | 13.1 |
|
||||
| 129 | 0.3.0 | | | **Explain `bin/rule_hook/rules.py`'s `OVERRIDE` pattern beside it: why the reason is lazy, what the lookahead splits and why `-->` is stripped.** | 1 | 1 | 1 | 1.1 by 4, 1.4 by 4 | 13.1 |
|
||||
| 114 | 0.3.0 | | decision | **Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost`, `stability_replay` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.** | 4 | 4 | 5 | 1.1 by 9, 1.4 by 9 | 12.3 |
|
||||
| 120 | 0.3.0 | | | **Rewrite larv-review §3's "Round cap" and "Re-entry" as one table that says, for each kind of fix, which passes rerun and what a round reviews.** | 3 | 4 | 5 | 1.1 by 6, 1.2 by 6, 1.4 by 6 | 11.9 |
|
||||
| 49 | 0.3.0 | | principle | **Check that `AGENTS.md` lists exactly the titles of `docs/decisions.md`.** | 1 | 2 | 2 | 1.1 by 6 | 10.9 |
|
||||
| 117 | 0.3.0 | | | **Give `todo_table.items` one way to say a plan has no items table, so no caller tells it from an empty plan by truthiness.** | 2 | 2 | 3 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 126 | 0.3.0 | | | **State when work has landed towards a release once, in `bin/larv_review/plan.py`, and have larv-review's help point to it.** | 1 | 2 | 2 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 131 | 0.3.0 | | | **Describe the transcript archive's layout in `bin/transcript_archive/`, which owns it, and have loop-claude's help point there.** | 1 | 2 | 1 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 133 | 0.3.0 | | | **Show, beside `bin/rule_hook/override_log.py`'s `this_chunks`, one example per case it decides: an amended or rebased commit, a renamed branch, and a commit on the default branch.** | 1 | 2 | 3 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 134 | 0.3.0 | | | **Name in `bin/shell_command.py`'s `segments` the shell forms it does not parse, so a caller knows what it misses.** | 1 | 2 | 3 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 154 | 0.3.0 | | defect | **Have `bin/shell_command.py`'s `segments` keep a command's words together across a redirect, so `git > out commit` reads as one git commit.** | 3 | 2 | 3 | 1.1 by 6 | 10.9 |
|
||||
| 157 | 0.3.0 | | | **Split `docs/decisions.md` → `todo.md: one table of items …`'s 2026-10-02 paragraph into one paragraph per choice.** | 1 | 2 | 1 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 106 | 0.3.0 | | | **Make larv-review's "Do not over-read a clean result" cite a rule that says Renovate or Dependabot owns dependency CVEs, or drop the citation.** | 1 | 1 | 1 | 1.1 by 5 | 10.9 |
|
||||
| 167 | 0.3.0 | | | **Refuse a `/security-review` or `/claude-security` Skill call below the project's rung, as rule-hook refuses such an Agent call.** | 2 | 2 | 3 | 1.1 by 4 | 7.3 |
|
||||
| 82 | 0.3.0 | | principle | **Sort `AGENTS.md`'s decision titles and `docs/decisions.md`'s entries by title.** | 1 | 4 | 2 | 1.1 by 4, 1.4 by 4 | 5.5 |
|
||||
| 162 | 0.3.0 | | owner | **Script each step that 0.2.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 3 | 3 | 1.4 by 4, 1.5 by 4 | 4.9 |
|
||||
| 165 | 0.3.0 | | | **Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant.** | 2 | 5 | 4 | 1.1 by 4, 1.4 by 3 | 3.0 |
|
||||
| 122 | 0.4.0 | | | **Assemble the judge's prompt in `bin/loop_claude/cycle/judge.py`, importing only wording the worker also sees, and call the worker "the worker" in it.** | 3 | 3 | 4 | 1.1 by 5, 1.4 by 5 | 10.3 |
|
||||
| 158 | 0.4.0 | | decision | **Measure the fixed cost of a chunk from loop-claude's archive, in tokens and hours against its items' S, and set F in `Score = 100 * V / (2**S + F)` to it.** | 2 | 3 | 4 | 1.1 by 5, 1.4 by 5 | 10.3 |
|
||||
| 116 | 0.4.0 | | | **Report L by the share of chunks with a 🔴, once 10 chunks that each shipped one item record finding counts and most record no 🔴.** | 1 | 2 | 2 | 1.1 by 5 | 9.1 |
|
||||
| 121 | 0.4.0 | | | **Check that `bin/round_titles.py`'s `PASSES` lists exactly the passes of larv-review §3's table.** | 1 | 2 | 2 | 1.1 by 5 | 9.1 |
|
||||
| 112 | 0.4.0 | | | **Count an item as shipped only where its chunk built it, so an item a chunk dropped from the plan gets no outcome.** | 2 | 3 | 5 | 1.1 by 6 | 8.2 |
|
||||
| 130 | 0.4.0 | | | **Move the vote counting out of `bin/larv_review/ruling.py`'s `votes`, so reading a votes file decides nothing.** | 2 | 3 | 3 | 1.1 by 4, 1.4 by 4 | 8.2 |
|
||||
| 125 | 0.4.0 | | decision | **Give "record", "outcome" and "places" one meaning each across `bin/`, and rename `bin/larv_review/git.py` by what it holds.** | 2 | 5 | 3 | 1.1 by 9, 1.4 by 9 | 7.4 |
|
||||
| 81 | 0.4.0 | | principle | **Have `workflow-bench/bench.py` return each run's failure as a value, so one failed clone does not hide the later runs' results.** | 2 | 2 | 3 | 1.1 by 4 | 7.3 |
|
||||
| 102 | 0.4.0 | | | **Give loop-claude's retry delay an environment override, so the black-box test stops replacing `time.sleep` through `sitecustomize`.** | 2 | 2 | 2 | 1.1 by 4 | 7.3 |
|
||||
| 66 | 0.4.0 | | principle | **Hold systems-architect's method thresholds only while the comprehension panel finds the flagged code hard.** | 4 | 3 | 5 | 1.1 by 5 | 6.8 |
|
||||
| 101 | 0.4.0 | | principle | **Have `larv_review/record.py`'s `read_seats` take the time, the commit check and the history as inputs, so it parses and decides without effects.** | 2 | 3 | 3 | 1.1 by 5 | 6.8 |
|
||||
| 111 | 0.4.0 | | | **Measure the chunks of a project that merges by squash or fast-forward, which `score-outcomes` skips today.** | 3 | 3 | 6 | 1.1 by 5 | 6.8 |
|
||||
| 113 | 0.4.0 | | | **Compute an item's S from the code it names, where it names any, and show it beside the finished items `score-outcomes` lists.** | 2 | 4 | 6 | 1.1 by 5, 1.4 by 5 | 6.8 |
|
||||
| 137 | 0.4.0 | | | **Archive a chunk's route as a record loop-claude reads back field by field, in place of the `Worker:` prose `ROUTED` parses, and name `stepped` and the judge's `bundled` by what they do.** | 3 | 4 | 4 | 1.1 by 4, 1.4 by 4 | 5.5 |
|
||||
| 110 | 0.4.0 | | | **Count, for each finished chunk, the later changes that fixed code it wrote, and show them beside its guessed E.** | 2 | 5 | 6 | 1.1 by 6, 1.4 by 6 | 4.9 |
|
||||
| 139 | 0.4.0 | | | **Split `bin/routing_figures/measure.py` by concern, and keep the figures' measured cycle and each raised call as fields loop-claude reads back, not as the prose it renders.** | 3 | 5 | 4 | 1.1 by 5, 1.4 by 5 | 4.1 |
|
||||
| 149 | 0.4.0 | | decision | **Fill each release by priority up to a capacity it states, in place of the bar, and move what does not fit to a later release.** | 4 | 5 | 5 | 1.1 by 7 | 3.8 |
|
||||
| 12 | 0.5.0 | | | **Run the test suite in CI on every PR, or record in `docs/decisions.md` why the local run is enough.** | 3 | 4 | 4 | 1.1 by 6 | 5.5 |
|
||||
| 105 | 0.5.0 | | decision | **Compare review rounds per pass and reading cost for the five chunks before and after the `worker:` decision moved the principles out of the worker's context, and file an item if either rose.** | 1 | 2 | 3 | 1.4 by 6 | 5.5 |
|
||||
| 135 | 0.5.0 | | | **Fingerprint only the modules loop-claude imports, so editing another tool's code no longer reloads a running loop.** | 3 | 3 | 4 | 1.1 by 4 | 5.5 |
|
||||
| 68 | 0.5.0 | | principle | **Test `workflow-bench/compare.py`'s blinding and `bench.py`'s run plan.** | 1 | 4 | 4 | 1.1 by 5 | 4.6 |
|
||||
| 4 | 0.5.0 | | | **Name each chunk's archive folder by the chunk's start time, so `<key>/` lists chunks in the order they ran.** | 3 | 3 | 4 | 1.1 by 3 | 4.1 |
|
||||
| 59 | 0.5.0 | | | **Bound a run of unfinished verdicts on one chunk, so a worker and judge that disagree cannot cycle unattended without end.** | 4 | 3 | 5 | 1.4 by 6 | 4.1 |
|
||||
| 123 | 0.5.0 | | | **Have `bin/comprehension_history.py` parse each history entry once into seats, votes and places, so `latest_places`, `worse_places` and `worse_votes` neither depend on table order nor split a label to find a seat.** | 3 | 5 | 5 | 1.1 by 5, 1.4 by 5 | 4.1 |
|
||||
| 132 | 0.5.0 | | | **Lift each dimension still under the comprehension floor after release 0.2.0's scoring run, at the places that run names.** | 3 | 5 | 5 | 1.1 by 5, 1.4 by 5 | 4.1 |
|
||||
| 124 | 0.5.0 | | decision | **Group the shared modules at `bin/`'s root into packages named by purpose, so the root holds only launchers and stand-alone tools.** | 4 | 6 | 3 | 1.1 by 9, 1.4 by 9 | 4.1 |
|
||||
| 156 | 0.5.0 | | decision | **Drop `rank_todo.py`'s ranking of an old-shape table once no project under `~/code` holds one.** | 3 | 4 | 3 | 1.1 by 4 | 3.6 |
|
||||
| 155 | 0.5.0 | | | **Lower the reading cost of the files a chunk reads most before its first commit.** | 3 | 5 | 5 | 1.4 by 6 | 1.6 |
|
||||
| 1 | 0.6.0 | | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 2 | 1 | 2 | | 0.0 |
|
||||
| 6 | 0.6.0 | | decision | **Move what `loop-claude` writes out of `~/.claude.json`, and ask the human about each write that must stay.** | 5 | 3 | 6 | 1.1 by 5 | 6.8 |
|
||||
| 5 | 0.6.0 | | decision | **Key a project on its checkout's full path, and refuse to start while another session uses that key.** | 5 | 5 | 7 | 1.1 by 6 | 3.3 |
|
||||
| 77 | 0.6.0 | | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 3 | 6 | 6 | 1.1 by 7, 1.4 by 7 | 3.2 |
|
||||
| 51 | 0.6.0 | | principle | **Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints.** | 3 | 5 | 4 | 1.1 by 5 | 2.7 |
|
||||
| 151 | 0.6.0 | | decision | **Keep 10 to 20 settled decisions per project as its goals' test set, and rerun them after a goal changes.** | 3 | 5 | 6 | 1.2 by 7 | 2.6 |
|
||||
| 52 | 0.6.0 | | principle | **Record why this checkout lives in Claude Code's own directory, or move it out and link it in.** | 6 | 6 | 6 | 1.1 by 4 | 1.2 |
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 152 | 0.2.0 | | decision | **Anchor each score level in `rules/todo-items.md` with words, and each project's with two or three items it shipped, and have a scorer compare a new item against them.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 7 | 6.4 |
|
||||
| 50 | 0.2.0 | | principle | **Grant the bench's runs only the tools a review needs, in place of skipping every permission.** | 2 | 2 | 1 | 3 | 2 | 1.1 by 3 | 5.5 |
|
||||
| 65 | 0.2.0 | | defect | **Archive each judge reply the loop rejects.** | 2 | 1 | 3 | 4 | 4 | 3 by 3 | 5.5 |
|
||||
| 153 | 0.2.0 | | decision | **Route the judge and the reviewers on the highest E + I in their chunk, an item the worker takes in included.** | 3 | 1 | 3 | 5 | 4 | 1.2 by 5 | 3.0 |
|
||||
| 161 | 0.2.0 | | | **Have loop-claude run a preflight script before each fresh worker and put its findings in the worker's prompt, so the worker no longer spends tool calls finding the chunk's mechanical facts.** | 4 | 2 | 3 | 7 | 6 | 1.1 by 4, 1.4 by 6, 1.5 by 6 | 2.9 |
|
||||
| 84 | 0.2.0 | | | **Check, at each release's scoring run, that seats in a paired ruling agree more often than two scoring runs on near-identical code do, as the paired-ruling decision assumes.** | 2 | 2 | 2 | 4 | 4 | 1.4 by 5 | 2.3 |
|
||||
| 168 | 0.2.0 | | | **Have the human answer an interrupt on loop-claude's terminal and resume the worker headless with the answer, opening the interactive session only when they ask for it.** | 4 | 2 | 3 | 7 | 5 | 1.5 by 8 | 1.8 |
|
||||
| 73 | 0.2.0 | | | **Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.** | 3 | 2 | 3 | 6 | 5 | 1.4 by 4 | 1.1 |
|
||||
| 159 | 0.2.0 | | decision | **Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.** | 4 | 2 | 3 | 7 | 6 | 1.4 by 7 | 1.1 |
|
||||
| 80 | 0.3.0 | | defect | **State in README → Install that every machine taking this checkout must be a sandbox, as `settings.json`'s auto mode assumes.** | 1 | 2 | 1 | 2 | 1 | 2 by 4 | 17.5 |
|
||||
| 21 | 0.3.0 | | defect | **Give `settings.json`'s model one writer, and sort its keys.** | 3 | 3 | 3 | 7 | 3 | 2 by 5 | 13.7 |
|
||||
| 29 | 0.3.0 | | | **Make the contributor a standing member of every project's audience, unless its `## Audience` says maintainability does not matter.** | 2 | 2 | 3 | 5 | 2 | 1.1 by 5, 1.4 by 5 | 13.7 |
|
||||
| 119 | 0.3.0 | | | **Retitle the comprehension-floor decision so it says a chunk lifting a project towards the floor works first on its lowest-scoring dimension, and that the floor blocks nothing.** | 1 | 2 | 1 | 2 | 1 | 1.1 by 4, 1.4 by 4 | 13.1 |
|
||||
| 127 | 0.3.0 | | | **Define seat, place, paired ruling, scoring run and the judge's records in `CLAUDE.md` → "Vocabulary".** | 1 | 2 | 3 | 4 | 1 | 1.1 by 4, 1.4 by 4 | 13.1 |
|
||||
| 129 | 0.3.0 | | | **Explain `bin/rule_hook/rules.py`'s `OVERRIDE` pattern beside it: why the reason is lazy, what the lookahead splits and why `-->` is stripped.** | 1 | 2 | 1 | 2 | 1 | 1.1 by 4, 1.4 by 4 | 13.1 |
|
||||
| 114 | 0.3.0 | | decision | **Run git through `git_repo.py` in `rule_hook`, `metric_bench`, `reading_cost`, `stability_replay` and `loop_claude`, keeping rule-hook's guards as an option, so one module owns how a tool runs git.** | 3 | 1 | 3 | 5 | 4 | 1.1 by 9, 1.4 by 9 | 12.3 |
|
||||
| 120 | 0.3.0 | | | **Rewrite larv-review §3's "Round cap" and "Re-entry" as one table that says, for each kind of fix, which passes rerun and what a round reviews.** | 2 | 2 | 3 | 5 | 4 | 1.1 by 6, 1.2 by 6, 1.4 by 6 | 11.9 |
|
||||
| 49 | 0.3.0 | | principle | **Check that `AGENTS.md` lists exactly the titles of `docs/decisions.md`.** | 1 | 1 | 1 | 1 | 2 | 1.1 by 6 | 10.9 |
|
||||
| 117 | 0.3.0 | | | **Give `todo_table.items` one way to say a plan has no items table, so no caller tells it from an empty plan by truthiness.** | 1 | 1 | 3 | 3 | 2 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 126 | 0.3.0 | | | **State when work has landed towards a release once, in `bin/larv_review/plan.py`, and have larv-review's help point to it.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 131 | 0.3.0 | | | **Describe the transcript archive's layout in `bin/transcript_archive/`, which owns it, and have loop-claude's help point there.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 133 | 0.3.0 | | | **Show, beside `bin/rule_hook/override_log.py`'s `this_chunks`, one example per case it decides: an amended or rebased commit, a renamed branch, and a commit on the default branch.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 134 | 0.3.0 | | | **Name in `bin/shell_command.py`'s `segments` the shell forms it does not parse, so a caller knows what it misses.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 154 | 0.3.0 | | defect | **Have `bin/shell_command.py`'s `segments` keep a command's words together across a redirect, so `git > out commit` reads as one git commit.** | 1 | 1 | 3 | 3 | 2 | 1.1 by 6 | 10.9 |
|
||||
| 157 | 0.3.0 | | | **Split `docs/decisions.md` → `todo.md: one table of items …`'s 2026-10-02 paragraph into one paragraph per choice.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 4, 1.4 by 4 | 10.9 |
|
||||
| 106 | 0.3.0 | | | **Make larv-review's "Do not over-read a clean result" cite a rule that says Renovate or Dependabot owns dependency CVEs, or drop the citation.** | 1 | 2 | 1 | 2 | 1 | 1.1 by 5 | 10.9 |
|
||||
| 167 | 0.3.0 | | | **Refuse a `/security-review` or `/claude-security` Skill call below the project's rung, as rule-hook refuses such an Agent call.** | 3 | 2 | 3 | 6 | 2 | 1.1 by 4 | 7.3 |
|
||||
| 172 | 0.3.0 | | | **Name the project's rung `project` across `bin/loop_claude`, in place of `anchor`, as `bin/model_routing/ladder.py` names it.** | 2 | 1 | 3 | 4 | 3 | 1.1 by 3, 1.4 by 3 | 6.2 |
|
||||
| 82 | 0.3.0 | | principle | **Sort `AGENTS.md`'s decision titles and `docs/decisions.md`'s entries by title.** | 2 | 2 | 1 | 3 | 4 | 1.1 by 4, 1.4 by 4 | 5.5 |
|
||||
| 173 | 0.3.0 | | | **Have rule-hook's `todo-rank-check` run a worktree's own `bin/rank_todo.py` when the worktree is of this repo, so a branch that changes `todo.md`'s shape commits without an override.** | 2 | 1 | 3 | 4 | 2 | 1.1 by 2, 1.4 by 2 | 5.5 |
|
||||
| 162 | 0.3.0 | | owner | **Script each step that 0.2.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 2 | 2 | 1 | 3 | 3 | 1.4 by 4, 1.5 by 4 | 4.9 |
|
||||
| 171 | 0.3.0 | | decision | **Group the routing figures' review rounds by the reviewers' rung as well as the worker's, so a call blames a rung only for rounds its own reviews raised.** | 3 | 2 | 2 | 5 | 4 | 1.2 by 3, 1.4 by 3 | 3.2 |
|
||||
| 165 | 0.3.0 | | | **Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant.** | 3 | 2 | 1 | 4 | 5 | 1.1 by 4, 1.4 by 3 | 3.0 |
|
||||
| 169 | 0.3.0 | | | **Have rule-hook's `todo-rank-check` catch a `todo.md` that the same Bash call edits and then commits.** | 3 | 1 | 3 | 5 | 3 | 1.1 by 2 | 2.7 |
|
||||
| 122 | 0.4.0 | | | **Assemble the judge's prompt in `bin/loop_claude/cycle/judge.py`, importing only wording the worker also sees, and call the worker "the worker" in it.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 5, 1.4 by 5 | 10.3 |
|
||||
| 158 | 0.4.0 | | decision | **Measure the fixed cost of a chunk from loop-claude's archive, in tokens and hours against its items' S, and set F in `Score = 100 * V / (2**S + F)` to it.** | 2 | 2 | 1 | 3 | 3 | 1.1 by 5, 1.4 by 5 | 10.3 |
|
||||
| 170 | 0.4.0 | | decision | **Re-tune `WORKER_STEPS` and `REVIEW_STEPS` once 10 chunks have shipped under them, from how well L, E and I order their review rounds and 🔴 findings.** | 2 | 2 | 3 | 5 | 2 | 1.1 by 3, 1.4 by 5 | 10.0 |
|
||||
| 116 | 0.4.0 | | | **Report L by the share of chunks with a 🔴, once 10 chunks that each shipped one item record finding counts and most record no 🔴.** | 1 | 1 | 2 | 2 | 2 | 1.1 by 5 | 9.1 |
|
||||
| 121 | 0.4.0 | | | **Check that `bin/round_titles.py`'s `PASSES` lists exactly the passes of larv-review §3's table.** | 1 | 1 | 1 | 1 | 2 | 1.1 by 5 | 9.1 |
|
||||
| 112 | 0.4.0 | | | **Count an item as shipped only where its chunk built it, so an item a chunk dropped from the plan gets no outcome.** | 3 | 2 | 2 | 5 | 3 | 1.1 by 6 | 8.2 |
|
||||
| 130 | 0.4.0 | | | **Move the vote counting out of `bin/larv_review/ruling.py`'s `votes`, so reading a votes file decides nothing.** | 2 | 1 | 2 | 3 | 3 | 1.1 by 4, 1.4 by 4 | 8.2 |
|
||||
| 125 | 0.4.0 | | decision | **Give "record", "outcome" and "places" one meaning each across `bin/`, and rename `bin/larv_review/git.py` by what it holds.** | 3 | 1 | 3 | 5 | 5 | 1.1 by 9, 1.4 by 9 | 7.4 |
|
||||
| 81 | 0.4.0 | | principle | **Have `workflow-bench/bench.py` return each run's failure as a value, so one failed clone does not hide the later runs' results.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 4 | 7.3 |
|
||||
| 102 | 0.4.0 | | | **Give loop-claude's retry delay an environment override, so the black-box test stops replacing `time.sleep` through `sitecustomize`.** | 1 | 1 | 2 | 2 | 2 | 1.1 by 4 | 7.3 |
|
||||
| 66 | 0.4.0 | | principle | **Hold systems-architect's method thresholds only while the comprehension panel finds the flagged code hard.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 5 | 6.8 |
|
||||
| 101 | 0.4.0 | | principle | **Have `larv_review/record.py`'s `read_seats` take the time, the commit check and the history as inputs, so it parses and decides without effects.** | 2 | 1 | 2 | 3 | 3 | 1.1 by 5 | 6.8 |
|
||||
| 111 | 0.4.0 | | | **Measure the chunks of a project that merges by squash or fast-forward, which `score-outcomes` skips today.** | 3 | 2 | 2 | 5 | 3 | 1.1 by 5 | 6.8 |
|
||||
| 113 | 0.4.0 | | | **Compute an item's S from the code it names, where it names any, and show it beside the finished items `score-outcomes` lists.** | 3 | 2 | 2 | 5 | 4 | 1.1 by 5, 1.4 by 5 | 6.8 |
|
||||
| 137 | 0.4.0 | | | **Archive a chunk's route as a record loop-claude reads back field by field, in place of the `Worker:` prose `ROUTED` parses, and name `stepped` and the judge's `bundled` by what they do.** | 4 | 2 | 3 | 7 | 4 | 1.1 by 4, 1.4 by 4 | 5.5 |
|
||||
| 110 | 0.4.0 | | | **Count, for each finished chunk, the later changes that fixed code it wrote, and show them beside its guessed E.** | 3 | 2 | 2 | 5 | 5 | 1.1 by 6, 1.4 by 6 | 4.9 |
|
||||
| 139 | 0.4.0 | | | **Split `bin/routing_figures/measure.py` by concern, and keep the figures' measured cycle and each raised call as fields loop-claude reads back, not as the prose it renders.** | 3 | 1 | 2 | 4 | 5 | 1.1 by 5, 1.4 by 5 | 4.1 |
|
||||
| 149 | 0.4.0 | | decision | **Fill each release by priority up to a capacity it states, in place of the bar, and move what does not fit to a later release.** | 4 | 2 | 3 | 7 | 5 | 1.1 by 7 | 3.8 |
|
||||
| 12 | 0.5.0 | | | **Run the test suite in CI on every PR, or record in `docs/decisions.md` why the local run is enough.** | 3 | 2 | 1 | 4 | 4 | 1.1 by 6 | 5.5 |
|
||||
| 105 | 0.5.0 | | decision | **Compare review rounds per pass and reading cost for the five chunks before and after the `worker:` decision moved the principles out of the worker's context, and file an item if either rose.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 6 | 5.5 |
|
||||
| 135 | 0.5.0 | | | **Fingerprint only the modules loop-claude imports, so editing another tool's code no longer reloads a running loop.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 4 | 5.5 |
|
||||
| 68 | 0.5.0 | | principle | **Test `workflow-bench/compare.py`'s blinding and `bench.py`'s run plan.** | 2 | 1 | 1 | 2 | 4 | 1.1 by 5 | 4.6 |
|
||||
| 4 | 0.5.0 | | | **Name each chunk's archive folder by the chunk's start time, so `<key>/` lists chunks in the order they ran.** | 3 | 2 | 3 | 6 | 3 | 1.1 by 3 | 4.1 |
|
||||
| 59 | 0.5.0 | | | **Bound a run of unfinished verdicts on one chunk, so a worker and judge that disagree cannot cycle unattended without end.** | 2 | 1 | 3 | 4 | 3 | 1.4 by 6 | 4.1 |
|
||||
| 123 | 0.5.0 | | | **Have `bin/comprehension_history.py` parse each history entry once into seats, votes and places, so `latest_places`, `worse_places` and `worse_votes` neither depend on table order nor split a label to find a seat.** | 2 | 1 | 2 | 3 | 5 | 1.1 by 5, 1.4 by 5 | 4.1 |
|
||||
| 132 | 0.5.0 | | | **Lift each dimension still under the comprehension floor after release 0.2.0's scoring run, at the places that run names.** | 3 | 2 | 2 | 5 | 5 | 1.1 by 5, 1.4 by 5 | 4.1 |
|
||||
| 124 | 0.5.0 | | decision | **Group the shared modules at `bin/`'s root into packages named by purpose, so the root holds only launchers and stand-alone tools.** | 4 | 1 | 3 | 6 | 6 | 1.1 by 9, 1.4 by 9 | 4.1 |
|
||||
| 156 | 0.5.0 | | decision | **Drop `rank_todo.py`'s ranking of an old-shape table once no project under `~/code` holds one.** | 2 | 1 | 3 | 4 | 4 | 1.1 by 4 | 3.6 |
|
||||
| 155 | 0.5.0 | | | **Lower the reading cost of the files a chunk reads most before its first commit.** | 3 | 2 | 2 | 5 | 5 | 1.4 by 6 | 1.6 |
|
||||
| 1 | 0.6.0 | | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 1 | 2 | 1 | 2 | 1 | | 0.0 |
|
||||
| 6 | 0.6.0 | | decision | **Move what `loop-claude` writes out of `~/.claude.json`, and ask the human about each write that must stay.** | 3 | 3 | 3 | 7 | 3 | 1.1 by 5 | 6.8 |
|
||||
| 5 | 0.6.0 | | decision | **Key a project on its checkout's full path, and refuse to start while another session uses that key.** | 4 | 2 | 3 | 7 | 5 | 1.1 by 6 | 3.3 |
|
||||
| 77 | 0.6.0 | | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 4 | 2 | 1 | 5 | 6 | 1.1 by 7, 1.4 by 7 | 3.2 |
|
||||
| 51 | 0.6.0 | | principle | **Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints.** | 2 | 1 | 3 | 4 | 5 | 1.1 by 5 | 2.7 |
|
||||
| 151 | 0.6.0 | | decision | **Keep 10 to 20 settled decisions per project as its goals' test set, and rerun them after a goal changes.** | 3 | 2 | 2 | 5 | 5 | 1.2 by 7 | 2.6 |
|
||||
| 52 | 0.6.0 | | principle | **Record why this checkout lives in Claude Code's own directory, or move it out and link it in.** | 4 | 3 | 4 | 9 | 6 | 1.1 by 4 | 1.2 |
|
||||
|
||||
## Details
|
||||
|
||||
### 166. Decide whether `docs/decisions.md` names rule-hook's refusal of a review launched below its rung, in `routing:` and in `rule-hook:`.
|
||||
|
||||
Item 163 (PR #295) has rule-hook refuse a review launch below the rung loop-claude gives its worker. The routing decision still says loop-claude's prompt has the worker launch reviewers, comprehension seats and the security pass on the project's rung, and does not name the hook. Recommendation: amend that paragraph to say rule-hook refuses a launch below the rung too. The decision was held before the chunk began, so the human decides. The `rule-hook:` entry's title says a marker overrides one rule, and `reviewer-rung` takes none. The amendment need not give the reason no marker waives the rule: the comment above `reviewer-rung` in bin/rule_hook/rules.py records it.
|
||||
|
||||
### 146. Score R as L + E + I − 2 from anchored levels, with R at least 7 when I is 5 and at least 5 when I is 4; give L, E and I their own columns and have `rank_todo.py` compute R; drop C; and route a task's worker on its L and its reviewers on its E + I.
|
||||
|
||||
Each level is about ×10 of a real quantity; `docs/decisions.md` → `todo.md: …` says why. The anchors, for `rules/todo-items.md`:
|
||||
|
||||
| Level | L — chance the change holds a defect | E — chance the defect escapes | I — damage, worst of availability, data, secrets, reach and reversibility |
|
||||
|---|---|---|---|
|
||||
| 1 | S 1–2, and nothing new | a test or the gate fails on it | one person's tooling or something cosmetic; a revert fixes it in minutes |
|
||||
| 2 | S 3–5, in one component, a pattern the repo already has | only review or a manual check sees it | one feature or service degraded; a revert within the hour |
|
||||
| 3 | one of: S 6 or more, several components, a new tool or pattern, live config or state | nothing checks it before users hit it | a service down, or every consumer affected; hours to restore |
|
||||
| 4 | two or more of level 3's, or a rewrite of code with known defects | — | everything down, a backup restore, or a secret or PII exposed |
|
||||
| 5 | — | — | irreversible: data lost for good, or a one-way door |
|
||||
|
||||
I's reach counts every project and session that runs the change, not the people in the README's audience. A change to a hook or a tool that every session runs is I level 3, "every consumer affected", even where one person uses it.
|
||||
|
||||
`rank_todo.py` ranks a table with C and without L, E and I until its project reshapes it, and says so, as it does for the old shape since PR #283, so no project's commits stop the day this lands. Reshaping is then that project's only work until done, as `CLAUDE.md` → "Working loop" says.
|
||||
|
||||
The same chunk drops C, so no reshaped table meets rung bands built for R + C. A task's worker routes on its L, and its reviewers on its E + I, never below the rungs the `routing:` decisions already set for them. A worker errs more the more it changes and the newer that is. Review must look harder the further a defect would escape and the more it would break. The bands (`docs/decisions.md` → `routing: one ladder…`) become bands over L and over E + I, spread across the values the rescored plans hold. The bands are the human's call, so the chunk proposes them and waits. The first bands are a placeholder: `score-outcomes` reports how well L, E and I each order finished chunks by review rounds and 🔴 findings, and the bands are re-tuned once 10 chunks have shipped under them.
|
||||
|
||||
`CLAUDE.md`'s delegation rule, `subagent-rung` and its `--plan`, loop-claude's chunk routing and prompts, the judge's rung, escalation trigger 1 and `score-outcomes` move from R + C to L and E + I. The routing decision's "S never moves a rung" no longer holds, since size sets L, so the chunk revises it. Once that chunk merges, append a `Reset` to `routing-calls-raised.md`, as `AGENTS.md` says of a band change, so the routing figures count only chunks routed on L and on E + I.
|
||||
|
||||
On 2026-10-16, over the 20 finished chunks that shipped one item, lines changed ordered chunks by review rounds at a rank correlation of +0.78 and by 🔴 findings at +0.74. Guessed C added nothing beyond guessed S, and guessed S matched the lines changed at only +0.26. So L rests on facts a scorer can check: the lines and components a change touches, and whether it adds a new tool or pattern or changes live config or state. Item 161's preflight lists the components, and the worker counts the size again once it has read the code.
|
||||
|
||||
### 152. Anchor each score level in `rules/todo-items.md` with words, and each project's with two or three items it shipped, and have a scorer compare a new item against them.
|
||||
|
||||
LLM ratings drift to the middle of an unanchored scale, and worked examples cut estimation error. A project's anchors are shipped items, cited by ID and commit, chosen from what `score-outcomes` measured for them.
|
||||
@@ -120,23 +97,23 @@ LLM ratings drift to the middle of an unanchored scale, and worked examples cut
|
||||
|
||||
A defect: goal 3, keeping unattended runs' transcripts backed up, does not hold while a rejected judge reply reaches only stderr. Raised by the principles audit of PR #215: the judge runs with `--no-session-persistence`, so the archive cannot answer the 3am question of why the loop rejected a verdict.
|
||||
|
||||
### 153. Run the judge on the rung the highest R + C in its chunk picks, an item the worker takes in included.
|
||||
### 153. Route the judge and the reviewers on the highest E + I in their chunk, an item the worker takes in included.
|
||||
|
||||
The judge reports the R + C of an item the worker takes in, and `routing.stepped` lifts the worker to the rung it picks, but `judge_rung` reads only the routed item's R + C, which the archived route line holds. So when the worker takes in a harder item, the judge stays on a lower rung than the work, where a missed interrupt lets the worker settle a decision that is lilleman's (goal 1.2). The route must carry the highest R + C taken in, archived so a rerun keeps it.
|
||||
The judge reports the L of an item the worker takes in, and `routing.stepped` lifts the worker to the rung it picks, but the judge and the reviewers route on the routed item's E + I alone, which the archived route line holds. So when the worker takes in an item whose defects would escape further or break more, the judge and the reviewers stay on a lower rung than that item needs: a missed interrupt lets the worker settle a decision that is lilleman's (goal 1.2), and a weaker review lets a defect through (goal 1.1). The judge must report the highest E + I taken in too, and the route must carry it, archived so a rerun keeps it; loop-claude then hands the reviewers' rung it picks to the worker's prompt and environment.
|
||||
|
||||
### 161. Have loop-claude run a preflight script before each fresh worker and put its findings in the worker's prompt, so the worker no longer spends tool calls finding the chunk's mechanical facts.
|
||||
|
||||
In item 140's chunk, the worker spent 5 min 46 s of 30 min, 16 of its 68 messages and 226k of its 491k fresh tokens before its first edit, through 12 Bash and 5 Read calls. About half of those found facts a program computes the same way in every chunk:
|
||||
|
||||
- the open PR and worktree for the chunk, the last tag and how many changes are unreleased;
|
||||
- the next release, the first plan item that does not wait, its R + C and its detail, after `rank_todo.py --check`;
|
||||
- the next release, the first plan item that does not wait, its L, its E + I and its detail, after `rank_todo.py --check`;
|
||||
- the items that may bundle with it under `CLAUDE.md` → "Working loop", "Work one chunk at a time, and make it as large as one purpose allows";
|
||||
- what `larv-review needs-scoring-run`, `reading-cost` and `larv-review places` print;
|
||||
- a worktree off a freshly fetched `origin/main`.
|
||||
|
||||
The preflight also lists the files and components the item's detail names, for the worker to judge the item's size from.
|
||||
|
||||
Compute these in one tool and give loop-claude's prompt its output as facts, so the worker reads them rather than finding them. The reading the chunk needs, such as the decision an item changes, stays the worker's. After its first read of the code, the worker counts the item's size again: the lines, files and components it will change. Where that count would pick another rung, a subagent on the project's rung re-scores the item. A re-score that picks a rung above the worker's is escalation trigger 1. Measure the next few chunks' time and fresh tokens before the first edit against item 140's figures. `bin/loop_claude/cycle/prompts.py`'s `off_the_projects_rung` is a place the comprehension seats found hard: put the preflight's facts in a function of their own rather than another branch there.
|
||||
Compute these in one tool and give loop-claude's prompt its output as facts, so the worker reads them rather than finding them. The reading the chunk needs, such as the decision an item changes, stays the worker's. After its first read of the code, the worker counts the item's size again: the lines, files and components it will change. Where that count would change the item's L, a subagent on the project's rung re-scores the item. A re-score whose L picks a rung above the worker's is escalation trigger 1. Measure the next few chunks' time and fresh tokens before the first edit against item 140's figures. The comprehension seats found the prompt assembly in `bin/loop_claude/cycle/prompts.py` hard: put the preflight's facts in a function of their own there.
|
||||
|
||||
### 84. Check, at each release's scoring run, that seats in a paired ruling agree more often than two scoring runs on near-identical code do, as the paired-ruling decision assumes.
|
||||
|
||||
@@ -148,14 +125,6 @@ Today an interrupt runs `claude --resume` on the terminal and waits for it to ex
|
||||
|
||||
The judge must treat an answer typed at the loop's terminal as the human's. `bin/claude_code/transcript.py`'s `render` labels every `-p` prompt `[loop]`, so an answer passed on headless would read as one quoted from the records. The judge would then never hold the chunk open until that answer is recorded. The change touches two places the comprehension seats found hard: `bin/loop_claude/cycle/judge.py`'s `run_judge` and `bin/loop_claude/loop/main.py`'s help. Leave each easier to read.
|
||||
|
||||
### 159. Run the prose pass on the lowest rung that a replay of archived prose rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.
|
||||
|
||||
lilleman asked, on 2026-10-16, for the prose pass to run on the worker's rung or lower. Today `bin/loop_claude/cycle/prompts.py`'s `off_the_projects_rung` keeps every reviewer on the project's rung while the worker runs below it, so a docs-only chunk such as item 140's, routed to Haiku 5.5, runs its prose reviewer on Opus 5.5.
|
||||
|
||||
A missed prose finding breaks no behaviour and a later change fixes it cheaply, so the prose pass is the likeliest reviewer to run cheaper without lowering quality (goal 1.4). Whether it does is measured, not assumed, as `docs/decisions.md` → `routing: the stability reviewer stays on the project's rung…` measures the stability reviewer: replay archived prose rounds on the code each reviewed, first on the project's rung, then on each rung below it, and take the lowest rung that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again. Generalise `bin/stability-replay` to take the pass, in place of a second replay tool. Record the run and the chosen rung in a `routing:` decision, valid until a new model joins the ladder or the prose reviewer's prompt changes.
|
||||
|
||||
`off_the_projects_rung` is a place the comprehension seats found hard: make it easier to read, such as by naming each reviewer's rung in one table, rather than adding a fourth branch.
|
||||
|
||||
### 73. Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.
|
||||
|
||||
2026-10-08, lilleman: rescored from A10/G8 to A4/G4 on the measurement below, so other items that lower token use go first.
|
||||
@@ -169,6 +138,12 @@ The reports would live in the session's scratchpad, such as `<scratchpad>/larv-r
|
||||
|
||||
This risks goal 1.1: a worker fixing from a one-line summary can miss context it needs.
|
||||
|
||||
### 159. Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.
|
||||
|
||||
lilleman asked, on 2026-10-16, for the prose pass to run on the worker's rung or lower, and the same day, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. Today `bin/loop_claude/cycle/prompts.py`'s `review_launches` keeps every reviewer at or above the project's rung whatever the item's E + I, so a docs-only chunk such as item 140's, routed to Haiku 5.5, runs its reviewers on Opus 5.5.
|
||||
|
||||
Whether a pass runs cheaper without lowering quality (goal 1.4 against 1.1) is measured, as `docs/decisions.md` → `routing: the stability reviewer runs no lower than the project's rung…` measures the stability reviewer: replay a pass's archived rounds on the code each reviewed, first on the project's rung, then on each rung below it, and take the lowest rung that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again. A missed prose finding breaks no behaviour and a later change fixes it cheaply, so replay the prose pass first. Generalise `bin/stability-replay` to take the pass, in place of a replay tool per pass. Record each run and the rung it chose in a `routing:` decision, valid until a new model joins the ladder or the pass's prompt changes. Give each pass a row of its own in `REVIEWS`, `review_launches`' table of review kinds in `bin/loop_claude/cycle/prompts.py`, rather than a branch.
|
||||
|
||||
### 80. State in README → Install that every machine taking this checkout must be a sandbox, as `settings.json`'s auto mode assumes.
|
||||
|
||||
A defect: goal 2, keeping the setup shareable between computers, does not hold while a new machine is never told it must be a sandbox. Raised by the principles audit of PR #224. The `settings.json` decision is valid while every machine taking this checkout is a sandbox, but Install asks for a sandbox only for `loop-claude`.
|
||||
@@ -245,20 +220,36 @@ Found by the goals audit of PR #245. `skills/larv-review/SKILL.md` §3 → "Secu
|
||||
|
||||
rule-hook checks Agent calls (item 163) but no Skill calls. A worker that invokes `/security-review` in its own context, instead of in a subagent, runs the pass on its own rung, and nothing blocks it. Check Skill calls too: `breaches_of` in bin/rule_hook/main.py takes a Skill call whose skill is `security-review` or `claude-security` as a review launch, and settings.json's PreToolUse matcher gains Skill. Found by the systems-architect's review of PR #295.
|
||||
|
||||
### 172. Name the project's rung `project` across `bin/loop_claude`, in place of `anchor`, as `bin/model_routing/ladder.py` names it.
|
||||
|
||||
The architecture pass of PR #296 found one rung under two names: `bin/model_routing/ladder.py` calls it `project`, while `bin/loop_claude/routing.py`, `cycle/prompts.py`, `cycle/run.py`'s `Cycle.anchor` and `loop/main.py` call it `anchor`, and gloss it as "anchor, the project's rung" in docstring after docstring. A reader works out that the two are one rung each time. Make `bin/loop_claude/routing.py`'s `first_route` easier to read.
|
||||
|
||||
### 82. Sort `AGENTS.md`'s decision titles and `docs/decisions.md`'s entries by title.
|
||||
|
||||
"Deterministic order", in `principles/technical-principles.md`, raised by the principles audit of PR #224: their order carries no meaning, so two chunks adding a decision conflict at the end, and a reader cannot find a title by scanning.
|
||||
|
||||
### 173. Have rule-hook's `todo-rank-check` run a worktree's own `bin/rank_todo.py` when the worktree is of this repo, so a branch that changes `todo.md`'s shape commits without an override.
|
||||
|
||||
Raised by the overrides pass of PR #296. The hook imports the main checkout's `rank_todo.py`, so while a branch of this repo changes the table's shape, every commit touching `todo.md` needs a `todo-rank-check` override: 33 log lines across four branches, the score-outcomes, plan-136, plan-148 and item-146 ones. The hook then checks nothing, and the rule rests on the worker chaining the branch's own check. Run another repo's `bin/rank_todo.py` never: that would run its code before every commit.
|
||||
|
||||
### 162. Script each step that 0.2.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.
|
||||
|
||||
lilleman asked, on 2026-10-16, for a standing audit each release, so the loop keeps moving work a program does as well from the model to scripts (goals 1.4 and 1.5). Item 161 scripts the steps found before the first edit in item 140's chunk. This audit reads the archived transcripts of the chunks that shipped release 0.2.0 for every step repeated by hand across chunks whose answer a program computes the same way, before, during and after review: a lookup, a check, a prompt assembled for a reviewer, a cleanup. File one item per such step, with the time and tokens it cost, into the next release with items left. Leave a step that needs judgement with the model.
|
||||
|
||||
The chunk that ships this item files a copy of it in the release after the one it audits, rewording "0.2.0" to the release that copy audits, and marks the copy `owner`, so the audit recurs each release until lilleman drops it.
|
||||
|
||||
### 171. Group the routing figures' review rounds by the reviewers' rung as well as the worker's, so a call blames a rung only for rounds its own reviews raised.
|
||||
|
||||
Raised by the goals audit of PR #296. `bin/routing_figures/measure.py` groups a chunk's review rounds by the rung its worker started on, and calls for a routing change when a lower rung took more of them. Since PR #296 the reviewers may run above the worker's rung, on the one their item's E + I picks, and no figure groups rounds by it, though the archived worker line names the E + I it follows from. A cheap worker on an item of high E + I gets stronger reviewers, who may raise more rounds, and a call could then blame the worker's rung for them: an interrupt of lilleman that settles nothing (goal 1.2).
|
||||
|
||||
### 165. Make this repo's tests assert the facts a prompt or help text carries in place of its sentences, and drop the tests that scan source or assert a constant.
|
||||
|
||||
A test audit across lilleman's projects on 2026-10-16 found 10 of 33 sampled tests in this repo pinning wording that is not the contract. Examples of these and other weak tests: `bin/loop_claude/test_loop_claude.py:165`, `:271` and most asserts in `:664` to `:687` pin whole sentences of the question, archive or prompt text, up to about 750 characters, and `:864` asserts help-text wording. `:906` and `:936` parse the source with `ast` and list each `Stop(...)` and `Failure(...)` call by function name. `bin/model_routing/test_ladder.py:16` asserts `len(LADDER) == 11`. Rewrite each to the technical principles' test bullet: assert the facts the text must carry, such as an item ID, a rung or a model argument, and delete a test no real bug could fail. Before and after rewriting a test, run the mutation tool on the code it covers: every mutant the old test killed, the new one kills.
|
||||
|
||||
### 169. Have rule-hook's `todo-rank-check` catch a `todo.md` that the same Bash call edits and then commits.
|
||||
|
||||
The hook judges a call before it runs, and `changed` in `bin/rule_hook/todo_check.py` compares `todo.md` with the index and HEAD as they stand then. A call that edits `todo.md` and commits it, such as a heredoc script followed by `git commit`, passes unchecked: PR #296's commit 149108e did, and the override marker it carried was never logged. larv-review's gate still runs `rank_todo.py --check`, so an unranked plan stops at merge rather than at commit. A git pre-commit hook would see the staged file.
|
||||
|
||||
### 122. Assemble the judge's prompt in `bin/loop_claude/cycle/judge.py`, importing only wording the worker also sees, and call the worker "the worker" in it.
|
||||
|
||||
All four seats of PR #257's scoring run named `bin/loop_claude/cycle/judge.py:16` `JUDGE_RULES`. The prompt is spliced from constants in `cycle/prompts.py` (`ANSWER_DESTINATIONS`, `ANSWERS_RECORDED`, `either`) and `project/records.py` (`HUMAN_RULE_NAMES`), so a reader rebuilds it from three files. It says "the agent" where the code and `CLAUDE.md` → "Vocabulary" say "worker". Wording the worker also sees, such as `ANSWER_DESTINATIONS` in `prompts.resumed`, may stay in `cycle/prompts.py`. Make `JUDGE_RULES` easier to read.
|
||||
@@ -267,9 +258,13 @@ All four seats of PR #257's scoring run named `bin/loop_claude/cycle/judge.py:16
|
||||
|
||||
F is 8 until then, about one S 3 change, lilleman's call in PR #283. A chunk costs more than its size: the worker reads about 130k tokens before its first commit whatever the item, and every chunk takes its review passes. Fit cost against 2**S from the finished chunks `score-outcomes` measures, in the same units `2**S` counts, and say in `docs/decisions.md` → `todo.md: …` how far the fit holds. A chunk bundling tiny items shares F between them.
|
||||
|
||||
### 170. Re-tune `WORKER_STEPS` and `REVIEW_STEPS` once 10 chunks have shipped under them, from how well L, E and I order their review rounds and 🔴 findings.
|
||||
|
||||
`docs/decisions.md` → `routing: one ladder…` calls the steps lilleman approved on 2026-10-16 a first guess. Once 10 chunks that each shipped one item scored with L, E and I have merged, read `score-outcomes`' rank correlations for L, E and I and the routed-chunks table of `routing-figures.md`, and propose new steps to lilleman, whose call they are.
|
||||
|
||||
### 116. Report L by the share of chunks with a 🔴, once 10 chunks that each shipped one item record finding counts and most record no 🔴.
|
||||
|
||||
`score-outcomes` reports risk as the median 🔴 count at each guessed R; after item 146, the part of R a 🔴 tests is L. Where most chunks have no 🔴, every median reads 0 and tells a scorer nothing. No round title held counts before PR #254, so how often a chunk has a 🔴 is not yet known.
|
||||
`score-outcomes` reports the median 🔴 count at each guessed R, and how well L orders chunks by 🔴 findings; the part of R a 🔴 tests is L. Where most chunks have no 🔴, every median reads 0 and the rank correlation rests on the few that have one, which tells a scorer little. No round title held counts before PR #254, so how often a chunk has a 🔴 is not yet known.
|
||||
|
||||
### 121. Check that `bin/round_titles.py`'s `PASSES` lists exactly the passes of larv-review §3's table.
|
||||
|
||||
@@ -311,15 +306,15 @@ Found by the principles audit of PR #242, against "Functional core, imperative s
|
||||
|
||||
### 113. Compute an item's S from the code it names, where it names any, and show it beside the finished items `score-outcomes` lists.
|
||||
|
||||
`score-outcomes` grounds R, S and C in finished chunks only. Where an item names files or units, their size bounds what the change can touch.
|
||||
`score-outcomes` grounds L, E, I and S in finished chunks only. Where an item names files or units, their size bounds what the change can touch.
|
||||
|
||||
### 137. Archive a chunk's route as a record loop-claude reads back field by field, in place of the `Worker:` prose `ROUTED` parses, and name `stepped` and the judge's `bundled` by what they do.
|
||||
|
||||
PR #262's paired ruling found the project reads worse: a rerun resumes on its rung only by parsing `Rung.__str__`'s and `Route.__str__`'s wording back with `bin/loop_claude/routing.py`'s `ROUTED`, a string contract over four files, where a reword silently routes afresh. Write the route as a structured line, such as JSON after `Worker:`, that `transcript_archive` reads into fields. `stepped` both raises to a bundled item's rung and counts towards a one-rung step; `Reply.bundled` names neither R + C nor highest. Make `bin/loop_claude/loop/main.py`'s `run_loop` easier to read where it threads the route.
|
||||
PR #262's paired ruling found the project reads worse: a rerun resumes on its rung only by parsing `Rung.__str__`'s and `Route.__str__`'s wording back with `bin/loop_claude/routing.py`'s `ROUTED`, a string contract over four files, where a reword silently routes afresh. Write the route as a structured line, such as JSON after `Worker:`, that `transcript_archive` reads into fields. `stepped` both raises to a bundled item's rung and counts towards a one-rung step; `Reply.bundled` names neither L nor highest. Make `bin/loop_claude/loop/main.py`'s `run_loop` easier to read where it threads the route.
|
||||
|
||||
### 110. Count, for each finished chunk, the later changes that fixed code it wrote, and show them beside its guessed E.
|
||||
|
||||
`score-outcomes` measures risk only by the 🔴 findings review caught. A serious bug review missed shows later, as a change that fixes code the chunk wrote: the defect that escaped, which E guesses (item 146).
|
||||
`score-outcomes` measures risk only by the 🔴 findings review caught. A serious bug review missed shows later, as a change that fixes code the chunk wrote: the defect that escaped, which E guesses.
|
||||
|
||||
### 139. Split `bin/routing_figures/measure.py` by concern, and keep the figures' measured cycle and each raised call as fields loop-claude reads back, not as the prose it renders.
|
||||
|
||||
@@ -327,7 +322,7 @@ The paired ruling on PR #265 ruled worse. Make these places easier to read: `mea
|
||||
|
||||
### 149. Fill each release by priority up to a capacity it states, in place of the bar, and move what does not fit to a later release.
|
||||
|
||||
A release's capacity is a sum of its items' C. Each project states its own; the human sets the first value per project. Nothing is deleted by score.
|
||||
A release's capacity is a sum of its items' cost, as the formula counts it. Each project states its own; the human sets the first value per project. Nothing is deleted by score.
|
||||
|
||||
### 12. Run the test suite in CI on every PR, or record in `docs/decisions.md` why the local run is enough.
|
||||
|
||||
@@ -375,7 +370,7 @@ While `rank_todo.py` ranks two shapes, projects follow two sets of scoring rules
|
||||
|
||||
### 155. Lower the reading cost of the files a chunk reads most before its first commit.
|
||||
|
||||
`bin/reading-cost` on 2026-10-16: the last 5 chunks' median is 189487 new input tokens before their first commit, the 5 before them 90662. Read most over the last 5, by opens and searches: `todo.md` 55, `docs/decisions.md` 23, `CHANGELOG.md` 12, `README.md` 12, `bin/test_rank_todo.py` 10, `AGENTS.md` 8, `bin/loop_claude/test_loop_claude.py` 7, `skills/larv-review/SKILL.md` 7, `agents/stability-reviewer.md` 6, `bin/rank_todo.py` 6. The chunk that ships this item runs `bin/reading-cost --reset`.
|
||||
`bin/reading-cost` on 2026-10-16: the last 5 chunks' median is 189873 new input tokens before their first commit, the 5 before them 90662. Read most over the last 5, by opens and searches: `todo.md` 56, `docs/decisions.md` 19, `CHANGELOG.md` 13, `README.md` 13, `bin/test_rank_todo.py` 10, `AGENTS.md` 8, `skills/larv-review/SKILL.md` 7, `agents/stability-reviewer.md` 6, `bin/loop_claude/test_loop_claude.py` 6, `bin/rank_todo.py` 6. The chunk that ships this item runs `bin/reading-cost --reset`.
|
||||
|
||||
### 1. Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user