diff --git a/CHANGELOG.md b/CHANGELOG.md index 4574b959..a18df30d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,16 +2,16 @@ ## Unreleased -- `score-outcomes` lists a project's score anchors: the shipped items its `docs/score-anchors.md` names for each level of L, E and I, each with what its chunk measured. `rules/todo-items.md` has a scorer compare a new item with them, and says how to choose two or three per level. Add the file to each project whose merge commits `score-outcomes` measures; elsewhere, score by the levels' words. - Every goal in `README.md` → Goals is marked `**Maintain:**`, a state to hold, or `**Achieve:**`, a direction to pursue. Each achieve goal has a signal, a counter-signal and a one-line reason under it. A change that breaks a maintain goal is redesigned or dropped, whatever it scores: `rules/todo-items.md` says so, and larv-review's goal alignment step no longer merges such a PR. `rules/goals-and-audience.md` states the form for every project's README. Expect larv-review to stop a PR on a maintain goal where it asked you before. - Reshape every project's `todo.md` to the new shape below: until it is reshaped, that is the project's only work, as it is while README.md lacks Goals and Audience, and `rank_todo.py` says so. First move every machine to this release: an older release rejects the new formula, so it fails on every reshaped `todo.md`. Then mark the README's goals `**Maintain:**` or `**Achieve:**`, score each item's L, E and I, and set a bar on the new scale below the score of every item to keep, since the old bar would drop most items. Until then, an old-shape `todo.md` still ranks by its own formula and weights. - `todo.md` ranks items by value per cost: this setup's own plan uses `Score = 100 * V / (2**S + 8)`, where 8 is a chunk's fixed cost, about one size-3 change, and `rank_todo.py --help` shows it. A Moves column replaces A, G and Goals, naming each achieve goal an item moves and how far, such as `1.1 by 8, 1.2 by 5`. V sums each named goal's weight times its move. `rank_todo.py` weighs each achieve goal by its position in README.md, the first heaviest, so the goal weights table goes. An item names the most specific goal it moves, never a goal beside its own sub-goal. `rules/todo-items.md` and `rank_todo.py --help` show the new shape. -- L, E and I replace C, and set R. L is the chance the change holds a defect, from 1 to 4; E the chance a defect escapes tests and review, from 1 to 3; I the damage it would do, from 1 to 5, counting every project and session that runs the change. `rules/todo-items.md` anchors each level in words, and `rank_todo.py` writes R as L + E + I − 2, at least 7 when I is 5 and at least 5 when I is 4. The table's score columns are `| L | E | I | R | S |`. `score-outcomes` takes `L E I S`, and reports how well L, E and I each order finished chunks by review rounds and 🔴 findings. +- L, E and I replace C, and set R. L is the chance the change holds a defect, from 1 to 4; E the chance a defect escapes tests and review, from 1 to 3; I the damage it would do, from 1 to 5, counting every project and session that runs the change. `rules/todo-items.md` defines each level in words, and `rank_todo.py` writes R as L + E + I − 2, at least 7 when I is 5 and at least 5 when I is 4. The table's score columns are `| L | E | I | R | S |`. `score-outcomes` takes `L E I S`, and reports how well L, E and I each order finished chunks by review rounds and 🔴 findings. - `loop-claude` routes a chunk's worker on its item's L, from two rungs below the project's rung at L 1 to one above it at L 4. It routes the reviewers on the item's E + I, one rung above the project's for each point past 4, never below the worker's rung. It runs the judge of an Opus 5.5 project on the rung E + I picks, never below the project's, and keeps it there when the worker steps up. The comprehension seats and the security pass stay on the project's rung. rule-hook holds each reviewer launch to the reviewers' rung, which `loop-claude` hands the worker in `LOOP_CLAUDE_REVIEWER_RUNG`. A delegated task's subagent takes the rung its L picks: `subagent-rung` takes L in place of R + C. Until a project reshapes its `todo.md`, every chunk there runs on the project's rung. Expect reviews of items whose defects would escape far or break much to cost more, and escalation from an item of L 4 to reach Fable 5.1 only with your yes. - An item that restores a maintain goal the project breaks today is exempt as `defect`, and its Moves may name that maintain goal, weighed as the heaviest achieve goal. A defect ranks by its score like any item. Only two kinds rank ahead of the rest of their release: questions first, since an answer may change other items, then security items. - The stability reviewer applies `principles/technical-principles.md` as written, in place of its own copies of it. Its test-data check now asks for UUID v7, where it asked for randomised UUIDs. - larv-review's principles audit is a pass of its own, and runs on every chunk that changes code, config or tests, not only on one with an architectural decision. A chunk of prose or planning alone skips it. - `rule-hook` refuses an Agent call that launches a reviewer, a comprehension seat or the security pass on a rung below the one `loop-claude` names for it, and takes no override: pass the model and effort the refusal names. +- `score-outcomes` lists a project's score anchors: the shipped items its `docs/score-anchors.md` names for each level of L, E and I, each with what its chunk measured. `rules/todo-items.md` has a scorer compare a new item with them, and says how to choose two or three per level. Add the file to each project that merges with merge commits; elsewhere, score by the level descriptions in `rules/todo-items.md`. ## 0.1.0 diff --git a/README.md b/README.md index 5a33196d..f3460ce0 100644 --- a/README.md +++ b/README.md @@ -89,7 +89,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a - `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format. - `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure. - `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit, Write and Agent call, and `rule-hook --help` states the rules and the override. -- `score-outcomes [L E I S]` lists the finished items whose guessed scores sit nearest the given ones, with what their chunks measured, the project's anchors for each level of L, E and I, the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings; `score-outcomes --help` states the measures. +- `score-outcomes [L E I S]` lists the finished items whose guessed scores sit nearest the given ones, and the shipped items the project's `docs/score-anchors.md` names for each level of L, E and I, each with what its chunk measured; then the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings; `score-outcomes --help` states the measures. - `stability-replay ` replays archived stability reviews of the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again; `stability-replay --help` states the replay. Run it only in a sandbox, like loop-claude: each replayed reviewer has Bash and skips every permission prompt, and an archived prompt may name paths on the host. - `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it. - `subagent-rung [--plan] ` prints the Agent tool's model and effort for a subagent's task, picked by its L from the project's model, and with `--plan` never below the project's rung; `subagent-rung --help` states the rule. diff --git a/bin/score_outcomes/anchors.py b/bin/score_outcomes/anchors.py index df2ef97b..c9443766 100644 --- a/bin/score_outcomes/anchors.py +++ b/bin/score_outcomes/anchors.py @@ -4,7 +4,6 @@ from todo_table import cells, LEVELS, parse_ids from failure import Failure from score_outcomes.outcomes import Outcome, render -# Where a project names the shipped items that show each level of L, E and I. ANCHORS_PATH = "docs/score-anchors.md" # A level, such as `L 1`; every scale tops out below 10. LEVEL = re.compile(r"([A-Z]) ([1-9])") @@ -23,7 +22,7 @@ def parse_anchors(text: str) -> Anchors | Failure: level, ids = LEVEL.fullmatch(pair[0]), parse_ids(pair[1]) if not level or int(level[2]) > LEVELS.get(level[1], 0) or not ids: scales = ", ".join(f"{name} 1 to {top}" for name, top in LEVELS.items()) - return Failure(f"{ANCHORS_PATH} cannot read the row {row}; each row names a level, {scales}, and the IDs of shipped items, such as | L 1 | 12, 14 |") + return Failure(f"cannot read the row {row} in {ANCHORS_PATH}; each row names a level, {scales}, and the IDs of shipped items, such as | L 1 | 12, 14 |") if pair[0] in dict(found): return Failure(f"{ANCHORS_PATH} names {pair[0]} twice") found.append((pair[0], ids)) @@ -31,11 +30,11 @@ def parse_anchors(text: str) -> Anchors | Failure: def anchor_lines(found: Anchors | Failure, outcomes: list[Outcome]) -> list[str]: - """Each level, then each of its anchors with what its chunk measured; why none shows where the file cannot be read, and where to name them where it names no level.""" + """Each level, then each of its anchors with what its chunk measured. Where the file cannot be read, one line saying why; where it names no level, one line saying how to choose anchors.""" if isinstance(found, Failure): return [f"- None: {found.reason}"] if not found: - return [f"- None: the project's {ANCHORS_PATH} names no level; ~/.claude/rules/todo-items.md says how to choose them."] + return [f"- None: the project has no {ANCHORS_PATH}, or it names no level; ~/.claude/rules/todo-items.md says how to choose anchors."] by_id = {outcome.item.id: outcome for outcome in outcomes} lines = [] for level, ids in found: diff --git a/bin/score_outcomes/main.py b/bin/score_outcomes/main.py index 7a046752..511205ba 100644 --- a/bin/score_outcomes/main.py +++ b/bin/score_outcomes/main.py @@ -31,8 +31,8 @@ measured, which items shipped with it, and its tagline. Then the project's anchors, the shipped items its docs/score-anchors.md names for each level of L, E and I, read from the checkout's working tree, each line as above, or that no merge shipped it. A -row that names no level on its scale, a level named twice, or an Items cell that is not a list of -IDs leaves the anchors out, saying why, and exits 1 after the rest of the report. +file that cannot be read, a row that does not hold a level on its scale and a list of IDs, or a +level named twice leaves the anchors out, saying why. Then the median of the outcome R and S each predict at each value they were guessed at: R against 🔴 findings, S against lines changed. And how well L, E and I each order chunks by review rounds @@ -42,7 +42,8 @@ item, since a chunk measures all its items at once, and reports a score only onc hold it and its outcomes. Exit 1, saying why, on bad arguments, when git cannot start, finds no checkout or reads a shallow -clone, or the default branch cannot be read. +clone, or the default branch cannot be read; or, after the rest of the report, when the anchors are +left out. """ import pathlib import re @@ -79,7 +80,6 @@ def main(argv: list[str]) -> int: return failed(outcomes) anchored = read_anchors() print(report_text(outcomes, anchored, dict(zip(GIVEN, map(int, argv))) if argv else None)) - # Anchors the tool cannot read leave the rest of the report standing, and still fail the run. return failed(anchored) if isinstance(anchored, Failure) else 0 diff --git a/bin/todo_table.py b/bin/todo_table.py index 60a4b0c5..ecbabeb3 100644 --- a/bin/todo_table.py +++ b/bin/todo_table.py @@ -231,7 +231,7 @@ def parse_scores(id_: str, named: dict[str, str], names: tuple[str, ...]) -> dic def parse_ids(cell: str) -> tuple[int, ...] | None: - """The item IDs a cell lists, such as `12, 14`, none where it is empty; None where it holds anything else.""" + """The item IDs a cell lists, such as `12, 14`: an empty tuple where the cell is empty, None where it holds anything else.""" if not re.fullmatch(r"([0-9]+(, [0-9]+)*)?", cell): return None return tuple(int(id_) for id_ in cell.split(", ") if id_) diff --git a/docs/score-anchors.md b/docs/score-anchors.md index 47bfb8b3..4680f553 100644 --- a/docs/score-anchors.md +++ b/docs/score-anchors.md @@ -1,6 +1,6 @@ # Score anchors -The items this repo shipped that show each level of L, E and I, which `rules/todo-items.md` defines. `bin/score-outcomes` lists each with what its chunk measured, for a scorer to compare a new item against. No item yet shows I 4 or I 5. +The items this repo shipped that show each level of L, E and I, which `rules/todo-items.md` defines. No item yet shows I 4 or I 5. | Level | Items | |---|---| diff --git a/rules/todo-items.md b/rules/todo-items.md index 76c2f939..33ded0b3 100644 --- a/rules/todo-items.md +++ b/rules/todo-items.md @@ -7,7 +7,7 @@ paths: - **`todo.md` is mandatory in every project, in the shape `rank_todo.py --help` shows**, which `~/.claude/bin/rank_todo.py` checks and ranks; its `--help` shows a valid file. `## Scoring` holds the formula and the bar; `## Items` is one table of every item; `## Details` holds each item's detail under its ID. Cite an item by its ID. The formula and bar are the project's own to tune. - **Order items only by score and the Blocked by column.** An item's Blocked by cell lists the IDs of the items that must be done before it starts, such as `12, 14`; `rank_todo.py --help` says how it ranks them. No text in `todo.md` outside the table orders items: where an item must go first, score it so, and say why in its detail. -- **Plan only on the project's rung or above.** loop-claude's prompt tells you when you run below it. Below it, never score or re-score an item, move it between releases, split a milestone into items, delete an item under the bar, rewrite a filed item's tagline or detail, or choose or replace an anchor. Draft each new item, and each change to a filed item, instead. Before your next commit, launch one subagent to apply the drafts, score them and set their release. Launch it on the rung `CLAUDE.md`'s delegation rule gives a subagent that files or scores items. Keep the scores and releases it writes. For a security finding, launch it at once. Any rung may remove a finished item's row and detail, and score a delegated task's L. A subagent launched for another task leaves `todo.md` alone. +- **Plan only on the project's rung or above.** loop-claude's prompt tells you when you run below it. Below it, never score or re-score an item, move it between releases, split a milestone into items, delete an item under the bar, rewrite a filed item's tagline or detail, or choose or replace an item in `docs/score-anchors.md`. Draft each new item, each change to a filed item, and each change to the anchors, instead. Before your next commit, launch one subagent to apply the drafts, score them and set their release. Launch it on the rung `CLAUDE.md`'s delegation rule gives a subagent that files or scores items. Keep the scores and releases it writes. For a security finding, launch it at once. Any rung may remove a finished item's row and detail, and score a delegated task's L. A subagent launched for another task leaves `todo.md` alone. - **An item's release allows the bump that semver read strictly asks for.** An item that changes what exists needs a major, or before 1.0.0 a minor. One that only adds needs a minor, and a fix a patch. - **Score an item when it is filed or edited.** S, the size of the change, is a whole number from 1 to 10. Score L, E and I by the levels below, each level about ten times as likely, or as damaging, as the one below it. `rank_todo.py` writes R, the risk that the change creates serious bugs or issues: L + E + I − 2, at least 7 when I is 5 and at least 5 when I is 4, since a severe enough consequence outranks how unlikely it is. A task's L picks its worker's rung, and its E + I its reviewers' rung. @@ -22,8 +22,8 @@ paths: - **I's reach counts every project and session that runs the change.** A change to a hook or a tool that every session runs is I level 3, every consumer affected, even where one person uses it. - **Name in Moves each achieve goal in `README.md` → `## Goals` the item moves, and how far,** from 1 to 10, judged by its value to the audience in `## Audience`: `1.1 by 8, 1.2 by 5`. Name the most specific goal it moves, and a parent goal only for a move none of its sub-goals covers, never beside its own sub-goal. Leave out every maintain goal, which vetoes, except on an item marked `defect` that restores it: name that goal, and how much of it is broken. `rank_todo.py --help` says how it weighs each goal. - **A table with C, A, G and Goals in place of L, E, I and Moves is in the old shape.** Reshaping it is the project's only work until done, as `CLAUDE.md` → "Working loop" says, and `rank_todo.py` says how. Until then, A is the item's value to the audience and G its value to the goals it cites. -- **Score L, E and I against the project's anchors, and check every score against finished chunks.** Guess the scores, then run `~/.claude/bin/score-outcomes ` in the project. It lists the finished items guessed nearest, the project's anchors, each with what its chunk measured, the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings. Compare the item with each anchor in turn: is it more or less likely to hold a defect, to let one escape, or to do damage? Give it the level whose anchors it matches; where a level has no anchor, judge by the table's words. Set S to the value guessed for the finished items whose outcome matches what you expect of this one. Keep your guess when no listed item resembles the one you are scoring. -- **Anchor each level of L, E and I with two or three items the project shipped,** in `docs/score-anchors.md`: a table whose Level column names a level, such as `L 1`, and whose Items column lists the anchors' IDs, such as `12, 14`. Choose an L anchor whose chunk `score-outcomes` measured as the level's words expect. Choose an E or I anchor by the level's words and what followed its release, since no tool measures either yet. Leave out a level no shipped item shows. When items scored at a level turn out about ten times more or less likely to hold, or to let escape, a defect, or to do damage, than its words say, replace that level's anchors. +- **Score L, E and I against the project's anchors, and check every score against finished chunks.** Guess the scores, then run `~/.claude/bin/score-outcomes ` in the project. It lists the finished items guessed nearest and the project's anchors, the shipped items its `docs/score-anchors.md` names for each level, each with what its chunk measured; then the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings. Compare the item with each anchor in turn: is it more or less likely to hold a defect, to let one escape, or to do damage? Give it the level whose anchors it matches; where a level has no anchor, judge by the level's words in the table above. Set S to the value guessed for the finished items whose outcome matches what you expect of this one. Keep your guess when no listed item resembles the one you are scoring. +- **Anchor each level of L, E and I with two or three items the project shipped,** in `docs/score-anchors.md`: a table whose Level column names a level, such as `L 1`, and whose Items column lists the anchors' IDs, such as `12, 14`. Choose an L anchor whose chunk, as `score-outcomes` measured it, fits the level's words: its lines changed and the components it touched. Choose an E or I anchor by the level's words and what followed its release, since no tool measures either yet. Leave out a level no shipped item shows. Replace a level's anchors when the items scored at it turn out about ten times more likely, or ten times less likely, than its words say: to hold a defect, to let one escape, or to do damage. - **Read the places the comprehension seats found hard before filing an item.** `~/.claude/bin/larv-review places`, run in the project, prints them. Where the item's change touches one, its detail says to make that place easier to read. - **An item scoring under the bar is deleted, with every item that needs it, and a finding under the bar is not filed.** The chunk's summary names each, with its score. - **The Exempt cell keeps an item whatever its score:** `security` for a security finding, `defect` for a 🔴 finding or an item that restores a maintain goal the project breaks today, `question` for a question only the human can answer, which ranks first in its release, `decision` for an item `docs/decisions.md` cites, `owner` for an item the human told you to keep, and, where no other reason fits, `principle` for a 🟡 finding against a file under `~/.claude/principles/`, whose item's detail names that principle. Every item `docs/decisions.md` cites carries one of these reasons.