Let an item name a maintain goal by the confidence a test, gate leg or probe adds, weighed 0.5 per point
CI / tests (pull_request) Successful in 2m7s

This commit is contained in:
2026-11-09 11:59:07 +01:00
parent f1967aca68
commit cef13bb354
7 changed files with 22 additions and 32 deletions
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- `rank_todo.py` now lets an item name a maintain goal by how much more confident a test, a gate leg or a probe makes the project that the goal holds, weighing it 0.5 per point, 1 to 10, judged against the confidence the project already has.
- `chunk-preflight` now quotes the detail of each item that may join the chunk, so a worker no longer opens `todo.md` to read it.
- `command-patterns [<version>]` prints a table of the shell command patterns a release's chunks ran.
- `deploy-release --cut` reports each step as it finishes, waits for the checks `main`'s protection requires and names them, deletes the merged release branch, and ends on one line saying the release is cut.
+4 -2
View File
@@ -46,8 +46,10 @@ A valid todo.md; ## Details and each item's detail are optional:
holds no scoring run, code health weighs 4 times its weight by position, and the other goals keep
theirs. A line `**Void:** code health, since <reason>.` in README.md's ## Goals drops code health,
and an item may not name it. A project must void it where its docs/decisions.md has a
`## review opt-out:` title naming the comprehension check. Only an item marked defect may name a
maintain goal, the one it restores, which weighs as the heaviest achieve goal.
`## review opt-out:` title naming the comprehension check. A maintain goal weighs
0.5 per point of a move that raises confidence in it, 1 to 10, for an item that adds a test, a
gate leg or a probe that fails where the goal breaks. An item marked defect that restores a
maintain goal weighs it as the heaviest achieve goal instead.
- The bar is judged on the Score as shown, to one decimal.
- A new item takes Next ID, and Next ID then goes up by one; every ID stays below it.
- An item's L, E and I are levels that ~/.claude/rules/todo-items.md defines: L from 1 to 4, E from 1
+9 -4
View File
@@ -20,6 +20,8 @@ from release_plan.release import version
GOAL_STEP = 0.97
# Code health weighs this many times its weight by position while the comprehension panel's overall is under FLOOR, or unmeasured; lilleman's choice, which docs/decisions.md records.
UNDER_FLOOR = 4
# A maintain goal weighs this much per point an item raises confidence in it, other than a defect that restores it; docs/decisions.md records the choice and its data.
MAINTAIN_WEIGHT = 0.5
# Any **Void:** line of README.md's ## Goals; ranked_goals accepts only `**Void:** code health, since <reason>.`, as ~/.claude/CLAUDE.md → "Shared goal" spells it.
VOID = re.compile(r"^\*\*Void:\*\* (.*)$", re.M)
# A docs/decisions.md title opting the project out of the comprehension check, as larv-review's SKILL.md → "Opt-out" spells it.
@@ -366,7 +368,7 @@ def readme_goals(readme: str) -> list[tuple[str, Mark]] | Failure:
def goal_weights(goals: list[tuple[str, Mark]], overall: decimal.Decimal | None) -> dict[str, float] | Failure:
"""Each achieve goal's weight: the one ranked k of n weighs GOAL_STEP ** (k - 1) / (1 + GOAL_STEP + ... + GOAL_STEP ** (n - 1)), so the weights sum to 1. Code health, where goals holds it, then weighs UNDER_FLOOR times that while the overall is under FLOOR or None, and the other goals keep their weights. Each maintain goal weighs as the heaviest achieve goal, since a broken one outranks any achieve goal; only a defect that restores it may name it. Every weight is 0 where goals holds no achieve goal, as where README.md voids code health and marks no goal **Achieve:**, so every item scores V 0 and only an exempt one passes the bar."""
"""Each achieve goal's weight: the one ranked k of n weighs GOAL_STEP ** (k - 1) / (1 + GOAL_STEP + ... + GOAL_STEP ** (n - 1)), so the weights sum to 1. Code health, where goals holds it, then weighs UNDER_FLOOR times that while the overall is under FLOOR or None, and the other goals keep their weights. Each maintain goal weighs as the heaviest achieve goal, since a broken one outranks any achieve goal, which is the weight of a defect that restores it; any other item weighs it at MAINTAIN_WEIGHT instead, in weight_of. Every weight is 0 where goals holds no achieve goal, as where README.md voids code health and marks no goal **Achieve:**, so every item scores V 0 and only an exempt one passes the bar."""
if unmarked := [number for number, mark in goals if mark is None]:
return Failure(f"README.md's goal {unmarked[0]} is marked neither **Maintain:** nor **Achieve:**; mark it, as ~/.claude/rules/goals-and-audience.md says, so rank_todo.py knows which goals score")
achieve = [number for number, mark in goals if mark == "Achieve"]
@@ -460,11 +462,16 @@ def score(formula: ast.Expression, item: Item, scoring: Scoring) -> float | Fail
term = max(scoring.weights[goal] for goal, _ in item.moves)
else:
# Only an OLD_SHAPE item names a goal without a move.
term = sum(scoring.weights[goal] * (by or 0) for goal, by in item.moves)
term = sum(weight_of(goal, item, scoring) * (by or 0) for goal, by in item.moves)
value = evaluate(formula, {**item.scores, goal_term(scoring.shape): term})
return value if math.isfinite(value) else Failure(f"the formula gives item {item.id} no finite score")
def weight_of(goal: str, item: Item, scoring: Scoring) -> float:
"""A maintain goal weighs MAINTAIN_WEIGHT for an item that raises confidence in it, and as the heaviest achieve goal only for a defect that restores it."""
return MAINTAIN_WEIGHT if scoring.goals[goal] == "Maintain" and item.exempt != "defect" else scoring.weights[goal]
def number(text: str | Failure, what: str) -> float | Failure:
if isinstance(text, Failure):
return text
@@ -519,8 +526,6 @@ def cited_goals_failure(item: Item, scoring: Scoring) -> Failure | None:
return Failure(f"item {item.id} names {CODE_HEALTH}, which README.md voids; drop it from the item's {MOVES}, or the void from README.md")
if goal not in scoring.goals:
return Failure(f"item {item.id} cites goal {goal}, which README.md's ## Goals does not number")
if scoring.goals[goal] == "Maintain" and item.exempt != "defect":
return Failure(f"item {item.id} cites goal {goal}, a maintain goal, which only a defect that restores it may name; name the achieve goals the item moves")
return None
+4 -1
View File
@@ -162,7 +162,6 @@ class Rank(unittest.TestCase):
def test_refuses_a_goal_it_cannot_weigh(self):
cases = {
(TODO.replace("2.1 by 8, 1 by 6", "1 by 8, 2 by 6"), README): "item 3 cites goal 2, a maintain goal, which only a defect that restores it may name; name the achieve goals the item moves",
(TODO.replace("2.1 by 8, 1 by 6", "2.1 by 8, 3 by 6"), README): "item 3 cites goal 3, which README.md's ## Goals does not number",
(TODO.replace("2.1 by 8, 1 by 6", "2.1 by 8, 2.1 by 6"), README): "item 3 cites goal 2.1 twice; name each goal once",
(TODO, README.replace("**Maintain:** Correct", "Correct")): "README.md's goal 2 is marked neither **Maintain:** nor **Achieve:**; mark it, as ~/.claude/rules/goals-and-audience.md says, so rank_todo.py knows which goals score",
@@ -180,6 +179,10 @@ class Rank(unittest.TestCase):
ranked = self.ranked(TODO.replace("| 2 | 0.1.0 | | | **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2.1 by 1 |", "| 2 | 0.1.0 | | defect | **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2 by 3 |"))
self.assertIn("| **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2 by 3 | 9.5 |", ranked.text)
def test_weighs_a_maintain_goal_an_item_raises_confidence_in_as_half_a_goal_per_point(self):
ranked = self.ranked(TODO.replace("| 2 | 0.1.0 | | | **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2.1 by 1 |", "| 2 | 0.1.0 | | | **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2 by 3 |"))
self.assertIn("| **Tighten the tests.** | 1 | 2 | 1 | 2 | 4 | 2 by 3 | 9.4 |", ranked.text)
def test_refuses_a_goal_named_beside_its_own_sub_goal(self):
readme = "## Goals\n\n" + VOID + "1. **Achieve:** A.\n 1. **Maintain:** B.\n 2. **Achieve:** C.\n2. **Achieve:** D.\n"
todo = TODO.replace("2.1 by 8, 1 by 6", "1.2 by 2, 1 by 3").replace("| 2.1 by 1 |", "| 2 by 1 |")
+3 -1
View File
@@ -466,7 +466,7 @@ Valid while the worker writes the tests that larv-review's reviewers hold to the
2026-10-15, lilleman: the scores follow a researched method, built by `todo.md` items 149 and 151 after PR #283 shipped the ranking, PR #296 the risk scores and PR #298 the anchors each project names for each level in `docs/score-anchors.md`; the evidence is in `docs/scoring-and-goals-research.md`. R is L + E + I − 2: the chance the change holds a defect, set first by its size; the chance the defect escapes tests, review and monitoring; and the worst impact dimension, each about ×10 per level. I5, irreversible, holds R at 7 or more, and I4 at 5 or more, since a severe enough consequence outranks how unlikely it is. R stays out of the ranking and sets how carefully an item is done; an item that lowers risk scores it as value. A task's L and E + I pick its rungs in place of R (2026-10-16, lilleman), as `routing: one ladder` records. There is no C column: what it scored beyond size, spread and novelty, which L already holds, is complexity, and that adds almost nothing once size is accounted for. There is no A column either, since the Audience says whose value the goals serve. Each README goal is `maintain`, which vetoes an item that breaks it, or `achieve`, with a signal, a counter-signal and a one-line reason, as this README's Goals mark them. Two parts are not built yet and hold once their items ship: each release fills by priority up to a capacity it states (item 149), and each project keeps 10 to 20 settled decisions to test its goals against (item 151).
2026-10-16, PR #283: items rank by `Score = 100 * V / (2**S + F)`. The 100 lets a score read to one decimal. 2**S is the cost. V sums, over the achieve goals an item's Moves cell names, each goal's weight times how far the item moves it, 1 to 10. A Moves column holds `1.1 by 8, 1.2 by 5`. An item that cites only maintain goals names the achieve goal it moves, or, as a defect restoring one, that maintain goal, as the defect paragraph below says.
2026-10-16, PR #283: items rank by `Score = 100 * V / (2**S + F)`. The 100 lets a score read to one decimal. 2**S is the cost. V sums, over the achieve goals an item's Moves cell names, each goal's weight times how far the item moves it, 1 to 10. A Moves column holds `1.1 by 8, 1.2 by 5`. An item names a maintain goal only as a defect restoring it, or by the confidence a test, gate leg or probe adds to it, as the two paragraphs below say.
2026-10-18, lilleman: `rank_todo.py` derives the goal weights from README.md. Each achieve goal, in reading order, weighs 0.97 times the one before it, and the weights sum to 1; `GOAL_STEP` sets the 0.97. lilleman wanted goals to weigh almost the same, since under the sum-reciprocal weight the goal ranked k of n achieve goals weighed (1/k) / (1 + 1/2 + … + 1/n), so the last of twelve goals weighed a twelfth of the first. The step departs from `docs/scoring-and-goals-research.md`, where sum-reciprocal weights came closest to the weights people gave.
@@ -478,6 +478,8 @@ Valid while the worker writes the tests that larv-review's reviewers hold to the
2026-10-16, lilleman, answering the review of PR #283. An item that restores a maintain goal the project breaks today is a defect: exempt as `defect`. Its Moves name that maintain goal and how much of it is broken, and the goal weighs as the heaviest achieve goal, since a broken maintain goal outranks any achieve goal. Every defect ranks by its score, so a trivial one waits behind valuable work. Only questions and then security findings rank first in their release: a question's answer may change other items, and a question waits for the human anyway, so ranking it first delays no chunk. An item names the most specific goal it moves, and a parent only for a move none of its sub-goals covers, so a sub-goal's move does not count twice; `rank_todo.py` refuses a goal named beside its own sub-goal. Each project reshapes its own `todo.md`; `CLAUDE.md` → "Working loop" makes the reshape the only work until done, as it makes writing missing premises. An item that makes this project easier to read moves code health, as the paragraph below says, and goal 1.4, since reading cost is measured in tokens per chunk.
2026-11-09, lilleman, answering the plan's question on maintain goals; claude proposed the numbers. An item that adds a test, a gate leg or a probe that fails where a maintain goal breaks may name that goal in Moves, by how much more confident the project becomes that the goal holds, 1 to 10, judged against the confidence it already has: 9 or 10 where nothing yet shows the goal holds, about 5 where a new thing is guarded in an area mostly covered, 1 where it only adds another way to check what is already checked. Without it, work that makes a held goal safer to rely on scored V 0 and ranked last; adf-codec's item 170, which bundles the package with four bundlers to back its goal 8, scored 0.0 beside items that move an achieve goal by 2. Such a goal weighs 0.5 per point, `MAINTAIN_WEIGHT` in `bin/release_plan/ranking.py`, not the heaviest achieve goal's weight, which suits a broken goal and is too much for a confidence gain, and which swings between about 0.8 and 0.2 as the overall crosses 7.0. The 0.5 comes from 150 distinct item and Moves pairs in 40 random `todo.md` commits from 2026-09-24 to 2026-11-09: the V their scores imply has quartiles 1.4, 2.6 and 4.5 and a median of 0.49 per point of a move, so a move of 10 is worth 5.0 (about the third quartile), 5 is worth 2.5 (the median) and 1 is worth 0.5 (under the tenth percentile, so only a cheap item ships). A defect that restores a broken maintain goal keeps the heaviest achieve goal's weight, and a maintain goal still vetoes an item that breaks it.
2026-10-18, lilleman. Serves goal 1.1: every project shares one achieve goal, code health, which `CLAUDE.md` → "Shared goal" states once and which replaces each project's own readability goal. It ranks first among a project's achieve goals, since lilleman rates it very important while the comprehension panel's overall is under 7.0. While that overall is under 7.0, or no scoring run has measured the project, it weighs four times its weight at 7.0 or above, as `UNDER_FLOOR` sets. The overall is the mean of the overall means of every scoring run on the newest scored commit in `docs/comprehension-history.md`, as `newest_commit_runs` finds them. Two runs of one head measure it twice, and the newest alone would let their noise flip the weight: 7.13 and 6.88 on one head of another project would have. The review of PR #324 raised this choice. Weighing code health four times as much leaves the other achieve goals' weights as they are. So crossing 7.0 changes only the scores of the items that move code health, and of the defects, since a broken maintain goal weighs as the heaviest achieve goal. `larv-review record-scoring` re-ranks `todo.md` after it appends a run. A project that opts out of the comprehension check voids the goal, as `review: a project decision may opt every chunk out …` records. A README voids the goal with `**Void:** code health, since <reason>.` under `## Goals`, and an item's Moves cell names it as in `code health by 6`. A panel of three readers seated as lilleman picked both 3-0: `code health` over `code`, `health`, `C` and `0`, and the bold `Void:` line over a sentence under `## Goals` or a line under `## Audience`. `score-outcomes` counts the counter-signal's misread findings per chunk, as `review: a reviewer under …` records. Valid while the comprehension panel's overall is how a project measures how hard it is to read.
2026-10-15, lilleman. Serves goal 1.4: loop-claude routes a chunk's first cycle on the item the chunk takes (`routing: a chunk's worker runs on the rung its item's L picks`). An item lists the items that must be done before it starts in a Blocked by column, and `rank_todo.py` ranks each blocker above every item it blocks. The table's order is `todo.md`'s only order: `rank_todo.py` fails on text beside the table under `## Items`, since a prose order there, "Release 0.3.0 takes items 125, 114 and 124 first", contradicted the table and loop-claude could not read it. A decision that wants an order sets the scores instead. An item waits while it is marked `question` or a blocker of it is still in the table, whatever the rows' order, so a release's first row that does not wait is the item a chunk takes. A blocker the table no longer holds was finished or deleted, and is dropped, so the cell shows what still blocks. A panel of three readers seated as lilleman picked the spelling: two ranked "Blocked by" first, as the cell lists only what still blocks, and all three chose bare IDs such as `12, 14` in a column right after Release, read before the tagline and kept apart from the scores. `rank_todo.py` adds the column to a table without it. The tools read cells by column name, so a later column leaves a reader on an older release working; `rank_todo.py`, which writes every column, fails on one it does not know rather than drop it.
+1 -1
View File
@@ -21,7 +21,7 @@ paths:
| 5 | — | — | irreversible: data lost for good, or a one-way door |
- **I's reach counts every project and session that runs the change.** A change to a hook or a tool that every session runs is I level 3, every consumer affected, even where one person uses it.
- **Name in Moves each achieve goal in `README.md` → `## Goals` the item moves, and how far,** from 1 to 10, judged by its value to the audience in `## Audience`: `1.1 by 8, 1.2 by 5`. Where the item moves code health, the goal in `CLAUDE.md` → "Shared goal", name it as in `code health by 6`, unless the README voids it. Name the most specific goal it moves, and a parent goal only for a move none of its sub-goals covers, never beside its own sub-goal. Leave out every maintain goal, which vetoes, except on an item marked `defect` that restores it: name that goal, and how much of it is broken. `rank_todo.py --help` says how it weighs each goal.
- **Name in Moves each achieve goal in `README.md` → `## Goals` the item moves, and how far,** from 1 to 10, judged by its value to the audience in `## Audience`: `1.1 by 8, 1.2 by 5`. Where the item moves code health, the goal in `CLAUDE.md` → "Shared goal", name it as in `code health by 6`, unless the README voids it. Name the most specific goal it moves, and a parent goal only for a move none of its sub-goals covers, never beside its own sub-goal. Name a maintain goal only where the item adds a test, a gate leg or a probe that fails where the goal breaks, or is a `defect` that restores it. For the test, gate leg or probe, the move is how much more confident the project becomes that the goal holds, judged against the confidence it has: 9 or 10 where nothing yet shows the goal holds, about 5 where a new thing is guarded in an area mostly covered, 1 where it only adds another way to check what is already checked. For the defect, it is how much of the goal is broken. A change that only claims the goal moves nothing. `rank_todo.py --help` says how it weighs each goal.
- **A table with C, A, G and Goals in place of L, E, I and Moves is in the old shape.** Reshaping it is the project's only work until done, as `CLAUDE.md` → "Working loop" says, and `rank_todo.py` says how. Until then, A is the item's value to the audience and G its value to the goals it cites.
- **Score L, E and I against the project's anchors, and check every score against finished chunks.** Guess the scores, then run `~/.claude/bin/score-outcomes <L> <E> <I> <S>` in the project. It lists the finished items guessed nearest and the project's anchors, the shipped items its `docs/score-anchors.md` names for each level, each with what its chunk measured; then the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings, L by the share of chunks with a 🔴 instead where most hold none. Compare the item with each anchor in turn: is it more or less likely to hold a defect, to let one escape, or to do damage? Give it the level whose anchors it matches; where a level has no anchor, judge by the level's words in the table above. Set S to the value guessed for the finished items whose outcome matches what you expect of this one. Keep your guess when no listed item resembles the one you are scoring.
- **Anchor each level of L, E and I with two or three items the project shipped,** in `docs/score-anchors.md`: a table whose Level column names a level, such as `L 1`, and whose Items column lists the anchors' IDs, such as `12, 14`. Choose an L anchor whose chunk, as `score-outcomes` measured it, fits the level's words: its lines changed and the components it touched. Choose an E anchor by the level's words and the later fixes `score-outcomes` counts for its chunk, the defects that escaped it. Choose an I anchor by the level's words and what followed its release, since no tool measures I yet. Leave out a level no shipped item shows. Replace a level's anchors when the items scored at it turn out about ten times more likely, or ten times less likely, than its words say: to hold a defect, to let one escape, or to do damage.
-23
View File
@@ -15,7 +15,6 @@
| 277 | 0.7.0 | | question | **Map the size of the code an item names onto S, on a scale the human picks, and have `score-outcomes` print that S for the item being scored and for each finished item it lists.** | 3 | 2 | 2 | 5 | 4 | 1.1 by 4, 1.4 by 3 | 2.4 |
| 1 | 0.7.0 | | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 1 | 2 | 1 | 2 | 1 | | 0.0 |
| 281 | 0.7.0 | | | **Remove a merged chunk's worktree and branch and fast-forward the main checkout with one command.** | 3 | 2 | 3 | 6 | 3 | 1.4 by 2, 1.5 by 2 | 1.6 |
| 274 | 0.7.0 | | | **Let an item name a maintain goal it raises confidence in, such as a test leg that guards it, and weigh that move against the achieve goals in `bin/release_plan/ranking.py`.** | 2 | 2 | 3 | 5 | 4 | 1 by 3 | 1.1 |
| 286 | 0.7.0 | | | **Lower the reading cost of the files a chunk reads most before its first commit.** | 3 | 2 | 2 | 5 | 5 | 1.4 by 4 | 1.0 |
| 284 | 0.8.0 | | owner | **Script each step that 0.7.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.6 |
| 77 | 0.8.0 | | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 4 | 2 | 1 | 5 | 6 | 1.1 by 7, 1.4 by 7 | 2.4 |
@@ -74,28 +73,6 @@ The README line covers the bench's transcripts in `workflow-bench/logs/` too (go
Audit of release 0.2.0's chunks (item 162): 31 chunks ran `git worktree remove`, `git branch -d` and `git pull --ff-only` by hand, 56 calls and about 5k command tokens. The program refuses a dirty main checkout, as `CLAUDE.md` → "Working with the codebase" says, and reports it.
### 274. Let an item name a maintain goal it raises confidence in, such as a test leg that guards it, and weigh that move against the achieve goals in `bin/release_plan/ranking.py`.
Today only a `defect` item may name a maintain goal, and only to restore one the project breaks
(`docs/decisions.md` → the `todo.md` entry of 2026-10-16). A maintain goal otherwise adds no value to
any item, so work that makes a held goal safer to rely on scores V 0 and ranks last: adf-codec's
item 170, which bundles the package with four bundlers to back its goal 8, runs on any JavaScript
engine, scores 0.0 beside items that move an achieve goal by 2.
Settle with the maintainer, and record in `docs/decisions.md`:
- How an item says it: a Moves entry naming the maintain goal with how far the item raises
confidence in it, 1 to 10, or a separate column.
- What it weighs. The maintain goal's weight now is the heaviest achieve goal's, which suits a broken
goal and is too much for a confidence gain; it needs its own factor, which the item measures or
asks for.
- Which items count: a test, a gate leg or a probe that would fail where the goal breaks. A change
that only claims the goal does not.
Then change `bin/release_plan/ranking.py` and its tests, `rank_todo.py --help`, and `rules/todo-items.md`'s sentences on
maintain goals and Moves. The decision of 2026-10-16 keeps its veto: an item that breaks a maintain
goal is still redesigned or dropped.
### 286. Lower the reading cost of the files a chunk reads most before its first commit.
`reading-cost` reports Rising: the last five chunks' median is 151,930 new input tokens before the first commit, against 103,215 for the five before them. Those five chunks opened or searched these files most: `todo.md` 42 times, `README.md` 16, `docs/decisions.md` 12, `AGENTS.md` 10, `bin/rank_todo.py` 9, `bin/release_plan/ranking.py` 8, `docs/reading-cost.md` 8, `CHANGELOG.md` 7, `rules/todo-items.md` 7 and `bin/rule_hook/test_rule_hook.py` 6.