Anchor each level of L, E and I with shipped items, and have score-outcomes list them for a scorer to compare against

This commit is contained in:
2026-10-16 19:37:55 +02:00
parent 2c67b4f2b3
commit 5d4f9d4c0b
12 changed files with 151 additions and 13 deletions
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- `score-outcomes` lists a project's score anchors: the shipped items its `docs/score-anchors.md` names for each level of L, E and I, each with what its chunk measured. `rules/todo-items.md` has a scorer compare a new item with them, and says how to choose two or three per level. Add the file to each project that has shipped items; until then, score by the levels' words.
- Every goal in `README.md` → Goals is marked `**Maintain:**`, a state to hold, or `**Achieve:**`, a direction to pursue. Each achieve goal has a signal, a counter-signal and a one-line reason under it. A change that breaks a maintain goal is redesigned or dropped, whatever it scores: `rules/todo-items.md` says so, and larv-review's goal alignment step no longer merges such a PR. `rules/goals-and-audience.md` states the form for every project's README. Expect larv-review to stop a PR on a maintain goal where it asked you before.
- Reshape every project's `todo.md` to the new shape below: until it is reshaped, that is the project's only work, as it is while README.md lacks Goals and Audience, and `rank_todo.py` says so. First move every machine to this release: an older release rejects the new formula, so it fails on every reshaped `todo.md`. Then mark the README's goals `**Maintain:**` or `**Achieve:**`, score each item's L, E and I, and set a bar on the new scale below the score of every item to keep, since the old bar would drop most items. Until then, an old-shape `todo.md` still ranks by its own formula and weights.
- `todo.md` ranks items by value per cost: this setup's own plan uses `Score = 100 * V / (2**S + 8)`, where 8 is a chunk's fixed cost, about one size-3 change, and `rank_todo.py --help` shows it. A Moves column replaces A, G and Goals, naming each achieve goal an item moves and how far, such as `1.1 by 8, 1.2 by 5`. V sums each named goal's weight times its move. `rank_todo.py` weighs each achieve goal by its position in README.md, the first heaviest, so the goal weights table goes. An item names the most specific goal it moves, never a goal beside its own sub-goal. `rules/todo-items.md` and `rank_todo.py --help` show the new shape.
+1 -1
View File
@@ -89,7 +89,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format.
- `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure.
- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit, Write and Agent call, and `rule-hook --help` states the rules and the override.
- `score-outcomes [L E I S]` lists the finished items whose guessed scores sit nearest the given ones, with what their chunks measured, the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings; `score-outcomes --help` states the measures.
- `score-outcomes [L E I S]` lists the finished items whose guessed scores sit nearest the given ones, with what their chunks measured, the project's anchors for each level of L, E and I, the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings; `score-outcomes --help` states the measures.
- `stability-replay <model>` replays archived stability reviews of the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again; `stability-replay --help` states the replay. Run it only in a sandbox, like loop-claude: each replayed reviewer has Bash and skips every permission prompt, and an archived prompt may name paths on the host.
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
- `subagent-rung [--plan] <model> <L>` prints the Agent tool's model and effort for a subagent's task, picked by its L from the project's model, and with `--plan` never below the project's rung; `subagent-rung --help` states the rule.
+1 -1
View File
@@ -43,7 +43,7 @@ A valid todo.md; ## Details and each item's detail are optional:
goal.
- The bar is judged on the Score as shown, to one decimal.
- A new item takes Next ID, and Next ID then goes up by one; every ID stays below it.
- An item's L, E and I are levels that ~/.claude/rules/todo-items.md anchors: L from 1 to 4, E from 1
- An item's L, E and I are levels that ~/.claude/rules/todo-items.md defines: L from 1 to 4, E from 1
to 3 and I from 1 to 5.
rank_todo.py writes its R, L + E + I - 2, at least 7 when I is 5 and at least 5 when I is 4. Its S
is a whole number from 1 to 10.
+39
View File
@@ -0,0 +1,39 @@
import re
from failure import Failure
from todo_table import LEVELS
from score_outcomes.outcomes import Outcome, render
# Where a project names the shipped items that show each level of L, E and I.
ANCHORS = "docs/score-anchors.md"
# One row of its table: a level, such as `L 1`, and the IDs of the items that show it.
ROW = re.compile(r"\|\s*([A-Z]) ([0-9]+)\s*\|\s*([0-9]+(?:\s*,\s*[0-9]+)*)\s*\|")
# The table's header and separator rows.
FRAME = re.compile(r"\|\s*Level\s*\|\s*Items\s*\||\|[-|\s]+\|")
def anchors(text: str) -> list[tuple[str, tuple[int, ...]]] | Failure:
"""Each level the anchors table names, such as `L 1`, with its items' IDs, in the file's order."""
found: list[tuple[str, tuple[int, ...]]] = []
for row in (line.strip() for line in text.splitlines() if line.strip().startswith("|")):
if FRAME.fullmatch(row):
continue
matched = ROW.fullmatch(row)
if not matched or not 1 <= int(matched[2]) <= LEVELS.get(matched[1], 0):
scales = ", ".join(f"{name} 1 to {top}" for name, top in LEVELS.items())
return Failure(f"{ANCHORS} cannot read the row {row}; each row names a level, {scales}, and the IDs of shipped items, such as | L 1 | 12, 14 |")
level = f"{matched[1]} {matched[2]}"
if level in dict(found):
return Failure(f"{ANCHORS} names {level} twice")
found.append((level, tuple(int(id_) for id_ in matched[3].split(","))))
return found
def anchor_lines(found: list[tuple[str, tuple[int, ...]]], outcomes: list[Outcome]) -> list[str]:
"""Each level, then each of its anchors with what its chunk measured."""
by_id = {outcome.item.id: outcome for outcome in outcomes}
lines = []
for level, ids in found:
lines.append(f"{level}:")
lines += [render(by_id[id_]) if id_ in by_id else f"- Item {id_}: no merge commit on the default branch shipped it." for id_ in ids]
return lines
+28 -2
View File
@@ -23,12 +23,17 @@ its table also had A, G and Goals. The chunk measures:
A project that merges without merge commits, by fast-forward or squash, has no chunk to measure.
Given L, E and I, on the levels ~/.claude/rules/todo-items.md anchors, and S, it works out R from
Given L, E and I, on the levels ~/.claude/rules/todo-items.md defines, and S, it works out R from
L, E and I, as rank_todo.py does, and lists the 5 finished items whose guesses sit closest, closest
first: the least mean squared difference over the scores both hold, so R and S alone for an item
scored before L, E and I, the newest first among equals. Each line gives what the item's chunk
measured, which items shipped with it, and its tagline.
Then the project's anchors, the shipped items its docs/score-anchors.md names for each level of L,
E and I, read from the checkout's working tree, each line as above, or that no merge shipped it. Exit 1
on a row that names no level on its scale, a level named twice, or an Items cell that is not a list
of IDs.
Then the median of the outcome R and S each predict at each value they were guessed at: R against
🔴 findings, S against lines changed. And how well L, E and I each order chunks by review rounds
past each pass's first and by 🔴 findings: Spearman's rank correlation, from -1 to +1, where +1
@@ -39,10 +44,12 @@ hold it and its outcomes.
Exit 1, saying why, on bad arguments, when git cannot start, finds no checkout or reads a shallow
clone, or the default branch cannot be read.
"""
import pathlib
import re
import sys
from git_repo import default_branch, file_at, full_history, git
from score_outcomes.anchors import anchor_lines, ANCHORS, anchors
from git_repo import default_branch, file_at, full_history, git, top
from failure import Failure
from todo_table import LEVELS, risk, TOP_SCORE
from markdown_text import listed
@@ -70,12 +77,18 @@ def main(argv: list[str]) -> int:
outcomes = read_outcomes(default)
if isinstance(outcomes, Failure):
return failed(outcomes)
anchored = read_anchors()
if isinstance(anchored, Failure):
return failed(anchored)
if argv:
given = dict(zip(GIVEN, map(int, argv)))
given["R"] = risk(given["L"], given["E"], given["I"])
print(f"Finished items nearest L {given['L']}, E {given['E']}, I {given['I']}, so R {given['R']}, and S {given['S']}, closest first:\n")
print("\n".join(render(outcome) for outcome in nearest(outcomes, given)) or "- None: no merge commit on the default branch shipped an item from todo.md. A project that merges by squash or fast-forward has none.")
print()
print(f"Anchors in {ANCHORS}, each level's shipped items:\n")
print("\n".join(anchor_lines(anchored, outcomes)) or f"- None: the project's {ANCHORS} names no level; ~/.claude/rules/todo-items.md says how to choose them.")
print()
print("Median outcome at each guessed score, among chunks that shipped one item:\n")
print("\n".join(report(outcomes)))
return 0
@@ -86,6 +99,19 @@ def failed(failure: Failure) -> int:
return 1
def read_anchors() -> list[tuple[str, tuple[int, ...]]] | Failure:
"""The checkout's anchors as its working tree holds them, so a branch that edits them shows its edit; none where it has no such file."""
root = top()
if isinstance(root, Failure):
return root
try:
return anchors((pathlib.Path(root) / ANCHORS).read_text())
except FileNotFoundError:
return []
except (OSError, UnicodeDecodeError) as failure:
return Failure(f"cannot read {ANCHORS}: {failure}")
def read_outcomes(default: str) -> list[Outcome] | Failure:
if partial := full_history():
return partial
+43
View File
@@ -7,6 +7,8 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from score_outcomes.anchors import anchor_lines, anchors
from failure import Failure
from test_env import isolated_env
from todo_table import Item
from score_outcomes.outcomes import nearest, Outcome, outcomes_of, rank_correlation, render, report, shipped
@@ -119,6 +121,23 @@ class Render(unittest.TestCase):
self.assertIn("1 round past each pass's first, 0 🔴, 0 🟡; merged 2026-10-13T10:00:00Z in aaaaaaa, with item 5. ", render(outcome(4, 2, 3, 4, rounds=1, shipped_with=(5,))))
ANCHORS = "# Score anchors\n\nWhy these.\n\n| Level | Items |\n|---|---|\n| L 1 | 4, 9 |\n| I 3 | 5 |\n"
class Anchors(unittest.TestCase):
def test_reads_each_levels_items_in_the_files_order(self):
self.assertEqual(anchors(ANCHORS), [("L 1", (4, 9)), ("I 3", (5,))])
def test_refuses_a_level_off_its_scale_a_score_without_levels_a_repeated_level_or_an_unreadable_row(self):
for row in ("| L 5 | 4 |", "| E 0 | 4 |", "| S 3 | 4 |", "| L 1 | 7 |", "| L 2 | four |", "| L 2 | |"):
with self.subTest(row):
self.assertIsInstance(anchors(ANCHORS + row + "\n"), Failure)
def test_lists_each_anchor_with_what_its_chunk_measured_and_says_which_it_cannot_find(self):
lines = anchor_lines([("L 1", (4, 9))], [outcome(4, 2, 3, None, findings=(1, 2))])
self.assertEqual(lines, ["L 1:", "- Item 4, R 2, S 3: 10 lines in 1 file, 1 round past each pass's first, 1 🔴, 2 🟡; merged 2026-10-13T10:00:00Z in aaaaaaa. **Item 4.**", "- Item 9: no merge commit on the default branch shipped it."])
class ScoreOutcomesTool(unittest.TestCase):
def setUp(self):
temp = tempfile.TemporaryDirectory()
@@ -155,6 +174,30 @@ class ScoreOutcomesTool(unittest.TestCase):
self.assertIn("- Item 4, R 2, S 3, C 10: 3 lines in 1 file, 0 rounds past each pass's first, 1 🔴, 2 🟡; merged ", ran.stdout)
self.assertNotIn("Item 5", ran.stdout)
def test_lists_the_projects_anchors_with_what_their_chunks_measured(self):
self.git("switch", "-q", "-c", "chunk")
self.commit("Do it", {"a.py": "1\n", "todo.md": todo(SECOND)})
self.git("switch", "-q", "main")
self.git("merge", "-q", "--no-ff", "-m", "Merge chunk", "chunk")
(self.checkout / "docs").mkdir()
(self.checkout / "docs" / "score-anchors.md").write_text(ANCHORS)
ran = self.run_tool("1", "2", "1", "3")
self.assertEqual(ran.returncode, 0, ran.stderr)
self.assertIn("Anchors in docs/score-anchors.md, each level's shipped items:\n\nL 1:\n- Item 4, R 2, S 3, C 10: 1 line in 1 file", ran.stdout)
self.assertIn("I 3:\n- Item 5: no merge commit on the default branch shipped it.", ran.stdout)
def test_says_where_anchors_go_when_the_project_has_none(self):
ran = self.run_tool()
self.assertEqual(ran.returncode, 0, ran.stderr)
self.assertIn("- None: the project's docs/score-anchors.md names no level; ~/.claude/rules/todo-items.md says how to choose them.", ran.stdout)
def test_refuses_anchors_it_cannot_read(self):
(self.checkout / "docs").mkdir()
(self.checkout / "docs" / "score-anchors.md").write_text(ANCHORS + "| L 9 | 4 |\n")
ran = self.run_tool()
self.assertEqual(ran.returncode, 1)
self.assertIn("docs/score-anchors.md", ran.stderr)
def test_takes_each_score_at_the_top_of_its_scale(self):
ran = self.run_tool("4", "3", "5", "10")
self.assertEqual(ran.returncode, 0, ran.stderr)
+1 -1
View File
@@ -4,7 +4,7 @@ import typing
from failure import Failure
# Each score a Shape names, and each goal's move, runs from 1 to TOP_SCORE, but for the levels L, E and I, which run from 1 to their LEVELS: ~/.claude/rules/todo-items.md anchors each level.
# Each score a Shape names, and each goal's move, runs from 1 to TOP_SCORE, but for the levels L, E and I, which run from 1 to their LEVELS: ~/.claude/rules/todo-items.md defines each level.
TOP_SCORE = 10
LEVELS = {"L": 4, "E": 3, "I": 5}
# The least R each impact level I allows, since a severe enough consequence outranks how unlikely it is.
+1 -1
View File
@@ -130,7 +130,7 @@ Valid while the ladder orders its rungs by competence.
2026-10-02, lilleman. Serves goal 1: findings were filed faster than chunks closed them, so the plan grew without telling what mattered. Each item is scored as `~/.claude/rules/todo-items.md` says; this repo's formula counts value highest and size least. An item under the bar is not kept; this repo's bar started at 10 and was set to 9 the same day, purging the 17 items under it. An Exempt cell keeps a row past the bar where a rule needs it kept. lilleman retired the two decisions whose plans the purge removed: the stats line in the transcript archive, and three benches before a workflow change merges. A finding under the bar is recorded nowhere, so a later review may raise it again, and the run rescores it. That costs one scoring; a ledger of dropped findings would regrow the list the bar caps. larv-review's "Round cap" → "Done" counts a finding named under the bar as settled, so one raised again holds no pass open. `rank_todo.py --check` is part of larv-review's gate, since only a check keeps the stored Score from going stale. The format is mandatory in every project, as the premises are (goal 1.3); the formula and bar are each project's own. A script ranks the table so a re-rank after each addition or edit is deterministic and cheap. A 🟡 against a technical principle is exempt as `principle`, because its audience value scores low and the bar would drop every one. A 🟢 against a principle is scored like any other finding. 2026-10-08, lilleman: the goal weights in `todo.md` depart from an even spread by position, which would weigh goal 4, leaking nothing, at 0.11. Valid while one human sets the bar and reads the ranking, and while the items marked `principle` stay few enough to rank.
2026-10-15, lilleman: the scores get a researched method, which `todo.md` items 149, 151 and 152 build, after PR #283 shipped the ranking and PR #296 the risk scores; the evidence is in `docs/scoring-and-goals-research.md`. R becomes L + E + I − 2: the chance the change holds a defect, set first by its size; the chance the defect escapes tests, review and monitoring; and the worst impact dimension, each about ×10 per level. I5, irreversible, holds R at 7 or more, and I4 at 5 or more, since a severe enough consequence outranks how unlikely it is. C goes from every table and tool: what it scored beyond size, spread and novelty, which L already holds, is complexity, which adds almost nothing once size is accounted for. Items rank by value per cost, V / 2^S, and V sums each cited achieve goal's sum-reciprocal weight times how far the item moves it. A goes, and the Audience says whose value the goals serve. R stays out of the ranking and sets how carefully an item is done; an item that lowers risk scores it as value. Each release fills by priority up to a capacity it states, in place of the bar. Each README goal is `maintain`, which vetoes an item that breaks it, or `achieve`, with a signal, a counter-signal and a one-line reason, as this README's Goals now mark them; each project keeps settled decisions to test its goals against. Each part of this method holds once the item that builds it ships. 2026-10-16, lilleman: a task's L and E + I pick its rungs in place of R, as `routing: one ladder` records.
2026-10-15, lilleman: the scores get a researched method, which `todo.md` items 149 and 151 build, after PR #283 shipped the ranking, PR #296 the risk scores and the PR shipping `docs/score-anchors.md` each level's anchors; the evidence is in `docs/scoring-and-goals-research.md`. R becomes L + E + I − 2: the chance the change holds a defect, set first by its size; the chance the defect escapes tests, review and monitoring; and the worst impact dimension, each about ×10 per level. I5, irreversible, holds R at 7 or more, and I4 at 5 or more, since a severe enough consequence outranks how unlikely it is. C goes from every table and tool: what it scored beyond size, spread and novelty, which L already holds, is complexity, which adds almost nothing once size is accounted for. Items rank by value per cost, V / 2^S, and V sums each cited achieve goal's sum-reciprocal weight times how far the item moves it. A goes, and the Audience says whose value the goals serve. R stays out of the ranking and sets how carefully an item is done; an item that lowers risk scores it as value. Each release fills by priority up to a capacity it states, in place of the bar. Each README goal is `maintain`, which vetoes an item that breaks it, or `achieve`, with a signal, a counter-signal and a one-line reason, as this README's Goals now mark them; each project keeps settled decisions to test its goals against. Each part of this method holds once the item that builds it ships. 2026-10-16, lilleman: a task's L and E + I pick its rungs in place of R, as `routing: one ladder` records.
2026-10-16, PR #283: items rank by `Score = 100 * V / 2**S`. 2**S is the cost; the 100 lets a score read to one decimal. V sums, over the achieve goals an item's Moves cell names, each goal's weight times how far the item moves it, 1 to 10. The goal ranked k of n achieve goals in reading order weighs (1/k) / (1 + 1/2 + … + 1/n), the sum-reciprocal weight. A Moves column replaces A, G and Goals, and `rank_todo.py` derives the weights from README.md, so the goal weights table and the 2026-10-08 weights are gone. An item that cited only maintain goals now names the achieve goal it moves, or, as a defect restoring one, that maintain goal, as the entry below says. A panel of three readers seated as lilleman picked the spelling: two put a Moves column holding `1.1 by 8, 1.2 by 5` first, and one put `1.1: 8` first; two chose `2**S`, since a Python reader takes `^` for XOR; all three chose one decimal of `100 * V / 2**S` over three significant digits or a log. The bar moved to 0.1, lilleman's call answering the review of PR #283: on one-decimal scores it drops an item scoring under 0.05, in practice one that moves no achieve goal, and item 149 revisits it. A table in the old shape still ranks by the formula and weights it states, and ranking it says to reshape it, so no other project's commits stop; reshaping it is that project's only work until done, as `CLAUDE.md` → "Working loop" says. PR #296 dropped the C column, and put L, E and I right before R, which `rank_todo.py` writes from them: of three readers seated as lilleman, two ranked that order first, since a reader meets each level before the R it builds and R sits beside S, and the third ranked it second, after an order with S before the levels. The old shape's ranking is valid until `todo.md` item 156.
+17
View File
@@ -520,3 +520,20 @@ Read most (opens and searches):
- `bin/rule_hook/rules.py` 3
- `AGENTS.md` 2
- `bin/loop_claude/cycle/worker.py` 2
## Reading cost, chunk eded4db7f44b, finished 2026-10-16T17:29:09Z
356291 new input tokens before the first commit.
Read most (opens and searches):
- `todo.md` 12
- `bin/loop_claude/cycle/judge.py` 5
- `bin/loop_claude/test_loop_claude.py` 5
- `bin/judge_replay/test_judge_replay.py` 4
- `bin/loop_claude/test_black_box.py` 4
- `bin/test_rank_todo.py` 4
- `bin/todo_table.py` 4
- `README.md` 3
- `bin/loop_claude/loop/main.py` 3
- `bin/rank_todo.py` 3
+16
View File
@@ -0,0 +1,16 @@
# Score anchors
The items this repo shipped that show each level of L, E and I, which `rules/todo-items.md` defines. `bin/score-outcomes` lists each with what its chunk measured, for a scorer to compare a new item against. No item yet shows I 4 or I 5.
| Level | Items |
|---|---|
| L 1 | 115, 67, 140 |
| L 2 | 141, 144, 97 |
| L 3 | 163, 143, 136 |
| L 4 | 148, 146, 96 |
| E 1 | 163, 148 |
| E 2 | 67, 140, 141 |
| E 3 | 71, 64 |
| I 1 | 115, 140 |
| I 2 | 58, 107 |
| I 3 | 163, 96, 148 |
+2 -1
View File
@@ -22,7 +22,8 @@ paths:
- **I's reach counts every project and session that runs the change.** A change to a hook or a tool that every session runs is I level 3, every consumer affected, even where one person uses it.
- **Name in Moves each achieve goal in `README.md` → `## Goals` the item moves, and how far,** from 1 to 10, judged by its value to the audience in `## Audience`: `1.1 by 8, 1.2 by 5`. Name the most specific goal it moves, and a parent goal only for a move none of its sub-goals covers, never beside its own sub-goal. Leave out every maintain goal, which vetoes, except on an item marked `defect` that restores it: name that goal, and how much of it is broken. `rank_todo.py --help` says how it weighs each goal.
- **A table with C, A, G and Goals in place of L, E, I and Moves is in the old shape.** Reshaping it is the project's only work until done, as `CLAUDE.md` → "Working loop" says, and `rank_todo.py` says how. Until then, A is the item's value to the audience and G its value to the goals it cites.
- **Check L, E, I and S against finished chunks.** Guess them, then run `~/.claude/bin/score-outcomes <L> <E> <I> <S>` in the project. It lists the finished items guessed nearest, what their chunks measured, the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings. Set each score to the value guessed for the finished items whose outcome matches what you expect of this one. Keep your guess when no listed item resembles the one you are scoring.
- **Score L, E and I against the project's anchors, and check every score against finished chunks.** Guess the scores, then run `~/.claude/bin/score-outcomes <L> <E> <I> <S>` in the project. It lists the finished items guessed nearest, the project's anchors, each with what its chunk measured, the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings. Compare the item with each anchor in turn: is it more or less likely to hold a defect, to let one escape, or to do damage? Give it the level whose anchors it matches; where a level has no anchor, judge by the table's words. Set S to the value guessed for the finished items whose outcome matches what you expect of this one. Keep your guess when no listed item resembles the one you are scoring.
- **Anchor each level of L, E and I with two or three items the project shipped,** in `docs/score-anchors.md`: a table whose Level column names a level, such as `L 1`, and whose Items column lists the anchors' IDs, such as `12, 14`. Choose each from what `score-outcomes` measured, whose outcome fits the level's words. Leave out a level no shipped item shows. When items scored at a level turn out about ten times more or less likely to fail, or to do damage, than its words say, replace that level's anchors.
- **Read the places the comprehension seats found hard before filing an item.** `~/.claude/bin/larv-review places`, run in the project, prints them. Where the item's change touches one, its detail says to make that place easier to read.
- **An item scoring under the bar is deleted, with every item that needs it, and a finding under the bar is not filed.** The chunk's summary names each, with its score.
- **The Exempt cell keeps an item whatever its score:** `security` for a security finding, `defect` for a 🔴 finding or an item that restores a maintain goal the project breaks today, `question` for a question only the human can answer, which ranks first in its release, `decision` for an item `docs/decisions.md` cites, `owner` for an item the human told you to keep, and, where no other reason fits, `principle` for a 🟡 finding against a file under `~/.claude/principles/`, whose item's detail names that principle. Every item `docs/decisions.md` cites carries one of these reasons.
+1 -6
View File
@@ -12,7 +12,6 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 152 | 0.2.0 | | decision | **Anchor each score level in `rules/todo-items.md` with words, and each project's with two or three items it shipped, and have a scorer compare a new item against them.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 7 | 6.4 |
| 50 | 0.2.0 | | principle | **Grant the bench's runs only the tools a review needs, in place of skipping every permission.** | 2 | 2 | 1 | 3 | 2 | 1.1 by 3 | 5.5 |
| 65 | 0.2.0 | | defect | **Archive each judge reply the loop rejects.** | 2 | 1 | 3 | 4 | 4 | 3 by 3 | 5.5 |
| 153 | 0.2.0 | | decision | **Route the judge and the reviewers on the highest E + I in their chunk, an item the worker takes in included.** | 3 | 1 | 3 | 5 | 4 | 1.2 by 5 | 3.0 |
@@ -85,10 +84,6 @@
## Details
### 152. Anchor each score level in `rules/todo-items.md` with words, and each project's with two or three items it shipped, and have a scorer compare a new item against them.
LLM ratings drift to the middle of an unanchored scale, and worked examples cut estimation error. A project's anchors are shipped items, cited by ID and commit, chosen from what `score-outcomes` measured for them.
### 50. Grant the bench's runs only the tools a review needs, in place of skipping every permission.
"Always grant explicit access to data", in `principles/technical-principles.md`: `workflow-bench/bench.py` starts claude with `--dangerously-skip-permissions`, and no rule confines it to a sandbox, so a run on the host reaches the host's credentials and other checkouts.
@@ -370,7 +365,7 @@ While `rank_todo.py` ranks two shapes, projects follow two sets of scoring rules
### 155. Lower the reading cost of the files a chunk reads most before its first commit.
`bin/reading-cost` on 2026-10-16: the last 5 chunks' median is 189873 new input tokens before their first commit, the 5 before them 90662. Read most over the last 5, by opens and searches: `todo.md` 56, `docs/decisions.md` 19, `CHANGELOG.md` 13, `README.md` 13, `bin/test_rank_todo.py` 10, `AGENTS.md` 8, `skills/larv-review/SKILL.md` 7, `agents/stability-reviewer.md` 6, `bin/loop_claude/test_loop_claude.py` 6, `bin/rank_todo.py` 6. The chunk that ships this item runs `bin/reading-cost --reset`.
`bin/reading-cost` on 2026-10-16: the last 5 chunks' median is 213946 new input tokens before their first commit, the 5 before them 133254. Read most over the last 5, by opens and searches: `todo.md` 61, `bin/test_rank_todo.py` 14, `docs/decisions.md` 14, `README.md` 11, `bin/loop_claude/test_loop_claude.py` 11, `CHANGELOG.md` 10, `AGENTS.md` 7, `bin/rank_todo.py` 7, `bin/todo_table.py` 7, `agents/stability-reviewer.md` 6. The chunk that ships this item runs `bin/reading-cost --reset`.
### 1. Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.