Score each item's complexity, and calibrate R, S and C against what finished chunks measured

This commit is contained in:
2026-10-13 11:19:30 +02:00
parent 445f47d5bc
commit 064bfe78f4
16 changed files with 463 additions and 87 deletions
+3
View File
@@ -44,6 +44,9 @@
!bin/rule-hook
!bin/rule_hook/
!bin/rule_hook/*.py
!bin/score-outcomes
!bin/score_outcomes/
!bin/score_outcomes/*.py
!bin/shell_command.py
!bin/statusline.py
!bin/test_*.py
+2
View File
@@ -2,6 +2,8 @@
## Unreleased
- Every `todo.md` item scores C, how hard the change is to get right, from 1 to 10, in a column after S. C stays out of the score. `rank_todo.py` fails on an items table without the column, and the rule hook then blocks committing that `todo.md`. Add the column to each project's `todo.md` and score every item's C, as `rank_todo.py --help` shows.
- The new `bin/score-outcomes` shows what finished chunks measured beside the R, S and C their items were guessed at: lines and files changed, review rounds, and 🔴 and 🟡 findings. It reads them from the default branch's merge commits, so a project that merges by squash or fast-forward shows none. Given a guess, it lists the 5 finished items guessed nearest, and for each score the median outcome at each guessed value, once 10 chunks of one item hold it. A session filing or editing an item runs it and scores R, S and C from what it shows. larv-review titles each round's commit with the round's 🔴 and 🟡 counts, `Answer stability round 1: 1 🔴, 2 🟡; …`, so the findings count from now on.
- The stability reviewer mutation-tests the lines each change adds or modifies: it makes small changes to that code, such as flipping a comparison. For each test to add or strengthen, it reports one 🟡 naming the changes that test would catch. It uses the project's documented mutation command, or else cargo-mutants, Infection, mutmut or Stryker. The technical principles replace "Test coverage should not decline" with "Every mutant of a changed line fails a test, unless the mutant leaves behaviour unchanged", and say what a valuable test is: it checks behaviour through the unit's interface, and fails when that behaviour breaks. README → "Tools" gives the mutmut command for `bin/`. Expect longer stability passes and more 🟡s about tests. In a project whose README documents no mutation command, the reviewer picks and installs a tool on each run; document one there to make the runs consistent.
- Each new `loop-claude` worker session's prompt now quotes the README's Goals and Audience. A line above them says that the chunk's `todo.md` items, or else the task, set its scope, and that the goals only choose how to deliver it. A session outside the loop reads Goals and Audience before starting a chunk, as `CLAUDE.md` now says.
- A session no longer reads the technical principles before writing code. It writes the simplest change that solves its item, test first wherever a test can check the change, and reads a principle only when the item changes it, a finding names it, or a decision or `todo.md` item it writes cites it. larv-review's reviewers hold the change to the principles, and a finding against one names it. Expect more findings from the prose and stability reviewers, since sessions no longer load the rules for comments and READMEs.
+2 -1
View File
@@ -72,6 +72,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
- `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format.
- `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure.
- `score-outcomes [R S C]` lists the finished items whose guessed R, S and C sit nearest the given scores, with what their chunks measured, and each score's median outcome; `score-outcomes --help` states the measures.
- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit and Write call, and `rule-hook --help` states the rules and the override.
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
@@ -90,7 +91,7 @@ pytest_add_cli_args=-k
Then, in the copy: `uvx --with pytest 'mutmut>=3,<4' run && uvx --with pytest 'mutmut>=3,<4' results`, and `uvx --with pytest 'mutmut>=3,<4' show <mutant>` to see a mutant's change. The config keys are mutmut 3's.
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, and `rule_hook` imports `rank_todo.py`'s check. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests; `rank_todo.py` and `statusline.py` are tools of one file each, tested beside them, `rule_hook` imports `rank_todo.py`'s check, and `score_outcomes` reads git and round titles through `larv_review`'s modules. The other non-test modules at `bin/`'s root, `claude_code/` and `transcript_archive/` serve more than one tool. `test_env.py` serves the tests, and `repo_checks/` tests the repo itself.
## Tracking a new file
+5 -5
View File
@@ -35,11 +35,11 @@ def said(overall: str, locality: str = "same") -> str:
def item(id_: int, release: tuple[int, int, int], tagline: str = "**Do it.**") -> Item:
return Item(id_, release, "", tagline, {"R": 1, "S": 1, "A": 1, "G": 1}, ("1",))
return Item(id_, release, "", tagline, {"R": 1, "S": 1, "C": 3, "A": 1, "G": 1}, ("1",))
def todo(*rows: str) -> str:
return "## Items\n\n| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |\n|---|---|---|---|---|---|---|---|---|---|\n" + "".join(f"{row}\n" for row in rows)
return "## Items\n\n| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |\n|---|---|---|---|---|---|---|---|---|---|---|\n" + "".join(f"{row}\n" for row in rows)
class Seats(unittest.TestCase):
@@ -188,9 +188,9 @@ class Rounds(unittest.TestCase):
class ScoringRunDue(unittest.TestCase):
def test_reads_each_rows_id_release_and_tagline(self):
self.assertEqual(items(todo("| 3 | 0.1.0 | | **Do it.** | 1 | 1 | 1 | 1 | 1 | 9.0 |")), [item(3, (0, 1, 0))])
self.assertEqual(items(todo("| 3 | 0.1.0 | | **Do it.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |")), [item(3, (0, 1, 0))])
self.assertIsInstance(items("# todo\n"), Failure)
self.assertIsInstance(items(todo("| 0 | 0.1.0 | | **Do it.** | 1 | 1 | 1 | 1 | 1 | 9.0 |")), Failure)
self.assertIsInstance(items(todo("| 0 | 0.1.0 | | **Do it.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |")), Failure)
def test_is_due_for_an_item_naming_the_floor(self):
plan = [item(1, (0, 1, 0), "**Lift the project to the Comprehension floor.**"), item(2, (0, 1, 0))]
@@ -226,7 +226,7 @@ class LarvReviewTool(unittest.TestCase):
self.checkout = root / "proj"
subprocess.run(["git", "init", "-q", "--bare", "-b", "main", str(root / "origin.git")], check=True, env=self.env)
subprocess.run(["git", "clone", "-q", str(root / "origin.git"), str(self.checkout)], check=True, env=self.env, capture_output=True)
self.commit("start", {"todo.md": todo("| 1 | 0.1.0 | | **Do it.** | 1 | 1 | 1 | 1 | 1 | 9.0 |", "| 2 | 0.2.0 | | **Then this.** | 1 | 1 | 1 | 1 | 1 | 9.0 |", "| 3 | 0.2.0 | | **And this.** | 1 | 1 | 1 | 1 | 1 | 9.0 |")})
self.commit("start", {"todo.md": todo("| 1 | 0.1.0 | | **Do it.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |", "| 2 | 0.2.0 | | **Then this.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |", "| 3 | 0.2.0 | | **And this.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |")})
self.git("push", "-q", "origin", "HEAD:main")
self.git("remote", "set-head", "origin", "main")
self.git("switch", "-q", "-c", "chunk")
+9 -6
View File
@@ -26,9 +26,9 @@ A valid todo.md; ## Details and each item's detail are optional:
## Items
| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 0.1.0 | | **Make the first thing work.** | 2 | 3 | 8 | 9 | 1 | 0 |
| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 0.1.0 | | **Make the first thing work.** | 2 | 3 | 4 | 8 | 9 | 1 | 0 |
## Details
@@ -40,7 +40,8 @@ A valid todo.md; ## Details and each item's detail are optional:
- The bar is judged on the Score as shown, to one decimal.
- A new item takes Next ID, and Next ID then goes up by one; every ID stays below it.
- The goal weights list README.md's numbered goals in order, a sub-goal such as 1.1 included.
- An item's R, S, A and G are whole numbers from 1 to 10; Goals lists goal numbers, comma separated.
- An item's R, S, C, A and G are whole numbers from 1 to 10; Goals lists goal numbers, comma separated.
C, how hard the change is to get right, stays out of the formula.
- An item's Exempt cell is empty, or names why the bar does not apply: decision, defect,
principle, question or security. An item docs/decisions.md cites as "`todo.md` item N" must be
exempt, and a row marked decision must be cited there.
@@ -57,7 +58,7 @@ import pathlib
import re
import sys
from todo_table import cells, Item, ITEMS_HEADER, parse_row, table_rows
from todo_table import cells, Item, ITEMS_HEADER, LEGACY_HEADER, parse_row, table_rows
from failure import Failure
from markdown_text import FENCE, sections
from file_writes import replace_file
@@ -186,6 +187,8 @@ def rank(text: str, readme: str, decisions: str) -> Ranked | Failure:
def read_scoring(text: str, lines: list[str], readme: str) -> Scoring | Failure:
if ITEMS_HEADER not in lines and "## Items" not in lines:
return Failure("todo.md is not in the scored-table format; rank_todo.py --help shows a valid one")
if LEGACY_HEADER in lines:
return Failure("todo.md's items table has no C column; add one after S, and score each item's C from 1 to 10 per rank_todo.py --help")
goals = readme_goals(readme)
if isinstance(goals, Failure):
return goals
@@ -367,7 +370,7 @@ def parse_item(row: str, weights: dict[str, float]) -> Item | Failure:
def render_item(item: Item, value: float) -> str:
release = ".".join(str(part) for part in item.release)
scores = " | ".join(str(item.scores[name]) for name in "RSAG")
scores = " | ".join(str(item.scores[name]) for name in "RSCAG")
return f"| {item.id} | {release} | {item.exempt} | {item.tagline} | {scores} | {', '.join(item.goals)} | {value:.1f} |"
+1 -1
View File
@@ -15,5 +15,5 @@ class Types(unittest.TestCase):
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
if shutil.which("uv") is None:
self.fail("uv on PATH fetches mypy; see README.md's Test line")
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "claude_code", "-m", "comprehension_history", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-p", "reading_cost", "-p", "rule_hook", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "--exclude", "/test_", "-p", "claude_code", "-m", "comprehension_history", "-p", "larv_review", "-p", "loop_claude", "-m", "markdown_text", "-p", "metric_bench", "-m", "project_key", "-m", "rank_todo", "-p", "reading_cost", "-p", "rule_hook", "-p", "score_outcomes", "-m", "shell_command", "-m", "statusline", "-m", "todo_table", "-p", "transcript_archive", "-m", "utc_stamp"], capture_output=True, cwd=ROOT / "bin", text=True)
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
+9
View File
@@ -0,0 +1,9 @@
#!/usr/bin/env python3
import pathlib
import sys
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from score_outcomes import main
sys.exit(main.main(sys.argv[1:]))
View File
+116
View File
@@ -0,0 +1,116 @@
"""Show what finished chunks measured beside the scores their items were guessed at, to score a new item by.
usage: score-outcomes [R S C]
Run it inside a project checkout or one of its worktrees. It fetches origin, when the checkout has
that remote, and reads origin's default branch, or else the local main or master; a failed fetch
reads origin as last fetched and says so on stderr.
Git history is the record. A chunk is a merge commit on the default branch's first-parent line
whose change removed items from todo.md; it shipped those items. Each item's guess is its R, S and
C in the todo.md the merge's first parent held, when the chunk began; C only where that todo.md
scored one. The chunk measures:
- lines changed: lines added plus lines deleted against the first parent, leaving out
docs/comprehension-history.md and docs/reading-cost.md, which tools append to;
- files: the files those lines are in;
- review rounds: each pass's rounds among the branch's `Answer <pass> round <n>:` commit titles,
summed, counted as `larv-review round-count` counts them;
- 🔴 and 🟡 findings: summed over those titles, each opening `<n> 🔴, <n> 🟡` after the colon,
such as `Answer stability round 1: 1 🔴, 2 🟡; flag a deleted test`. Unknown where a round title
lacks the counts, or the branch has no round.
A project that merges without merge commits, by fast-forward or squash, has no chunk to measure.
Given R, S and C, it lists the 5 finished items whose guesses sit closest, closest first: the
smallest sum of squared differences, C counted only where the item had one, the newest first among
equals. Each line gives what the item's chunk measured, which items shipped with it, and its tagline.
Then, for each score, the median of the outcome it predicts at each value it was guessed at: R
against 🔴 findings, S against lines changed, C against review rounds. It counts only chunks that
shipped one item, since a chunk measures all its items at once, and reports a score only once 10
such chunks hold it and its outcome.
Exit 1, saying why, on bad arguments, when git cannot start, finds no checkout or reads a shallow
clone, or the default branch cannot be read.
"""
import re
import sys
from failure import Failure
from larv_review.git import default_branch, file_at, git
from score_outcomes.outcomes import guesses, nearest, Outcome, render, report, round_findings, rounds, shipped
# Tools append to these records; their lines are not the chunk's work.
RECORDS = ("docs/comprehension-history.md", "docs/reading-cost.md")
def main(argv: list[str]) -> int:
if argv[:1] in (["-h"], ["--help"]):
print(__doc__.strip())
return 0
if argv and (len(argv) != 3 or not all(re.fullmatch(r"[1-9]|10", arg) for arg in argv)):
print(f"score-outcomes: cannot read the arguments {' '.join(argv)}; usage: score-outcomes [R S C], each a whole number from 1 to 10", file=sys.stderr)
return 1
outcomes = read_outcomes()
if isinstance(outcomes, Failure):
print(f"score-outcomes: {outcomes.reason}", file=sys.stderr)
return 1
if argv:
given = dict(zip("RSC", map(int, argv)))
print(f"Finished items nearest R {given['R']}, S {given['S']}, C {given['C']}, closest first:\n")
print("\n".join(render(outcome) for outcome in nearest(outcomes, given)) or "- None: no merge commit on the default branch removed an item from todo.md.")
print()
print("Median outcome at each guessed score, among chunks that shipped one item:\n")
print("\n".join(report(outcomes)))
return 0
def read_outcomes() -> list[Outcome] | Failure:
found = default_branch(stale_ok=True)
if isinstance(found, Failure):
return found
default, note = found
if note:
print(f"score-outcomes: {note}", file=sys.stderr)
shallow = git("rev-parse", "--is-shallow-repository")
if isinstance(shallow, Failure):
return shallow
if shallow.strip() == "true":
return Failure("a shallow clone lacks the merges it measures; run `git fetch --unshallow origin`")
merges = git("log", "--first-parent", "--merges", "--format=%H %h %cs", default, "--", "todo.md")
if isinstance(merges, Failure):
return merges
outcomes: list[Outcome] = []
for line in merges.splitlines():
measured = measure(*line.split())
if isinstance(measured, Failure):
return measured
outcomes += measured
return outcomes
def measure(merge: str, short: str, merged: str) -> list[Outcome] | Failure:
"""What one merge measured, for each item it removed from todo.md; none where either todo.md holds no items table."""
texts = [file_at(f"{merge}^1", "todo.md"), file_at(merge, "todo.md")]
if failed := next((text for text in texts if isinstance(text, Failure)), None):
return failed
before, after = (guesses(text) for text in texts if isinstance(text, str))
if before is None or after is None:
return []
items = shipped(before, after)
if not items:
return []
numstat = git("diff", "--numstat", f"{merge}^1", merge, "--", ".", *(f":(exclude){record}" for record in RECORDS))
if isinstance(numstat, Failure):
return numstat
changed = [row.split("\t") for row in numstat.splitlines()]
# Git counts a binary file's lines as "-".
lines = sum(int(count) for row in changed for count in row[:2] if count.isdigit())
titles = git("log", "--format=%s", f"{merge}^1..{merge}^2")
if isinstance(titles, Failure):
return titles
findings = round_findings(titles.splitlines())
red, yellow = findings if findings else (None, None)
ids = [item.id for item in items]
return [Outcome(item, short, merged, tuple(id_ for id_ in ids if id_ != item.id), lines, len(changed), rounds(titles.splitlines()), red, yellow) for item in items]
+120
View File
@@ -0,0 +1,120 @@
import dataclasses
import re
import statistics
from typing import Callable
from larv_review.rounds import round_counts, ROUND
ENOUGH = 10
FINDINGS = re.compile(r"([0-9]+) 🔴, ([0-9]+) 🟡(;|$)")
@dataclasses.dataclass(frozen=True)
class Guess:
"""An item's tagline and its R, S and C as the plan held them, C only where the plan scored one."""
id: int
tagline: str
scores: dict[str, int]
@dataclasses.dataclass(frozen=True)
class Outcome:
"""What the merge that shipped an item measured; the findings are unknown where a round title lacks their counts."""
guess: Guess
merge: str
merged: str
shipped_with: tuple[int, ...]
lines: int
files: int
rounds: int
red: int | None
yellow: int | None
# Each score, and the outcome it predicts.
MEASURES: dict[str, tuple[str, Callable[[Outcome], int | None]]] = {
"R": ("🔴 findings", lambda outcome: outcome.red),
"S": ("lines changed", lambda outcome: outcome.lines),
"C": ("review rounds", lambda outcome: outcome.rounds),
}
def guesses(todo: str) -> dict[int, Guess] | None:
"""Each item of a todo.md's items table by ID, from any shape of table with ID, Item, R and S columns; None without such a table. A row it cannot read is left out."""
lines = todo.splitlines()
for index, line in enumerate(lines):
header = split(line)
if {"ID", "Item", "R", "S"} <= set(header):
break
else:
return None
found = {}
for row in lines[index + 2:]:
if not row.startswith("|"):
break
named = dict(zip(header, split(row)))
if len(split(row)) != len(header) or not all(re.fullmatch(r"[0-9]+", named.get(name, "0")) for name in ("ID", "R", "S", "C")):
continue
scores: dict[str, int] = {name: int(named[name]) for name in ("R", "S", "C") if name in named}
found[int(named["ID"])] = Guess(int(named["ID"]), named["Item"], scores)
return found
def split(row: str) -> list[str]:
return [cell.strip() for cell in re.split(r"(?<!\\)\|", row.strip())[1:-1]] if row.startswith("|") else []
def shipped(before: dict[int, Guess], after: dict[int, Guess]) -> list[Guess]:
return [guess for id_, guess in sorted(before.items()) if id_ not in after]
def round_findings(titles: list[str]) -> tuple[int, int] | None:
"""The 🔴 and 🟡 findings summed over each `Answer <pass> round <n>:` title, which opens with `<n> 🔴, <n> 🟡`; None where a round title does not, or no title is a round."""
counts = []
for title in titles:
if named := ROUND.match(title):
found = FINDINGS.match(title[named.end():].lstrip())
if not found:
return None
counts.append((int(found.group(1)), int(found.group(2))))
return (sum(red for red, _ in counts), sum(yellow for _, yellow in counts)) if counts else None
def rounds(titles: list[str]) -> int:
return sum(round_counts(titles).values())
def nearest(outcomes: list[Outcome], given: dict[str, int], count: int = 5) -> list[Outcome]:
"""The outcomes whose guesses sit closest to the given scores, newest first among equals; C counts only where both hold one."""
def distance(outcome: Outcome) -> int:
return sum((value - outcome.guess.scores[name]) ** 2 for name, value in given.items() if name in outcome.guess.scores)
newest_first = sorted(outcomes, key=lambda outcome: outcome.merged, reverse=True)
return sorted(newest_first, key=distance)[:count]
def report(outcomes: list[Outcome]) -> list[str]:
"""For each score, the median outcome of each value it was guessed at, among chunks that shipped one item; a chunk measures several items at once."""
single = [outcome for outcome in outcomes if not outcome.shipped_with]
lines = []
for name, (what, measure) in MEASURES.items():
held = [(outcome.guess.scores[name], value) for outcome in single if name in outcome.guess.scores and (value := measure(outcome)) is not None]
if len(held) < ENOUGH:
chunks = "chunk that shipped one item records" if len(held) == 1 else "chunks that shipped one item record"
lines.append(f"- {name}, {what}: {len(held)} {chunks} {name} and {what}; the report needs {ENOUGH}.")
continue
values = sorted({guess for guess, _ in held})
medians = [(guess, [value for at, value in held if at == guess]) for guess in values]
lines.append(f"- {name}, {what}: " + ", ".join(f"{guess} → {statistics.median(found):g} ({plural(len(found), 'chunk')})" for guess, found in medians))
return lines
def render(outcome: Outcome) -> str:
scores = ", ".join(f"{name} {value}" for name, value in outcome.guess.scores.items())
findings = "findings unknown" if outcome.red is None else f"{outcome.red} 🔴, {outcome.yellow} 🟡"
others = outcome.shipped_with
together = f", with item{'s' if len(others) > 1 else ''} {', '.join(map(str, others))}" if others else ""
return f"- Item {outcome.guess.id}, {scores}: {plural(outcome.lines, 'line')} in {plural(outcome.files, 'file')}, {plural(outcome.rounds, 'review round')}, {findings}; merged {outcome.merged} in {outcome.merge}{together}. {outcome.guess.tagline}"
def plural(count: int, noun: str) -> str:
return f"{count} {noun}{'' if count == 1 else 's'}"
+115
View File
@@ -0,0 +1,115 @@
import os
import pathlib
import subprocess
import sys
import tempfile
import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from score_outcomes.outcomes import Guess, guesses, nearest, Outcome, report, round_findings, shipped
from test_env import isolated_env
TOOL = pathlib.Path(__file__).resolve().parents[1] / "score-outcomes"
HEADER = "| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |\n|---|---|---|---|---|---|---|---|---|---|---|\n"
LEGACY = "| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |\n|---|---|---|---|---|---|---|---|---|---|\n"
def todo(*rows: str, header: str = HEADER) -> str:
return "# todo\n\n## Items\n\n" + header + "".join(f"{row}\n" for row in rows)
def outcome(id_: int, r: int, s: int, c: int | None, lines: int = 10, rounds: int = 1, red: int | None = 0, merge: str = "aaaaaaa", shipped_with: tuple[int, ...] = ()) -> Outcome:
scores = {"R": r, "S": s} | ({"C": c} if c else {})
return Outcome(Guess(id_, f"**Item {id_}.**", scores), merge, "2026-10-13", shipped_with, lines, 1, rounds, red, 0 if red is not None else None)
class Guesses(unittest.TestCase):
def test_reads_each_items_r_s_and_c_and_an_older_table_without_c(self):
self.assertEqual(guesses(todo("| 4 | 0.1.0 | | **Do it.** | 2 | 3 | 5 | 6 | 7 | 1 | 9.0 |")), {4: Guess(4, "**Do it.**", {"R": 2, "S": 3, "C": 5})})
self.assertEqual(guesses(todo("| 4 | 0.1.0 | | **A \\| B.** | 2 | 3 | 6 | 7 | 1 | 9.0 |", header=LEGACY)), {4: Guess(4, "**A \\| B.**", {"R": 2, "S": 3})})
def test_reads_no_plan_from_a_file_without_an_items_table(self):
self.assertIsNone(guesses("# todo\n\n- [ ] Do it.\n"))
self.assertEqual(guesses(todo()), {})
def test_skips_a_row_it_cannot_read(self):
self.assertEqual(guesses(todo("| x | 0.1.0 | | **Do it.** | 2 | 3 | 5 | 6 | 7 | 1 | 9.0 |", "| 5 | 0.1.0 | | **Too few.** | 2 |")), {})
def test_ships_each_item_the_change_removed(self):
before = guesses(todo("| 4 | 0.1.0 | | **Do it.** | 2 | 3 | 5 | 6 | 7 | 1 | 9.0 |", "| 5 | 0.1.0 | | **Then this.** | 2 | 3 | 5 | 6 | 7 | 1 | 9.0 |"))
after = guesses(todo("| 5 | 0.1.0 | | **Then this.** | 2 | 3 | 5 | 6 | 7 | 1 | 9.0 |"))
assert before is not None and after is not None
self.assertEqual(shipped(before, after), [before[4]])
class RoundFindings(unittest.TestCase):
def test_sums_the_counts_each_round_title_opens_with(self):
self.assertEqual(round_findings(["Answer stability round 1: 1 🔴, 2 🟡; flag a deleted test", "Answer prose round 1: 0 🔴, 3 🟡", "Merge origin/main"]), (1, 5))
def test_knows_no_count_when_a_round_title_or_every_round_lacks_one(self):
self.assertIsNone(round_findings(["Answer stability round 1: 1 🔴, 2 🟡; fix", "Answer prose round 1: fix the README"]))
self.assertIsNone(round_findings(["Do the work"]))
class Nearest(unittest.TestCase):
def test_lists_the_closest_guesses_first_and_counts_c_only_where_an_item_had_one(self):
found = [outcome(1, 9, 9, 9), outcome(2, 3, 4, None), outcome(3, 3, 4, 9), outcome(4, 3, 5, 5)]
self.assertEqual([each.guess.id for each in nearest(found, {"R": 3, "S": 4, "C": 5}, 3)], [2, 4, 3])
class Report(unittest.TestCase):
def test_gives_each_guessed_values_median_outcome_once_ten_single_item_chunks_hold_it(self):
found = [outcome(id_, 2, 1 + id_ % 2, 4, lines=10 * id_, rounds=id_, merge=f"m{id_}") for id_ in range(1, 11)]
lines = report(found + [outcome(11, 2, 2, None, lines=1000, red=None, merge="mx", shipped_with=(12,))])
self.assertIn("- R, 🔴 findings: 2 → 0 (10 chunks)", lines)
self.assertIn("- S, lines changed: 1 → 60 (5 chunks), 2 → 50 (5 chunks)", lines)
self.assertIn("- C, review rounds: 4 → 5.5 (10 chunks)", lines)
def test_says_how_many_chunks_it_has_under_ten(self):
self.assertIn("- C, review rounds: 1 chunk that shipped one item records C and review rounds; the report needs 10.", report([outcome(1, 2, 2, 4), outcome(2, 2, 2, None)]))
class ScoreOutcomesTool(unittest.TestCase):
def setUp(self):
temp = tempfile.TemporaryDirectory()
self.addCleanup(temp.cleanup)
root = pathlib.Path(os.path.realpath(temp.name))
self.env = {**isolated_env(), "GIT_CONFIG_GLOBAL": os.devnull, "HOME": str(root)}
self.checkout = root / "proj"
self.checkout.mkdir()
self.git("init", "-q", "-b", "main")
self.commit("start", {"todo.md": todo("| 1 | 0.1.0 | | **Do it.** | 2 | 3 | 5 | 6 | 7 | 1 | 9.0 |", "| 2 | 0.1.0 | | **Then this.** | 4 | 6 | 8 | 6 | 7 | 1 | 9.0 |")})
def git(self, *args):
return subprocess.run(["git", "-c", "user.name=t", "-c", "user.email=t@t", *args], cwd=self.checkout, check=True, env=self.env, capture_output=True, text=True).stdout
def commit(self, title, files):
for name, text in files.items():
(self.checkout / name).parent.mkdir(parents=True, exist_ok=True)
(self.checkout / name).write_text(text)
self.git("add", name)
self.git("commit", "-q", "-m", title)
def run_tool(self, *args):
return subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, cwd=self.checkout, env=self.env, text=True)
def test_measures_each_merged_chunk_that_removed_an_item(self):
self.git("switch", "-q", "-c", "chunk")
self.commit("Do it", {"a.py": "1\n2\n3\n", "todo.md": todo("| 2 | 0.1.0 | | **Then this.** | 4 | 6 | 8 | 6 | 7 | 1 | 9.0 |")})
self.commit("Answer stability round 1: 1 🔴, 2 🟡; fix", {"a.py": "1\n2\n4\n", "docs/reading-cost.md": "x\n" * 50})
self.git("switch", "-q", "main")
self.git("merge", "-q", "--no-ff", "-m", "Merge chunk", "chunk")
ran = self.run_tool("2", "3", "5")
self.assertEqual(ran.returncode, 0, ran.stderr)
self.assertIn("- Item 1, R 2, S 3, C 5: 4 lines in 2 files, 1 review round, 1 🔴, 2 🟡; merged", ran.stdout)
self.assertNotIn("Item 2", ran.stdout)
def test_refuses_a_score_outside_one_to_ten(self):
ran = self.run_tool("2", "11", "5")
self.assertEqual(ran.returncode, 1)
self.assertIn("usage: score-outcomes [R S C]", ran.stderr)
if __name__ == "__main__":
unittest.main()
+21 -16
View File
@@ -33,13 +33,13 @@ TODO = """# todo
## Items
| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 0.2.0 | | **Ship the later thing.** | 1 | 1 | 9 | 9 | 1 | 0 |
| 2 | 0.1.0 | | **Tighten the tests.** | 2 | 4 | 5 | 5 | 2.1 | 0 |
| 3 | 0.1.0 | | **Make it fast, `a \\| b`.** | 3 | 8 | 6 | 8 | 2.1, 1 | 0 |
| 4 | 0.1.0 | security | **Rotate the leaked key.** | 9 | 2 | 1 | 1 | 2.1 | 0 |
| 5 | 0.1.0 | decision | **Move the writes out.** | 5 | 5 | 4 | 5 | 2.1 | 0 |
| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 0.2.0 | | **Ship the later thing.** | 1 | 1 | 3 | 9 | 9 | 1 | 0 |
| 2 | 0.1.0 | | **Tighten the tests.** | 2 | 4 | 3 | 5 | 5 | 2.1 | 0 |
| 3 | 0.1.0 | | **Make it fast, `a \\| b`.** | 3 | 8 | 3 | 6 | 8 | 2.1, 1 | 0 |
| 4 | 0.1.0 | security | **Rotate the leaked key.** | 9 | 2 | 3 | 1 | 1 | 2.1 | 0 |
| 5 | 0.1.0 | decision | **Move the writes out.** | 5 | 5 | 3 | 4 | 5 | 2.1 | 0 |
## Details
@@ -63,11 +63,11 @@ class Rank(unittest.TestCase):
def test_scores_and_sorts_by_release_then_security_then_score_and_orders_the_details_to_match(self):
ranked = self.ranked()
self.assertIn("""| 4 | 0.1.0 | security | **Rotate the leaked key.** | 9 | 2 | 1 | 1 | 2.1 | -6.8 |
| 3 | 0.1.0 | | **Make it fast, `a \\| b`.** | 3 | 8 | 6 | 8 | 2.1, 1 | 23.0 |
| 2 | 0.1.0 | | **Tighten the tests.** | 2 | 4 | 5 | 5 | 2.1 | 10.3 |
| 5 | 0.1.0 | decision | **Move the writes out.** | 5 | 5 | 4 | 5 | 2.1 | 5.1 |
| 1 | 0.2.0 | | **Ship the later thing.** | 1 | 1 | 9 | 9 | 1 | 34.8 |
self.assertIn("""| 4 | 0.1.0 | security | **Rotate the leaked key.** | 9 | 2 | 3 | 1 | 1 | 2.1 | -6.8 |
| 3 | 0.1.0 | | **Make it fast, `a \\| b`.** | 3 | 8 | 3 | 6 | 8 | 2.1, 1 | 23.0 |
| 2 | 0.1.0 | | **Tighten the tests.** | 2 | 4 | 3 | 5 | 5 | 2.1 | 10.3 |
| 5 | 0.1.0 | decision | **Move the writes out.** | 5 | 5 | 3 | 4 | 5 | 2.1 | 5.1 |
| 1 | 0.2.0 | | **Ship the later thing.** | 1 | 1 | 3 | 9 | 9 | 1 | 34.8 |
""", ranked.text)
self.assertTrue(ranked.text.endswith("## Details\n\n### 3. Make it fast, `a | b`.\n\nFast detail.\n\n### 1. Ship the later thing.\n\nLater detail.\n\n## Notes\n\nKept as written.\n"))
self.assertEqual(ranked.under_bar, [])
@@ -75,7 +75,7 @@ class Rank(unittest.TestCase):
def test_holds_each_item_a_decision_cites_and_says_how_to_settle_a_finished_one(self):
self.assertIsInstance(self.ranked(decisions=DECISIONS + "Until `todo.md` item 4 lands.\n"), rank_todo.Ranked)
gone = self.ranked(TODO.replace("| 5 | 0.1.0 | decision | **Move the writes out.** | 5 | 5 | 4 | 5 | 2.1 | 0 |\n", ""))
gone = self.ranked(TODO.replace("| 5 | 0.1.0 | decision | **Move the writes out.** | 5 | 5 | 3 | 4 | 5 | 2.1 | 0 |\n", ""))
self.assertEqual(gone.reason, "docs/decisions.md cites \"`todo.md` item 5\", which todo.md no longer holds; revise that decision to say what holds now")
def test_names_each_item_under_the_bar_that_is_not_exempt(self):
@@ -94,16 +94,21 @@ class Rank(unittest.TestCase):
self.assertIn("| 1 | 1.00 |\n| 2 | 0.75 |\n| 2.1 | 0.50 |\n| 3 | 0.25 |", failure.reason)
def test_refuses_a_formula_that_is_more_than_arithmetic_over_the_scores(self):
for formula in ("__import__('os')", "R.real", "X + 1", "R ** 2", "R +", "R / (S - 2)", "R\x00", "1" * 400, " + ".join(["R"] * 300), "-" * 100000 + "R"):
for formula in ("__import__('os')", "R.real", "X + 1", "C + 1", "R ** 2", "R +", "R / (S - 2)", "R\x00", "1" * 400, " + ".join(["R"] * 300), "-" * 100000 + "R"):
with self.subTest(formula=formula):
self.assertIsInstance(self.ranked(TODO.replace("-R - S/4 + 2*A + 2*G*W", formula)), Failure)
def test_asks_for_a_c_column_in_a_table_without_one(self):
legacy = TODO.split("## Items")[0] + "## Items\n\n| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |\n|---|---|---|---|---|---|---|---|---|---|\n| 2 | 0.1.0 | | **Tighten the tests.** | 2 | 4 | 5 | 5 | 2.1 | 0 |\n"
self.assertEqual(self.ranked(legacy, decisions="").reason, "todo.md's items table has no C column; add one after S, and score each item's C from 1 to 10 per rank_todo.py --help")
def test_refuses_a_file_off_the_format(self):
row = "| 2 | 0.1.0 | | **Tighten the tests.** | 2 | 4 | 5 | 5 | 2.1 | 0 |"
row = "| 2 | 0.1.0 | | **Tighten the tests.** | 2 | 4 | 3 | 5 | 5 | 2.1 | 0 |"
broken = {
row: row.replace("0.1.0", "0.1"),
"| 2 | 4 | 3 | 5 | 5 |": "| 2 | 4 | 11 | 5 | 5 |",
row + "\n": row.replace("5 | 5", "5 | ²") + "\n",
"| 2.1 | 0 |\n\n## Details": "| 2.1 | 0 |\n\n| 9 | 0.1.0 | | **Stray.** | 1 | 1 | 1 | 1 | 1 | 0 |\n\n## Details",
"| 2.1 | 0 |\n\n## Details": "| 2.1 | 0 |\n\n| 9 | 0.1.0 | | **Stray.** | 1 | 1 | 3 | 1 | 1 | 1 | 0 |\n\n## Details",
"| 4 | 0.1.0 | security |": "| 4 | 0.1.0 | later |",
"| 5 | 0.1.0 | decision |": "| 5 | 0.1.0 | |",
"`Next ID = 6`": "`Next ID = 5`",
+6 -4
View File
@@ -4,7 +4,9 @@ import re
from failure import Failure
EXEMPTIONS = ("", "decision", "defect", "principle", "question", "security")
ITEMS_HEADER = "| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |"
ITEMS_HEADER = "| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |"
# The items header before C joined it, which rank_todo.py asks to have C added.
LEGACY_HEADER = "| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |"
@dataclasses.dataclass(frozen=True)
@@ -57,10 +59,10 @@ def cells(row: str, count: int) -> list[str] | Failure:
def parse_row(row: str) -> Item | Failure:
"""An item row, every cell checked but the goals it cites."""
found = cells(row, 10)
found = cells(row, 11)
if isinstance(found, Failure):
return found
id_, release, exempt, tagline, r, s, a, g, goals, _ = found
id_, release, exempt, tagline, r, s, c, a, g, goals, _ = found
if not re.fullmatch(r"[0-9]+", id_) or int(id_) < 1:
return Failure(f"the ID {id_!r} is not a positive whole number: {row}")
if not re.fullmatch(r"[0-9]+\.[0-9]+\.[0-9]+", release):
@@ -70,7 +72,7 @@ def parse_row(row: str) -> Item | Failure:
if not re.fullmatch(r"\*\*[^*].*[^*]\*\*", tagline):
return Failure(f"item {id_}'s Item cell is not one bold tagline: {tagline}")
scores = {}
for name, value in zip("RSAG", (r, s, a, g)):
for name, value in zip("RSCAG", (r, s, c, a, g)):
if not re.fullmatch(r"[0-9]+", value) or not 1 <= int(value) <= 10:
return Failure(f"item {id_}'s {name} {value!r} is not a whole number from 1 to 10")
scores[name] = int(value)
+3 -1
View File
@@ -7,11 +7,13 @@ paths:
- **`todo.md` is mandatory in every project, in one shape**, which `~/.claude/bin/rank_todo.py` checks and ranks; its `--help` shows a valid file. `## Scoring` holds the formula, the bar and the goal weights; `## Items` is one table of every item; `## Details` holds each item's detail under its ID. Cite an item by its ID. The formula, bar and weights are the project's own to tune.
- **An item's release allows the bump that semver read strictly asks for.** An item that changes what exists needs a major, or before 1.0.0 a minor. One that only adds needs a minor, and a fix a patch.
- **Score an item when it is filed or edited.** Each of R, S, A and G is a whole number from 1 to 10:
- **Score an item when it is filed or edited.** Each of R, S, C, A and G is a whole number from 1 to 10:
- **R** — the risk that the change creates serious bugs or issues.
- **S** — the size of the change.
- **C** — how hard the change is to get right: novel design, subtle invariants, many interacting parts. A huge rename is low C; a ten-line lock fix can be high. C stays out of the score.
- **A** — its value to the audience in `README.md` → `## Audience`.
- **G** — its value to the goals it cites in the Goals column; W is the weight of the heaviest of them.
- **Score R, S and C from finished chunks wherever they hold a measurement.** Guess them, then run `~/.claude/bin/score-outcomes <R> <S> <C>` in the project. It lists the finished items guessed nearest and what their chunks measured, and the median outcome at each guessed value. Move each score to where items of a measured outcome like the one you expect sit. Keep a guess only where it prints no finished item.
- **An item scoring under the bar is deleted, with every item that needs it, and a finding under the bar is not filed.** The chunk's summary names each, with its score.
- **The Exempt cell keeps an item whatever its score:** `security` for a security finding, `defect` for a 🔴 finding, `question` for a question only the human can answer, `decision` for an item `docs/decisions.md` cites, and, where no other reason fits, `principle` for a 🟡 finding against a file under `~/.claude/principles/`, whose item's detail names that principle. Every item `docs/decisions.md` cites carries one of these reasons.
- **Unless the project has a reason to differ, weigh goals by position, the first heaviest.** When the weights stop listing the README's goals, `rank_todo.py` fails and prints those weights.
+1 -1
View File
@@ -66,7 +66,7 @@ Give a reviewer without Bash the branch diff and `git ls-files`'s output in its
- **Round.** A round is one launch of a pass's reviewer: the first, each re-entry, and the rerun after the architect returns Stage 0 questions ("Architecture" below). A rescore is a round only as "Architecture" → "Fresh" says.
- **Architecture round.** The architecture pass's first round is the architect (unless skipped), the principles audit and the comprehension check, which "Opt-out" or "Architecture" → "Suspended" may skip. A later round relaunches each reviewer whose 🔴 or 🟡 the previous round's fixes answer, the comprehension check only as "Architecture" → "Fresh" says. Where those fixes carry an architectural decision, the architect and the principles audit run again.
- **Auto-reviewer.** The external auto-reviewer (§4) is a pass of its own, each review it posts a round.
- **Count.** Close each round with one commit that holds the round's fixes and is titled `Answer <pass> round <n>: …`, `<pass>` being the pass's name in lower case, such as `full scan`. A round that changed nothing gets an empty commit (`git commit --allow-empty`). Push the commits when §4 runs next, where there is a remote. `~/.claude/bin/larv-review round-count` prints each pass's count and marks a pass at the cap. The auto-reviewer's count is the number of its reviews on the PR, less its reviews of a `Rescore:` commit.
- **Count.** Close each round with one commit that holds the round's fixes and is titled `Answer <pass> round <n>: <n> 🔴, <n> 🟡; …`, `<pass>` being the pass's name in lower case, such as `full scan`, and the counts the 🔴 and 🟡 findings the round reported, such as `Answer stability round 1: 1 🔴, 2 🟡; flag a deleted test`. `~/.claude/bin/score-outcomes` reads the counts to calibrate the plan's scores. A round that changed nothing gets an empty commit (`git commit --allow-empty`). Push the commits when §4 runs next, where there is a remote. `~/.claude/bin/larv-review round-count` prints each pass's count and marks a pass at the cap. The auto-reviewer's count is the number of its reviews on the PR, less its reviews of a `Rescore:` commit.
- **Later rounds.** A rerun pass reviews only what changed since it last reviewed: `git diff <the commit it last reviewed, or else the commit before this round's fixes> HEAD`, since the fixes may be unpushed. The comprehension check's seats still read the whole project. From round 2, prompt an agent under `~/.claude/agents/` with its own last round's findings verbatim too.
- **Done.** A pass is done when its latest round leaves no 🔴 or 🟡 standing: none was reported, or each was shown factually wrong, declined, filed, or named under the bar per "Findings", or, for the prose pass, fixed. Resolve that round's 🟢s and ❓s per "Findings" too.
- **Cap.** A pass takes at most three rounds per chunk. Instead of a fourth, list what is open and ask the user. For the auto-reviewer, ask before the push that would summon its fourth, and hold the fixes unpushed until the answer.
+50 -52
View File
@@ -6,7 +6,7 @@
`Bar = 9`
`Next ID = 110`
`Next ID = 112`
| Goal | W |
|---|---|
@@ -22,64 +22,56 @@
## Items
| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |
|---|---|---|---|---|---|---|---|---|---|
| 98 | 0.1.0 | | **Score R and S from a measurement wherever one exists, and calibrate each guessed score against the finished chunk's measured outcome.** | 2 | 5 | 6 | 7 | 1.1, 1.4 | 22.6 |
| 99 | 0.1.0 | | **Drop the comprehension panel as a merge gate, and keep recording its scores and the places in the code its seats found hard, for later measurement and for other reviewers and agents to improve.** | 4 | 5 | 7 | 7 | 1.1, 1.4, 1.5 | 22.6 |
| 87 | 0.1.0 | | **Score every `todo.md` item's complexity C from 1 to 10, and have `rank_todo.py` fail on an item without one.** | 3 | 4 | 6 | 7 | 1.1, 1.4 | 21.9 |
| 97 | 0.1.0 | | **Narrow each review pass to the findings worth a fix, so a pass converges in its second round.** | 5 | 4 | 7 | 7 | 1.1, 1.4, 1.5 | 21.9 |
| 109 | 0.1.0 | | **Lower the reading cost of the files a chunk reads most before its first commit.** | 2 | 4 | 5 | 6 | 1.4 | 18.5 |
| 93 | 0.2.0 | | **Let a chunk's worker start below the project's model, once the archive shows chunks on a lower rung took no more review rounds than chunks on the project's model.** | 3 | 2 | 6 | 7 | 1.1, 1.4 | 22.4 |
| 94 | 0.2.0 | | **Every 20th cycle, have loop-claude measure token spend and routing outcomes, file them here, and interrupt the human only when they call for a routing change.** | 3 | 5 | 6 | 7 | 1.1, 1.4 | 21.6 |
| 88 | 0.2.0 | | **Define one routing ladder of (model, effort) rungs ordered by a published ranking of their competence, and map a task's R + C to a rung relative to the project's model.** | 3 | 4 | 6 | 7 | 1.4 | 21.4 |
| 90 | 0.2.0 | | **Route each reviewer by the chunk's R + C, once a replay of past chunks shows the routed reviewer finds the 🔴 and 🟡 findings on record.** | 4 | 5 | 6 | 7 | 1.4 | 20.2 |
| 91 | 0.2.0 | | **Start each chunk's worker on the rung its item's R + C picks, never below the project's model, and step it one rung up after two unfinished verdicts on one chunk.** | 4 | 5 | 6 | 7 | 1.4 | 20.2 |
| 105 | 0.2.0 | decision | **Compare review rounds per pass and reading cost for the five chunks before and after the `worker:` decision moved the principles out of the worker's context, and file an item if either rose.** | 1 | 2 | 5 | 6 | 1.4 | 20.0 |
| 92 | 0.2.0 | | **Have the worker route each subagent by its task's R + C, and delegate a task only when the worker needs just its conclusion.** | 2 | 2 | 5 | 6 | 1.4 | 19.0 |
| 89 | 0.2.0 | | **Run the judge on the rung its chunk's R + C picks, once a replay of archived transcripts shows it returns the verdicts on record.** | 3 | 4 | 5 | 6 | 1.4 | 17.5 |
| 1 | 0.3.0 | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 3 | 5 | 8 | 9 | 3, 4 | 27.0 |
| 62 | 0.3.0 | defect | **Lift Locality, Self-sufficiency and the overall to the comprehension floor, at the places the seats name.** | 4 | 6 | 6 | 8 | 1.1 | 22.3 |
| 21 | 0.3.0 | | **Give `settings.json`'s model one writer, and sort its keys.** | 3 | 3 | 6 | 7 | 1.3 | 21.8 |
| 49 | 0.3.0 | principle | **Check that `AGENTS.md` lists exactly the titles of `docs/decisions.md`.** | 1 | 2 | 5 | 6 | 1.3 | 20.1 |
| 59 | 0.3.0 | | **Bound a run of unfinished verdicts on one chunk, so a worker and judge that disagree cannot cycle unattended without end.** | 3 | 3 | 6 | 6 | 1.4 | 19.8 |
| 12 | 0.3.0 | | **Run the test suite in CI on every PR, or record in `docs/decisions.md` why the local run is enough.** | 2 | 4 | 4 | 6 | 1.1 | 16.9 |
| 80 | 0.3.0 | | **State in README → Install that every machine taking this checkout must be a sandbox, as `settings.json`'s auto mode assumes.** | 1 | 1 | 5 | 5 | 4 | 16.8 |
| 5 | 0.3.0 | decision | **Key a project on its checkout's full path, and refuse to start while another session uses that key.** | 5 | 5 | 5 | 6 | 1.1 | 15.6 |
| 84 | 0.3.0 | | **Check, at each release's scoring run, that seats in a paired ruling agree more often than two scoring runs on near-identical code do, as the paired-ruling decision assumes.** | 2 | 2 | 4 | 5 | 1.4 | 15.1 |
| 67 | 0.3.0 | principle | **Point stability-reviewer at the technical principles in place of its own restatement of them.** | 2 | 3 | 4 | 5 | 1.3 | 14.9 |
| 106 | 0.3.0 | | **Make larv-review's "Do not over-read a clean result" cite a rule that says Renovate or Dependabot owns dependency CVEs, or drop the citation.** | 1 | 1 | 3 | 5 | 1.1 | 14.7 |
| 66 | 0.3.0 | principle | **Hold systems-architect's method thresholds only while the comprehension panel finds the flagged code hard.** | 3 | 3 | 4 | 5 | 1.1 | 14.2 |
| 101 | 0.3.0 | principle | **Have `larv_review/record.py`'s `read_seats` take the time, the commit check and the history as inputs, so it parses and decides without effects.** | 3 | 3 | 4 | 5 | 1.1 | 14.2 |
| 29 | 0.3.0 | | **Make the contributor a standing member of every project's audience, unless its `## Audience` says maintainability does not matter.** | 2 | 2 | 3 | 5 | 1.1, 1.4 | 13.4 |
| 68 | 0.3.0 | principle | **Test `workflow-bench/compare.py`'s blinding and `bench.py`'s run plan.** | 2 | 4 | 3 | 5 | 1.1 | 12.9 |
| 65 | 0.3.0 | | **Archive each judge reply the loop rejects.** | 3 | 4 | 4 | 5 | 3 | 12.5 |
| 82 | 0.3.0 | principle | **Sort `AGENTS.md`'s decision titles and `docs/decisions.md`'s entries by title.** | 1 | 2 | 3 | 4 | 1.3 | 12.3 |
| 73 | 0.3.0 | | **Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.** | 3 | 4 | 4 | 4 | 1.4 | 11.7 |
| 6 | 0.3.0 | decision | **Move what `loop-claude` writes out of `~/.claude.json`, and ask the human about each write that must stay.** | 5 | 5 | 4 | 5 | 1.1 | 11.7 |
| 51 | 0.3.0 | principle | **Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints.** | 3 | 5 | 3 | 5 | 1.1 | 11.7 |
| 102 | 0.3.0 | | **Give loop-claude's retry delay an environment override, so the black-box test stops replacing `time.sleep` through `sitecustomize`.** | 2 | 2 | 3 | 4 | 1.1 | 11.4 |
| 50 | 0.3.0 | principle | **Grant the bench's runs only the tools a review needs, in place of skipping every permission.** | 2 | 2 | 3 | 4 | 4 | 9.9 |
| 81 | 0.3.0 | principle | **Have `workflow-bench/bench.py` return each run's failure as a value, so one failed clone does not hide the later runs' results.** | 2 | 2 | 3 | 4 | 4 | 9.9 |
| 4 | 0.3.0 | | **Name each chunk's archive folder by the chunk's start time, so `<key>/` lists chunks in the order they ran.** | 3 | 2 | 4 | 3 | 3 | 9.6 |
| 52 | 0.3.0 | principle | **Record why this checkout lives in Claude Code's own directory, or move it out and link it in.** | 6 | 6 | 3 | 5 | 2 | 7.5 |
| 77 | 0.4.0 | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 3 | 6 | 6 | 7 | 1.1, 1.4 | 21.4 |
| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |
|---|---|---|---|---|---|---|---|---|---|---|
| 99 | 0.1.0 | | **Drop the comprehension panel as a merge gate, and keep recording its scores and the places in the code its seats found hard, for later measurement and for other reviewers and agents to improve.** | 4 | 5 | 4 | 7 | 7 | 1.1, 1.4, 1.5 | 22.6 |
| 97 | 0.1.0 | | **Narrow each review pass to the findings worth a fix, so a pass converges in its second round.** | 5 | 4 | 6 | 7 | 7 | 1.1, 1.4, 1.5 | 21.9 |
| 109 | 0.1.0 | | **Lower the reading cost of the files a chunk reads most before its first commit.** | 2 | 4 | 4 | 5 | 6 | 1.4 | 18.5 |
| 93 | 0.2.0 | | **Let a chunk's worker start below the project's model, once the archive shows chunks on a lower rung took no more review rounds than chunks on the project's model.** | 3 | 2 | 6 | 6 | 7 | 1.1, 1.4 | 22.4 |
| 94 | 0.2.0 | | **Every 20th cycle, have loop-claude measure token spend and routing outcomes, file them here, and interrupt the human only when they call for a routing change.** | 3 | 5 | 6 | 6 | 7 | 1.1, 1.4 | 21.6 |
| 88 | 0.2.0 | | **Define one routing ladder of (model, effort) rungs ordered by a published ranking of their competence, and map a task's R + C to a rung relative to the project's model.** | 3 | 4 | 5 | 6 | 7 | 1.4 | 21.4 |
| 90 | 0.2.0 | | **Route each reviewer by the chunk's R + C, once a replay of past chunks shows the routed reviewer finds the 🔴 and 🟡 findings on record.** | 4 | 5 | 8 | 6 | 7 | 1.4 | 20.2 |
| 91 | 0.2.0 | | **Start each chunk's worker on the rung its item's R + C picks, never below the project's model, and step it one rung up after two unfinished verdicts on one chunk.** | 4 | 5 | 6 | 6 | 7 | 1.4 | 20.2 |
| 105 | 0.2.0 | decision | **Compare review rounds per pass and reading cost for the five chunks before and after the `worker:` decision moved the principles out of the worker's context, and file an item if either rose.** | 1 | 2 | 3 | 5 | 6 | 1.4 | 20.0 |
| 92 | 0.2.0 | | **Have the worker route each subagent by its task's R + C, and delegate a task only when the worker needs just its conclusion.** | 2 | 2 | 5 | 5 | 6 | 1.4 | 19.0 |
| 110 | 0.2.0 | | **Count, for each finished chunk, the later changes that fixed code it wrote, and show them beside its guessed R.** | 2 | 4 | 6 | 5 | 6 | 1.1, 1.4 | 18.9 |
| 89 | 0.2.0 | | **Run the judge on the rung its chunk's R + C picks, once a replay of archived transcripts shows it returns the verdicts on record.** | 3 | 4 | 7 | 5 | 6 | 1.4 | 17.5 |
| 111 | 0.2.0 | | **Measure the chunks of a project that merges by squash or fast-forward, which `score-outcomes` skips today.** | 3 | 3 | 5 | 4 | 5 | 1.3 | 13.9 |
| 1 | 0.3.0 | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 3 | 5 | 2 | 8 | 9 | 3, 4 | 27.0 |
| 62 | 0.3.0 | defect | **Lift Locality, Self-sufficiency and the overall to the comprehension floor, at the places the seats name.** | 4 | 6 | 6 | 6 | 8 | 1.1 | 22.3 |
| 21 | 0.3.0 | | **Give `settings.json`'s model one writer, and sort its keys.** | 3 | 3 | 4 | 6 | 7 | 1.3 | 21.8 |
| 49 | 0.3.0 | principle | **Check that `AGENTS.md` lists exactly the titles of `docs/decisions.md`.** | 1 | 2 | 2 | 5 | 6 | 1.3 | 20.1 |
| 59 | 0.3.0 | | **Bound a run of unfinished verdicts on one chunk, so a worker and judge that disagree cannot cycle unattended without end.** | 3 | 3 | 5 | 6 | 6 | 1.4 | 19.8 |
| 12 | 0.3.0 | | **Run the test suite in CI on every PR, or record in `docs/decisions.md` why the local run is enough.** | 2 | 4 | 3 | 4 | 6 | 1.1 | 16.9 |
| 80 | 0.3.0 | | **State in README → Install that every machine taking this checkout must be a sandbox, as `settings.json`'s auto mode assumes.** | 1 | 1 | 1 | 5 | 5 | 4 | 16.8 |
| 5 | 0.3.0 | decision | **Key a project on its checkout's full path, and refuse to start while another session uses that key.** | 5 | 5 | 7 | 5 | 6 | 1.1 | 15.6 |
| 84 | 0.3.0 | | **Check, at each release's scoring run, that seats in a paired ruling agree more often than two scoring runs on near-identical code do, as the paired-ruling decision assumes.** | 2 | 2 | 5 | 4 | 5 | 1.4 | 15.1 |
| 67 | 0.3.0 | principle | **Point stability-reviewer at the technical principles in place of its own restatement of them.** | 2 | 3 | 2 | 4 | 5 | 1.3 | 14.9 |
| 106 | 0.3.0 | | **Make larv-review's "Do not over-read a clean result" cite a rule that says Renovate or Dependabot owns dependency CVEs, or drop the citation.** | 1 | 1 | 1 | 3 | 5 | 1.1 | 14.7 |
| 66 | 0.3.0 | principle | **Hold systems-architect's method thresholds only while the comprehension panel finds the flagged code hard.** | 3 | 3 | 4 | 4 | 5 | 1.1 | 14.2 |
| 101 | 0.3.0 | principle | **Have `larv_review/record.py`'s `read_seats` take the time, the commit check and the history as inputs, so it parses and decides without effects.** | 3 | 3 | 3 | 4 | 5 | 1.1 | 14.2 |
| 29 | 0.3.0 | | **Make the contributor a standing member of every project's audience, unless its `## Audience` says maintainability does not matter.** | 2 | 2 | 3 | 3 | 5 | 1.1, 1.4 | 13.4 |
| 68 | 0.3.0 | principle | **Test `workflow-bench/compare.py`'s blinding and `bench.py`'s run plan.** | 2 | 4 | 4 | 3 | 5 | 1.1 | 12.9 |
| 65 | 0.3.0 | | **Archive each judge reply the loop rejects.** | 3 | 4 | 3 | 4 | 5 | 3 | 12.5 |
| 82 | 0.3.0 | principle | **Sort `AGENTS.md`'s decision titles and `docs/decisions.md`'s entries by title.** | 1 | 2 | 2 | 3 | 4 | 1.3 | 12.3 |
| 73 | 0.3.0 | | **Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.** | 3 | 4 | 4 | 4 | 4 | 1.4 | 11.7 |
| 6 | 0.3.0 | decision | **Move what `loop-claude` writes out of `~/.claude.json`, and ask the human about each write that must stay.** | 5 | 5 | 6 | 4 | 5 | 1.1 | 11.7 |
| 51 | 0.3.0 | principle | **Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints.** | 3 | 5 | 5 | 3 | 5 | 1.1 | 11.7 |
| 102 | 0.3.0 | | **Give loop-claude's retry delay an environment override, so the black-box test stops replacing `time.sleep` through `sitecustomize`.** | 2 | 2 | 2 | 3 | 4 | 1.1 | 11.4 |
| 50 | 0.3.0 | principle | **Grant the bench's runs only the tools a review needs, in place of skipping every permission.** | 2 | 2 | 3 | 3 | 4 | 4 | 9.9 |
| 81 | 0.3.0 | principle | **Have `workflow-bench/bench.py` return each run's failure as a value, so one failed clone does not hide the later runs' results.** | 2 | 2 | 3 | 3 | 4 | 4 | 9.9 |
| 4 | 0.3.0 | | **Name each chunk's archive folder by the chunk's start time, so `<key>/` lists chunks in the order they ran.** | 3 | 2 | 3 | 4 | 3 | 3 | 9.6 |
| 52 | 0.3.0 | principle | **Record why this checkout lives in Claude Code's own directory, or move it out and link it in.** | 6 | 6 | 4 | 3 | 5 | 2 | 7.5 |
| 77 | 0.4.0 | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 3 | 6 | 6 | 6 | 7 | 1.1, 1.4 | 21.4 |
## Details
### 98. Score R and S from a measurement wherever one exists, and calibrate each guessed score against the finished chunk's measured outcome.
A score that can be computed from the code it touches, such as the files an item names, comes from that computation. Otherwise size and risk are estimated from finished work, the way reference-class forecasting estimates a project from earlier projects like it. When a chunk merges, record its measured outcome beside the item's guessed R and S: lines and files changed, review rounds, 🔴 and 🟡 findings, and fixes the code needed later. Show the scorer, when it files an item, the finished items whose guessed scores sat nearest and what they measured. Once enough chunks are recorded, report how far each guess was from its outcome, so a scorer whose S runs high can be corrected. Item 87's C is scored and calibrated the same way.
### 99. Drop the comprehension panel as a merge gate, and keep recording its scores and the places in the code its seats found hard, for later measurement and for other reviewers and agents to improve.
No published study validates simulated reader panels as a measure of how easy code is to understand, and item 84's test of whether the panel's signal beats its noise has not run. Keep running the panel where it runs, and keep `docs/comprehension-history.md`: each run's scores, and each place a seat named with what it found hard there. No paired ruling, floor or ratchet blocks a merge. The architecture pass and the stability reviewer read the latest places named, and each chunk that touches one of them improves it or says in its PR summary why not. A worker filing items reads them too.
This revises the `comprehension floor:` decision and the `review:` decisions on the paired ruling, the scoring run, suspending and opting out of the comprehension check, and it turns item 62 from a defect into an ordinary item scored by its value. A project's own gate rule, such as the one in fejkdata's `AGENTS.md`, is that project's to change.
### 87. Score every `todo.md` item's complexity C from 1 to 10, and have `rank_todo.py` fail on an item without one.
C is how hard the change is to get right: novel design, subtle invariants, many interacting parts. It differs from R, the chance the change creates serious bugs, and from S, its size: a huge rename is low C, a ten-line lock fix can be high C. C routes work (items 88–92) and stays out of the Score formula. Add the column to `rank_todo.py`'s valid shape and its `--help`, backfill every item in this repo, and state in `~/.claude/rules/todo-items.md` that C is scored when an item is filed, like R, S, A and G, so no chunk starts on an unscored item.
### 97. Narrow each review pass to the findings worth a fix, so a pass converges in its second round.
Anthropic's best-practices page: "A reviewer prompted to find gaps will usually report some, even when the work is sound… Chasing every finding leads to over-engineering." 320 of this repo's 876 commits answer a review round.
@@ -108,8 +100,6 @@ Needs 88, 91, 92.
One ladder, the same in every project (goal 1.3), read by loop-claude, larv-review and the worker. A rung is a model and an effort; the rungs run from Haiku at low effort to the strongest model at max effort. Order the rungs by a published source, cited in the decision recording the ladder; where no source orders two rungs, such as Sonnet at xhigh against Opus at medium, ask the human. R + C moves a task up or down from the rung holding the project's model at its default effort: a low sum steps down, a high sum steps up, the thresholds set in the decision. S never moves a rung: a huge task with low R and C runs cheap. Leave out the comprehension panel and the security pass, which stay on the project model. Record each route in loop-claude's archive, so the outcomes can tune the thresholds.
Needs 87.
### 90. Route each reviewer by the chunk's R + C, once a replay of past chunks shows the routed reviewer finds the 🔴 and 🟡 findings on record.
larv-review passes the rung through the Agent tool's `model` and `effort`. A reviewer never steps down when the worker has stepped down on the same chunk, so a cheap worker always meets a reviewer on the project model or above.
@@ -132,12 +122,20 @@ State in `CLAUDE.md` when the worker delegates: a search or read that spans many
Needs 88.
### 110. Count, for each finished chunk, the later changes that fixed code it wrote, and show them beside its guessed R.
`score-outcomes` measures R only by the 🔴 findings review caught. A serious bug review missed shows later, as a change that fixes code the chunk wrote. Item 98 asked for both; this is the second half.
### 89. Run the judge on the rung its chunk's R + C picks, once a replay of archived transcripts shows it returns the verdicts on record.
loop-claude starts the judge itself (`bin/loop_claude/cycle/judge.py`), so it routes without any skill change. Replay the judge on archived cycles at the candidate rung, and switch only when its interrupts and verdicts match those on record.
Needs 88.
### 111. Measure the chunks of a project that merges by squash or fast-forward, which `score-outcomes` skips today.
`score-outcomes` reads a chunk from a merge commit: its branch's round titles, and its change against the first parent. visthus and fejkdata merge without merge commits, so their scorers see no finished item.
### 1. Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.
It lives under `~/.claude`, which `.gitignore` keeps from the remote, so a lost disk loses every unattended run's record. Back it up only to a destination private to me and encrypted at rest: the worker runs with permissions bypassed, so the archived tool results can hold credentials it read.