Leave items a chunk drops by a 'Drop item' title out of score-outcomes, check this repo's branches with their own rank_todo.py, and test the replays' model, floor and checkout

This commit is contained in:
2026-11-09 01:14:42 +01:00
parent 19c09e0526
commit bd96cb7c03
11 changed files with 128 additions and 29 deletions
+2
View File
@@ -10,6 +10,8 @@
- This setup's own plan now ranks items by `Score = 100 * V / (2**S + 32)`, up from 8: F, a chunk's fixed cost, is 32, about one size-5 change, fitted to ten finished chunks' tokens, so small items rank lower against large ones. `docs/decisions.md` holds the fit.
- `loop-claude` now keeps the cycle it last measured the routing figures at, and each routing call it raised, in `routing-state.json` at the archive's root, and reads them back from there in place of `routing-figures.md` and `routing-calls-raised.md`. The two markdown files hold what they held before. From them, loop-claude now reads only a `Reset` line in `routing-calls-raised.md`, which still resets the figures. The first loop to close a cycle after the upgrade measures the figures at once. If the figures still make a call raised since the latest reset, it puts that call to you once more.
- `loop-claude` and the routing figures no longer read a `Worker:` line written as a sentence, as versions before 0.4.0 archived. A rerun routes a chunk whose archive still ends in one afresh, and the figures leave it out of the routed chunks and charge its dollars to a row named by the sentence.
- `score-outcomes` no longer credits an item a chunk removed from `todo.md` without building it with that chunk's measurements. Remove such an item in a commit whose title opens `Drop item 69: <why>`, or `Drop items 69, 70 and 71: <why>`, and `score-outcomes` leaves it out. Items dropped before this release still count as shipped.
- rule-hook's `todo-rank-check` now checks a branch of this setup's repo with that branch's own `bin/rank_todo.py`, so a branch that changes `todo.md`'s shape commits without an override. Any other repo is still checked with `~/.claude`'s.
## 0.6.0
+13 -6
View File
@@ -14,6 +14,8 @@ NOT_WORK = ("todo.md", HISTORY, RECORD)
LOG = ("log", "--format=%x00%H %ct %at %P%x1f%h%x1f%s", "--name-only", "--diff-merges=first-parent")
# One commit's record in LOG's output, without the NUL that opens it.
LOG_RECORD = re.compile(r"([0-9a-f]+) ([0-9]+) ([0-9]+) ([0-9a-f ]*)\x1f([0-9a-f]+)\x1f(.*)", re.DOTALL)
# A commit title that drops items unbuilt opens so: `Drop item 69: <why>`, or `Drop items 69, 70 and 71: <why>`. A panel of three readers seated as the audience picked it 3-0, over the words anywhere in a title and over a trailer, since a log skimmed by its titles shows it first.
DROP = re.compile(r"Drop items? ([0-9]+(?:, [0-9]+)*(?: and [0-9]+)?):")
# How a chunk reached the default branch; rebase and push leave no merge commit to bound it.
Landing = typing.Literal["merge", "rebase", "push"]
@@ -43,7 +45,7 @@ class Span:
@dataclasses.dataclass(frozen=True)
class BranchChunk(Span):
"""A Span with two more fields: shipped holds each todo.md item its change removed, as scored before the change; fixed_by names the later chunks that fixed code it shipped, newest first."""
"""A Span with two more fields: shipped holds each todo.md item its change removed and no title of it dropped, as scored before the change; fixed_by names the later chunks that fixed code it shipped, newest first."""
shipped: tuple[Item, ...]
fixed_by: tuple[str, ...]
@@ -135,15 +137,20 @@ def owners(runs: list[list[Commit]], commits: dict[str, Commit]) -> dict[str, in
return owner
def shipped(before: str, after: str) -> list[Item]:
"""Each item the todo.md before a change holds and the todo.md after it does not, as scored before; none where either holds no items table it can read."""
def shipped(before: str, after: str, titles: tuple[str, ...] = ()) -> list[Item]:
"""Each item the todo.md before a change holds and the todo.md after it does not, as scored before, less those a title of the change drops; none where either holds no items table it can read."""
planned, left = items(before), items(after)
if not isinstance(planned, list) or not isinstance(left, list):
return []
kept = {item.id for item in left}
kept = {item.id for item in left} | dropped(titles)
return sorted((item for item in planned if item.id not in kept), key=lambda item: item.id)
def dropped(titles: tuple[str, ...]) -> set[int]:
"""The IDs of the items the titles drop, each title opening as DROP reads."""
return {int(number) for title in titles if (found := DROP.match(title)) for number in re.findall(r"[0-9]+", found[1])}
def plan_revisions(branch: Branch) -> list[str]:
"""The revisions whose todo.md shipped_by needs: for each chunk that brought a commit changing todo.md, other than the branch's oldest chunk, the commit before it and its tip."""
changed = sorted({branch.owner[commit.hash] for commit in branch.commits.values() if "todo.md" in commit.files})
@@ -156,11 +163,11 @@ def plan_revisions(branch: Branch) -> list[str]:
def shipped_by(branch: Branch, plans: dict[str, str]) -> list[tuple[Item, ...]]:
"""The items each of the branch's chunks shipped, in the order of its chunks. plans holds todo.md's text at each revision plan_revisions names. A chunk with a revision plans lacks changed no plan, so it shipped none."""
"""The items each of the branch's chunks shipped, in the order of its chunks. plans holds todo.md's text at each revision plan_revisions names. A chunk with a revision plans lacks changed no plan, so it shipped none; an item a title of the chunk drops it did not ship either."""
found: list[tuple[Item, ...]] = []
for chunk in branch.chunks:
if chunk.before is not None and chunk.before in plans and chunk.tip in plans:
found.append(tuple(shipped(plans[chunk.before], plans[chunk.tip])))
found.append(tuple(shipped(plans[chunk.before], plans[chunk.tip], chunk.titles)))
else:
found.append(())
return found
+19
View File
@@ -57,6 +57,18 @@ class Shipped(unittest.TestCase):
older = "| 4 | 0.1.0 | | **A \\| B.** | 2 | 3 | 6 | 7 | 1 | 9.0 |"
self.assertEqual(shipped(todo(older, header=WITHOUT_C), todo(header=WITHOUT_C))[0].scores, {"R": 2, "S": 3, "A": 6, "G": 7})
def test_leaves_out_each_item_a_title_opening_drop_item_names(self):
third = "| 7 | 0.1.0 | | **And that.** | 4 | 6 | 8 | 6 | 7 | 1 | 9.0 |"
eighth = "| 8 | 0.1.0 | | **Or this.** | 4 | 6 | 8 | 6 | 7 | 1 | 9.0 |"
before = todo(FIRST, SECOND, third, eighth)
self.assertEqual([found.id for found in shipped(before, todo(), ("Build item 4", "Drop item 5: item 4 covers it"))], [4, 7, 8])
self.assertEqual([found.id for found in shipped(before, todo(), ("Drop items 5, 7 and 8: moot",))], [4])
self.assertEqual([found.id for found in shipped(before, todo(), ("Drop items 5 and 7: moot",))], [4, 8])
def test_ships_an_item_a_title_drops_only_mid_line_or_without_the_colon(self):
titles = ("Keep the steps and drop item 4, with its measurement", "Drop item 5 later", "Drop item 45: another")
self.assertEqual([found.id for found in shipped(todo(FIRST, SECOND), todo(), titles)], [4, 5])
def test_ships_nothing_where_either_plan_has_no_table_it_can_read(self):
self.assertEqual(shipped(todo(FIRST), "# todo\n\n- [ ] Do it.\n"), [])
self.assertEqual(shipped(todo(FIRST, "| x | 0.1.0 | | **Bad.** | 2 | 3 | 5 | 6 | 7 | 1 | 9.0 |"), todo()), [])
@@ -278,6 +290,13 @@ class BranchChunks(unittest.TestCase):
self.assertEqual((found[1].tip, found[2].tip), (self.git("rev-parse", "HEAD^").strip(), self.git("rev-parse", "HEAD^^").strip()))
self.assertEqual((found[1].before, found[2].before), (found[2].tip, None))
def test_a_chunk_that_drops_an_item_by_a_commit_title_ships_it_not(self):
self.merge("four", "Do four", {"b.py": "x\ny\n", "todo.md": todo(SECOND, FIX)}, "2026-10-13T10:00:00Z")
self.merge("six", "Drop item 6: item 4 fixed it", {"b.py": "x\nY\n", "todo.md": todo(SECOND)}, "2026-10-14T10:00:00Z")
found = self.read()
self.assertNotIsInstance(found, Failure)
self.assertEqual([([item.id for item in found.shipped], found.fixed_by) for found in found[:2]], [([], ()), ([4], ())])
def test_fails_where_git_cannot_read_the_branch(self):
self.assertEqual(branch_chunks("main", lambda *args: Failure("no log")), Failure("no log"))
+8
View File
@@ -151,6 +151,14 @@ class Main(unittest.TestCase):
self.assertEqual([(model, effort) for model, effort, _ in self.calls[::2]][-1], ("claude-sonnet-5-5", "high"))
self.assertTrue(out.endswith("returns every verdict the reference judge reproduces: claude-sonnet-5-5 at high effort\n"), out)
def test_a_fable_project_tries_no_rung_below_opus_at_max_and_its_report_names_fable(self):
(self.home / ".claude" / "loop-claude-models.json").write_text(json.dumps({"proj": "claude-opus-5-5", "other": "claude-fable-5-1"}))
code, out, err = self.run_main(["claude-fable-5-1"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
self.assertEqual(code, 0, err)
self.assertEqual([(model, effort) for model, effort, _ in self.calls][-1], ("claude-opus-5-5", "max"))
self.assertTrue(out.splitlines()[0].startswith("Replayed 1 verdicts of claude-fable-5-1 projects"), out)
self.assertTrue(out.endswith("returns every verdict the reference judge reproduces: claude-opus-5-5 at max effort\n"), out)
def test_stops_at_the_projects_rung_and_names_none_when_it_misses(self):
code, out, _ = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted and rung == REFERENCE, f"said {INTERRUPTED if interrupted else FINISHED}"))
self.assertEqual(code, 0)
+5
View File
@@ -179,6 +179,7 @@ class Main(unittest.TestCase):
self.write_chunk(key, "c1", "2026-10-14T10:00:00Z")
self.reviews = []
self.agents = set()
self.matched_in = []
def write_chunk(self, key, chunk, at, began="2026-10-14T09:15:00Z"):
"""A chunk that began at began, whose stability round began at at."""
@@ -215,6 +216,7 @@ class Main(unittest.TestCase):
self.agents.add(args[args.index("--agent") + 1])
result = review(rung)
else:
self.matched_in.append(cwd)
reviewed = Rung(*re.search(r"review on (\S+) (\S+)", prompt).groups())
result = found(reviewed)
result = result if "api_error_status" in result else {"type": "result", "structured_output": {"found": [n + 1 for n in result["found"]]}}
@@ -238,6 +240,9 @@ class Main(unittest.TestCase):
self.assertIn("The reference reviewer, claude-opus-5-5 at medium effort, found 2 of them.", out)
self.assertIn(f" {self.checkout} 2026-10-14T10:00:00Z: 🟡 slow", out)
self.assertTrue(out.endswith("finds: none below the project's rung\n"), out)
# Claude matches each replay's findings in the round's checkout, and the report names the model replayed.
self.assertEqual(set(self.matched_in), {str(self.checkout)})
self.assertTrue(out.startswith("Replayed 1 stability rounds of claude-opus-5-5 projects,"), out)
def test_replays_the_prose_pass_with_the_prose_reviewer(self):
code, out, err = self.run_main(["prose", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
+1 -1
View File
@@ -80,7 +80,7 @@ RULES: dict[RuleName, Rule] = {
"a docker build, docker buildx build or docker compose build, and a run of cargo mutants, Infection, mutmut or Stryker, the mutation tools stability-reviewer names, whether direct, through a launcher such as uvx, or inside docker run, docker exec or docker compose run or exec. A gate script is not checked, since no program can tell it from a light one",
),
"todo-rank-check": Rule(
"while todo.md has changes, staged or not, a git commit waits until ~/.claude/bin/rank_todo.py --check passes on it",
"while todo.md has changes, staged or not, a git commit waits until rank_todo.py --check passes on it: the checkout's own bin/rank_todo.py where the checkout is of this setup's repo, else ~/.claude/bin/rank_todo.py",
"a git commit in a checkout whose todo.md has changes",
),
}
+44
View File
@@ -18,6 +18,7 @@ from rule_hook.call import Call, parsed_call
from git_cli.git_command import commit_message, CommitCall, commits
from release_plan.test_ranking import DECISIONS, README, TODO
from rule_hook.git import guarded_git
from rule_hook.todo_check import todo_breaches
from test_env import isolated_env
from rule_hook.override_log import LogLine, parsed_entry, this_chunks
from rule_hook.reviewer_rung import reviewer_breaches, skill_breaches
@@ -850,3 +851,46 @@ class Hook(unittest.TestCase):
self.assertIn("reviewer-report\n The rule: a reviewer writes nothing but its report", helped)
self.assertRegex(helped, r" reviewer-report\n( .*\n){2} Takes no override\.\n")
self.assertRegex(helped, r" image-pin\n( .*\n){2} \S", "a rule a marker waives says nothing of overrides")
class TodoRankCheck(unittest.TestCase):
"""A checkout of the repo the hook runs from checks todo.md with its own bin/rank_todo.py; any other checkout with the hook's."""
STUB = "import sys\nprint('own script checked ' + sys.argv[-1], file=sys.stderr)\nsys.exit(int(sys.argv[1] == '--check'))\n"
def setUp(self):
self.folder = tempfile.mkdtemp()
self.addCleanup(subprocess.run, ["rm", "-rf", self.folder])
self.own = f"{self.folder}/own"
pathlib.Path(self.own).mkdir()
git(self.own, "init", "-q", "-b", "main")
git(self.own, "commit", "-q", "--allow-empty", "-m", "root")
git(self.folder, "clone", "-q", self.own, "clone")
self.other = f"{self.folder}/other"
pathlib.Path(self.other).mkdir()
git(self.other, "init", "-q", "-b", "main")
git(self.other, "commit", "-q", "--allow-empty", "-m", "another root")
def changed_plan(self, checkout: str, stub: str) -> str:
pathlib.Path(checkout, "bin").mkdir()
pathlib.Path(checkout, "bin", "rank_todo.py").write_text(stub)
pathlib.Path(checkout, "todo.md").write_text("# todo\n")
return checkout
def details(self, checkout: str) -> list[str]:
return [breach.detail for breach in todo_breaches(checkout, pathlib.Path(self.own))]
def test_a_checkout_of_the_hooks_repo_runs_its_own_rank_todo(self):
checkout = self.changed_plan(f"{self.folder}/clone", self.STUB)
self.assertEqual(self.details(checkout), [f"own script checked {checkout}/todo.md"])
def test_its_own_rank_todo_passing_passes(self):
self.assertEqual(self.details(self.changed_plan(f"{self.folder}/clone", "")), [])
def test_its_own_rank_todo_failing_silently_is_named(self):
checkout = self.changed_plan(f"{self.folder}/clone", "import sys\nsys.exit(3)\n")
self.assertEqual(self.details(checkout), [f"{checkout}/bin/rank_todo.py --check failed with exit 3 and printed nothing"])
def test_another_repos_rank_todo_never_runs(self):
reasons = self.details(self.changed_plan(self.other, self.STUB))
self.assertTrue(reasons)
self.assertNotIn("own script", " ".join(reasons))
+32 -2
View File
@@ -1,5 +1,7 @@
import hashlib
import pathlib
import subprocess
import sys
from rule_hook.rules import Breach
from failure import Failure
@@ -7,12 +9,40 @@ from rule_hook.git import guarded_git
from release_plan import ranking
def todo_breaches(directory: str) -> list[Breach]:
# The checkout this hook runs from: bin/rule_hook/ sits two folders below it.
OWN = pathlib.Path(__file__).resolve().parents[2]
# rank_todo.py --check takes about a tenth of a second; the hook must answer well within Claude Code's minute.
OWN_CHECK_SECONDS = 20
def todo_breaches(directory: str, own: pathlib.Path = OWN) -> list[Breach]:
top = guarded_git(directory, "rev-parse", "--show-toplevel")
if isinstance(top, Failure) or not (pathlib.Path(top) / "todo.md").is_file():
return []
todo = pathlib.Path(top) / "todo.md"
return [Breach("todo-rank-check", line, top) for line in ranking.check_failures(todo)] if changed(top, todo) else []
return [Breach("todo-rank-check", line, top) for line in check_failures(top, todo, own)] if changed(top, todo) else []
def check_failures(top: str, todo: pathlib.Path, own: pathlib.Path) -> list[str]:
"""Why rank_todo.py --check fails on todo. A checkout of the repo the hook runs from checks with its own bin/rank_todo.py, so a branch that changes the table's shape is held to its own rules. Any other checkout checks with the hook's ranking: running its script would run its code before every commit."""
script = pathlib.Path(top) / "bin" / "rank_todo.py"
if not script.is_file() or not same_repo(top, own):
return ranking.check_failures(todo)
try:
ran = subprocess.run([sys.executable, str(script), "--check", str(todo)], capture_output=True, cwd=top, text=True, timeout=OWN_CHECK_SECONDS)
except subprocess.TimeoutExpired:
return [f"{script} --check did not finish in {OWN_CHECK_SECONDS} seconds"]
except OSError as failure:
return [f"{script} --check could not run: {failure}"]
if ran.returncode == 0:
return []
return ran.stderr.splitlines() or [f"{script} --check failed with exit {ran.returncode} and printed nothing"]
def same_repo(top: str, own: pathlib.Path) -> bool:
"""Whether the checkout's history and the hook's share a root commit."""
theirs, mine = guarded_git(top, "rev-list", "--max-parents=0", "HEAD"), guarded_git(str(own), "rev-list", "--max-parents=0", "HEAD")
return not isinstance(theirs, Failure) and not isinstance(mine, Failure) and bool(set(theirs.split()) & set(mine.split()))
def changed(top: str, todo: pathlib.Path) -> bool:
+2 -1
View File
@@ -9,7 +9,8 @@ reads origin as last fetched and says so on stderr.
Git history is the record. The default branch's first-parent line is cut into chunks by one rule: a
merge commit is a chunk of its own, and any other commit joins the run of the newer commit above it when
both share a commit time, else starts a chunk. A chunk whose change removed items from todo.md, and changed more
than todo.md and the records below, shipped those items. Each item's guess is the scores it held in
than todo.md and the records below, shipped those items, less any a commit title of the chunk drops by
opening `Drop item 69: <why>` or `Drop items 69, 70 and 71: <why>`. Each item's guess is the scores it held in
the todo.md the chunk started from: L, E, I, R and S. An item scored before L, E and I replaced C
holds R and S, and C too where its table also had A, G and Goals. The chunk measures:
+1
View File
@@ -28,3 +28,4 @@ paths:
- **An item scoring under the bar is deleted, with every item that needs it, and a finding under the bar is not filed.** The chunk's summary names each, with its score.
- **The Exempt cell keeps an item whatever its score:** `security` for a security finding, `defect` for a 🔴 finding or an item that restores a maintain goal the project breaks today, `question` for a question only the human can answer, which ranks first in its release, `decision` for an item `docs/decisions.md` cites, `owner` for an item the human told you to keep, and, where no other reason fits, `principle` for a 🟡 finding against a file under `~/.claude/principles/`, whose item's detail names that principle. Every item `docs/decisions.md` cites carries one of these reasons.
- **A maintain goal vetoes an item that breaks it,** whatever the item scores: redesign the item so it holds the goal, or drop it.
- **Remove an item that no chunk built in a commit titled `Drop item <ID>: <why>`,** or `Drop items 12, 14 and 15: <why>`. Otherwise `score-outcomes` counts it as shipped by the chunk that removed it, measured by that chunk's work. A deleted, vetoed, superseded or moot item is such an item; a finished one is not.
+1 -19
View File
@@ -14,7 +14,6 @@
|---|---|---|---|---|---|---|---|---|---|---|---|
| 277 | 0.7.0 | | question | **Map the size of the code an item names onto S, on a scale the human picks, and have `score-outcomes` print that S for the item being scored and for each finished item it lists.** | 3 | 2 | 2 | 5 | 4 | 1.1 by 4, 1.4 by 3 | 2.4 |
| 1 | 0.7.0 | | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 1 | 2 | 1 | 2 | 1 | | 0.0 |
| 112 | 0.7.0 | | | **Count an item as shipped only where its chunk built it, so an item a chunk dropped from the plan gets no outcome.** | 3 | 2 | 2 | 5 | 3 | 1.1 by 6 | 2.5 |
| 276 | 0.7.0 | | | **Let a worker fix on its own rung a `standards/quality.md` finding whose fix changes only prose, so escalation trigger 4 calls for a higher rung only where code or config must change.** | 2 | 2 | 3 | 5 | 3 | 1.2 by 4, 1.4 by 2 | 2.4 |
| 279 | 0.7.0 | | | **Add an item to `todo.md` with one command that reads `Next ID`, bumps it, and inserts the row and its detail.** | 3 | 1 | 3 | 5 | 3 | 1.4 by 3, 1.5 by 3 | 2.3 |
| 116 | 0.7.0 | | | **Report L by the share of chunks with a 🔴, once 10 chunks that each shipped one item record finding counts and most record no 🔴.** | 1 | 1 | 2 | 2 | 2 | 1.1 by 5 | 2.3 |
@@ -22,14 +21,12 @@
| 282 | 0.7.0 | | | **Write the branch diff and what `larv-review places` prints into the chunk's review folder, and print each reviewer's prompt header, with one `larv-review` command.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 4, 1.5 by 3 | 2.3 |
| 12 | 0.7.0 | | | **Run the test suite in CI on every PR, or record in `docs/decisions.md` why the local run is enough.** | 3 | 2 | 1 | 4 | 4 | 1.1 by 6 | 2.1 |
| 278 | 0.7.0 | | | **Remove a finished item's row and detail from `todo.md` and add its changelog bullet under `## Unreleased` with one command, given the item's ID and the bullet.** | 3 | 1 | 3 | 5 | 3 | 1.4 by 3, 1.5 by 2 | 2.0 |
| 173 | 0.7.0 | | | **Have rule-hook's `todo-rank-check` run a worktree's own `bin/rank_todo.py` when the worktree is of this repo, so a branch that changes `todo.md`'s shape commits without an override.** | 2 | 1 | 3 | 4 | 2 | 1.1 by 2, 1.4 by 2 | 1.8 |
| 285 | 0.7.0 | | | **Cut a worker's opens and searches of `todo.md` from the median 7 per chunk to at most 3, by finding in the last five chunks' archived transcripts what each read was for and removing its cause.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 4 | 1.6 |
| 281 | 0.7.0 | | | **Remove a merged chunk's worktree and branch and fast-forward the main checkout with one command.** | 3 | 2 | 3 | 6 | 3 | 1.4 by 2, 1.5 by 2 | 1.6 |
| 283 | 0.7.0 | | | **Print, for the archived chunks of a release, a table of its shell command patterns, such as `tea pr create`: the chunks that ran each, its calls, command tokens, result tokens and tool time, so an audit reads one table.** | 3 | 2 | 1 | 4 | 3 | 1.4 by 2, 1.5 by 2 | 1.6 |
| 272 | 0.7.0 | | owner | **Make `deploy-release --cut` say when each step finishes, end on one line saying the release is cut and `~/.claude` is on it, and delete the merged release branch.** | 1 | 1 | 1 | 1 | 2 | 1.5 by 3 | 1.3 |
| 274 | 0.7.0 | | | **Let an item name a maintain goal it raises confidence in, such as a test leg that guards it, and weigh that move against the achieve goals in `bin/release_plan/ranking.py`.** | 2 | 2 | 3 | 5 | 4 | 1 by 3 | 1.1 |
| 286 | 0.7.0 | | | **Lower the reading cost of the files a chunk reads most before its first commit.** | 3 | 2 | 2 | 5 | 5 | 1.4 by 4 | 1.0 |
| 275 | 0.7.0 | | | **Make the `Main` tests of `judge-replay` and `review-replay` fail when the replays ignore the project's model, its lowest worker rung or the round's checkout.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 2 | 0.9 |
| 284 | 0.8.0 | | owner | **Script each step that 0.7.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.6 |
| 77 | 0.8.0 | | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 4 | 2 | 1 | 5 | 6 | 1.1 by 7, 1.4 by 7 | 2.4 |
| 273 | 0.8.0 | | decision | **Re-tune the E + I steps of `REVIEW_STEPS` for 6 to 8 once 10 finished chunks have scored an E + I of 6 or more, from how well E + I orders their review rounds and 🔴 findings.** | 1 | 2 | 2 | 3 | 2 | 1.1 by 2, 1.4 by 3 | 2.3 |
@@ -83,10 +80,6 @@ It lives under `~/.claude`, which `.gitignore` keeps from the remote, so a lost
The README line covers the bench's transcripts in `workflow-bench/logs/` too (goal 3).
### 112. Count an item as shipped only where its chunk built it, so an item a chunk dropped from the plan gets no outcome.
`score-outcomes` counts every item a merge removed from `todo.md` as shipped. A merge that changed only the plan ships nothing, but one that built an item and also dropped another gives the dropped item the built one's outcome, as PR #253 did with item 69. Mark a dropped item in the change that drops it, such as by the commit or merge title, and leave it out.
### 276. Let a worker fix on its own rung a `standards/quality.md` finding whose fix changes only prose, so escalation trigger 4 calls for a higher rung only where code or config must change.
On 2026-11-08, PR #432's prose round 1 reported two findings against `standards/quality.md` → "A false claim is a bug": a docstring at `bin/release_plan/todo_table.py:61` that named the wrong module, and a sentence in `docs/decisions.md`. That fired escalation trigger 4, and the worker asked to hand both fixes to an opus max subagent. The human answered: "Just prose changes, run the fix on the same rung." Write that into trigger 4 in `CLAUDE.md` → "Working with the codebase", into larv-review's `SKILL.md` §3 → "Rung up", and into the `docs/decisions.md` entry `review: a reviewer or comprehension seat that rules the work below standard…`, whose background says work below standard gets a more capable rung to fix it. A comment or docstring counts as prose, as larv-review §3 says.
@@ -117,10 +110,6 @@ When CI runs a rule check, such as one of `bin/rule-hook`'s, let `<!-- rule-over
Audit of release 0.2.0's chunks (item 162): 16 chunks edited `CHANGELOG.md` and at least 10 deleted an item's row and its `### <ID>.` detail from `todo.md`, each time with a Python heredoc written by hand. The CHANGELOG edits alone were 33 calls, about 9k command tokens and 6 minutes of tool time. The program removes the row and the detail, creates `## Unreleased` when it is missing, and inserts the bullet; the bullet's wording stays with the model.
### 173. Have rule-hook's `todo-rank-check` run a worktree's own `bin/rank_todo.py` when the worktree is of this repo, so a branch that changes `todo.md`'s shape commits without an override.
Raised by the overrides pass of PR #296. The hook imports the main checkout's `release_plan/ranking.py`, so while a branch of this repo changes the table's shape, every commit touching `todo.md` needs a `todo-rank-check` override: 33 log lines across four branches, the score-outcomes, plan-136, plan-148 and item-146 ones. The hook then checks nothing, and the rule rests on the worker chaining the branch's own check. Run another repo's `bin/rank_todo.py` never: that would run its code before every commit.
### 285. Cut a worker's opens and searches of `todo.md` from the median 7 per chunk to at most 3, by finding in the last five chunks' archived transcripts what each read was for and removing its cause.
Item 105 compared the five chunks before and after the `worker:` decision, merged 2026-10-13 at 06:49Z in 21b060e, matching chunks to PRs by finish time. Reading cost, the new input tokens before the first commit, rose: median 56,802 before and 136,132 after. The ranges, 46k to 172k and 50k to 140k, overlap. Review rounds per pass did not rise: 2.26 before and 1.52 after, among the five multi-pass chunks on each side.
@@ -165,17 +154,10 @@ goal is still redesigned or dropped.
### 286. Lower the reading cost of the files a chunk reads most before its first commit.
`reading-cost` reports Rising: the last five chunks' median is 151,930 new input tokens before the first commit, against 84,028 for the five before them. Those five chunks opened or searched these files most: `todo.md` 40 times, `bin/loop_claude/test_loop_claude.py` 15, `bin/loop_claude/cycle/judge.py` 10, `bin/routing_figures/test_routing_figures.py` 9, `README.md` 8, `docs/decisions.md` 8, `AGENTS.md` 7, `bin/chunk_preflight/choices.py` 7, `bin/rule_hook/repeat_read.py` 7 and `bin/review_replay/test_review_replay.py` 6.
`reading-cost` reports Rising: the last five chunks' median is 151,930 new input tokens before the first commit, against 86,720 for the five before them. Those five chunks opened or searched these files most: `todo.md` 47 times, `bin/loop_claude/test_loop_claude.py` 15, `docs/decisions.md` 11, `bin/loop_claude/cycle/judge.py` 10, `bin/routing_figures/test_routing_figures.py` 9, `README.md` 8, `docs/reading-cost.md` 8, `AGENTS.md` 7, `bin/rule_hook/repeat_read.py` 7 and `bin/review_replay/test_review_replay.py` 6.
Item 285 takes the `todo.md` reads; this item takes the rest. Find why a worker opens each of them before its first commit, and cut that cause, such as a test file too large to read in parts.
### 275. Make the `Main` tests of `judge-replay` and `review-replay` fail when the replays ignore the project's model, its lowest worker rung or the round's checkout.
Found by the stability review of item 219's chunk, findings 1 and 2; both gaps predate it. Mutants that survive today:
- `bin/judge_replay/main.py` `main`: passing `None` in place of `lowest_worker_rung(project)` to `down_from`, since every `Main` test in `bin/judge_replay/test_judge_replay.py` uses an Opus 5.5 project, whose lowest worker rung already is the ladder's bottom. No test checks that the report's first line names the model passed as `report(project.model, ...)` either. Add a `Main` test with a Fable 5.1 project asserting that the replays stop at Opus 5.5 at max effort and that the first line names the model.
- `bin/review_replay/main.py` `findings_found`: passing `None` in place of `found.root` to `match`, since the `Main` tests' fake claude in `bin/review_replay/test_review_replay.py` ignores the folder a match call runs in. The model `main` passes to `report` is unchecked too. Record the folder each match call runs in and assert it is the round's checkout, and assert that the first output line names the model.
### 284. Script each step that 0.7.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.
lilleman asked, on 2026-10-16, for a standing audit each release, so the loop keeps moving work a program does as well from the model to scripts (goals 1.4 and 1.5). This audit reads the archived transcripts of the chunks that shipped release 0.7.0 for every step that chunks repeated by hand, before, during or after review. It keeps the steps whose answer a program computes the same way each time, such as a lookup, a check, a prompt assembled for a reviewer, a cleanup. File one item per such step, with the time and tokens it cost, into the next release with items left. Leave a step that needs judgement with the model. Run item 283's tool first, once it has shipped.