Merge pull request 'Anchor each level of L, E and I with shipped items, and have score-outcomes list them for a scorer to compare against' (#298) from item-152 into main
This commit was merged in pull request #298.
This commit is contained in:
+2
-1
@@ -5,12 +5,13 @@
|
||||
- Every goal in `README.md` → Goals is marked `**Maintain:**`, a state to hold, or `**Achieve:**`, a direction to pursue. Each achieve goal has a signal, a counter-signal and a one-line reason under it. A change that breaks a maintain goal is redesigned or dropped, whatever it scores: `rules/todo-items.md` says so, and larv-review's goal alignment step no longer merges such a PR. `rules/goals-and-audience.md` states the form for every project's README. Expect larv-review to stop a PR on a maintain goal where it asked you before.
|
||||
- Reshape every project's `todo.md` to the new shape below: until it is reshaped, that is the project's only work, as it is while README.md lacks Goals and Audience, and `rank_todo.py` says so. First move every machine to this release: an older release rejects the new formula, so it fails on every reshaped `todo.md`. Then mark the README's goals `**Maintain:**` or `**Achieve:**`, score each item's L, E and I, and set a bar on the new scale below the score of every item to keep, since the old bar would drop most items. Until then, an old-shape `todo.md` still ranks by its own formula and weights.
|
||||
- `todo.md` ranks items by value per cost: this setup's own plan uses `Score = 100 * V / (2**S + 8)`, where 8 is a chunk's fixed cost, about one size-3 change, and `rank_todo.py --help` shows it. A Moves column replaces A, G and Goals, naming each achieve goal an item moves and how far, such as `1.1 by 8, 1.2 by 5`. V sums each named goal's weight times its move. `rank_todo.py` weighs each achieve goal by its position in README.md, the first heaviest, so the goal weights table goes. An item names the most specific goal it moves, never a goal beside its own sub-goal. `rules/todo-items.md` and `rank_todo.py --help` show the new shape.
|
||||
- L, E and I replace C, and set R. L is the chance the change holds a defect, from 1 to 4; E the chance a defect escapes tests and review, from 1 to 3; I the damage it would do, from 1 to 5, counting every project and session that runs the change. `rules/todo-items.md` anchors each level in words, and `rank_todo.py` writes R as L + E + I − 2, at least 7 when I is 5 and at least 5 when I is 4. The table's score columns are `| L | E | I | R | S |`. `score-outcomes` takes `L E I S`, and reports how well L, E and I each order finished chunks by review rounds and 🔴 findings.
|
||||
- L, E and I replace C, and set R. L is the chance the change holds a defect, from 1 to 4; E the chance a defect escapes tests and review, from 1 to 3; I the damage it would do, from 1 to 5, counting every project and session that runs the change. `rules/todo-items.md` defines each level in words, and `rank_todo.py` writes R as L + E + I − 2, at least 7 when I is 5 and at least 5 when I is 4. The table's score columns are `| L | E | I | R | S |`. `score-outcomes` takes `L E I S`, and reports how well L, E and I each order finished chunks by review rounds and 🔴 findings.
|
||||
- `loop-claude` routes a chunk's worker on its item's L, from two rungs below the project's rung at L 1 to one above it at L 4. It routes the reviewers on the item's E + I, one rung above the project's for each point past 4, never below the worker's rung. It runs the judge of an Opus 5.5 project on the rung E + I picks, never below the project's, and keeps it there when the worker steps up. The comprehension seats and the security pass stay on the project's rung. rule-hook holds each reviewer launch to the reviewers' rung, which `loop-claude` hands the worker in `LOOP_CLAUDE_REVIEWER_RUNG`. A delegated task's subagent takes the rung its L picks: `subagent-rung` takes L in place of R + C. Until a project reshapes its `todo.md`, every chunk there runs on the project's rung. Expect reviews of items whose defects would escape far or break much to cost more, and escalation from an item of L 4 to reach Fable 5.1 only with your yes.
|
||||
- An item that restores a maintain goal the project breaks today is exempt as `defect`, and its Moves may name that maintain goal, weighed as the heaviest achieve goal. A defect ranks by its score like any item. Only two kinds rank ahead of the rest of their release: questions first, since an answer may change other items, then security items.
|
||||
- The stability reviewer applies `principles/technical-principles.md` as written, in place of its own copies of it. Its test-data check now asks for UUID v7, where it asked for randomised UUIDs.
|
||||
- larv-review's principles audit is a pass of its own, and runs on every chunk that changes code, config or tests, not only on one with an architectural decision. A chunk of prose or planning alone skips it.
|
||||
- `rule-hook` refuses an Agent call that launches a reviewer, a comprehension seat or the security pass on a rung below the one `loop-claude` names for it, and takes no override: pass the model and effort the refusal names.
|
||||
- `score-outcomes` lists a project's score anchors: the shipped items its `docs/score-anchors.md` names for each level of L, E and I, each with what its chunk measured. `rules/todo-items.md` has a scorer compare a new item with them, and says how to choose two or three per level. Add the file to each project that merges with merge commits; elsewhere, score by the level descriptions in `rules/todo-items.md`.
|
||||
|
||||
## 0.1.0
|
||||
|
||||
|
||||
@@ -89,7 +89,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format.
|
||||
- `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure.
|
||||
- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit, Write and Agent call, and `rule-hook --help` states the rules and the override.
|
||||
- `score-outcomes [L E I S]` lists the finished items whose guessed scores sit nearest the given ones, with what their chunks measured, the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings; `score-outcomes --help` states the measures.
|
||||
- `score-outcomes [L E I S]` lists the finished items whose guessed scores sit nearest the given ones, and the shipped items the project's `docs/score-anchors.md` names for each level of L, E and I, each with what its chunk measured; then the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings; `score-outcomes --help` states the measures.
|
||||
- `stability-replay <model>` replays archived stability reviews of the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again; `stability-replay --help` states the replay. Run it only in a sandbox, like loop-claude: each replayed reviewer has Bash and skips every permission prompt, and an archived prompt may name paths on the host.
|
||||
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
|
||||
- `subagent-rung [--plan] <model> <L>` prints the Agent tool's model and effort for a subagent's task, picked by its L from the project's model, and with `--plan` never below the project's rung; `subagent-rung --help` states the rule.
|
||||
|
||||
+1
-1
@@ -43,7 +43,7 @@ A valid todo.md; ## Details and each item's detail are optional:
|
||||
goal.
|
||||
- The bar is judged on the Score as shown, to one decimal.
|
||||
- A new item takes Next ID, and Next ID then goes up by one; every ID stays below it.
|
||||
- An item's L, E and I are levels that ~/.claude/rules/todo-items.md anchors: L from 1 to 4, E from 1
|
||||
- An item's L, E and I are levels that ~/.claude/rules/todo-items.md defines: L from 1 to 4, E from 1
|
||||
to 3 and I from 1 to 5.
|
||||
rank_todo.py writes its R, L + E + I - 2, at least 7 when I is 5 and at least 5 when I is 4. Its S
|
||||
is a whole number from 1 to 10.
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
import re
|
||||
|
||||
from todo_table import cells, LEVELS, parse_ids
|
||||
from failure import Failure
|
||||
from score_outcomes.outcomes import Outcome, render
|
||||
|
||||
ANCHORS_PATH = "docs/score-anchors.md"
|
||||
# A level, such as `L 1`; every scale tops out below 10.
|
||||
LEVEL = re.compile(r"([A-Z]) ([1-9])")
|
||||
# Each level the anchors table names, such as `L 1`, with its items' IDs, in the file's order.
|
||||
Anchors = list[tuple[str, tuple[int, ...]]]
|
||||
|
||||
|
||||
def parse_anchors(text: str) -> Anchors | Failure:
|
||||
found: Anchors = []
|
||||
for row in (line.strip() for line in text.splitlines() if line.strip().startswith("|")):
|
||||
pair = cells(row, 2)
|
||||
if isinstance(pair, Failure):
|
||||
return Failure(f"{ANCHORS_PATH}: {pair.reason}")
|
||||
if pair == ["Level", "Items"] or all(re.fullmatch(r":?-+:?", cell) for cell in pair):
|
||||
continue
|
||||
level, ids = LEVEL.fullmatch(pair[0]), parse_ids(pair[1])
|
||||
if not level or int(level[2]) > LEVELS.get(level[1], 0) or not ids:
|
||||
scales = ", ".join(f"{name} 1 to {top}" for name, top in LEVELS.items())
|
||||
return Failure(f"cannot read the row {row} in {ANCHORS_PATH}; each row names a level, {scales}, and the IDs of shipped items, such as | L 1 | 12, 14 |")
|
||||
if pair[0] in dict(found):
|
||||
return Failure(f"{ANCHORS_PATH} names {pair[0]} twice")
|
||||
found.append((pair[0], ids))
|
||||
return found
|
||||
|
||||
|
||||
def anchor_lines(found: Anchors | Failure, outcomes: list[Outcome]) -> list[str]:
|
||||
"""Each level, then each of its anchors with what its chunk measured. Where the file cannot be read, one line saying why; where it names no level, one line saying how to choose anchors."""
|
||||
if isinstance(found, Failure):
|
||||
return [f"- None: {found.reason}"]
|
||||
if not found:
|
||||
return [f"- None: the project has no {ANCHORS_PATH}, or it names no level; ~/.claude/rules/todo-items.md says how to choose anchors."]
|
||||
by_id = {outcome.item.id: outcome for outcome in outcomes}
|
||||
lines = []
|
||||
for level, ids in found:
|
||||
lines.append(f"{level}:")
|
||||
lines += [render(by_id[id_]) if id_ in by_id else f"- Item {id_}: no merge commit on the default branch shipped it." for id_ in ids]
|
||||
return lines
|
||||
+39
-12
@@ -23,12 +23,17 @@ its table also had A, G and Goals. The chunk measures:
|
||||
|
||||
A project that merges without merge commits, by fast-forward or squash, has no chunk to measure.
|
||||
|
||||
Given L, E and I, on the levels ~/.claude/rules/todo-items.md anchors, and S, it works out R from
|
||||
Given L, E and I, on the levels ~/.claude/rules/todo-items.md defines, and S, it works out R from
|
||||
L, E and I, as rank_todo.py does, and lists the 5 finished items whose guesses sit closest, closest
|
||||
first: the least mean squared difference over the scores both hold, so R and S alone for an item
|
||||
scored before L, E and I, the newest first among equals. Each line gives what the item's chunk
|
||||
measured, which items shipped with it, and its tagline.
|
||||
|
||||
Then the project's anchors, the shipped items its docs/score-anchors.md names for each level of L,
|
||||
E and I, read from the checkout's working tree, each line as above, or that no merge shipped it. A
|
||||
file that cannot be read, a row that does not hold a level on its scale and a list of IDs, or a
|
||||
level named twice leaves the anchors out, saying why.
|
||||
|
||||
Then the median of the outcome R and S each predict at each value they were guessed at: R against
|
||||
🔴 findings, S against lines changed. And how well L, E and I each order chunks by review rounds
|
||||
past each pass's first and by 🔴 findings: Spearman's rank correlation, from -1 to +1, where +1
|
||||
@@ -37,12 +42,15 @@ item, since a chunk measures all its items at once, and reports a score only onc
|
||||
hold it and its outcomes.
|
||||
|
||||
Exit 1, saying why, on bad arguments, when git cannot start, finds no checkout or reads a shallow
|
||||
clone, or the default branch cannot be read.
|
||||
clone, or the default branch cannot be read; or, after the rest of the report, when the anchors are
|
||||
left out.
|
||||
"""
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
from git_repo import default_branch, file_at, full_history, git
|
||||
from score_outcomes.anchors import anchor_lines, Anchors, ANCHORS_PATH, parse_anchors
|
||||
from git_repo import default_branch, file_at, full_history, git, top
|
||||
from failure import Failure
|
||||
from todo_table import LEVELS, risk, TOP_SCORE
|
||||
from markdown_text import listed
|
||||
@@ -70,15 +78,21 @@ def main(argv: list[str]) -> int:
|
||||
outcomes = read_outcomes(default)
|
||||
if isinstance(outcomes, Failure):
|
||||
return failed(outcomes)
|
||||
if argv:
|
||||
given = dict(zip(GIVEN, map(int, argv)))
|
||||
given["R"] = risk(given["L"], given["E"], given["I"])
|
||||
print(f"Finished items nearest L {given['L']}, E {given['E']}, I {given['I']}, so R {given['R']}, and S {given['S']}, closest first:\n")
|
||||
print("\n".join(render(outcome) for outcome in nearest(outcomes, given)) or "- None: no merge commit on the default branch shipped an item from todo.md. A project that merges by squash or fast-forward has none.")
|
||||
print()
|
||||
print("Median outcome at each guessed score, among chunks that shipped one item:\n")
|
||||
print("\n".join(report(outcomes)))
|
||||
return 0
|
||||
anchored = read_anchors()
|
||||
print(report_text(outcomes, anchored, dict(zip(GIVEN, map(int, argv))) if argv else None))
|
||||
return failed(anchored) if isinstance(anchored, Failure) else 0
|
||||
|
||||
|
||||
def report_text(outcomes: list[Outcome], anchored: Anchors | Failure, given: dict[str, int] | None) -> str:
|
||||
"""The report: the finished items nearest the given scores, where any are given, then the anchors, then the medians."""
|
||||
sections = []
|
||||
if given is not None:
|
||||
r = risk(given["L"], given["E"], given["I"])
|
||||
near = [render(outcome) for outcome in nearest(outcomes, given | {"R": r})] or ["- None: no merge commit on the default branch shipped an item from todo.md. A project that merges by squash or fast-forward has none."]
|
||||
sections.append([f"Finished items nearest L {given['L']}, E {given['E']}, I {given['I']}, so R {r}, and S {given['S']}, closest first:", "", *near])
|
||||
sections.append([f"Anchors in {ANCHORS_PATH}, each level's shipped items:", "", *anchor_lines(anchored, outcomes)])
|
||||
sections.append(["Median outcome at each guessed score, among chunks that shipped one item:", "", *report(outcomes)])
|
||||
return "\n\n".join("\n".join(section) for section in sections)
|
||||
|
||||
|
||||
def failed(failure: Failure) -> int:
|
||||
@@ -86,6 +100,19 @@ def failed(failure: Failure) -> int:
|
||||
return 1
|
||||
|
||||
|
||||
def read_anchors() -> Anchors | Failure:
|
||||
"""The checkout's anchors as its working tree holds them, so a branch that edits them shows its edit; none where it has no such file."""
|
||||
root = top()
|
||||
if isinstance(root, Failure):
|
||||
return root
|
||||
try:
|
||||
return parse_anchors((pathlib.Path(root) / ANCHORS_PATH).read_text())
|
||||
except FileNotFoundError:
|
||||
return []
|
||||
except (OSError, UnicodeDecodeError) as failure:
|
||||
return Failure(f"cannot read {ANCHORS_PATH}: {failure}")
|
||||
|
||||
|
||||
def read_outcomes(default: str) -> list[Outcome] | Failure:
|
||||
if partial := full_history():
|
||||
return partial
|
||||
|
||||
@@ -7,9 +7,12 @@ import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from score_outcomes.anchors import anchor_lines, parse_anchors
|
||||
from failure import Failure
|
||||
from test_env import isolated_env
|
||||
from todo_table import Item
|
||||
from score_outcomes.outcomes import nearest, Outcome, outcomes_of, rank_correlation, render, report, shipped
|
||||
from score_outcomes.main import report_text
|
||||
|
||||
TOOL = pathlib.Path(__file__).resolve().parents[1] / "score-outcomes"
|
||||
HEADER = "| ID | Release | Exempt | Item | R | S | C | A | G | Goals | Score |\n|---|---|---|---|---|---|---|---|---|---|---|\n"
|
||||
@@ -119,6 +122,65 @@ class Render(unittest.TestCase):
|
||||
self.assertIn("1 round past each pass's first, 0 🔴, 0 🟡; merged 2026-10-13T10:00:00Z in aaaaaaa, with item 5. ", render(outcome(4, 2, 3, 4, rounds=1, shipped_with=(5,))))
|
||||
|
||||
|
||||
ANCHORS_TEXT = "# Score anchors\n\nWhy these.\n\n| Level | Items |\n|---|---|\n| L 1 | 4, 9 |\n| I 3 | 5 |\n"
|
||||
|
||||
|
||||
class Anchors(unittest.TestCase):
|
||||
def test_reads_each_levels_items_in_the_files_order(self):
|
||||
self.assertEqual(parse_anchors(ANCHORS_TEXT), [("L 1", (4, 9)), ("I 3", (5,))])
|
||||
|
||||
def test_reads_each_level_at_the_top_of_its_scale(self):
|
||||
self.assertEqual(parse_anchors(ANCHORS_TEXT + "| L 4 | 4 |\n| E 3 | 4 |\n| I 5 | 4 |\n")[2:], [("L 4", (4,)), ("E 3", (4,)), ("I 5", (4,))])
|
||||
|
||||
def test_refuses_a_level_off_its_scale_a_score_without_levels_or_an_unreadable_row_naming_the_row(self):
|
||||
for row in ("| L 5 | 4 |", "| E 0 | 4 |", "| S 1 | 4 |", "| L 01 | 4 |", "| L 2 | four |", "| L 2 | |", "| L 2 | 4,5 |", "| L 2 | 4 | 5 |"):
|
||||
with self.subTest(row):
|
||||
failure = parse_anchors(ANCHORS_TEXT + row + "\n")
|
||||
self.assertIsInstance(failure, Failure)
|
||||
self.assertIn("docs/score-anchors.md", failure.reason)
|
||||
self.assertIn(row, failure.reason)
|
||||
|
||||
def test_refuses_a_repeated_level_naming_it(self):
|
||||
self.assertEqual(parse_anchors(ANCHORS_TEXT + "| L 1 | 7 |\n"), Failure("docs/score-anchors.md names L 1 twice"))
|
||||
|
||||
def test_lists_each_anchor_with_what_its_chunk_measured_and_says_which_it_cannot_find(self):
|
||||
lines = anchor_lines([("L 1", (4, 9))], [outcome(4, 2, 3, None, findings=(1, 2))])
|
||||
self.assertEqual(lines, ["L 1:", "- Item 4, R 2, S 3: 10 lines in 1 file, 1 round past each pass's first, 1 🔴, 2 🟡; merged 2026-10-13T10:00:00Z in aaaaaaa. **Item 4.**", "- Item 9: no merge commit on the default branch shipped it."])
|
||||
|
||||
def test_points_to_the_rules_where_the_project_names_no_anchors(self):
|
||||
self.assertIn("docs/score-anchors.md", anchor_lines([], [])[0])
|
||||
self.assertIn("~/.claude/rules/todo-items.md", anchor_lines([], [])[0])
|
||||
|
||||
def test_says_why_it_shows_no_anchors_it_cannot_read(self):
|
||||
self.assertEqual(anchor_lines(Failure("docs/score-anchors.md names L 1 twice"), []), ["- None: docs/score-anchors.md names L 1 twice"])
|
||||
|
||||
|
||||
class ReportText(unittest.TestCase):
|
||||
def test_lists_the_nearest_by_r_then_the_anchors_then_the_medians(self):
|
||||
found = [outcome(1, 5, 3, None, merged="2026-10-13T10:00:00Z"), outcome(2, 1, 3, None, merged="2026-10-12T10:00:00Z")]
|
||||
too_few = "chunks that shipped one item record"
|
||||
self.assertEqual(report_text(found, [("L 1", (2,))], {"L": 1, "E": 1, "I": 1, "S": 3}), f"""Finished items nearest L 1, E 1, I 1, so R 1, and S 3, closest first:
|
||||
|
||||
- Item 2, R 1, S 3: 10 lines in 1 file, 1 round past each pass's first, 0 🔴, 0 🟡; merged 2026-10-12T10:00:00Z in aaaaaaa. **Item 2.**
|
||||
- Item 1, R 5, S 3: 10 lines in 1 file, 1 round past each pass's first, 0 🔴, 0 🟡; merged 2026-10-13T10:00:00Z in aaaaaaa. **Item 1.**
|
||||
|
||||
Anchors in docs/score-anchors.md, each level's shipped items:
|
||||
|
||||
L 1:
|
||||
- Item 2, R 1, S 3: 10 lines in 1 file, 1 round past each pass's first, 0 🔴, 0 🟡; merged 2026-10-12T10:00:00Z in aaaaaaa. **Item 2.**
|
||||
|
||||
Median outcome at each guessed score, among chunks that shipped one item:
|
||||
|
||||
- R, 🔴 findings: 2 {too_few} R and its outcome; the report needs 10.
|
||||
- S, lines changed: 2 {too_few} S and its outcome; the report needs 10.
|
||||
- L, by review rounds past each pass's first and 🔴 findings: 0 {too_few} L and both outcomes; the report needs 10.
|
||||
- E, by review rounds past each pass's first and 🔴 findings: 0 {too_few} E and both outcomes; the report needs 10.
|
||||
- I, by review rounds past each pass's first and 🔴 findings: 0 {too_few} I and both outcomes; the report needs 10.""")
|
||||
|
||||
def test_lists_no_nearest_items_where_no_scores_are_given(self):
|
||||
self.assertTrue(report_text([], [], None).startswith("Anchors in docs/score-anchors.md"))
|
||||
|
||||
|
||||
class ScoreOutcomesTool(unittest.TestCase):
|
||||
def setUp(self):
|
||||
temp = tempfile.TemporaryDirectory()
|
||||
@@ -155,6 +217,26 @@ class ScoreOutcomesTool(unittest.TestCase):
|
||||
self.assertIn("- Item 4, R 2, S 3, C 10: 3 lines in 1 file, 0 rounds past each pass's first, 1 🔴, 2 🟡; merged ", ran.stdout)
|
||||
self.assertNotIn("Item 5", ran.stdout)
|
||||
|
||||
def test_lists_the_projects_anchors_with_what_their_chunks_measured(self):
|
||||
self.git("switch", "-q", "-c", "chunk")
|
||||
self.commit("Do it", {"a.py": "1\n", "todo.md": todo(SECOND)})
|
||||
self.git("switch", "-q", "main")
|
||||
self.git("merge", "-q", "--no-ff", "-m", "Merge chunk", "chunk")
|
||||
(self.checkout / "docs").mkdir()
|
||||
(self.checkout / "docs" / "score-anchors.md").write_text(ANCHORS_TEXT)
|
||||
ran = self.run_tool("1", "2", "1", "3")
|
||||
self.assertEqual(ran.returncode, 0, ran.stderr)
|
||||
self.assertIn("Anchors in docs/score-anchors.md, each level's shipped items:\n\nL 1:\n- Item 4, R 2, S 3, C 10: 1 line in 1 file", ran.stdout)
|
||||
self.assertIn("I 3:\n- Item 5: no merge commit on the default branch shipped it.", ran.stdout)
|
||||
|
||||
def test_refuses_anchors_it_cannot_read(self):
|
||||
(self.checkout / "docs").mkdir()
|
||||
(self.checkout / "docs" / "score-anchors.md").write_text(ANCHORS_TEXT + "| L 9 | 4 |\n")
|
||||
ran = self.run_tool()
|
||||
self.assertEqual(ran.returncode, 1)
|
||||
self.assertIn("docs/score-anchors.md", ran.stderr)
|
||||
self.assertIn("Median outcome at each guessed score", ran.stdout)
|
||||
|
||||
def test_takes_each_score_at_the_top_of_its_scale(self):
|
||||
ran = self.run_tool("4", "3", "5", "10")
|
||||
self.assertEqual(ran.returncode, 0, ran.stderr)
|
||||
|
||||
+10
-3
@@ -4,7 +4,7 @@ import typing
|
||||
|
||||
from failure import Failure
|
||||
|
||||
# Each score a Shape names, and each goal's move, runs from 1 to TOP_SCORE, but for the levels L, E and I, which run from 1 to their LEVELS: ~/.claude/rules/todo-items.md anchors each level.
|
||||
# Each score a Shape names, and each goal's move, runs from 1 to TOP_SCORE, but for the levels L, E and I, which run from 1 to their LEVELS: ~/.claude/rules/todo-items.md defines each level.
|
||||
TOP_SCORE = 10
|
||||
LEVELS = {"L": 4, "E": 3, "I": 5}
|
||||
# The least R each impact level I allows, since a severe enough consequence outranks how unlikely it is.
|
||||
@@ -230,10 +230,17 @@ def parse_scores(id_: str, named: dict[str, str], names: tuple[str, ...]) -> dic
|
||||
return scores
|
||||
|
||||
|
||||
def parse_blockers(id_: str, cell: str) -> tuple[int, ...] | Failure:
|
||||
def parse_ids(cell: str) -> tuple[int, ...] | None:
|
||||
"""The item IDs a cell lists, such as `12, 14`: an empty tuple where the cell is empty, None where it holds anything else."""
|
||||
if not re.fullmatch(r"([0-9]+(, [0-9]+)*)?", cell):
|
||||
return None
|
||||
return tuple(int(id_) for id_ in cell.split(", ") if id_)
|
||||
|
||||
|
||||
def parse_blockers(id_: str, cell: str) -> tuple[int, ...] | Failure:
|
||||
blocked_by = parse_ids(cell)
|
||||
if blocked_by is None:
|
||||
return Failure(f"item {id_}'s {BLOCKED_BY} {cell!r} is not empty or IDs separated by \", \"")
|
||||
blocked_by = tuple(int(blocker) for blocker in cell.split(", ") if blocker)
|
||||
if int(id_) in blocked_by:
|
||||
return Failure(f"item {id_} is blocked by itself; drop {id_} from its {BLOCKED_BY}")
|
||||
return blocked_by
|
||||
|
||||
@@ -2014,3 +2014,22 @@ Inherited architect, decided by:
|
||||
- Shape: `bin/todo_table.py:21` `Shape`: names `optional`, `risk_columns` and `derived` as fields where `C` was hard-coded, though its seven-argument constructors take counting
|
||||
- Shape: `bin/model_routing/route.py:16` `ArchivedWorker`: the file doubles, almost all of it for old R + C archive lines
|
||||
- Self-sufficiency: `bin/score_outcomes/outcomes.py:37` `SHOWN`: its comment says C shows on any item scored before L, E and I, though only a table with A, G and Goals keeps it
|
||||
|
||||
## 2026-10-16T18:11:57Z, PR #298 at 3f8f30a, against 2c67b4f
|
||||
|
||||
Ruling: same
|
||||
|
||||
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|
||||
|---|---|---|---|---|---|
|
||||
| Mid A | same | better | same | same | better |
|
||||
| Inherited architect | same | same | same | same | same |
|
||||
| Maintainability senior | same | same | same | same | same |
|
||||
| Junior A | same | better | same | same | better |
|
||||
|
||||
Mid A, decided by:
|
||||
|
||||
- Locality, Overall: `bin/score_outcomes/main.py:86` `report_text`: main only fetches the inputs and sets the exit code, and the report's sections are built apart from the git and fetch code
|
||||
|
||||
Junior A, decided by:
|
||||
|
||||
- Locality, Overall: `bin/score_outcomes/main.py:86` `report_text`: the report is one pure function that states its section order and no longer writes R into the caller's dict, so one section changes without tracing the print order through main
|
||||
|
||||
+1
-1
@@ -130,7 +130,7 @@ Valid while the ladder orders its rungs by competence.
|
||||
|
||||
2026-10-02, lilleman. Serves goal 1: findings were filed faster than chunks closed them, so the plan grew without telling what mattered. Each item is scored as `~/.claude/rules/todo-items.md` says; this repo's formula counts value highest and size least. An item under the bar is not kept; this repo's bar started at 10 and was set to 9 the same day, purging the 17 items under it. An Exempt cell keeps a row past the bar where a rule needs it kept. lilleman retired the two decisions whose plans the purge removed: the stats line in the transcript archive, and three benches before a workflow change merges. A finding under the bar is recorded nowhere, so a later review may raise it again, and the run rescores it. That costs one scoring; a ledger of dropped findings would regrow the list the bar caps. larv-review's "Round cap" → "Done" counts a finding named under the bar as settled, so one raised again holds no pass open. `rank_todo.py --check` is part of larv-review's gate, since only a check keeps the stored Score from going stale. The format is mandatory in every project, as the premises are (goal 1.3); the formula and bar are each project's own. A script ranks the table so a re-rank after each addition or edit is deterministic and cheap. A 🟡 against a technical principle is exempt as `principle`, because its audience value scores low and the bar would drop every one. A 🟢 against a principle is scored like any other finding. 2026-10-08, lilleman: the goal weights in `todo.md` depart from an even spread by position, which would weigh goal 4, leaking nothing, at 0.11. Valid while one human sets the bar and reads the ranking, and while the items marked `principle` stay few enough to rank.
|
||||
|
||||
2026-10-15, lilleman: the scores get a researched method, which `todo.md` items 149, 151 and 152 build, after PR #283 shipped the ranking and PR #296 the risk scores; the evidence is in `docs/scoring-and-goals-research.md`. R becomes L + E + I − 2: the chance the change holds a defect, set first by its size; the chance the defect escapes tests, review and monitoring; and the worst impact dimension, each about ×10 per level. I5, irreversible, holds R at 7 or more, and I4 at 5 or more, since a severe enough consequence outranks how unlikely it is. C goes from every table and tool: what it scored beyond size, spread and novelty, which L already holds, is complexity, which adds almost nothing once size is accounted for. Items rank by value per cost, V / 2^S, and V sums each cited achieve goal's sum-reciprocal weight times how far the item moves it. A goes, and the Audience says whose value the goals serve. R stays out of the ranking and sets how carefully an item is done; an item that lowers risk scores it as value. Each release fills by priority up to a capacity it states, in place of the bar. Each README goal is `maintain`, which vetoes an item that breaks it, or `achieve`, with a signal, a counter-signal and a one-line reason, as this README's Goals now mark them; each project keeps settled decisions to test its goals against. Each part of this method holds once the item that builds it ships. 2026-10-16, lilleman: a task's L and E + I pick its rungs in place of R, as `routing: one ladder` records.
|
||||
2026-10-15, lilleman: the scores get a researched method, which `todo.md` items 149 and 151 build, after PR #283 shipped the ranking, PR #296 the risk scores and PR #298 the anchors each project names for each level in `docs/score-anchors.md`; the evidence is in `docs/scoring-and-goals-research.md`. R becomes L + E + I − 2: the chance the change holds a defect, set first by its size; the chance the defect escapes tests, review and monitoring; and the worst impact dimension, each about ×10 per level. I5, irreversible, holds R at 7 or more, and I4 at 5 or more, since a severe enough consequence outranks how unlikely it is. C goes from every table and tool: what it scored beyond size, spread and novelty, which L already holds, is complexity, which adds almost nothing once size is accounted for. Items rank by value per cost, V / 2^S, and V sums each cited achieve goal's sum-reciprocal weight times how far the item moves it. A goes, and the Audience says whose value the goals serve. R stays out of the ranking and sets how carefully an item is done; an item that lowers risk scores it as value. Each release fills by priority up to a capacity it states, in place of the bar. Each README goal is `maintain`, which vetoes an item that breaks it, or `achieve`, with a signal, a counter-signal and a one-line reason, as this README's Goals now mark them; each project keeps settled decisions to test its goals against. Each part of this method holds once the item that builds it ships. 2026-10-16, lilleman: a task's L and E + I pick its rungs in place of R, as `routing: one ladder` records.
|
||||
|
||||
2026-10-16, PR #283: items rank by `Score = 100 * V / 2**S`. 2**S is the cost; the 100 lets a score read to one decimal. V sums, over the achieve goals an item's Moves cell names, each goal's weight times how far the item moves it, 1 to 10. The goal ranked k of n achieve goals in reading order weighs (1/k) / (1 + 1/2 + … + 1/n), the sum-reciprocal weight. A Moves column replaces A, G and Goals, and `rank_todo.py` derives the weights from README.md, so the goal weights table and the 2026-10-08 weights are gone. An item that cited only maintain goals now names the achieve goal it moves, or, as a defect restoring one, that maintain goal, as the entry below says. A panel of three readers seated as lilleman picked the spelling: two put a Moves column holding `1.1 by 8, 1.2 by 5` first, and one put `1.1: 8` first; two chose `2**S`, since a Python reader takes `^` for XOR; all three chose one decimal of `100 * V / 2**S` over three significant digits or a log. The bar moved to 0.1, lilleman's call answering the review of PR #283: on one-decimal scores it drops an item scoring under 0.05, in practice one that moves no achieve goal, and item 149 revisits it. A table in the old shape still ranks by the formula and weights it states, and ranking it says to reshape it, so no other project's commits stop; reshaping it is that project's only work until done, as `CLAUDE.md` → "Working loop" says. PR #296 dropped the C column, and put L, E and I right before R, which `rank_todo.py` writes from them: of three readers seated as lilleman, two ranked that order first, since a reader meets each level before the R it builds and R sits beside S, and the third ranked it second, after an order with S before the levels. The old shape's ranking is valid until `todo.md` item 156.
|
||||
|
||||
|
||||
@@ -520,3 +520,20 @@ Read most (opens and searches):
|
||||
- `bin/rule_hook/rules.py` 3
|
||||
- `AGENTS.md` 2
|
||||
- `bin/loop_claude/cycle/worker.py` 2
|
||||
|
||||
## Reading cost, chunk eded4db7f44b, finished 2026-10-16T17:29:09Z
|
||||
|
||||
356291 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `todo.md` 12
|
||||
- `bin/loop_claude/cycle/judge.py` 5
|
||||
- `bin/loop_claude/test_loop_claude.py` 5
|
||||
- `bin/judge_replay/test_judge_replay.py` 4
|
||||
- `bin/loop_claude/test_black_box.py` 4
|
||||
- `bin/test_rank_todo.py` 4
|
||||
- `bin/todo_table.py` 4
|
||||
- `README.md` 3
|
||||
- `bin/loop_claude/loop/main.py` 3
|
||||
- `bin/rank_todo.py` 3
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
# Score anchors
|
||||
|
||||
The items this repo shipped that show each level of L, E and I, which `rules/todo-items.md` defines. No item yet shows I 4 or I 5.
|
||||
|
||||
| Level | Items |
|
||||
|---|---|
|
||||
| L 1 | 115, 140 |
|
||||
| L 2 | 97, 141, 144 |
|
||||
| L 3 | 136, 143, 163 |
|
||||
| L 4 | 96, 146, 148 |
|
||||
| E 1 | 92, 146 |
|
||||
| E 2 | 67, 140, 141 |
|
||||
| E 3 | 64, 71 |
|
||||
| I 1 | 115, 140 |
|
||||
| I 2 | 58, 107 |
|
||||
| I 3 | 96, 148, 163 |
|
||||
+3
-2
@@ -7,7 +7,7 @@ paths:
|
||||
|
||||
- **`todo.md` is mandatory in every project, in the shape `rank_todo.py --help` shows**, which `~/.claude/bin/rank_todo.py` checks and ranks; its `--help` shows a valid file. `## Scoring` holds the formula and the bar; `## Items` is one table of every item; `## Details` holds each item's detail under its ID. Cite an item by its ID. The formula and bar are the project's own to tune.
|
||||
- **Order items only by score and the Blocked by column.** An item's Blocked by cell lists the IDs of the items that must be done before it starts, such as `12, 14`; `rank_todo.py --help` says how it ranks them. No text in `todo.md` outside the table orders items: where an item must go first, score it so, and say why in its detail.
|
||||
- **Plan only on the project's rung or above.** loop-claude's prompt tells you when you run below it. Below it, never score or re-score an item, move it between releases, split a milestone into items, delete an item under the bar, or rewrite a filed item's tagline or detail. Draft each new item, and each change to a filed item, instead. Before your next commit, launch one subagent to apply the drafts, score them and set their release. Launch it on the rung `CLAUDE.md`'s delegation rule gives a subagent that files or scores items. Keep the scores and releases it writes. For a security finding, launch it at once. Any rung may remove a finished item's row and detail, and score a delegated task's L. A subagent launched for another task leaves `todo.md` alone.
|
||||
- **Plan only on the project's rung or above.** loop-claude's prompt tells you when you run below it. Below it, never score or re-score an item, move it between releases, split a milestone into items, delete an item under the bar, rewrite a filed item's tagline or detail, or choose or replace an item in `docs/score-anchors.md`. Draft each new item, each change to a filed item, and each change to the anchors, instead. Before your next commit, launch one subagent to apply the drafts, score them and set their release. Launch it on the rung `CLAUDE.md`'s delegation rule gives a subagent that files or scores items. Keep the scores and releases it writes. For a security finding, launch it at once. Any rung may remove a finished item's row and detail, and score a delegated task's L. A subagent launched for another task leaves `todo.md` alone.
|
||||
- **An item's release allows the bump that semver read strictly asks for.** An item that changes what exists needs a major, or before 1.0.0 a minor. One that only adds needs a minor, and a fix a patch.
|
||||
- **Score an item when it is filed or edited.** S, the size of the change, is a whole number from 1 to 10. Score L, E and I by the levels below, each level about ten times as likely, or as damaging, as the one below it. `rank_todo.py` writes R, the risk that the change creates serious bugs or issues: L + E + I − 2, at least 7 when I is 5 and at least 5 when I is 4, since a severe enough consequence outranks how unlikely it is. A task's L picks its worker's rung, and its E + I its reviewers' rung.
|
||||
|
||||
@@ -22,7 +22,8 @@ paths:
|
||||
- **I's reach counts every project and session that runs the change.** A change to a hook or a tool that every session runs is I level 3, every consumer affected, even where one person uses it.
|
||||
- **Name in Moves each achieve goal in `README.md` → `## Goals` the item moves, and how far,** from 1 to 10, judged by its value to the audience in `## Audience`: `1.1 by 8, 1.2 by 5`. Name the most specific goal it moves, and a parent goal only for a move none of its sub-goals covers, never beside its own sub-goal. Leave out every maintain goal, which vetoes, except on an item marked `defect` that restores it: name that goal, and how much of it is broken. `rank_todo.py --help` says how it weighs each goal.
|
||||
- **A table with C, A, G and Goals in place of L, E, I and Moves is in the old shape.** Reshaping it is the project's only work until done, as `CLAUDE.md` → "Working loop" says, and `rank_todo.py` says how. Until then, A is the item's value to the audience and G its value to the goals it cites.
|
||||
- **Check L, E, I and S against finished chunks.** Guess them, then run `~/.claude/bin/score-outcomes <L> <E> <I> <S>` in the project. It lists the finished items guessed nearest, what their chunks measured, the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings. Set each score to the value guessed for the finished items whose outcome matches what you expect of this one. Keep your guess when no listed item resembles the one you are scoring.
|
||||
- **Score L, E and I against the project's anchors, and check every score against finished chunks.** Guess the scores, then run `~/.claude/bin/score-outcomes <L> <E> <I> <S>` in the project. It lists the finished items guessed nearest and the project's anchors, the shipped items its `docs/score-anchors.md` names for each level, each with what its chunk measured; then the median outcome at each guessed R and S, and how well L, E and I each order the chunks by review rounds and 🔴 findings. Compare the item with each anchor in turn: is it more or less likely to hold a defect, to let one escape, or to do damage? Give it the level whose anchors it matches; where a level has no anchor, judge by the level's words in the table above. Set S to the value guessed for the finished items whose outcome matches what you expect of this one. Keep your guess when no listed item resembles the one you are scoring.
|
||||
- **Anchor each level of L, E and I with two or three items the project shipped,** in `docs/score-anchors.md`: a table whose Level column names a level, such as `L 1`, and whose Items column lists the anchors' IDs, such as `12, 14`. Choose an L anchor whose chunk, as `score-outcomes` measured it, fits the level's words: its lines changed and the components it touched. Choose an E or I anchor by the level's words and what followed its release, since no tool measures either yet. Leave out a level no shipped item shows. Replace a level's anchors when the items scored at it turn out about ten times more likely, or ten times less likely, than its words say: to hold a defect, to let one escape, or to do damage.
|
||||
- **Read the places the comprehension seats found hard before filing an item.** `~/.claude/bin/larv-review places`, run in the project, prints them. Where the item's change touches one, its detail says to make that place easier to read.
|
||||
- **An item scoring under the bar is deleted, with every item that needs it, and a finding under the bar is not filed.** The chunk's summary names each, with its score.
|
||||
- **The Exempt cell keeps an item whatever its score:** `security` for a security finding, `defect` for a 🔴 finding or an item that restores a maintain goal the project breaks today, `question` for a question only the human can answer, which ranks first in its release, `decision` for an item `docs/decisions.md` cites, `owner` for an item the human told you to keep, and, where no other reason fits, `principle` for a 🟡 finding against a file under `~/.claude/principles/`, whose item's detail names that principle. Every item `docs/decisions.md` cites carries one of these reasons.
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 152 | 0.2.0 | | decision | **Anchor each score level in `rules/todo-items.md` with words, and each project's with two or three items it shipped, and have a scorer compare a new item against them.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 7 | 6.4 |
|
||||
| 50 | 0.2.0 | | principle | **Grant the bench's runs only the tools a review needs, in place of skipping every permission.** | 2 | 2 | 1 | 3 | 2 | 1.1 by 3 | 5.5 |
|
||||
| 65 | 0.2.0 | | defect | **Archive each judge reply the loop rejects.** | 2 | 1 | 3 | 4 | 4 | 3 by 3 | 5.5 |
|
||||
| 153 | 0.2.0 | | decision | **Route the judge and the reviewers on the highest E + I in their chunk, an item the worker takes in included.** | 3 | 1 | 3 | 5 | 4 | 1.2 by 5 | 3.0 |
|
||||
@@ -85,10 +84,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 152. Anchor each score level in `rules/todo-items.md` with words, and each project's with two or three items it shipped, and have a scorer compare a new item against them.
|
||||
|
||||
LLM ratings drift to the middle of an unanchored scale, and worked examples cut estimation error. A project's anchors are shipped items, cited by ID and commit, chosen from what `score-outcomes` measured for them.
|
||||
|
||||
### 50. Grant the bench's runs only the tools a review needs, in place of skipping every permission.
|
||||
|
||||
"Always grant explicit access to data", in `principles/technical-principles.md`: `workflow-bench/bench.py` starts claude with `--dangerously-skip-permissions`, and no rule confines it to a sandbox, so a run on the host reaches the host's credentials and other checkouts.
|
||||
@@ -370,7 +365,7 @@ While `rank_todo.py` ranks two shapes, projects follow two sets of scoring rules
|
||||
|
||||
### 155. Lower the reading cost of the files a chunk reads most before its first commit.
|
||||
|
||||
`bin/reading-cost` on 2026-10-16: the last 5 chunks' median is 189873 new input tokens before their first commit, the 5 before them 90662. Read most over the last 5, by opens and searches: `todo.md` 56, `docs/decisions.md` 19, `CHANGELOG.md` 13, `README.md` 13, `bin/test_rank_todo.py` 10, `AGENTS.md` 8, `skills/larv-review/SKILL.md` 7, `agents/stability-reviewer.md` 6, `bin/loop_claude/test_loop_claude.py` 6, `bin/rank_todo.py` 6. The chunk that ships this item runs `bin/reading-cost --reset`.
|
||||
`bin/reading-cost` on 2026-10-16: the last 5 chunks' median is 213946 new input tokens before their first commit, the 5 before them 133254. Read most over the last 5, by opens and searches: `todo.md` 61, `bin/test_rank_todo.py` 14, `docs/decisions.md` 14, `README.md` 11, `bin/loop_claude/test_loop_claude.py` 11, `CHANGELOG.md` 10, `AGENTS.md` 7, `bin/rank_todo.py` 7, `bin/todo_table.py` 7, `agents/stability-reviewer.md` 6. The chunk that ships this item runs `bin/reading-cost --reset`.
|
||||
|
||||
### 1. Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user