Measure what would lower review's token use: bench static metrics against the panel's hardest places, and rescore item 73 by its reports' measured cost

This commit is contained in:
2026-10-08 21:28:38 +02:00
parent c7c5ec083f
commit 382472bd39
13 changed files with 617 additions and 21 deletions
+3
View File
@@ -24,6 +24,9 @@
!bin/loop_claude/**/
!bin/loop_claude/**/*.py
!bin/markdown_sections.py
!bin/metric-bench
!bin/metric_bench/
!bin/metric_bench/*.py
!bin/rank_todo.py
!bin/reading-cost
!bin/reading_cost/
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- The new `bin/metric-bench` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest: cognitive complexity, length, shared state and change spread, each scored by how well it ranks the named functions above the rest. No metric enters review yet.
- larv-review blocks a merge when simulated readers rule that the PR makes the project read worse than the last measured commit: the default branch's newest merged change that added a scoring run to `docs/comprehension-history.md`, or whose last ruling there is better. A ruling of same leaves that commit in place, so a worse ruling may name places earlier chunks wrote. In each architecture pass Mid A and the inherited architect vote better, same or worse; when they disagree, the maintainability senior and Junior A vote too. A ruling needs a lead of two or more votes, else it is same, and a vote that names no place counts as same. A worse ruling never ships: after three rounds larv-review asks you for another round, a no leaves the PR unmerged, and so does an unattended run. The four-seat scoring run checks the comprehension floor only when a chunk ships or removes a release's last plan item, or ships an item naming the floor; between those runs the floor blocks nothing. A score under the floor blocks the merge until fixed or until you decide to ship; a decision in `docs/decisions.md` that lets every chunk ship under the floor counts as yours.
- larv-review reruns fewer passes after a fix. A fix reruns the pass whose 🔴 or 🟡 it answers, plus stability when the fix changes behaviour, security when it opens an untrusted-input boundary or touches authn/authz, secrets or crypto, and architecture when it carries an architectural decision. A rerun reviews only what changed since that pass last reviewed. A fix for a 🟢, a ❓ or any prose finding reruns no pass, and prose takes one round.
- larv-review runs one security pass on each change: the full scan alone when it runs, else `/security-review`.
+1
View File
@@ -66,6 +66,7 @@ Rename `## Unreleased` in `CHANGELOG.md` to the version, commit, tag that commit
## Tools
- `loop-claude <task>` runs a project's next chunk cycle after cycle; `loop-claude --help` states its contract.
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
- `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format.
- `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure.
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
+9
View File
@@ -0,0 +1,9 @@
#!/usr/bin/env python3
import pathlib
import sys
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from metric_bench import main
sys.exit(main.main(sys.argv[1:]))
View File
+69
View File
@@ -0,0 +1,69 @@
import collections
import dataclasses
import statistics
from typing import Callable
from metric_bench.places import Place
from metric_bench.python_units import Unit
METRICS: dict[str, Callable[[Unit, dict[str, int]], int]] = {
"cognitive complexity": lambda unit, spread: unit.cognitive,
"length": lambda unit, spread: unit.length,
"shared state": lambda unit, spread: unit.shared_state,
"change spread": lambda unit, spread: spread.get(unit.path, 0),
}
@dataclasses.dataclass(frozen=True)
class Score:
auc: float
top_k: float
@dataclasses.dataclass(frozen=True)
class RunScores:
units: int
named: int
unmeasured: int
scores: dict[str, Score]
def co_changed(log: str) -> dict[str, int]:
"""For each file, how many other files changed in a commit with it; log is `git log --format=format:@@ --name-only`."""
partners: dict[str, set[str]] = collections.defaultdict(set)
for commit in log.split("@@"):
files = set(commit.split())
for path in files:
partners[path] |= files - {path}
return {path: len(others) for path, others in partners.items()}
def bench(units: list[Unit], places: list[Place], spread: dict[str, int]) -> RunScores | None:
"""How well each metric ranks the units the seats named above the rest; None when no named place is a measured unit."""
names = {(place.path, place.name) for place in places}
named = [unit for unit in units if (unit.path, unit.name) in names]
if not named or len(named) == len(units):
return None
rest = [unit for unit in units if (unit.path, unit.name) not in names]
measured = {(unit.path, unit.name) for unit in named}
unmeasured = len({place for place in places if (place.path, place.name) not in measured})
scores = {}
for metric, value in METRICS.items():
scores[metric] = Score(auc([value(unit, spread) for unit in named], [value(unit, spread) for unit in rest]), top_k(units, named, lambda unit: value(unit, spread)))
return RunScores(len(units), len(named), unmeasured, scores)
def auc(named: list[int], rest: list[int]) -> float:
"""The chance a named unit outranks one not named, a tie counting half: 0.5 is a coin toss, 1.0 a perfect ranking."""
wins = sum((a > b) + 0.5 * (a == b) for a in named for b in rest)
return wins / (len(named) * len(rest))
def top_k(units: list[Unit], named: list[Unit], value: Callable[[Unit], int]) -> float:
"""The share of the metric's top k units the seats named, k being how many they named; a tie at the cut counts in order of the units."""
top = sorted(units, key=value, reverse=True)[:len(named)]
return sum(unit in named for unit in top) / len(named)
def medians(runs: list[RunScores]) -> dict[str, Score]:
return {metric: Score(statistics.median(run.scores[metric].auc for run in runs), statistics.median(run.scores[metric].top_k for run in runs)) for metric in METRICS}
+99
View File
@@ -0,0 +1,99 @@
"""Bench cheap static metrics against the places a project's comprehension panel named as hardest.
usage: metric-bench [history]
Run it inside a project checkout. history defaults to docs/comprehension-history.md there. For each
scoring run in it, at the commit the seats read, it measures every Python function and method git
tracks, test files left out: cognitive complexity (SonarSource's white paper, read for Python),
length in lines, shared state (attributes on self another method of the class also writes, plus
names declared global or nonlocal), and change spread (how many other files changed in a commit with
the unit's file, up to that commit). It then prints, for each metric, how well it ranks the units
the seats named above the rest: AUC (0.5 is a coin toss, 1.0 a perfect ranking) and top k (the share
of the metric's k highest units the seats named, k being how many they named). A last row gives the
medians. Places that name no Python function or method, such as a section of a markdown file, are
counted as unmeasured.
Exit 1, saying why, on unknown arguments, or when git cannot start or finds no checkout, the
history cannot be read, a run's commit is missing, or no scoring run names a measured unit.
"""
import pathlib
import subprocess
import sys
from metric_bench.bench import bench, co_changed, medians, METRICS, RunScores, Score
from failure import Failure
from metric_bench.places import resolved, scoring_runs
from metric_bench.python_units import python_units, Unit
def main(argv: list[str]) -> int:
if argv in (["-h"], ["--help"]):
print(__doc__.strip())
return 0
if len(argv) > 1 or argv[:1] and argv[0].startswith("-"):
print(f"unknown arguments {' '.join(argv)}; usage: metric-bench [history]", file=sys.stderr)
return 1
top = git(["rev-parse", "--show-toplevel"], None)
if isinstance(top, Failure):
return failed(top)
history = pathlib.Path(argv[0]) if argv else pathlib.Path(top.strip(), "docs", "comprehension-history.md")
try:
text = history.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError) as failure:
return failed(Failure(f"could not read {history}: {failure}"))
rows = []
for run in scoring_runs(text):
tracked = git(["ls-tree", "-r", "--name-only", run.commit], top.strip())
if isinstance(tracked, Failure):
return failed(tracked)
units = commit_units(run.commit, tracked.split("\n"), top.strip())
if isinstance(units, Failure):
return failed(units)
log = git(["log", "--no-merges", "--format=format:@@", "--name-only", run.commit], top.strip())
if isinstance(log, Failure):
return failed(log)
if scores := bench(units, resolved(run.places, tracked.split("\n")), co_changed(log)):
rows.append((run.heading, scores))
if not rows:
return failed(Failure(f"no scoring run in {history} names a Python function or method the bench measures"))
print(table(rows))
return 0
def commit_units(commit: str, tracked: list[str], top: str) -> list[Unit] | Failure:
units = []
for path in tracked:
name = path.rsplit("/", 1)[-1]
if path.endswith(".py") and not name.startswith("test_") and not name.endswith("_test.py") and "tests" not in path.split("/")[:-1]:
source = git(["show", f"{commit}:{path}"], top)
if isinstance(source, Failure):
return source
units += python_units(path, source)
return units
def table(rows: list[tuple[str, RunScores]]) -> str:
head = ["Scoring run", "Units", "Named", "Unmeasured", *METRICS]
lines = ["| " + " | ".join(head) + " |", "|" + "---|" * len(head)]
for heading, run in rows:
lines.append("| " + " | ".join([heading, str(run.units), str(run.named), str(run.unmeasured), *(cell(run.scores[metric]) for metric in METRICS)]) + " |")
medianed = medians([run for _, run in rows])
lines.append("| " + " | ".join(["Median", "", "", "", *(cell(medianed[metric]) for metric in METRICS)]) + " |")
return "\n".join(lines)
def cell(score: Score) -> str:
return f"AUC {score.auc:.2f}, top k {score.top_k:.2f}"
def git(args: list[str], cwd: str | None) -> str | Failure:
try:
ran = subprocess.run(["git", *args], capture_output=True, cwd=cwd, encoding="utf-8", errors="replace")
except OSError as failure:
return Failure(f"could not start git: {failure.strerror}")
return ran.stdout if ran.returncode == 0 else Failure(f"git {' '.join(args)}: {ran.stderr.strip()}")
def failed(failure: Failure) -> int:
print(failure.reason, file=sys.stderr)
return 1
+70
View File
@@ -0,0 +1,70 @@
import dataclasses
import os
import re
from markdown_sections import sections
RUN = re.compile(r"^(.+), (.+) at ([0-9a-f]{7,40})$")
SEAT_PLACES = re.compile(r"^.+, (hardest first|would restructure first):$")
# `path:line` or `:line`, then the unit's name where one follows.
PLACE = re.compile(r"`([^`\s]*):\d+`(?: `(\w+)`)?")
@dataclasses.dataclass(frozen=True)
class Place:
path: str
name: str | None
@dataclasses.dataclass(frozen=True)
class ScoringRun:
heading: str
commit: str
places: list[Place]
def scoring_runs(history: str) -> list[ScoringRun]:
"""Each scoring run in a comprehension history, with the places its seats named, as they wrote them; paired rulings name what changed, not what is hard, and are left out."""
runs = []
for heading, bodies in sections(history).items():
if (named := RUN.match(heading)) and ", against " not in heading:
runs.append(ScoringRun(heading, named.group(3), [place for body in bodies for place in seat_places(body)]))
return runs
def seat_places(body: str) -> list[Place]:
places = []
listing = False
for line in body.splitlines():
if SEAT_PLACES.match(line):
listing = True
elif listing and line.startswith("- "):
places += bullet_places(line)
elif not line.startswith(" "):
listing = False
return places
def bullet_places(bullet: str) -> list[Place]:
"""A `:line` with no path is in the file named before it in the bullet."""
places = []
path = ""
for match in PLACE.finditer(bullet):
path = match.group(1) or path
if path:
places.append(Place(path, match.group(2)))
return places
def resolved(places: list[Place], tracked: list[str]) -> list[Place]:
"""Seats shorten a path once they have named its folder: `cycle/run.py` after `bin/loop_claude/cycle/states.py`. Each place resolves to the one tracked file it names, or, where several match, the one sharing most of the path resolved before it; a place naming no tracked file is dropped."""
found = []
before = ""
for place in places:
candidates = [path for path in tracked if path == place.path or path.endswith("/" + place.path)]
if len(candidates) > 1:
candidates = [max(candidates, key=lambda path: len(os.path.commonprefix([path, before])))]
if candidates:
before = candidates[0]
found.append(Place(before, place.name))
return found
+151
View File
@@ -0,0 +1,151 @@
import ast
import collections
import dataclasses
FUNCTION = (ast.FunctionDef, ast.AsyncFunctionDef)
@dataclasses.dataclass(frozen=True)
class Unit:
path: str
name: str
cognitive: int
length: int
shared_state: int
def python_units(path: str, source: str) -> list[Unit]:
"""Each module-level function and each method in source, with its metrics; a source that does not parse has none."""
try:
tree = ast.parse(source)
except (SyntaxError, ValueError):
return []
units = []
for node in tree.body:
if isinstance(node, FUNCTION):
units.append(unit(path, node, 0))
elif isinstance(node, ast.ClassDef):
shared = shared_writes(node)
units += [unit(path, method, shared[method.name]) for method in node.body if isinstance(method, FUNCTION)]
return units
def unit(path: str, node: ast.FunctionDef | ast.AsyncFunctionDef, shared: int) -> Unit:
globals_written = sum(len(statement.names) for statement in ast.walk(node) if isinstance(statement, ast.Global | ast.Nonlocal))
return Unit(path, node.name, cognitive(node), (node.end_lineno or node.lineno) - node.lineno + 1, shared + globals_written)
def shared_writes(cls: ast.ClassDef) -> collections.Counter[str]:
"""For each method, how many of the attributes it writes on self another method of the class writes too."""
writers = collections.defaultdict(set)
for method in cls.body:
if isinstance(method, FUNCTION):
for node in ast.walk(method):
for target in written(node):
if isinstance(target, ast.Attribute) and isinstance(target.value, ast.Name) and target.value.id == "self":
writers[target.attr].add(method.name)
return collections.Counter(method for methods in writers.values() if len(methods) > 1 for method in methods)
def written(node: ast.AST) -> list[ast.expr]:
if isinstance(node, ast.Assign):
return node.targets
return [node.target] if isinstance(node, ast.AugAssign | ast.AnnAssign) else []
# SonarSource's "Cognitive Complexity" white paper (G. Ann Campbell, 2017), read for Python: a comprehension counts as a loop and each of its `if` clauses as a condition.
def cognitive(function: ast.FunctionDef | ast.AsyncFunctionDef) -> int:
counter = Cognitive(function.name)
counter.within(function.body, 0)
return counter.score
class Cognitive(ast.NodeVisitor):
def __init__(self, name: str) -> None:
self.name = name
self.nesting = 0
self.score = 0
def within(self, nodes: list, deeper: int = 1) -> None:
self.nesting += deeper
for node in nodes:
self.visit(node)
self.nesting -= deeper
def structure(self) -> None:
self.score += 1 + self.nesting
def visit_If(self, node: ast.If) -> None:
self.structure()
self.branch(node)
def branch(self, node: ast.If) -> None:
self.visit(node.test)
self.within(node.body)
if len(node.orelse) == 1 and isinstance(node.orelse[0], ast.If):
self.score += 1
self.branch(node.orelse[0])
elif node.orelse:
self.score += 1
self.within(node.orelse)
def loop(self, node: ast.For | ast.AsyncFor | ast.While, head: ast.expr) -> None:
self.structure()
self.visit(head)
self.within(node.body)
if node.orelse:
self.score += 1
self.within(node.orelse)
def visit_For(self, node: ast.For | ast.AsyncFor) -> None:
self.loop(node, node.iter)
visit_AsyncFor = visit_For
def visit_While(self, node: ast.While) -> None:
self.loop(node, node.test)
def visit_Try(self, node: ast.Try | ast.TryStar) -> None:
self.within(node.body, 0)
for handler in node.handlers:
self.structure()
self.within(handler.body)
self.within(node.orelse, 0)
self.within(node.finalbody, 0)
visit_TryStar = visit_Try
def visit_Match(self, node: ast.Match) -> None:
self.structure()
self.visit(node.subject)
for case in node.cases:
self.within(case.body)
def visit_IfExp(self, node: ast.IfExp) -> None:
self.structure()
self.within([node.test, node.body, node.orelse])
def visit_BoolOp(self, node: ast.BoolOp) -> None:
self.score += 1
self.generic_visit(node)
def comprehension(self, node: ast.ListComp | ast.SetComp | ast.DictComp | ast.GeneratorExp) -> None:
for generator in node.generators:
self.structure()
self.score += len(generator.ifs)
self.within(list(ast.iter_child_nodes(node)))
visit_DictComp = visit_GeneratorExp = visit_ListComp = visit_SetComp = comprehension
def visit_FunctionDef(self, node: ast.FunctionDef | ast.AsyncFunctionDef) -> None:
self.within(node.body)
visit_AsyncFunctionDef = visit_FunctionDef
def visit_Lambda(self, node: ast.Lambda) -> None:
self.within([node.body])
def visit_Call(self, node: ast.Call) -> None:
if isinstance(node.func, ast.Name) and node.func.id == self.name:
self.score += 1
self.generic_visit(node)
+1 -1
View File
@@ -8,7 +8,7 @@ CITATION = re.compile(r'`?CLAUDE\.md`? → ("[^"]+"(?:,? (?:and )?"[^"]+")*)')
HEADING = re.compile(r"^#+ (.+?)(?: \(.*\))?$", re.MULTILINE)
LEAD = re.compile(r"^(?:- )?\*\*(.+?)(?:\*\*|[:.](?=\*\*| )| —)", re.MULTILINE)
NAME = re.compile(r'"([^"]+)"')
STRAY = re.compile(r'CLAUDE\.md(?!`? → ")[^\n"]{1,5}"')
STRAY = re.compile(r'CLAUDE\.md(?!`? → "|:\d)[^\n"]{1,5}"')
class Citations(unittest.TestCase):
+182
View File
@@ -0,0 +1,182 @@
import ast
import os
import pathlib
import subprocess
import sys
import tempfile
import textwrap
import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from metric_bench.bench import auc, bench, co_changed
from metric_bench.places import Place, resolved, scoring_runs
from metric_bench.python_units import cognitive, python_units, Unit
from test_loop_claude_black_box import host_env
TOOL = pathlib.Path(__file__).resolve().parent / "metric-bench"
def scored(source: str) -> int:
function = ast.parse(textwrap.dedent(source)).body[0]
assert isinstance(function, ast.FunctionDef)
return cognitive(function)
class Cognitive(unittest.TestCase):
def test_scores_the_white_papers_sum_of_primes_without_its_labelled_continue(self):
self.assertEqual(scored("""
def sum_of_primes(limit):
total = 0
for i in range(1, limit + 1):
for j in range(2, i):
if i % j == 0:
break
total += i
return total
"""), 6)
def test_counts_elif_and_else_flat_and_a_nested_ternary_by_its_depth(self):
self.assertEqual(scored("""
def f(a):
if a:
return 1 if a > 1 else 2
elif a is None:
return 3
else:
return 4
"""), 5)
def test_counts_each_boolean_operator_sequence_a_handler_a_match_and_recursion(self):
self.assertEqual(scored("""
def f(a, b, c):
try:
if a and b and c or a:
match a:
case 1:
return f(b, c, a)
except ValueError:
pass
"""), 1 + 2 + 2 + 1 + 1)
def test_counts_a_comprehension_as_a_loop_and_its_conditions(self):
self.assertEqual(scored("""
def f(rows):
return [row for row in rows if row if row > 1]
"""), 3)
class Units(unittest.TestCase):
def test_measures_functions_and_methods_with_the_state_methods_share(self):
units = python_units("a.py", textwrap.dedent("""
COUNT = 0
def bump():
global COUNT
COUNT += 1
class Box:
def __init__(self):
self.size = 0
def grow(self):
self.size += 1
self.name = "x"
"""))
self.assertEqual(units, [Unit("a.py", "bump", 0, 3, 1), Unit("a.py", "__init__", 0, 2, 1), Unit("a.py", "grow", 0, 3, 1)])
def test_a_source_that_does_not_parse_has_no_units(self):
self.assertEqual(python_units("a.py", "def ("), [])
class Places(unittest.TestCase):
HISTORY = textwrap.dedent("""
# Comprehension history
## 2026-10-08T12:50:49Z, PR #224 at a58a0a8
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|---|---|---|---|---|---|
| Mean | 7 | 7 | 7 | 7 | 7 |
Junior A, hardest first:
- `skills/x/SKILL.md:77` "Architecture"
- `bin/tool/cycle/states.py:69` `transition`, with `cycle/run.py:55` `step` and `:90` `close`
## 2026-10-08T18:37:21Z, PR #228 at 2682668, against 8c9bf09
Mid A, decided by:
- `bin/tool/cycle/run.py:1` `step`
""")
def test_reads_each_scoring_runs_places_and_leaves_paired_rulings_out(self):
runs = scoring_runs(self.HISTORY)
self.assertEqual([(run.heading, run.commit) for run in runs], [("2026-10-08T12:50:49Z, PR #224 at a58a0a8", "a58a0a8")])
self.assertEqual(runs[0].places, [Place("skills/x/SKILL.md", None), Place("bin/tool/cycle/states.py", "transition"), Place("cycle/run.py", "step"), Place("cycle/run.py", "close")])
def test_resolves_a_shortened_path_to_the_file_nearest_the_one_before_it(self):
tracked = ["bin/tool/cycle/states.py", "bin/tool/cycle/run.py", "bin/other/cycle/run.py"]
places = [Place("bin/tool/cycle/states.py", "transition"), Place("cycle/run.py", "step"), Place("gone.py", "x")]
self.assertEqual(resolved(places, tracked), [Place("bin/tool/cycle/states.py", "transition"), Place("bin/tool/cycle/run.py", "step")])
class Bench(unittest.TestCase):
def test_auc_is_one_for_a_perfect_ranking_and_counts_a_tie_half(self):
self.assertEqual(auc([5, 4], [1, 2]), 1.0)
self.assertEqual(auc([3], [3]), 0.5)
def test_co_changed_counts_each_files_distinct_partners(self):
self.assertEqual(co_changed("@@\na.py\nb.py\n@@\na.py\nc.py\n@@\na.py"), {"a.py": 2, "b.py": 1, "c.py": 1})
def test_scores_the_named_units_and_counts_places_no_unit_measures(self):
units = [Unit("a.py", "hard", 9, 40, 0), Unit("a.py", "easy", 1, 5, 0), Unit("b.py", "plain", 2, 6, 0)]
run = bench(units, [Place("a.py", "hard"), Place("doc.md", None)], {"a.py": 1})
assert run is not None
self.assertEqual((run.units, run.named, run.unmeasured), (3, 1, 1))
self.assertEqual((run.scores["cognitive complexity"].auc, run.scores["cognitive complexity"].top_k), (1.0, 1.0))
self.assertEqual(run.scores["shared state"].auc, 0.5)
def test_a_run_naming_no_measured_unit_is_not_scored(self):
self.assertIsNone(bench([Unit("a.py", "f", 1, 1, 0)], [Place("doc.md", None)], {}))
class BlackBox(unittest.TestCase):
def project(self, history: str) -> str:
directory = tempfile.TemporaryDirectory()
self.addCleanup(directory.cleanup)
top = directory.name
env = host_env()
for args in (["init", "-q"], ["config", "user.email", "t@t"], ["config", "user.name", "t"]):
subprocess.run(["git", *args], cwd=top, check=True, env=env)
pathlib.Path(top, "a.py").write_text("def hard(a):\n\tif a:\n\t\tfor x in a:\n\t\t\tif x:\n\t\t\t\treturn x\n\n\ndef easy():\n\treturn 1\n")
pathlib.Path(top, "test_a.py").write_text("def test_hard():\n\tif 1:\n\t\tpass\n")
subprocess.run(["git", "add", "."], cwd=top, check=True, env=env)
subprocess.run(["git", "commit", "-qm", "a"], cwd=top, check=True, env=env)
commit = subprocess.run(["git", "rev-parse", "--short", "HEAD"], cwd=top, check=True, capture_output=True, text=True, env=env).stdout.strip()
os.makedirs(pathlib.Path(top, "docs"))
pathlib.Path(top, "docs", "comprehension-history.md").write_text(history.replace("COMMIT", commit))
return top
def run_tool(self, top: str, *args: str) -> subprocess.CompletedProcess:
return subprocess.run([sys.executable, str(TOOL), *args], cwd=top, capture_output=True, text=True, env=host_env())
def test_prints_each_metrics_score_on_each_run_and_the_medians(self):
top = self.project("# Comprehension history\n\n## 2026-10-08T12:50:49Z, PR #1 at COMMIT\n\nJunior A, hardest first:\n\n- `a.py:1` `hard`\n")
ran = self.run_tool(top)
self.assertEqual(ran.returncode, 0, ran.stderr)
self.assertIn("| 2026-10-08T12:50:49Z, PR #1 at ", ran.stdout)
self.assertIn("| 2 | 1 | 0 | AUC 1.00, top k 1.00 | AUC 1.00, top k 1.00 | AUC 0.50,", ran.stdout)
self.assertIn("| Median |", ran.stdout)
def test_exits_1_when_no_run_names_a_measured_unit(self):
top = self.project("# Comprehension history\n\n## 2026-10-08T12:50:49Z, PR #1 at COMMIT\n\nJunior A, hardest first:\n\n- `README.md:1` \"Install\"\n")
ran = self.run_tool(top)
self.assertEqual((ran.returncode, ran.stdout), (1, ""))
self.assertIn("no scoring run in", ran.stderr)
def test_exits_1_naming_a_history_it_cannot_read_and_unknown_arguments(self):
top = self.project("")
self.assertIn("could not read", self.run_tool(top, "missing.md").stderr)
self.assertEqual(self.run_tool(top, "--x").returncode, 1)
+1 -1
View File
@@ -11,5 +11,5 @@ class Types(unittest.TestCase):
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
if shutil.which("uv") is None:
self.fail("uv on PATH fetches mypy; see README.md's Test line")
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "-m", "rank_todo", "-p", "reading_cost", "-m", "statusline", "-p", "loop_claude"], capture_output=True, cwd=ROOT / "bin", text=True)
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "-p", "metric_bench", "-m", "rank_todo", "-p", "reading_cost", "-m", "statusline", "-p", "loop_claude"], capture_output=True, cwd=ROOT / "bin", text=True)
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
+30 -19
View File
@@ -24,8 +24,6 @@
| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |
|---|---|---|---|---|---|---|---|---|---|
| 73 | 0.1.0 | | **Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.** | 3 | 4 | 10 | 8 | 1.4 | 31.4 |
| 77 | 0.1.0 | | **Measure each project with cheap static metrics — cognitive complexity, how far a change spreads, state shared or mutated across methods — and keep only those the panel's history shows tracking comprehension.** | 3 | 7 | 9 | 8 | 1.1, 1.4 | 29.1 |
| 78 | 0.1.0 | | **Generate comprehension questions with deterministic scripts — from symbols, the call graph and the tests, each answer checkable by a tool — and bench the scores an agent gets on them against the panel, reading cost and static metrics.** | 3 | 7 | 9 | 8 | 1.1, 1.4 | 29.1 |
| 85 | 0.1.0 | decision | **Move larv-review's mechanical rules — the paired ruling's base and vote count, each pass's round count, the scoring run's trigger and the comprehension history's format — into one tested tool under `bin/`, so the skill says only when to run it and what each answer means.** | 4 | 7 | 8 | 9 | 1.1 | 28.1 |
| 1 | 0.1.0 | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 3 | 5 | 8 | 9 | 3, 4 | 27.0 |
@@ -45,32 +43,17 @@
| 68 | 0.1.0 | principle | **Test `workflow-bench/compare.py`'s blinding and `bench.py`'s run plan.** | 2 | 4 | 3 | 5 | 1.1 | 12.9 |
| 65 | 0.1.0 | | **Archive each judge reply the loop rejects.** | 3 | 4 | 4 | 5 | 3 | 12.5 |
| 82 | 0.1.0 | principle | **Sort `AGENTS.md`'s decision titles and `docs/decisions.md`'s entries by title.** | 1 | 2 | 3 | 4 | 1.3 | 12.3 |
| 73 | 0.1.0 | | **Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.** | 3 | 4 | 4 | 4 | 1.4 | 11.7 |
| 6 | 0.1.0 | decision | **Move what `loop-claude` writes out of `~/.claude.json`, and ask the human about each write that must stay.** | 5 | 5 | 4 | 5 | 1.1 | 11.7 |
| 51 | 0.1.0 | principle | **Hand loop-claude's terminal output to an injected effect, so no storage or retry function prints.** | 3 | 5 | 3 | 5 | 1.1 | 11.7 |
| 50 | 0.1.0 | principle | **Grant the bench's runs only the tools a review needs, in place of skipping every permission.** | 2 | 2 | 3 | 4 | 4 | 9.9 |
| 81 | 0.1.0 | principle | **Have `workflow-bench/bench.py` return each run's failure as a value, so one failed clone does not hide the later runs' results.** | 2 | 2 | 3 | 4 | 4 | 9.9 |
| 4 | 0.1.0 | | **Name each chunk's archive folder by the chunk's start time, so `<key>/` lists chunks in the order they ran.** | 3 | 2 | 4 | 3 | 3 | 9.6 |
| 52 | 0.1.0 | principle | **Record why this checkout lives in Claude Code's own directory, or move it out and link it in.** | 6 | 6 | 3 | 5 | 2 | 7.5 |
| 77 | 0.2.0 | | **Rerun `bin/metric-bench` once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 2 | 4 | 6 | 7 | 1.1, 1.4 | 22.9 |
## Details
### 73. Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.
The human ranks review cost first. Across 54 archived chunks the worker's own thread was 54% of spend, mostly cache reads: its context averaged about 350k tokens a turn and peaked near 700k, filled by reviewer reports, each of which arrived twice (the spawn result and the task notification).
The reports would live in the session's scratchpad, such as `<scratchpad>/larv-review/prose-round-1.md`, so no project's checkout sees them, and round counting and the `Answer <pass> round <n>:` commits stay as they are. A later round gets the path of that pass's earlier report in place of the findings pasted in. The worker opens the file only for the findings it fixes.
Measure first, from loop-claude's archive: how often a report still reaches the worker twice (PR #226's prose report, on 2026-10-08, arrived once), and how much of each report the worker acts on. Then choose how a report reaches the file:
- **Write access.** Grant each reviewer Write. No reviewer under `agents/` holds it today, and an agent's tool list cannot confine Write to one folder. Check whether a PreToolUse hook can tell a subagent's call from the worker's and deny a Write outside the scratchpad; without such a hook, only the prompt keeps its writes inside the scratchpad.
- **A wrapper.** A general-purpose subagent launches the reviewer with its tools unchanged, saves the report and returns only the summary lines. Claude Code's documentation says a subagent cannot start another; drop this route unless a test shows one can.
This risks goal 1.1: a worker fixing from a one-line summary can miss context it needs.
### 77. Measure each project with cheap static metrics — cognitive complexity, how far a change spreads, state shared or mutated across methods — and keep only those the panel's history shows tracking comprehension.
The human wants these explored and benched against the panel, with a global push toward a good state in every project. Candidates: cognitive complexity per function; how far a change spreads (files and modules a typical chunk touches, and files that change together in git history); state shared or mutated across methods (fields written by more than one method of a type, package-level variables written outside init, lack of cohesion between methods). Function and file length are warnings to look into, never limits: a plain long function can be simple, and a hard one may need five-line pieces. One harness under `bin/` serves every project; which metrics a project tracks may differ by project.
### 78. Generate comprehension questions with deterministic scripts — from symbols, the call graph and the tests, each answer checkable by a tool — and bench the scores an agent gets on them against the panel, reading cost and static metrics.
A hand-written question bank goes stale as the project grows and leaves everything it does not ask about unmeasured. A script that samples symbols at random each run ("what calls X?", "where is Y validated?", "which test fails if Z changes?") covers the whole project over time, and the call graph, grep and the tests hold the answers, so no judge is needed. Each answer names the code it rests on, so a question whose code changed is regenerated. Keep the method only where the bench shows it tracks the panel at a lower cost.
@@ -157,6 +140,19 @@ Goal 3, raised by the principles audit of PR #215: the judge runs with `--no-ses
"Deterministic order", in `principles/technical-principles.md`, raised by the principles audit of PR #224: their order carries no meaning, so two chunks adding a decision conflict at the end, and a reader cannot find a title by scanning.
### 73. Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.
2026-10-08, lilleman: rescored from A10/G8 to A4/G4 on the measurement below, so other items that lower token use go first.
Measured over the 68 transcripts in loop-claude's archive on 2026-10-08, matching each Agent call to its tool result and to any `<task-notification>` naming its tool-use id. No report reached the worker twice: 813 arrived once as the hand-back, and 363 once as a notification after Claude Code ran the agent in the background. Reports took a median 19k tokens per chunk, 8% of the worker's peak context (24% at most). Lines carrying a severity mark are 9–24% of a reviewer's report, and the worker would still open the file for each 🔴 or 🟡 it fixes. A third of the report volume is the comprehension seats', whose votes and places the worker copies into the history. File reads fill more of the worker's context: `sed -n` output is 30% of its Bash results and `grep` output 11%, which item 78 and `reading-cost` target.
The reports would live in the session's scratchpad, such as `<scratchpad>/larv-review/prose-round-1.md`, so no project's checkout sees them. A later round gets the path of that pass's earlier report in place of the findings pasted in. How a report reaches the file:
- **Write access.** Grant each reviewer Write. No reviewer under `agents/` holds it today, an agent's tool list cannot confine Write to one folder, and `~/.claude/CLAUDE.md` → "Working with the codebase" gives a reviewer that only reads no edits. Check whether a PreToolUse hook can tell a subagent's call from the worker's and deny a Write outside the scratchpad.
- **A wrapper.** A general-purpose subagent launches the reviewer, saves the report and returns only the summary lines. Claude Code's documentation says a subagent cannot start another; drop this route unless a test shows one can.
This risks goal 1.1: a worker fixing from a one-line summary can miss context it needs.
### 6. Move what `loop-claude` writes out of `~/.claude.json`, and ask the human about each write that must stay.
Claude Code owns that file and writes it. `trust` reads it, changes it and writes it back, so a write that lands in between is lost.
@@ -180,3 +176,18 @@ Today the folder is the SHA-256 of the transcript's first timestamped line, whic
### 52. Record why this checkout lives in Claude Code's own directory, or move it out and link it in.
"One owner per value" and "Prefer the vendor-neutral option", in `principles/technical-principles.md`: git, Claude Code and loop-claude all write `~/.claude`, kept apart only by the `.gitignore` allowlist, and the repo's records live in a vendor directory.
### 77. Rerun `bin/metric-bench` once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.
2026-10-08, lilleman: no metric enters review until a rerun on more data shows it tracking the panel.
`bin/metric-bench` measures each Python function and method at each scoring run's commit, and scores how well each metric ranks the units the seats named above the rest. On 2026-10-08 only this repo's history named places: five scoring runs, all that day. The medians:
| Metric | AUC | Top k |
|---|---|---|
| Cognitive complexity | 0.66 | 0.26 |
| Length | 0.74 | 0.18 |
| Shared state | 0.50 | 0.06 |
| Change spread | 0.64 | 0.17 |
AUC is 0.5 for a coin toss. Shared state cannot be judged here, since no unit in `bin/` writes state another method shares. Every run's first place was a section of `skills/larv-review/SKILL.md`, which no code metric sees. The rerun needs places from a project in Go or TypeScript, the languages most other projects use, so the bench first measures those languages, run through each project's own tooling. Function and file length stay warnings to look into, never limits: a plain long function can be simple, and a hard one may need five-line pieces.