Merge pull request 'Count seat agreement in paired rulings against scoring runs' (#303) from item-84 into main

This commit was merged in pull request #303.
This commit is contained in:
2026-10-17 21:38:13 +02:00
14 changed files with 237 additions and 28 deletions
+1 -1
View File
@@ -13,7 +13,7 @@ Decisions live in `docs/decisions.md`:
- review: a pass gets at most three rounds per chunk; instead of a fourth, the run asks the human
- review: a reviewer or comprehension seat that rules the work below standard may ask to rerun once, one rung up, in the same round; the ruling that stands is an escalation trigger for the worker, never for the judge
- review: a fix for a 🔴 or 🟡 reruns the pass that found it and each pass whose trigger names fixes like it; from round 2 a pass reviews only the answers to its last round's findings and any other pass's fix that re-entered it; a fix for a 🟢, a ❓ or a prose finding reruns none
- review: the comprehension check records how hard a project is to read and blocks no merge: a paired ruling against the last measured commit where the architecture pass ran, and the four-seat scoring run when a release's items run out or an item names the floor
- review: the comprehension check records how hard a project is to read and blocks no merge: a paired ruling against the last measured commit where the architecture pass ran, and the four-seat scoring run, once when an item names the floor and twice on one head when a release's items run out
- review: a project decision may suspend the comprehension check for the plan items it names
- review: a project decision may opt every chunk out of the architect, the product-owner pass or the comprehension check
- review: the principles audit runs on every chunk that changes code, config or tests, and on no other chunk
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- A scoring run due because a release's items ran out costs four more seats: it is done twice on one head. `larv-review agreement` compares how often Mid A and the inherited architect vote alike in paired rulings with how often their overall scores move the same way between the two runs. After 10 such pairs it gives a verdict, and on `Assumption fails:` larv-review asks you whether to keep the paired ruling.
- `workflow-bench/bench.py` gives each run a fixed list of allowed tools. Before, it skipped every permission check. A run may launch agents, run `/security-review`, edit its clone (not the clone's `.git/` or `.claude/`) and its report, and read its clone and the copy of `~/.claude` it benches. For git it gets one command, `workflow-bench/review_git.py`, and no bare `git`. The wrapper runs `log`, `diff`, `show`, `status`, `ls-files`, `add` and `commit -m` on the run's clone, whatever the working directory. It allows only the options it lists for each, and refuses an argument that names a path outside the clone. git runs with no global or system config, hook, pager or editor. A run loads no settings or MCP servers from `~/.claude` or from the case's clone, so their hooks and extra directories do not apply to it. Every other call is denied. Expect a benched review that needs another git command, or to run a project's code or tests, to say in its report that it could not.
- Every goal in `README.md` → Goals is marked `**Maintain:**`, a state to hold, or `**Achieve:**`, a direction to pursue. Each achieve goal has a signal, a counter-signal and a one-line reason under it. A change that breaks a maintain goal is redesigned or dropped, whatever it scores: `rules/todo-items.md` says so, and larv-review's goal alignment step no longer merges such a PR. `rules/goals-and-audience.md` states the form for every project's README. Expect larv-review to stop a PR on a maintain goal where it asked you before.
- Reshape every project's `todo.md` to the new shape below: until it is reshaped, that is the project's only work, as it is while README.md lacks Goals and Audience, and `rank_todo.py` says so. First move every machine to this release: an older release rejects the new formula, so it fails on every reshaped `todo.md`. Then mark the README's goals `**Maintain:**` or `**Achieve:**`, score each item's L, E and I, and set a bar on the new scale below the score of every item to keep, since the old bar would drop most items. Until then, an old-shape `todo.md` still ranks by its own formula and weights.
+1 -1
View File
@@ -84,7 +84,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `chunk-preflight` finds the mechanical facts a chunk starts from and makes its worktree; `loop-claude` runs it before it routes each chunk, and `chunk-preflight --help` lists the facts.
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints the lowest rung that, like each rung above it, returns them; `judge-replay --help` states the replay.
- `larv-review <command>` holds larv-review's mechanical rules: the paired ruling's base and vote count, each pass's round count, the scoring run's trigger, and the comprehension history's format and latest places; `larv-review --help` states them.
- `larv-review <command>` holds larv-review's mechanical rules: the paired ruling's base and vote count, each pass's round count, the scoring run's trigger, and the comprehension history's format and latest places, and its count of how often seats agree; `larv-review --help` states them.
- `loop-claude <task>` runs a project's next chunk cycle after cycle; `loop-claude --help` states its contract.
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
- `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format.
+20 -10
View File
@@ -79,7 +79,7 @@ class ScoringRun:
means: list[decimal.Decimal]
# A paired ruling as entry_text writes it. A scoring run's heading has no `, against`; it has no Ruling line, ends its table with a Mean row, and labels each seat's places `<seat>, hardest first:` or, for the architect, `<seat>, would restructure first:`.
# A paired ruling as entry_text writes it. A scoring run's heading has no `, against`; it has no Ruling line, ends its table with a Mean row, and labels each seat's places `<seat>, hardest first:` or, for the architect, `<seat>, would restructure first:`. A ruling recorded before PR #232 has a `| Seat | Vote |` table, whose one cell is the overall vote.
#
# ## 2026-10-09T09:47:25Z, PR #232 at 324d650, against 12d6d75
#
@@ -104,10 +104,12 @@ def entries(history: str) -> list[Entry]:
def latest_places(history: str) -> str | None:
"""The places seats found hard from the newest entry, by time, that is a scoring run or holds scoring seats' places: in it and each later entry, each ruled place opening with a dimension its seat's vote counted as worse, and every scoring seat's place. Each entry's places come under its heading, oldest first; with no such entry, it starts at the oldest entry. None when no place counts."""
"""The places seats found hard from the oldest entry on the commit of the newest entry, by time, that is a scoring run or holds scoring seats' places: in it and each later entry, each ruled place opening with a dimension its seat's vote counted as worse, and every scoring seat's place. Each entry's places come under its heading, oldest first; with no such entry, it starts at the oldest entry. None when no place counts."""
found = sorted(entries(history), key=lambda entry: entry.time)
newest = max((index for index, entry in enumerate(found) if not entry.paired or entry.scored_places), default=0)
parts = [placed(entry, worse_places(entry) + labelled_bullets(entry.body, SCORED_PLACES)) for entry in found[newest:]]
scored = [index for index, entry in enumerate(found) if not entry.paired or entry.scored_places]
newest_commit = found[scored[-1]].commit if scored else None
start = next((index for index in scored if found[index].commit == newest_commit), 0)
parts = [placed(entry, worse_places(entry) + labelled_bullets(entry.body, SCORED_PLACES)) for entry in found[start:]]
return "\n".join(part for part in parts if part) or None
@@ -125,13 +127,16 @@ def placed(entry: Entry, labelled: list[tuple[str, list[str]]]) -> str:
def worse_votes(body: str) -> dict[str, set[str]]:
"""Each seat's dimensions, Overall included, whose vote in the ruling's table counted as worse; `same (voted worse, …)` counted as same. The votes table comes first, so a seat's row in a later scores table is not its vote."""
found: dict[str, set[str]] = {}
return {name: {dimension for dimension, vote in zip(DIMENSIONS, cells) if vote == "worse"} for name, cells in table_rows(body).items() if len(cells) == len(DIMENSIONS)}
def table_rows(body: str) -> dict[str, list[str]]:
"""Each table row's cells after its first, keyed by that first cell; where rows share a first cell, the first row wins. Header and Mean rows are included."""
found: dict[str, list[str]] = {}
for line in body.splitlines():
if not line.startswith("| "):
continue
name, *votes = [cell.strip() for cell in line.strip("|").split("|")]
if len(votes) == len(DIMENSIONS):
found.setdefault(name, {dimension for dimension, vote in zip(DIMENSIONS, votes) if vote == "worse"})
if line.startswith("| "):
name, *cells = [cell.strip() for cell in line.strip("|").split("|")]
found.setdefault(name, cells)
return found
@@ -205,6 +210,11 @@ def vote_text(dimension: str, said: str, counted: str) -> str:
return f"same (voted {said}, naming no place)" if dimension == DIMENSIONS[-1] else f"same (voted {said}, no place naming {dimension})"
def counted_vote(cell: str) -> str:
"""The vote a ruling's table cell counted as, which is its first word, as vote_text writes it."""
return cell.split()[0]
def scoring_entry(title: str, scoring: ScoringRun) -> str:
return entry_text(title, [], [scores_table(scoring)], scoring_labels(scoring.seats))
+46
View File
@@ -0,0 +1,46 @@
import decimal
from comprehension_history import counted_vote, entries, Entry, table_rows
from larv_review.seats import FIRST_SEATS
# Matches score-outcomes, which waits for 10 finished chunks before it shows a median.
MIN_PAIRS = 10
def overall_cells(entry: Entry) -> dict[str, str]:
"""Each table row's last non-empty cell, by its first cell: a seat's overall vote in a ruling table, whether it has five votes or the older single `Vote`, or its overall score in a scoring table."""
return {name: cells[-1] for name, cells in table_rows(entry.body).items() if cells and cells[-1]}
def ruling_agreement(found: list[Entry]) -> list[bool]:
"""For each paired ruling that holds both of the first seats' overall votes, whether they counted alike."""
first, second = FIRST_SEATS
cells = [overall_cells(entry) for entry in found if entry.paired]
return [counted_vote(each[first]) == counted_vote(each[second]) for each in cells if first in each and second in each]
def run_agreement(found: list[Entry]) -> list[bool]:
"""For each commit scored at least twice, whether the first two seats' overall scores moved the same way, up, down or not at all, from its first run to its second. Both runs read one tree, so the pair is the ruling those seats would make from absolute scores on code that did not change."""
first, second = FIRST_SEATS
runs: dict[str, list[dict[str, decimal.Decimal]]] = {}
for entry in sorted((each for each in found if not each.paired), key=lambda each: each.time):
cells = overall_cells(entry)
if first in cells and second in cells:
runs.setdefault(entry.commit, []).append({first: decimal.Decimal(cells[first]), second: decimal.Decimal(cells[second])})
pairs = [(each[0], each[1]) for each in runs.values() if len(each) >= 2]
return [direction(before[first], after[first]) == direction(before[second], after[second]) for before, after in pairs]
def direction(before: decimal.Decimal, after: decimal.Decimal) -> int:
return (after > before) - (after < before)
def agreement_report(history: str) -> str:
"""Whether the first two seats agree more often in a paired ruling than across two scoring runs on one commit, as the comprehension check's decision assumes. It opens `Unchecked:` until the history holds MIN_PAIRS pairs of runs and one paired ruling, then `Assumption holds:` or `Assumption fails:`; equal rates fail."""
found = entries(history)
ruled, scored = ruling_agreement(found), run_agreement(found)
counts = f"{' and '.join(FIRST_SEATS)} voted alike overall in {sum(ruled)} of {len(ruled)} paired rulings, and their overall scores moved alike between two scoring runs on one commit in {sum(scored)} of {len(scored)} pairs."
if len(scored) < MIN_PAIRS or not ruled:
return f"Unchecked: {counts} A verdict needs {MIN_PAIRS} commits each scored twice and one paired ruling."
holds = sum(ruled) / len(ruled) > sum(scored) / len(scored)
return f"Assumption {'holds' if holds else 'fails'}: {counts}"
+20 -7
View File
@@ -1,13 +1,13 @@
"""Hold larv-review's mechanical rules: the paired ruling's base and vote count, each pass's rounds, the scoring run's trigger, and the comprehension history's format and latest places.
"""Hold larv-review's mechanical rules: the paired ruling's base and vote count, each pass's rounds, the scoring run's trigger, and the comprehension history's format and latest places, and its count of how often seats agree.
usage: larv-review ruling-base | round-count | needs-scoring-run [<item ID>...] | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places
usage: larv-review ruling-base | round-count | needs-scoring-run [<item ID>...] | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement
The comprehension check records how hard a project is to read, two ways, and blocks no merge. In a
paired ruling, simulated readers, the seats, read the branch's head beside a base and vote better,
same or worse, overall and on each dimension. In a scoring run, four seats score the project from 1
to 10. Each seat names the places it found hard.
Run it inside a project checkout or one of its worktrees. Each command but record-ruling and places
Run it inside a project checkout or one of its worktrees. Each command but record-ruling, places and agreement
first fetches origin, when the checkout has that remote, and takes origin's default branch, or else the
local main or master, as the default branch. When the fetch fails, round-count reads origin as
last fetched and says so on stderr; the other three exit 1. The history is
@@ -31,6 +31,9 @@ landed when the chunk ships one of the release's items, or when the release is t
default branch's todo.md names and the default branch has a commit after its newest v* tag, or has
no v* tag.
A second scoring run of one head names itself `<PR or branch> rerun at <commit>`, so record-scoring
does not refuse it as a repeat of the first.
A seats file's first line names what the seats read, `<PR or branch> at <commit>`, ending
`, against <base>` in a votes file. Each seat's line follows, opening with its label written
exactly: `Junior A:`, `Mid A:`, `Maintainability senior:` or `Inherited architect:`. Its places
@@ -66,11 +69,18 @@ record-scoring takes the scores file of a run needs-scoring-run calls, whose fir
and their means, rounded half up to two decimals, prints the means and, when a mean is under the
floor of {floor}, a line opening `Floor:` naming each one.
places prints the places seats found hard since the newest scoring run: that run's places, and,
places prints the places seats found hard since the newest scored commit: the places of each scoring
run on it, and,
in each later ruling, each place opening with a dimension, or Overall, that its seat voted worse.
A ruling recorded with scoring seats' places counts as a scoring run. Each entry's places come
under its heading, oldest first. With no scoring run, it reads every ruling.
agreement pairs the first two scoring runs recorded on one commit. In each pair, a seat's overall
score goes up, goes down or stays the same. It checks the assumption behind the paired ruling: Mid A
and the inherited architect vote alike overall in a paired ruling more often than their scores go the
same way in a pair. It prints both counts. It opens `Unchecked:` until the history holds {pairs} such
pairs and one paired ruling, then `Assumption holds:` or `Assumption fails:`, and exits 0 each time.
Each record is appended to the history, created when missing, under a heading of the current UTC
time and the seats file's first line. Each commit that line names must be in the checkout. It exits 1
when the history already holds an entry on what that line reads.
@@ -90,15 +100,16 @@ from failure import Failure
from larv_review.scoring import FLOOR
from larv_review.git import history_changes
from larv_review.ruling_base import measured_commit
from larv_review.agreement import MIN_PAIRS
from larv_review.plan import plan_reasons
from larv_review.record import record_ruling, record_scoring, show_places
from larv_review.record import record_ruling, record_scoring, show_agreement, show_places
USAGE = "usage: larv-review ruling-base | round-count | needs-scoring-run [<item ID>...] | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places"
USAGE = "usage: larv-review ruling-base | round-count | needs-scoring-run [<item ID>...] | record-ruling <votes file> | record-scoring <scores file> [<item ID>...] | places | agreement"
def main(argv: list[str]) -> int:
if argv in (["-h"], ["--help"]):
print(__doc__.strip().format(cap=CAP, floor=FLOOR))
print(__doc__.strip().format(cap=CAP, floor=FLOOR, pairs=MIN_PAIRS))
return 0
command, *given = argv or [""]
if command == "ruling-base" and not given:
@@ -113,6 +124,8 @@ def main(argv: list[str]) -> int:
done = with_default_branch(lambda default: record_scoring(default, pathlib.Path(given[0]), [int(id_) for id_ in given[1:]]), stale_ok=False)
elif command == "places" and not given:
done = show_places()
elif command == "agreement" and not given:
done = show_agreement()
elif command in ("needs-scoring-run", "record-scoring") and given:
named = given[1:] if command == "record-scoring" else given
print(f"{command} takes item IDs, the numbers in todo.md's ID column, not {' '.join(id_ for id_ in named if not re.fullmatch(r'[0-9]+', id_))}; {USAGE}", file=sys.stderr)
+1
View File
@@ -19,6 +19,7 @@ def scoring_run_due(before: list[Item], after: list[Item], ships: list[int], lan
next_release = min((item.release for item in before), default=None)
for release in emptied(before, after, shipped):
if any(item.release == release for item in shipped) or (release == next_release and landed):
# SKILL.md → "Second run" keys on this wording.
reasons.append(f"leaves release {'.'.join(map(str, release))} with no plan item")
return reasons
+9
View File
@@ -1,6 +1,7 @@
import pathlib
import time
from larv_review.agreement import agreement_report
from comprehension_history import appended, DIMENSIONS, heading, Heading, HISTORY, latest_places, recorded, ruling_entry, scoring_entry, ScoringRun, vote_text
from git_repo import commit_exists, top
from larv_review.ruling import counted, overall, ruling, split, votes
@@ -85,6 +86,14 @@ def show_places() -> Failure | None:
return None
def show_agreement() -> Failure | None:
history = history_file()
if isinstance(history, Failure):
return history
print(agreement_report(history[1]))
return None
def history_file() -> tuple[pathlib.Path, str] | Failure:
"""The checkout's history and its text, empty when the file is missing."""
checkout = top()
+95
View File
@@ -10,6 +10,7 @@ import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from larv_review.agreement import agreement_report, ruling_agreement, run_agreement
from comprehension_history import appended, entries, HistoryChange, latest_places, Place, recorded, ruling_entry, scoring_entry, ScoringRun, Seat
import comprehension_history
from larv_review.ruling import counted, ruling, split, votes
@@ -155,6 +156,93 @@ class History(unittest.TestCase):
fell = ruled.replace("2026-10-10T12:00:00Z", "2026-10-09T12:00:00Z") + "\n| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |\n|---|---|---|---|---|---|\n| Mid A | 7 | 6 | 7 | 7 | 7 |\n| Mean | 7 | 6 | 7 | 7 | 7 |\n\nJunior A, hardest first:\n\n- `bin/j.py:5` `s`: wide\n"
self.assertEqual(latest_places(appended(appended("", scored), fell)), hard_since.replace("2026-10-10T12:00:00Z", "2026-10-09T12:00:00Z") + "\nJunior A, hardest first:\n\n- `bin/j.py:5` `s`: wide\n")
def test_names_the_places_of_every_run_on_the_newest_scored_commit(self):
run = "## {time}, {pr} at {commit}\n\nJunior A, hardest first:\n\n- `bin/b.py:{line}` `h`: long\n"
first, rerun, other = (run.format(time=time, pr=pr, commit=commit, line=line) for time, pr, commit, line in (("2026-10-08T12:00:00Z", "PR #1", "aaaaaaa", 1), ("2026-10-08T13:00:00Z", "PR #1 rerun", "aaaaaaa", 2), ("2026-10-09T12:00:00Z", "PR #2", "bbbbbbb", 3)))
self.assertEqual(latest_places(appended(appended("", first), rerun)), first + "\n" + rerun)
self.assertEqual(latest_places(appended(appended(appended("", first), rerun), other)), other)
def scored(time: str, pr: str, commit: str, mid: str, architect: str) -> str:
"""A scoring run's entry as the history writes it, with the two seats' overall scores."""
run = ScoringRun([], {"Mid A": ["7", "7", "7", "7", mid], "Inherited architect": ["7", "7", "7", "7", architect]}, [decimal.Decimal("7.00")] * 5)
return scoring_entry(f"{time}, {pr} at {commit}", run)
def ruled(time: str, pr: str, mid: str, architect: str, voted: str | None = None) -> str:
"""A paired ruling's entry as the history writes it; `voted` is what Mid A said overall when it counted as `mid`."""
counts = {"Mid A": ["same"] * 4 + [mid], "Inherited architect": ["same"] * 4 + [architect]}
said = {**counts, "Mid A": ["same"] * 4 + [voted or mid]}
return ruling_entry(f"{time}, {pr} at ccccccc, against aaaaaaa", [Seat("Mid A", "", []), Seat("Inherited architect", "", [])], said, counts, "same")
def pairs_of_runs(alike: int, apart: int) -> list[str]:
"""Each pair is two runs on one commit: Mid A and the architect both up, or Mid A up and the architect not moving."""
return [scored(f"2026-10-02T00:{index:02d}:{later}Z", "PR #1 rerun" if later else "PR #1", f"{index:07x}", "7.5" if later else "7", "7.5" if later and index < alike else "7") for index in range(alike + apart) for later in (0, 1)]
class Agreement(unittest.TestCase):
def test_counts_the_rulings_whose_two_seats_voted_alike_overall(self):
old = "## 2026-10-08T18:37:21Z, PR #228 at 2682668, against 8c9bf09\n\nRuling: worse\n\n| Seat | Vote |\n|---|---|\n| Mid A | worse |\n| Inherited architect | worse |\n"
history = "\n".join([ruled("2026-10-01T00:00:00Z", "PR #1", "same", "same"), ruled("2026-10-02T00:00:00Z", "PR #2", "same", "same", voted="worse"), ruled("2026-10-03T00:00:00Z", "PR #3", "worse", "better"), old])
self.assertEqual(ruling_agreement(entries(history)), [True, True, False, True])
def test_compares_the_two_seats_across_the_first_two_runs_on_one_commit(self):
both_up = [scored("2026-10-01T00:00:00Z", "PR #1", "aaaaaaa", "7", "7"), scored("2026-10-02T00:00:00Z", "PR #1 rerun", "aaaaaaa", "7.5", "7.5")]
one_up = [scored("2026-10-03T00:00:00Z", "PR #2", "bbbbbbb", "7", "7"), scored("2026-10-04T00:00:00Z", "PR #2 rerun", "bbbbbbb", "7.5", "7")]
self.assertEqual(run_agreement(entries("\n".join([*both_up, *one_up]))), [True, False])
def test_counts_opposite_moves_and_a_move_against_none_as_different(self):
opposite = [scored("2026-10-01T00:00:00Z", "PR #1", "aaaaaaa", "7", "7"), scored("2026-10-02T00:00:00Z", "PR #1 rerun", "aaaaaaa", "7.5", "6.5")]
down_and_none = [scored("2026-10-03T00:00:00Z", "PR #2", "bbbbbbb", "7", "7"), scored("2026-10-04T00:00:00Z", "PR #2 rerun", "bbbbbbb", "6.5", "7")]
both_down = [scored("2026-10-05T00:00:00Z", "PR #3", "ccccccc", "7", "7"), scored("2026-10-06T00:00:00Z", "PR #3 rerun", "ccccccc", "6.5", "6.5")]
self.assertEqual(run_agreement(entries("\n".join([*opposite, *down_and_none, *both_down]))), [False, False, True])
def test_leaves_out_commits_scored_once_and_runs_of_one_pr_on_different_commits(self):
history = "\n".join([scored("2026-10-01T00:00:00Z", "PR #1", "aaaaaaa", "7", "7"), scored("2026-10-02T00:00:00Z", "PR #1", "bbbbbbb", "7.5", "7.5")])
self.assertEqual(run_agreement(entries(history)), [])
def test_takes_the_two_earliest_runs_of_a_commit_whatever_order_the_file_holds_them(self):
history = "\n".join([scored("2026-10-03T00:00:00Z", "PR #1 again", "aaaaaaa", "6", "7"), scored("2026-10-01T00:00:00Z", "PR #1", "aaaaaaa", "7", "7"), scored("2026-10-02T00:00:00Z", "PR #1 rerun", "aaaaaaa", "7.5", "7.5")])
self.assertEqual(run_agreement(entries(history)), [True])
def test_skips_a_run_whose_table_lists_neither_seat(self):
mean_only = scoring_entry("2026-10-01T12:00:00Z, PR #1 at aaaaaaa", ScoringRun([], {}, [decimal.Decimal("7.00")] * 5))
history = "\n".join([mean_only, scored("2026-10-01T00:00:00Z", "PR #1 rerun", "aaaaaaa", "7", "7"), scored("2026-10-02T00:00:00Z", "PR #1 again", "aaaaaaa", "7.5", "7.5")])
self.assertEqual(run_agreement(entries(history)), [True])
def test_skips_an_entry_listing_one_seat_or_an_empty_overall(self):
one_seat = "## 2026-10-01T00:00:00Z, PR #1 at aaaaaaa, against bbbbbbb\n\nRuling: same\n\n| Seat | Overall |\n|---|---|\n| Mid A | same |\n"
empty = "## 2026-10-02T00:00:00Z, PR #2 at ccccccc, against bbbbbbb\n\nRuling: same\n\n| Seat | Overall |\n|---|---|\n| Mid A | same |\n| Inherited architect | |\n"
self.assertEqual(ruling_agreement(entries("\n".join([one_seat, empty]))), [])
half = "## 2026-10-03T00:00:00Z, PR #3 at ddddddd\n\n| Seat | Overall |\n|---|---|\n| Mid A | 7 |\n"
self.assertEqual(run_agreement(entries("\n".join([half, half.replace("PR #3", "PR #3 rerun").replace("T00:00:00Z", "T01:00:00Z")]))), [])
def test_records_a_rerun_beside_the_run_it_repeats(self):
history = appended("", scored("2026-10-01T00:00:00Z", "PR #1", "aaaaaaa", "7", "7"))
self.assertTrue(recorded(history, "PR #1 at aaaaaaa"))
self.assertFalse(recorded(history, "PR #1 rerun at aaaaaaa"))
def test_gives_no_verdict_before_ten_pairs_of_runs_and_a_ruling(self):
ruling = ruled("2026-10-20T00:00:00Z", "PR #2", "same", "same")
self.assertIn("Unchecked: ", agreement_report("\n".join([*pairs_of_runs(9, 0), ruling])))
self.assertIn("Unchecked: ", agreement_report("\n".join(pairs_of_runs(10, 0))))
def test_holds_when_the_seats_agree_in_a_higher_share_of_rulings_than_of_pairs_of_runs(self):
rulings = [ruled(f"2026-10-20T00:00:0{index}Z", f"PR #{index + 2}", "same", "same") for index in range(3)]
report = agreement_report("\n".join([*pairs_of_runs(5, 5), *rulings]))
self.assertEqual(report.split(":")[0], "Assumption holds")
self.assertIn("3 of 3 paired rulings", report)
self.assertIn("5 of 10 pairs", report)
def test_fails_when_the_two_shares_are_equal(self):
rulings = [ruled("2026-10-20T00:00:00Z", "PR #2", "same", "same"), ruled("2026-10-20T00:00:01Z", "PR #3", "worse", "same")]
self.assertEqual(agreement_report("\n".join([*pairs_of_runs(5, 5), *rulings])).split(":")[0], "Assumption fails")
def test_fails_when_the_seats_agree_in_a_lower_share_of_rulings_than_of_pairs_of_runs(self):
rulings = [ruled("2026-10-20T00:00:00Z", "PR #2", "same", "same"), ruled("2026-10-20T00:00:01Z", "PR #3", "same", "same"), ruled("2026-10-20T00:00:02Z", "PR #4", "worse", "same")]
self.assertEqual(agreement_report("\n".join([*pairs_of_runs(8, 2), *rulings])).split(":")[0], "Assumption fails")
class Base(unittest.TestCase):
def test_takes_the_newest_change_adding_an_entry(self):
@@ -337,6 +425,13 @@ class LarvReviewTool(unittest.TestCase):
outside = subprocess.run([sys.executable, str(TOOL), "places"], capture_output=True, cwd=self.checkout.parent, env=self.env, text=True)
self.assertEqual((outside.returncode, outside.stdout), (1, ""))
def test_prints_how_often_seats_agree_in_rulings_and_across_runs_on_one_commit(self):
history = "\n".join([scored("2026-10-01T00:00:00Z", "PR #1", "aaaaaaa", "7", "7"), scored("2026-10-02T00:00:00Z", "PR #1 rerun", "aaaaaaa", "7.5", "7"), ruled("2026-10-03T00:00:00Z", "PR #2", "same", "same")])
self.commit("Record the checks", {"docs/comprehension-history.md": appended("", history)})
ran = self.run_tool("agreement")
self.assertEqual((ran.returncode, ran.stdout), (0, "Unchecked: Mid A and Inherited architect voted alike overall in 1 of 1 paired rulings, and their overall scores moved alike between two scoring runs on one commit in 0 of 1 pairs. A verdict needs 10 commits each scored twice and one paired ruling.\n"), ran.stderr)
self.assertIn("unknown arguments", self.run_tool("agreement", "x").stderr)
def test_refuses_a_ruling_file_that_names_no_base_or_a_commit_the_checkout_lacks(self):
ran = self.run_tool("record-ruling", self.seats_file(f"PR #2 at {self.start}\n\nMid A: {said('same')}\n\nInherited architect: {said('same')}\n"))
self.assertEqual(ran.returncode, 1)
+16
View File
@@ -2103,3 +2103,19 @@ Junior A, decided by:
- Shape, Locality: `bin/todo_table.py:212` `standing`: the table parser grew from 247 to 346 lines and now decides how a release stands through a five-valued literal whose three-line comment must be decoded first, where 46b79e7's `routed_item` stated the pick in six lines
- Locality: `bin/chunk_preflight/main.py:30` `__doc__`: the preflight runs reading-cost in the new worktree, and larv-review's step now depends on whether the prompt shows that, so one action is split across a tool, a prompt and a skill
- Shape: `bin/chunk_preflight/gathered.py:1` `gathered`: the name covers the data but not the pure choices it also holds, and `main.py` imports 12 names from it on one line
## 2026-10-17T19:35:31Z, PR #303 at 85cbd6b, against 712dabd
Ruling: same
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|---|---|---|---|---|---|
| Mid A | same | same | same | same | same |
| Inherited architect | same | worse | same | same | same |
Inherited architect, decided by:
- Locality: `bin/comprehension_history.py:106` `latest_places`: one `max` over the entries became three dependent steps, and the `rerun at` rule in SKILL.md is needed to see why the start point moved
- Locality: `skills/larv-review/SKILL.md:118` "Second run" and `bin/larv_review/plan.py:22` `scoring_run_due`: the second run keys on the exact wording of a reason string, with no test or import linking the two
- Locality: `bin/larv_review/agreement.py:10` `overall_cells`: the last cell means overall only through `table_rows`' first-row-wins rule and the pre-#232 `Vote` table, described at `comprehension_history.py:82`
- Locality: `bin/larv_review/main.py:81` `__doc__`: the help hard-codes 10 pairs where `MIN_PAIRS` holds the number
+3 -1
View File
@@ -74,12 +74,14 @@ Valid while the ladder orders its rungs by competence.
2026-10-13, lilleman through todo item 97, implemented by claude: from round 2 a pass reviews only how the chunk answered its previous round, each fix and each dismissal with its reason, and reports only one that is plainly wrong or worse than the change it answered. A fix from another pass that re-enters it is still reviewed as a first round would review it, limited to its lines. Reviewing all that changed had a reviewer find new gaps in each round's fixes, so a pass ran to its cap. Serves goals 1.4 and 1.5. Valid while round-2 findings outside a round's answers keep changing nothing a user would notice.
## review: the comprehension check records how hard a project is to read and blocks no merge: a paired ruling against the last measured commit where the architecture pass ran, and the four-seat scoring run when a release's items run out or an item names the floor
## review: the comprehension check records how hard a project is to read and blocks no merge: a paired ruling against the last measured commit where the architecture pass ran, and the four-seat scoring run, once when an item names the floor and twice on one head when a release's items run out
2026-10-08, lilleman. Serves goals 1.1 and 1.4. Two independent four-seat runs moved up to 0.4 on near-identical code, so a fall within that noise forced rounds that changed nothing (PR #224). A seat reading both versions judges the difference on the same material, so the noise the versions share cancels. No conventional significance level lets a sign test settle with four votes, so the human chose a lead of two votes over every other ruling, same included. Two seats that agree settle it, so a PR that leaves the reading as it was costs two seats. A third seat alone can never settle it, so the third and fourth launch together. The human chose the last measured commit on the architect's finding that with a merge base as the base, chunks without an architecture pass, and a suspended restructure's steps, could each lower comprehension unseen. Reading the default branch one merged change at a time keeps the rule working for merge commits and squash merges alike. The scoring run also runs on a chunk shipping an item that names the floor, so the chunk that lifts a project to the floor can prove it. A chunk that removes a release's last item without shipping it runs it too, so no release ends unmeasured. Valid while seats reading both versions agree more often than two absolute runs do.
2026-10-13, lilleman. No published study validates simulated reader panels as a measure of how easy code is to understand, and no test of whether the panel's signal beats its noise has run. So no ruling, score or floor blocks a merge. The runs go on, so the history keeps the scores and the places the seats found hard, for later measurement. The architect, the stability reviewer and a session filing an item read the newest places, since those places are what a chunk can improve. With nothing to block, the check runs once per chunk, on the branch as review left it, in place of a run in each architecture round and a rescore after later commits. A fallen dimension no longer calls the four scoring seats either: their scores only decided the block. The base is now the newest merged change that adds any ruling or scoring run: with worse rulings merging, a base left behind them would charge one chunk's ruling with earlier chunks' falls. The newest places are the newest scoring run's and each later ruling's places a seat voted worse, so they change with each ruling. A chunk that leaves a touched place as hard says why in its PR summary, not in a decline's record, which would outlive the place.
2026-10-17, lilleman. Serves goal 1.1. Nothing checked the 2026-10-08 validity condition. The history's successive scoring runs of one PR read changed code, not near-identical code, so they cannot stand for two absolute runs. So each release's scoring run is done twice on the same head, the second recorded under `<PR> rerun at <commit>`, and `larv-review agreement` compares two rates: how often Mid A and the inherited architect vote alike overall in paired rulings, and how often their overall scores move alike between two scoring runs on one commit. It gives no verdict until the history holds 10 such pairs of runs, as `score-outcomes` waits for 10 chunks before it shows a median. The human chose the second run's cost, four more seats per release (goal 1.4), over a weaker check on the old pairs.
## review: a project decision may suspend the comprehension check for the plan items it names
2026-10-04, lilleman. Serves goal 1.4. In a restructure, each step's scoring run measures code the next step deletes. It costs a run and, below the floor, blocks the step, yet says nothing about what the restructure buys. The scoring run after the last step measures that. 2026-10-08: the paired ruling replaced the scoring run in most chunks, and the suspension covers it too; the ruling after the last step compares with the commit measured before the first step. 2026-10-13, lilleman: the comprehension check blocks nothing now, so a suspension saves only the runs. Valid while such a decision names its items and the plan item that ends it.
+17
View File
@@ -603,3 +603,20 @@ Read most (opens and searches):
- `AGENTS.md` 1
- `README.md` 1
- `bin/loop_claude/cycle/judge.py` 1
## Reading cost, chunk 4b3c8b8b3ee9, finished 2026-10-17T18:43:14Z
173360 new input tokens before the first commit.
Read most (opens and searches):
- `bin/loop_claude/test_loop_claude.py` 9
- `bin/rank_todo.py` 6
- `bin/loop_claude/test_black_box.py` 5
- `todo.md` 5
- `README.md` 4
- `.gitignore` 3
- `CHANGELOG.md` 2
- `bin/loop_claude/routing.py` 2
- `bin/markdown_text.py` 2
- `bin/todo_table.py` 2
+7 -3
View File
@@ -45,7 +45,7 @@ A comment is prose, not code. Passes run in this order: line-level polish on cod
Give a reviewer without Bash the branch diff and `git ls-files`'s output in its prompt: some harnesses grant it Read alone.
`~/.claude/bin/larv-review` holds this section's mechanical rules: the round count, the paired ruling's base and vote count, the scoring run's trigger, and the history's format and latest places. Run it in the chunk's worktree; `~/.claude/bin/larv-review --help` states each rule. On exit 1, repair what its message names and run it again; when you cannot, ask the user.
`~/.claude/bin/larv-review` holds this section's mechanical rules: the round count, the paired ruling's base and vote count, the scoring run's trigger, and the history's format and latest places, and its count of how often seats agree. Run it in the chunk's worktree; `~/.claude/bin/larv-review --help` states each rule. On exit 1, repair what its message names and run it again; when you cannot, ask the user.
**Precedent, then ask** — for the architecture pass and the full scan:
@@ -112,11 +112,15 @@ The comprehension check records how hard the project is to read, and blocks no m
- **Head.** The head holds `origin/<default branch>`, merged in at §4 step 1, so it differs from the default branch only by this branch's change.
- **Seats.** Mid A and the inherited architect. Write what they read and their replies into a votes file in your scratchpad, in the shape `~/.claude/bin/larv-review --help` shows, and run `~/.claude/bin/larv-review record-ruling <votes file>`. On `Split:`, launch the maintainability senior and Junior A together, add their lines, and run it again.
**Scoring run**, when `~/.claude/bin/larv-review needs-scoring-run <the IDs of the items the chunk ships, if any>` prints `Scoring run due:`. Run `needs-scoring-run` before removing the chunk's items from `todo.md`. Junior A, Mid A, the maintainability senior and the inherited architect each score the project at the head from 1 to 10 overall and on four dimensions: Navigation, Locality, Shape and Self-sufficiency. Write their lines into a scores file and run `~/.claude/bin/larv-review record-scoring <scores file> <the same item IDs>`, which appends the run to the history and prints the four seats' means.
**Scoring run**, when `~/.claude/bin/larv-review needs-scoring-run <the IDs of the items the chunk ships, if any>` prints `Scoring run due:`. Run `needs-scoring-run` before removing the chunk's items from `todo.md`.
- **Seats.** Junior A, Mid A, the maintainability senior and the inherited architect each score the project at the head from 1 to 10 overall and on four dimensions: Navigation, Locality, Shape and Self-sufficiency. Write their lines into a scores file and run `~/.claude/bin/larv-review record-scoring <scores file> <the same item IDs>`, which appends the run to the history and prints the four seats' means.
- **Second run.** Where `needs-scoring-run` printed that the chunk `leaves release … with no plan item`, launch the same four seats afresh on the same head. Their scores file's first line is `<PR or branch> rerun at <commit>`. Record it with the same `record-scoring` command.
- **Agreement.** After the second run, run `~/.claude/bin/larv-review agreement` and put its counts in the summary. On `Assumption fails:`, ask the user whether to keep the paired ruling or change the decision `review: the comprehension check records …`. Record the answer as a decision in the repo whose history was counted, with the counts that led to the ask. A later run asks again only when the rulings' rate falls, or the runs' rate rises, against the recorded counts.
**Record.** `docs/comprehension-history.md` keeps every paired ruling and scoring run, which `larv-review` appends; never edit one. Commit what it appends in one commit titled `Record the comprehension check: …`, push it, and wait for the forge's checks (§4 step 4), not for an auto-review; it re-enters no pass. Resolve a merge conflict in the file by putting the default branch's entries first. Put each ruling with its base and votes, and each run's five scores, in the summary.
**Places.** Give systems-architect and stability-reviewer what `~/.claude/bin/larv-review places` prints: the places seats found hard since the newest scoring run. Ask each to report, as a 🟡, each of those places the change touches without making it easier to read, matching a place by its path and the function or section it names, since its line may have moved. Fix such a 🟡, or say in the PR summary why not; that reason settles it in place of a decline's record.
**Places.** Give systems-architect and stability-reviewer what `~/.claude/bin/larv-review places` prints: the places seats found hard since the newest scored commit. Ask each to report, as a 🟡, each of those places the change touches without making it easier to read, matching a place by its path and the function or section it names, since its line may have moved. Fix such a 🟡, or say in the PR summary why not; that reason settles it in place of a decline's record.
**Suspended.** A decision in the project's `docs/decisions.md`, held before the chunk began, may name plan items to ship without a comprehension check. A named item counts as `todo.md` worded it before the chunk began. The decision must name the later plan item whose chunk ends the suspension with a comprehension check. A chunk may ship named items only with each other. Such a chunk skips the comprehension check, and its summary cites the decision. A chunk due a scoring run runs it, even when every item it ships is named.
-5
View File
@@ -12,7 +12,6 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 84 | 0.2.0 | | | **Check, at each release's scoring run, that seats in a paired ruling agree more often than two scoring runs on near-identical code do, as the paired-ruling decision assumes.** | 2 | 2 | 2 | 4 | 4 | 1.4 by 5 | 2.3 |
| 168 | 0.2.0 | | | **Have the human answer an interrupt on loop-claude's terminal and resume the worker headless with the answer, opening the interactive session only when they ask for it.** | 4 | 2 | 3 | 7 | 5 | 1.5 by 8 | 1.8 |
| 73 | 0.2.0 | | | **Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.** | 3 | 2 | 3 | 6 | 5 | 1.4 by 4 | 1.1 |
| 159 | 0.2.0 | | decision | **Run each review pass on the lowest rung that a replay of its archived rounds shows finding every 🔴 and 🟡 the project's rung finds, even below the worker's rung.** | 4 | 2 | 3 | 7 | 6 | 1.4 by 7 | 1.1 |
@@ -84,10 +83,6 @@
## Details
### 84. Check, at each release's scoring run, that seats in a paired ruling agree more often than two scoring runs on near-identical code do, as the paired-ruling decision assumes.
Raised by the principles audit of PR #228: the `review: the comprehension check records …` entry in `docs/decisions.md` is valid only while that holds, and `docs/comprehension-history.md` keeps every vote, yet nothing reads the agreement rate. Scoring runs on near-identical code are rare now, so compare with those already in the history, such as PR #224's and PR #225's, or make a second run for the check.
### 168. Have the human answer an interrupt on loop-claude's terminal and resume the worker headless with the answer, opening the interactive session only when they ask for it.
Today an interrupt runs `claude --resume` on the terminal and waits for it to exit (`bin/loop_claude/cycle/interrupt.py`). Once the human answers, the worker carries on in that interactive session, so the loop waits until the human exits it, often for the rest of the chunk. Instead, the loop prints the question and reads the answer on its own terminal. The next cycle resumes the worker headless with a prompt like `prompts.answered`'s, saying the human gave the answer. A word typed in place of an answer, such as `/talk`, opens today's interactive session, for when the human wants to read the diff or discuss first. An empty answer stops the loop, as an interrupt with nothing typed does today. A release ready to cut still opens the interactive session. Bench the word, and the line that asks for the answer, as `CLAUDE.md` says.