Answer prose round 1: allow a bare :line in the history's format, date the shortened paths, say slots for top k, list the notes in --help, say no run could be scored, plainer item 85, 73 and 77 text, and the tools' path line

This commit is contained in:
2026-10-08 23:11:36 +02:00
parent a69cfd2624
commit 5cc37dd861
7 changed files with 25 additions and 22 deletions
+1 -1
View File
@@ -65,7 +65,7 @@ Rename `## Unreleased` in `CHANGELOG.md` to the version, commit, tag that commit
## Tools
Call each by its path, such as `~/.claude/bin/metric-bench`, from the project's checkout; Install puts only `loop-claude` on `PATH`.
Install puts only `loop-claude` on `PATH`; call the others by their path, such as `~/.claude/bin/metric-bench`, from the project's checkout.
- `loop-claude <task>` runs a project's next chunk cycle after cycle; `loop-claude --help` states its contract.
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
+3 -3
View File
@@ -69,14 +69,14 @@ def auc(named: list[int], rest: list[int]) -> float:
def top_k(named: list[int], rest: list[int]) -> float:
"""The share of the metric's k highest units the seats named, k being how many they named. Units tied at the cut share its remaining seats evenly, as a random order of them would."""
"""The share of the metric's k highest units the seats named, k being how many they named. Units tied at the cut share its remaining slots evenly, as a random order of them would."""
k = len(named)
cut = sorted(named + rest, reverse=True)[k - 1]
above = sum(value > cut for value in named)
seats = k - above - sum(value > cut for value in rest)
slots = k - above - sum(value > cut for value in rest)
tied_named = sum(value == cut for value in named)
tied = tied_named + sum(value == cut for value in rest)
return (above + seats * tied_named / tied) / k
return (above + slots * tied_named / tied) / k
def medians(runs: list[RunScores]) -> dict[str, Score]:
+9 -8
View File
@@ -3,7 +3,7 @@
usage: metric-bench [history]
history defaults to docs/comprehension-history.md in the checkout it runs in, and the bench reads
the checkout that holds the history. For each scoring run in it, at the commit the seats read, it
the checkout that holds the history. For each scoring run in it, at the commit the panel read, it
measures each module-level function and each method of a module-level class in the Python files git
tracks, leaving out test_*.py, *_test.py and anything under a tests/ folder:
@@ -14,19 +14,20 @@ tracks, leaving out test_*.py, *_test.py and anything under a tests/ folder:
- change spread: how many other files changed in a commit with the unit's file, up to that commit,
leaving out commits of more than 30 files.
It then prints, for each metric, how well it ranks the units the seats named above the rest: AUC
It then prints, for each metric, how well it ranks the units the panel named above the rest: AUC
(0.5 is a coin toss, 1.0 a perfect ranking) and top k (the share of the metric's k highest units
the seats named, k being how many they named; Chance is what top k scores at random). A metric that
the panel named, k being how many it named; Chance is what top k scores at random). A metric that
gave every unit one value reads "constant". A last row gives the medians. Unmeasured counts the
distinct places that name no Python function or method, such as a section of a markdown file, or no
single tracked file.
Below the table it names each run it skipped, its commit not in the checkout, each run it could not
score, its places naming no measured unit or every one, and each file that does not parse on this
Python, left out of its run.
Below the table it names:
- each run it skipped because its commit is not in the checkout;
- each run it could not score because its places name no measured unit, or every one;
- each file left out of its run because it does not parse on this Python.
Exit 1, saying why, on unknown arguments, or when git cannot start or finds no checkout, the history
cannot be read, or no scoring run names a measured unit.
cannot be read, or no scoring run could be scored.
"""
import pathlib
import statistics
@@ -55,7 +56,7 @@ def main(argv: list[str]) -> int:
return failed(benched)
rows, notes = benched
if not rows:
return failed(Failure(f"no scoring run in {history} names a Python function or method the bench measures; its format: ~/.claude/skills/larv-review/SKILL.md → \"History\"{noted(notes)}"))
return failed(Failure(f"no scoring run in {history} could be scored; its format: ~/.claude/skills/larv-review/SKILL.md → \"History\"{noted(notes)}"))
print(table(rows) + noted(notes))
return 0
+1 -1
View File
@@ -58,7 +58,7 @@ def bullet_places(bullet: str) -> list[Place]:
def resolved(places: list[Place], tracked: list[str]) -> tuple[list[Place], list[Place]]:
"""Seats shorten a path once they have named its folder: `cycle/run.py` after `bin/loop_claude/cycle/states.py`. Each place resolves to the one tracked file it names, or, where several match, the one in the folder of the place resolved before it; a place naming no file, or several, is dropped. Returns the resolved places and the dropped ones."""
"""Entries written before paths were written from the repository's root shorten a path once they have named its folder: `cycle/run.py` after `bin/loop_claude/cycle/states.py`. Each place resolves to the one tracked file it names, or, where several match, the one in the folder of the place resolved before it; a place naming no file, or several, is dropped. Returns the resolved places and the dropped ones."""
found = []
dropped = []
before = ""
+2 -2
View File
@@ -82,9 +82,9 @@ def cognitive(function: ast.FunctionDef | ast.AsyncFunctionDef) -> int:
return inside(function.body, 0, function.name)
# One case per node type, as the white paper's table has one row per structure: split, the rules would scatter.
# One case per node type, as the white paper gives one rule per structure.
def score(node: ast.AST, nesting: int, name: str) -> int:
"""The cognitive complexity node adds at depth nesting, its children included; name is the function's own, a call to it counting as recursion."""
"""The cognitive complexity `node` adds, its children included, at nesting depth `nesting`; a call to `name`, the function's own, counts as recursion."""
if isinstance(node, ast.If):
return 1 + nesting + branches(node, nesting, name)
if isinstance(node, ast.For | ast.AsyncFor | ast.While):
+1 -1
View File
@@ -108,7 +108,7 @@ Fix a worse ruling at the places the seats voting worse name, and the floor at t
Two files of the project keep the records:
- **History.** `docs/comprehension-history.md` keeps every paired ruling and scoring run, appended and never edited. Each opens with a `## <UTC time, ISO 8601>, <PR or branch> at <commit>` heading naming when the seats finished and the commit they read; a paired ruling's heading ends `, against <base>`. A paired ruling holds a `Ruling: <better, same or worse>` line, a table of each seat's vote, and under each seat, labelled `<seat>, decided by:`, its places. A scoring run holds a table of each seat's five scores, with a last row holding the four seats' mean of each, and under each seat, labelled `<seat>, hardest first:` or, for the architect, `<seat>, would restructure first:`, its places. A place is `path:line`, the path written from the repository's root, and the unit's name. `bin/metric_bench/places.py` reads this format: change both together. Appending re-enters no pass. Resolve a merge conflict in the file by putting the default branch's entries first.
- **History.** `docs/comprehension-history.md` keeps every paired ruling and scoring run, appended and never edited. Each opens with a `## <UTC time, ISO 8601>, <PR or branch> at <commit>` heading naming when the seats finished and the commit they read; a paired ruling's heading ends `, against <base>`. A paired ruling holds a `Ruling: <better, same or worse>` line, a table of each seat's vote, and under each seat, labelled `<seat>, decided by:`, its places. A scoring run holds a table of each seat's five scores, with a last row holding the four seats' mean of each, and under each seat, labelled `<seat>, hardest first:` or, for the architect, `<seat>, would restructure first:`, its places. A place is `path:line` and the unit's name, the path written from the repository's root. A bare `:line` is in the path named before it in the same bullet. `bin/metric_bench/places.py` reads this format: change both together. Appending re-enters no pass. Resolve a merge conflict in the file by putting the default branch's entries first.
- **Entry.** `docs/decisions.md` keeps one entry whose title opens `comprehension floor:`. It holds each decision to ship under the floor, with its scores.
A `comprehension floor:` entry written before the history may still hold a scoring run's five scores. The next scoring run moves them, and only them, into the history as its first run: a heading of the date or time the entry gives and whatever it names the run by, its commit included where named, and a table with the mean alone. The rest of the entry stays.
+8 -6
View File
@@ -50,7 +50,7 @@
| 81 | 0.1.0 | principle | **Have `workflow-bench/bench.py` return each run's failure as a value, so one failed clone does not hide the later runs' results.** | 2 | 2 | 3 | 4 | 4 | 9.9 |
| 4 | 0.1.0 | | **Name each chunk's archive folder by the chunk's start time, so `<key>/` lists chunks in the order they ran.** | 3 | 2 | 4 | 3 | 3 | 9.6 |
| 52 | 0.1.0 | principle | **Record why this checkout lives in Claude Code's own directory, or move it out and link it in.** | 6 | 6 | 3 | 5 | 2 | 7.5 |
| 77 | 0.2.0 | | **Rerun `bin/metric-bench` once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 2 | 4 | 6 | 7 | 1.1, 1.4 | 22.9 |
| 77 | 0.2.0 | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 3 | 6 | 6 | 7 | 1.1, 1.4 | 21.4 |
## Details
@@ -60,7 +60,7 @@ A hand-written question bank goes stale as the project grows and leaves everythi
### 85. Move larv-review's mechanical rules — the paired ruling's base and vote count, each pass's round count, the scoring run's trigger and the comprehension history's format — into one tested tool under `bin/`, so the skill says only when to run it and what each answer means.
2026-10-08, lilleman: next after the items that lower token use (73, 77, 78). Later that day lilleman rescored 73 below this item, and 77 moved to 0.2.0 to wait for data, so only 78 goes first. Every comprehension run since PR #224 named `skills/larv-review/SKILL.md` §3 "Architecture" the hardest place, and PR #228's paired ruling ruled worse on rules a program should hold: the base's `git log` search and fallbacks, the vote arithmetic, a trigger written out three times, and the round count PR #225's seats said only prose enforces. PR #228's reviewers found about twenty edge cases in that text, among them squash against merge commits, entry order, a vote with no place, and a trigger judged before items are removed; tests can hold each, prose could not. Keep the skill's judgement calls in prose, and give the tool the history's format, so the base search does not hang on the wording of a line. The history's reader in `bin/metric_bench/places.py` moves into that tool, so the format has one reader.
2026-10-08, lilleman: next after item 78, which lowers token use. Every comprehension run since PR #224 named `skills/larv-review/SKILL.md` §3 "Architecture" the hardest place, and PR #228's paired ruling ruled worse on rules a program should hold: the base's `git log` search and fallbacks, the vote arithmetic, a trigger written out three times, and the round count PR #225's seats said only prose enforces. PR #228's reviewers found about twenty edge cases in that text, among them squash against merge commits, entry order, a vote with no place, and a trigger judged before items are removed; tests can hold each, prose could not. Keep the skill's judgement calls in prose, and give the tool the history's format, so the base search does not hang on the wording of a line. The history's reader in `bin/metric_bench/places.py` moves into that tool, so the format has one reader.
### 1. Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.
@@ -148,7 +148,7 @@ Measured over the 68 transcripts in loop-claude's archive on 2026-10-08, matchin
The reports would live in the session's scratchpad, such as `<scratchpad>/larv-review/prose-round-1.md`, so no project's checkout sees them. A later round gets the path of that pass's earlier report in place of the findings pasted in. How a report reaches the file:
- **Write access.** Grant each reviewer Write. No reviewer under `agents/` holds it today, an agent's tool list cannot confine Write to one folder, and `~/.claude/CLAUDE.md` → "Working with the codebase" gives a reviewer that only reads no edits. Check whether a PreToolUse hook can tell a subagent's call from the worker's and deny a Write outside the scratchpad.
- **Write access.** Grant each reviewer Write. No reviewer under `agents/` holds it today, an agent's tool list cannot confine Write to one folder, and `~/.claude/CLAUDE.md` → "Working with the codebase" denies edits to a reviewer that only reads. Check whether a PreToolUse hook can tell a subagent's call from the worker's and deny a Write outside the scratchpad.
- **A wrapper.** A general-purpose subagent launches the reviewer, saves the report and returns only the summary lines. Claude Code's documentation says a subagent cannot start another; drop this route unless a test shows one can.
This risks goal 1.1: a worker fixing from a one-line summary can miss context it needs.
@@ -177,11 +177,13 @@ Today the folder is the SHA-256 of the transcript's first timestamped line, whic
"One owner per value" and "Prefer the vendor-neutral option", in `principles/technical-principles.md`: git, Claude Code and loop-claude all write `~/.claude`, kept apart only by the `.gitignore` allowlist, and the repo's records live in a vendor directory.
### 77. Rerun `bin/metric-bench` once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.
### 77. Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.
2026-10-08, lilleman: no metric enters review until a rerun on more data shows it tracking the panel.
`bin/metric-bench` measures each Python function and method at each scoring run's commit, and scores how well each metric ranks the units the seats named above the rest. On 2026-10-08 only this repo's history named places: five scoring runs, all that day. The medians:
Before the rerun, set the AUC and top-k lift over chance that count as tracking, from a published threshold, so the rerun ends in a decision. The rerun needs places from a project in Go or TypeScript, the languages most other projects use, so the bench first measures those languages, run through each project's own tooling. Function and file length stay warnings to look into, never limits: a plain long function can be simple, and a hard one may need five-line pieces.
On 2026-10-08 only this repo's history named places: five scoring runs, all that day. The medians:
| Metric | AUC | Top k |
|---|---|---|
@@ -190,4 +192,4 @@ Today the folder is the SHA-256 of the transcript's first timestamped line, whic
| Shared state | constant | constant |
| Change spread | 0.67 | 0.18 |
AUC is 0.5 for a coin toss, and top k scores 0.10 by chance. Shared state cannot be judged here: every unit in `bin/` scores 0 on it. Before the rerun, set the AUC and top-k lift over chance that count as tracking, from a published threshold, so the rerun ends in a decision. Every run's first place was a section of `skills/larv-review/SKILL.md`, which no code metric sees. The rerun needs places from a project in Go or TypeScript, the languages most other projects use, so the bench first measures those languages, run through each project's own tooling. Function and file length stay warnings to look into, never limits: a plain long function can be simple, and a hard one may need five-line pieces.
AUC is 0.5 for a coin toss, and top k scores 0.10 by chance. Shared state cannot be judged here: every unit in `bin/` scores 0 on it. Every run's first place was a section of `skills/larv-review/SKILL.md`, which no code metric sees.