Merge pull request 'Bench questions a script answers from the code against the panel's hardest places, and keep only the result' (#230) from item-78 into main

This commit was merged in pull request #230.
This commit is contained in:
2026-10-09 00:48:47 +02:00
+15 -7
View File
@@ -24,7 +24,6 @@
| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |
|---|---|---|---|---|---|---|---|---|---|
| 78 | 0.1.0 | | **Generate comprehension questions with deterministic scripts — from symbols, the call graph and the tests, each answer checkable by a tool — and bench the scores an agent gets on them against the panel, reading cost and static metrics.** | 3 | 7 | 9 | 8 | 1.1, 1.4 | 29.1 |
| 85 | 0.1.0 | decision | **Move larv-review's mechanical rules — the paired ruling's base and vote count, each pass's round count, the scoring run's trigger and the comprehension history's format — into one tested tool under `bin/`, so the skill says only when to run it and what each answer means.** | 4 | 7 | 8 | 9 | 1.1 | 28.1 |
| 1 | 0.1.0 | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 3 | 5 | 8 | 9 | 3, 4 | 27.0 |
| 62 | 0.1.0 | defect | **Lift Locality, Self-sufficiency and the overall to the comprehension floor, at the places the seats name.** | 4 | 6 | 6 | 8 | 1.1 | 22.3 |
@@ -54,13 +53,9 @@
## Details
### 78. Generate comprehension questions with deterministic scripts — from symbols, the call graph and the tests, each answer checkable by a tool — and bench the scores an agent gets on them against the panel, reading cost and static metrics.
A hand-written question bank goes stale as the project grows and leaves everything it does not ask about unmeasured. A script that samples symbols at random each run ("what calls X?", "where is Y validated?", "which test fails if Z changes?") covers the whole project over time, and the call graph, grep and the tests hold the answers, so no judge is needed. Each answer names the code it rests on, so a question whose code changed is regenerated. Keep the method only where the bench shows it tracks the panel at a lower cost.
### 85. Move larv-review's mechanical rules — the paired ruling's base and vote count, each pass's round count, the scoring run's trigger and the comprehension history's format — into one tested tool under `bin/`, so the skill says only when to run it and what each answer means.
2026-10-08, lilleman: next after item 78, which lowers token use. Every comprehension run since PR #224 named `skills/larv-review/SKILL.md` §3 "Architecture" the hardest place, and PR #228's paired ruling ruled worse on rules a program should hold: the base's `git log` search and fallbacks, the vote arithmetic, a trigger written out three times, and the round count PR #225's seats said only prose enforces. PR #228's reviewers found about twenty edge cases in that text, among them squash against merge commits, entry order, a vote with no place, and a trigger judged before items are removed; tests can hold each, prose could not. Keep the skill's judgement calls in prose, and give the tool the history's format, so the base search does not hang on the wording of a line. The history's reader in `bin/metric_bench/places.py` moves into that tool, so the format has one reader.
2026-10-08, lilleman: Every comprehension run since PR #224 named `skills/larv-review/SKILL.md` §3 "Architecture" the hardest place, and PR #228's paired ruling ruled worse on rules a program should hold: the base's `git log` search and fallbacks, the vote arithmetic, a trigger written out three times, and the round count PR #225's seats said only prose enforces. PR #228's reviewers found about twenty edge cases in that text, among them squash against merge commits, entry order, a vote with no place, and a trigger judged before items are removed; tests can hold each, prose could not. Keep the skill's judgement calls in prose, and give the tool the history's format, so the base search does not hang on the wording of a line. The history's reader in `bin/metric_bench/places.py` moves into that tool, so the format has one reader.
### 1. Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.
@@ -144,7 +139,7 @@ Goal 3, raised by the principles audit of PR #215: the judge runs with `--no-ses
2026-10-08, lilleman: rescored from A10/G8 to A4/G4 on the measurement below, so other items that lower token use go first.
Measured over the 68 transcripts in loop-claude's archive on 2026-10-08, matching each Agent call to its tool result and to any `<task-notification>` naming its tool-use id. No report reached the worker twice: 813 arrived once as the hand-back, and 363 once as a notification after Claude Code ran the agent in the background. Reports took a median 19k tokens per chunk, 8% of the worker's peak context (24% at most). Lines carrying a severity mark are 9–24% of a reviewer's report, and the worker would still open the file for each 🔴 or 🟡 it fixes. A third of the report volume is the comprehension seats', whose votes and places the worker copies into the history. File reads fill more of the worker's context: `sed -n` output is 30% of its Bash results and `grep` output 11%, which item 78 and `reading-cost` target.
Measured over the 68 transcripts in loop-claude's archive on 2026-10-08, matching each Agent call to its tool result and to any `<task-notification>` naming its tool-use id. No report reached the worker twice: 813 arrived once as the hand-back, and 363 once as a notification after Claude Code ran the agent in the background. Reports took a median 19k tokens per chunk, 8% of the worker's peak context (24% at most). Lines carrying a severity mark are 9–24% of a reviewer's report, and the worker would still open the file for each 🔴 or 🟡 it fixes. A third of the report volume is the comprehension seats', whose votes and places the worker copies into the history. File reads fill more of the worker's context: `sed -n` output is 30% of its Bash results and `grep` output 11%, which `reading-cost` targets.
The reports would live in the session's scratchpad, such as `<scratchpad>/larv-review/prose-round-1.md`, so no project's checkout sees them. A later round gets the path of that pass's earlier report in place of the findings pasted in. How a report reaches the file:
@@ -193,3 +188,16 @@ On 2026-10-08 only this repo's history named places: five scoring runs, all that
| Change spread | 0.67 | 0.18 |
AUC is 0.5 for a coin toss, and top k scores 0.10 by chance. Shared state cannot be judged here: every unit in `bin/` scores 0 on it. Every run's first place was a section of `skills/larv-review/SKILL.md`, which no code metric sees.
On 2026-10-09 a question bench asked a headless claude, allowed only Read, Grep and Glob, three questions a script answers from the syntax tree: which units call a unit, which units it calls, and which test files refer to it. It asked them about every unit the history names whose name is unique, plus as many units the history does not name. Five of the history's six scoring runs named such a unit. The tool is in commit 083cc78; it was deleted after the run.
Claude's misses ranked the named units at a coin toss. Top k scores 0.50 by chance on these units:
| Metric | AUC | Top k |
|---|---|---|
| Question misses | 0.49 | 0.50 |
| Cognitive complexity | 0.66 | 0.59 |
| Length | 0.75 | 0.69 |
| Change spread | 0.57 | 0.54 |
Claude's answers overlapped the truth by a median Jaccard index of 0.76. A run took in 62,643 new input tokens and cost $0.70; it did not record which model answered. Grep answers these questions whatever the code's shape, so no question enters review; a new question bench would need questions that take understanding the code, not finding it.