Move item 309 to 0.13.0, since agreement has 6 of the 10 commits it needs scored twice #579

Merged
lilleman merged 3 commits from item-309 into main 2026-11-17 07:39:37 +01:00
+21 -19
View File
@@ -14,7 +14,6 @@
|---|---|---|---|---|---|---|---|---|---|---|---|
| 387 | 0.12.0 | | | **Bind a project's rung to its ladder in one value, so the routing functions and `Cycle` take one argument for both.** | 3 | 1 | 3 | 5 | 5 | code health by 3 | 2.9 |
| 338 | 0.12.0 | | owner | **Script each step that 0.9.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.3 |
| 309 | 0.12.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.2 |
| 299 | 0.12.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
| 311 | 0.12.0 | | | **Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.0 |
| 316 | 0.12.0 | | | **Launch comprehension seats as an agent type of their own under `agents/`, without `CLAUDE.md` and the skill and agent listings a general-purpose subagent carries.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
@@ -26,6 +25,7 @@
| 327 | 0.12.0 | | | **Have `larv-review` print a round commit's title prefix, counted from the round's report, and have rule-hook refuse a round title whose counts differ from it.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 1, 1.4 by 1 | 0.6 |
| 365 | 0.13.0 | | | **Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
| 368 | 0.13.0 | | | **Call a release review due, call a release ready to cut, and tell whether a chunk owes a scoring run, for a plan outside the checkout, from the project repo's history or from Jira.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
| 309 | 0.13.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.2 |
| 366 | 0.13.0 | | | **Count the chunks of a project whose plan sits outside the checkout in the routing figures, from the project repo's history or from Jira.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.4 by 2 | 1.1 |
| 395 | 0.13.0 | | decision | **Keep each block a save didn't change, with its localId, and the spacing paragraphs beside it as Jira or Confluence holds them.** | 3 | 2 | 2 | 5 | 5 | 1.6 by 4 | 0.8 |
| 367 | 0.13.0 | | | **Check that every write to a Jira plan leaves it ranked, as rule-hook's `todo-rank-check` checks `todo.md` today.** | 2 | 2 | 3 | 5 | 3 | 1.6 by 2 | 0.7 |
@@ -97,24 +97,6 @@ The 0.8.0 audit filed four steps, items 334, 335, 336 and 337, and found most ot
The chunk that ships this item files a copy of this audit in release 0.15.0, the release after this item's own, to audit releases 0.10.0 to 0.14.0, rewording "0.9.0" to them, and marks the copy `owner`, so the audit recurs each release until lilleman drops it.
### 309. Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.
Seats cost $1,143 of the $9,154 measured from 2026-09-30 to 2026-11-11 (12.5%), scoring runs $470 of that. A scoring run today is four seats, run twice on one head: eight seat runs. Two seats on the second pass make it six, and two on both passes four.
Why it should hold quality: the seats are one model with different personas, so their errors are correlated, and a correlated extra seat adds little. `larv-review agreement` showed on 2026-11-11 that Mid A and the inherited architect voted alike in 58 of 75 paired rulings, while their scores moved alike between two runs on one commit in only 1 of 4 pairs: seats agree with each other more than a run agrees with its rerun, so the rerun carries more new information than the third and fourth seat. Nine judges from seven model families carried about two independent votes (arXiv 2605.29800), and ten with error correlation 0.21 carried 3.5 (arXiv 2609.22512).
Decide once `agreement` has its 10 commits scored twice. Keep four seats on any pass that files a floor item, since its places come from seat reports. Revise the decision `review: the comprehension check records how hard a project is to read…`, which names the four-seat run.
Touches:
- `bin/larv_review/seats.py`: `SCORING_SEATS`, `scores`
- `bin/larv_review/history_commands.py`: `record_scoring`
- `bin/larv_review/scoring_setup.py`: `SEATS`, `prepare_scoring`
- `bin/larv_review/main.py`: `(module)` (small)
- `bin/larv_review/test_larv_review.py`
- `bin/larv_review/test_scoring_setup.py`
- `skills/larv-review/SKILL.md`: `### Comprehension check`
- `skills/comprehension-panel/SKILL.md`: `## The scoring run` (small)
### 299. Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.
The judge's LLM still decides three things: the verdict kind, the escalation it quotes, and whether a release is ready to cut (`docs/decisions.md` → `routing: loop-claude reads the items a chunk takes …`). This item replays the verdict kind. No single rule found so far separates interrupts: a "?" in the worker's final message appears in 48 interrupts and misses 66. A smaller LLM already failed the replay (`docs/decisions.md` → `routing: on a model judge-replay covered, …`: Sonnet returned another kind on 3 interrupts, Haiku missed 7). Untested are combined code rules, a decision table, and a small classifier trained and run on this machine on the archived verdicts (372 finished, 234 interrupt, 42 unfinished, 20 quoted). Run each offline, nothing sent to an outside service (goal 4), and count disagreements by kind; the bar is zero missed interrupts. A hosted decision model is out of scope, since it would send transcripts to an outside service.
@@ -261,6 +243,26 @@ Read the plan's earlier texts from the project repo's commits, which item 345 wr
The comprehension seats found `next_release` hard to read: the tag cases, dated headings, renames and `named_releases` meet in one function. Make it easier to read while there.
### 309. Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.
Seats cost $1,143 of the $9,154 measured from 2026-09-30 to 2026-11-11 (12.5%), scoring runs $470 of that. A scoring run today is four seats, run twice on one head: eight seat runs. Two seats on the second pass make it six, and two on both passes four.
Why it should hold quality: the seats are one model with different personas, so their errors are correlated, and a correlated extra seat adds little. `larv-review agreement` showed on 2026-11-11 that Mid A and the inherited architect voted alike in 58 of 75 paired rulings, while their scores moved alike between two runs on one commit in only 1 of 4 pairs: seats agree with each other more than a run agrees with its rerun, so the rerun carries more new information than the third and fourth seat. Nine judges from seven model families carried about two independent votes (arXiv 2605.29800), and ten with error correlation 0.21 carried 3.5 (arXiv 2609.22512).
Decide once `larv-review agreement` prints `Assumption holds:` or `Assumption fails:`; it needs 10 commits each scored twice before it can. Keep four seats on any pass that files a floor item, since its places come from seat reports. Revise the decision `review: the comprehension check records how hard a project is to read…`, which names the four-seat run.
lilleman moved it from 0.12.0 to 0.13.0 on 2026-11-17, when 6 commits had been scored twice.
Touches:
- `bin/larv_review/seats.py`: `SCORING_SEATS`, `scores`
- `bin/larv_review/history_commands.py`: `record_scoring`
- `bin/larv_review/scoring_setup.py`: `SEATS`, `prepare_scoring`
- `bin/larv_review/main.py`: `(module)` (small)
- `bin/larv_review/test_larv_review.py`
- `bin/larv_review/test_scoring_setup.py`
- `skills/larv-review/SKILL.md`: `### Comprehension check`
- `skills/comprehension-panel/SKILL.md`: `## The scoring run` (small)
### 366. Count the chunks of a project whose plan sits outside the checkout in the routing figures, from the project repo's history or from Jira.
One of 0.13.0's history ports, per lilleman's call of 2026-11-12.