Answer prose round 1: 0 🔴, 13 🟡 (0 misread); state the decision's figures as the trials ran them, fix item 289's tagline and wording and the scripts' docstrings and comments
CI / tests (pull_request) Successful in 2m3s

This commit was merged in pull request #456.
This commit is contained in:
2026-11-09 22:51:02 +01:00
parent f99537976a
commit b58916ae71
6 changed files with 10 additions and 10 deletions
+1 -1
View File
@@ -230,7 +230,7 @@ Valid while the ladder orders its rungs by competence.
## review: each pass keeps a reviewer of its own
2026-11-09, lilleman asked for the trials. Serves goals 1.1 and 1.4. Four merges were replayed on archived first rounds, beside the separate reviewers run twice to measure the noise between runs (`reviewer-merge-trials/README.md`). Stability, product-owner and prose as one reviewer cost $9.77 against $17, and recovered 15 of 34 prose findings against 22 and 23. The architect and the principles auditor as one cost $15.62 against $29.40, and recovered 5 of 17 principles findings against 10, losing both 🔴. Stability and product-owner as one cost a third less, and recovered 4 and 5 of 16 product-owner findings against 7 and 10. Each time, the merged reviewer did the first brief's work and wrote little for the second. Merging auditors that check the same kind of thing lost no findings: `review: the principles audit runs only in the release review, …` records that trial. Valid until a new model joins the ladder or a reviewer's definition changes: replay the merge then, with the scripts in `reviewer-merge-trials/`.
2026-11-09, lilleman asked for the trials. Serves goals 1.1 and 1.4. Merged reviewers were replayed on archived first rounds, beside the separate reviewers (`reviewer-merge-trials/README.md`). Each merge cost a third to a half less, and in each, one absorbed pass lost findings. Stability, product-owner and prose as one reviewer recovered 15 of 34 prose findings, against 22 and 23 for two separate runs. The architect and the principles auditor as one recovered 5 of 17 principles findings, against 10 for one separate run, and lost both principles 🔴. Stability and product-owner as one, run twice, recovered 4 and 5 of 16 product-owner findings, against 7 and 10 for two separate runs. Folding override-reviewer into stability had no data: no archived override round holds a 🔴 or 🟡. Valid until a new model joins the ladder or a reviewer's definition changes: replay the merge then, with the scripts in `reviewer-merge-trials/`.
## review: each reviewer under `agents/` writes its full report into the chunk's review folder under `~/.local/state/larv-review/` and returns the worker its verdict and one line per finding; rule-hook refuses its Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`
+1 -1
View File
@@ -1,4 +1,4 @@
"""Match each variant's replay reports to the archived findings, and list new findings. Usage: analyze.py <candidate> <variant-rep>..."""
"""Match each variant's replay reports to the archived findings, and list new findings. Usage: analyze.py <candidate> <variant>-<run>..."""
import concurrent.futures, json, pathlib, pickle, re, sys
sys.path.insert(0, str(pathlib.Path.home() / "code/claude-config/bin"))
from claude_code.reply import claude_reply
+2 -2
View File
@@ -1,4 +1,4 @@
"""Run analyze.py's matching and new-finding count, and also sum what those calls cost. Usage: [BASELINE=<candidate>] analyze_metered.py <candidate> <variant-rep>... [case ids...]"""
"""Run analyze.py's matching and new-finding count, and also sum what those calls cost. Usage: [BASELINE=<candidate>] analyze_metered.py <candidate> <variant>-<run>... [case ids...]"""
import pathlib, runpy, sys, threading
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
import analyze, review_replay.main as rm
@@ -11,7 +11,7 @@ def metered(*a, **k):
return r
rm.claude_reply = analyze.claude_reply = metered
import json, os
# BASELINE=wave3: the reports are reused from that candidate, so their new findings are too, kept where they sit under one of this candidate's passes; no second call counts them again.
# BASELINE=wave3: the reports come from candidate 1, so this reuses its new-finding counts for this candidate's passes and makes no call to count them again.
BASE = os.environ.get("BASELINE")
if BASE: analyze.new_findings = lambda arch, text: []
def reused_new(res):
+1 -1
View File
@@ -1,4 +1,4 @@
"""List every archived reviewer run (with or without findings) per chunk, with its round number from the prompt."""
"""Pickle to runs.pkl one flat list of the archived reviewer runs of the projects on claude-opus-5-5, with or without findings, each with its round number from the prompt."""
import json, pathlib, pickle, re, sys
sys.path.insert(0, str(pathlib.Path.home() / "code/claude-config/bin"))
from claude_code.entries import AssistantTurn, parse_entry, Text, UserText
+2 -2
View File
@@ -1,4 +1,4 @@
"""Replay archived review rounds separately and merged; save reports, costs and matches. Usage: driver.py <candidate> <variant> <rep> [case ids...]"""
"""Replay archived review rounds separately or merged, and save each replay's reply, reports and cost; analyze.py matches them. Usage: driver.py <candidate> <sep|merged> <run> [case ids...]"""
import concurrent.futures, json, os, pathlib, pickle, re, shutil, sys, tempfile
sys.path.insert(0, str(pathlib.Path.home() / "code/claude-config/bin"))
from claude_code.reply import claude_reply, Limited
@@ -36,7 +36,7 @@ def prepared(rnd, code, copies, clone=None, cutoff=None):
copy_reports(cutoff or rnd, folder, copies.reports)
p = redirected(rnd.prompt, HOME, folder, code.path, copies)
if isinstance(p, Failure): return p
# Inputs rewritten after the round: ls-files is regenerated from the clone; a places list cannot be, so it says so.
# Inputs rewritten after the round: ls-files is regenerated from the clone; a places list cannot be, so its file says it is missing.
for name in lost_inputs(p, copies.reports, lambda n: (pathlib.Path(copies.reports) / n).exists()):
target = pathlib.Path(copies.reports) / name
if name == "ls-files.txt" and clone:
+3 -3
View File
@@ -14,7 +14,7 @@
|---|---|---|---|---|---|---|---|---|---|---|---|
| 277 | 0.7.0 | | question | **Map the size of the code an item names onto S, on a scale the human picks, and have `score-outcomes` print that S for the item being scored and for each finished item it lists.** | 3 | 2 | 2 | 5 | 4 | 1.1 by 4, 1.4 by 3 | 2.4 |
| 1 | 0.7.0 | | question | **Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.** | 1 | 2 | 1 | 2 | 1 | | 0.0 |
| 289 | 0.7.0 | | defect | **Replay each archived round on the commit with the newest author time by the round's start, in place of the first such commit in git log order.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3, 1.1 by 2 | 2.3 |
| 289 | 0.7.0 | | defect | **Pick each replay's commits by the newest author time, in place of the first such commit in git log order.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3, 1.1 by 2 | 2.3 |
| 281 | 0.7.0 | | | **Remove a merged chunk's worktree and branch and fast-forward the main checkout with one command.** | 3 | 2 | 3 | 6 | 3 | 1.4 by 2, 1.5 by 2 | 1.6 |
| 286 | 0.7.0 | | | **Lower the reading cost of the files a chunk reads most before its first commit.** | 3 | 2 | 2 | 5 | 5 | 1.4 by 4 | 1.0 |
| 287 | 0.8.0 | | | **Move the forge helpers `Forge`, `forge_of`, `runner`, `open_pull` and `gitea` into one module that `chunk_preflight`, `chunk_pr`, `deploy_release` and `larv_review` all import from, so no tool imports another tool's internals for them.** | 3 | 1 | 3 | 5 | 5 | code health by 3 | 3.4 |
@@ -72,9 +72,9 @@ It lives under `~/.claude`, which `.gitignore` keeps from the remote, so a lost
The README line covers the bench's transcripts in `workflow-bench/logs/` too (goal 3).
### 289. Replay each archived round on the commit with the newest author time by the round's start, in place of the first such commit in git log order.
### 289. Pick each replay's commits by the newest author time, in place of the first such commit in git log order.
`authored_by` in `bin/review_replay/rounds.py` takes the first commit in log order authored at or before a time. A commit authored early but merged late, such as a dependency bump, sits high in log order, so it wins over the commits the round actually reviewed. `code_of` in `bin/review_replay/main.py` uses it for both the reviewed commit and the clone's base. Found on 2026-11-09 by the reviewer-merge trials (`reviewer-merge-trials/README.md`): two plainpages rounds would have replayed at a commit older than their own base. Fix as the trials' driver did: take the commit with the newest author time at or before the time, log order breaking ties. Archived replays, and the routing decisions that cite them, may have reviewed the wrong code; once fixed, say in the PR which of those decisions' replays would change.
`authored_by` in `bin/review_replay/rounds.py` takes the first commit in log order authored at or before a time. A commit authored early but merged late, such as a dependency bump, sits high in log order, so it wins over the commits the round actually reviewed. `code_of` in `bin/review_replay/main.py` uses it for both the reviewed commit and the clone's base. Found on 2026-11-09 by the reviewer-merge trials (`reviewer-merge-trials/README.md`): two plainpages rounds would have replayed at a commit older than their own base. Fix as the trials' driver did: take the commit with the newest author time at or before the time, log order breaking ties. Archived replays may have reviewed the wrong code, and so may the replays the routing decisions cite; once fixed, say in the PR which of those decisions' replays would change.
### 281. Remove a merged chunk's worktree and branch and fast-forward the main checkout with one command.