Answer product-owner round 1: 0 🔴, 5 🟡 (0 misread); hedge the verdicts that lack a noise run, and complete the rerun steps: run from the folder, reuse candidate 1's separate reports, change the model in two places, replay the tracked definitions, name the commit; prose's order and term fixes ride along

This commit is contained in:
2026-11-09 22:50:56 +01:00
parent 3bf00dace7
commit f99537976a
+86 -88
View File
@@ -3,40 +3,40 @@
These trials ask one question: can larv-review run fewer, merged reviewers without losing findings?
A merged reviewer is one agent that does the work of two or three separate reviewers in one run.
It costs less, because it reads the change once.
## Verdicts
No merge tested is safe. Each one lost the findings of one pass it absorbed, while the other pass held.
| Candidate | Reviewers merged | Verdict |
|---|---|---|
| 1 | stability + product-owner + prose | Unsafe. Prose findings drop well outside the noise. |
| 2 | systems-architect + principles-auditor | Unsafe. Principles findings halve, and both principles 🔴 are lost. |
| 3 | override-reviewer folded into stability | No data. No archived override round has a 🔴 or 🟡. |
| 4 | stability + product-owner, prose kept separate | Unsafe. Product-owner findings drop outside the noise; stability holds. |
The three-way and architect merges cut cost about in half. The stability and product-owner merge cuts it by about a third.
They ran on 2026-11-09. `docs/decisions.md` → `review: each pass keeps a reviewer of its own` records what was decided from them.
## Terms
- **Pass:** one kind of review, run by one reviewer agent. The passes here are stability, product-owner, prose, architecture and principles.
- **Pass:** one kind of review, run by one reviewer agent: stability, product-owner, prose, architecture or principles.
- **Round:** one run of one pass on one change, archived with its report.
- **Archived finding:** a 🔴 (blocker) or 🟡 (should fix) that an archived round reported.
- **Recovered:** a replay's report raises an archived finding again, as 🔴 or 🟡.
- **New finding:** a 🔴 or 🟡 in a replay's report that matches no archived finding. New findings are counted only; nobody checked whether they are real.
- **Noise:** how much two runs of the same reviewer on the same round differ.
- **Noise:** how much two runs of the same reviewer on the same rounds differ.
## Verdicts
No merge tested is safe. In each, one absorbed pass held and another lost findings.
| Candidate | Reviewers merged | Verdict |
|---|---|---|
| 1 | stability + product-owner + prose | Unsafe. Prose fell to 15 of 34, below both separate runs (22 and 23). The merged reviewer ran once. |
| 2 | systems-architect + principles-auditor | Likely unsafe. Principles fell to 5 of 17, against 10, and both principles 🔴 were lost. No noise run. |
| 3 | override-reviewer folded into stability | No data. No archived override round has a 🔴 or 🟡. |
| 4 | stability + product-owner, prose kept separate | Unsafe. Product-owner fell to 4–5 of 16, against 7–10; the ranges don't overlap. Stability held. Each variant ran twice. |
Candidates 1 and 2 cut cost about in half. Candidate 4 cut it by about a third.
## Method
1. Pick archived round-1 review rounds where every pass under test ran on the same chunk.
2. Replay each round with review-replay's own functions, from `bin/review_replay/` and `bin/rung_replay/` in claude-config.
3. Each replay runs on Opus 5.5 at medium effort.
4. Each replay runs in a throwaway clone of the project at the reviewed commit.
5. Each replay gets a copy of the review folder holding only the reports written before the case's first round. So no variant sees another pass's findings.
6. The separate variant runs each reviewer unchanged, with the worker's archived brief.
7. The merged variant passes one agent through `--agents`. Its definition holds the full text of every reviewer it replaces, under a short preamble. The preamble asks for one report with one heading per pass. Its prompt joins the worker's archived briefs to each pass.
8. review-replay's matcher, on Opus 5.5 at medium, counts which archived findings each report recovers.
9. A second call on the same rung lists the report's new findings.
1. Pick archived first rounds where every pass under test ran on the same chunk.
2. Replay each round with review-replay's own functions, from `bin/review_replay/` and `bin/rung_replay/`.
3. Each replay runs on Opus 5.5 at medium effort, in a throwaway clone of the project at the reviewed commit.
4. Each replay gets a copy of the review folder holding only the reports written before the case's first round, so no variant sees another pass's findings.
5. The separate variant runs each reviewer unchanged, with the worker's archived brief.
6. The merged variant runs one agent, defined for the run through claude's `--agents` flag. Its definition holds the full text of every reviewer it replaces, under a short preamble asking for one report with one heading per pass. Its prompt joins the worker's archived briefs to each pass.
7. review-replay's matcher, on Opus 5.5 at medium, counts which archived findings each report recovers.
8. A second call on the same rung counts the report's new findings.
### The noise lesson
@@ -44,32 +44,28 @@ Two runs of the same separate reviewers on the same rounds differ a lot.
In candidate 1, run 1 recovered 47 of 77 findings and run 2 recovered 52.
Run 1 found 8 that run 2 missed, and run 2 found 13 that run 1 missed.
So one run per variant cannot tell a small loss from chance.
A verdict needs a noise figure: at least two runs of each variant.
Only a loss larger than the spread between runs counts.
A firm verdict needs two runs of each variant, and a loss larger than the spread between them.
Candidate 4 has that. Candidate 1's merged reviewer ran once, but its prose loss is far below both separate runs. Candidate 2 has no noise run at all.
### The reviewed-commit bug
### The reviewed-commit fix
review-replay must find the commit the archived reviewer read.
It took the first commit in `git log` order authored by the round's time.
A commit authored early but merged late, such as a dependency bump, then wins.
Two plainpages rounds would have been replayed at a commit older than their own base.
The fix, in `driver.py` → `code_of`, takes the commit with the newest author time by then, with log order breaking ties.
Every trial here used the fix.
Archived replays made with the unfixed review-replay may have reviewed the wrong code.
Every trial used a fix for a review-replay defect, filed as `todo.md` item 289: `driver.py` → `code_of` takes the commit with the newest author time, log order breaking ties.
Without it, two plainpages rounds would have replayed at a commit older than their own base.
## Samples
Candidates 1 and 4 use the same 6 chunks. The commit is the one each replay reviewed; the base is the default branch as the chunk began.
The reviewed commit is the one each replay read; the base is the default branch as the chunk began.
Candidates 1 and 4 use these 6 chunks.
| Project | Chunk | Reviewed commit | Base |
|---|---|---|---|
| adf-codec | item-45 | ca6016b4b6627159f369f3ce047ebb4d1727a95a | 9dfa14a095222c5b8fd4d4d435e2940720483feb |
| claude-config | item-230 | b37d03056c3277cd4a460c91cbbbaa025e7fd672 | 2a07feebecbb7b73ff692bfad9f6f5519423fb84 |
| plainpages | item-2 | dd39c7a07ad3e497792a645a0c94c4eb496db9e4 | 34dfbea156649fd54ff4ea17221d98d19d0d53df |
| plainpages | item-23 | 80070762e7337fbdd21a59ebce2cb75d98dcd2f6 | 01f368d663e9b66e2ba488de4b75d64559106b63 |
| fejkdata | item-223 | a3967a01956bb67d25a83caf8f9be45426cec837 | f40bdd0dcefc01b821feae72d86e70ad606d4c99 |
| log | item-7 | 08fe3562230affe87ca5aa4661f199831189de91 | 08e8945eaa182fcf06689337d8140ebf91b2a2e7 |
| adf-codec | item-45 | ca6016b4b6627159f369f3ce047ebb4d1727a95a | 9dfa14a095222c5b8fd4d4d435e2940720483feb |
| plainpages | item-2 | dd39c7a07ad3e497792a645a0c94c4eb496db9e4 | 34dfbea156649fd54ff4ea17221d98d19d0d53df |
| plainpages | item-23 | 80070762e7337fbdd21a59ebce2cb75d98dcd2f6 | 01f368d663e9b66e2ba488de4b75d64559106b63 |
Candidate 2 uses round 1 of the 5 release reviews where both the architect and the auditor ran.
@@ -81,43 +77,10 @@ Candidate 2 uses round 1 of the 5 release reviews where both the architect and t
| plainpages-plugin-fastjira | release-review-0.2.0 | 3ffedfee6203a95053f12ef674b0ae69d903555b | 6345fc75fb24562e8a975eccae05a865d33fc003 |
| plainpages-plugin-fastjira | release-review-0.3.0 | 30a4a61b5bde493ce0d76d10dda281b94a14aad2 | 468d89c9186a6b4bad3bbd372e2e9035e47fc7b6 |
## Candidate 4: stability + product-owner, prose kept separate
Candidate 4 merges only the two passes that held up inside candidate 1's merge.
It uses candidate 1's 6 chunks, without their prose rounds.
They hold 43 archived findings: stability 27, product-owner 16. Two are 🔴, one per pass.
The separate figures reuse candidate 1's runs 1 and 2. Nothing was rerun for them.
Their stability and product-owner reports were matched again, against these 43 findings only.
So no prose report can recover a finding here, unlike candidate 1's figures.
The merged pair ran twice, so it has its own noise figure.
| | Separate, run 1 | Separate, run 2 | Merged, run 1 | Merged, run 2 |
|---|---|---|---|---|
| Recovered, both passes (of 43) | 20 | 28 | 20 | 17 |
| Stability (of 27) | 13 | 18 | 15 | 13 |
| **Product-owner (of 16)** | **7** | **10** | **5** | **4** |
| 🔴 recovered (of 2) | 0 | 2 | 2 | 2 |
| 🔴/🟡 written under stability | 19 | 19 | 19 | 16 |
| **🔴/🟡 written under product-owner** | **8** | **9** | **2** | **3** |
| New 🔴/🟡 | 6 | 6 | 4 | 4 |
| Cost | $12.28 | $12.76 | **$8.49** | **$8.04** |
| Tokens | 13.0M | 14.7M | 11.0M | 9.7M |
- Separate runs recover 20 to 28. Merged runs recover 17 to 20.
- Stability stays inside the noise: 13 to 18 separate, 13 to 15 merged.
- Product-owner does not: 7 to 10 separate, 4 to 5 merged. The ranges don't overlap.
- The merged reviewer writes about a third as many product-owner findings.
- The merged runs missed 12 to 14 of separate run 2's recoveries, half of them product-owner.
- Both merged runs recovered both 🔴. Separate run 1 recovered neither.
- The two merged runs differ by 9 and 6 findings each way, about as much as the separate runs.
- The separate runs' new-finding counts are candidate 1's, kept where they sit under stability or product-owner. They were counted against all 77 archived findings, not these 43.
**Verdict: unsafe.** Product-owner findings fall outside the noise in both merged runs. The sample is small, 16 product-owner findings. But the drop has the same shape as in candidates 1 and 2: the merged reviewer does the first brief's job and writes little for the second. The merge saves about a third of the cost, less than the half the larger merges saved.
## Candidate 1: stability + product-owner + prose
The 6 chunks hold 77 archived findings: stability 27, product-owner 16, prose 34. Four are 🔴.
Here each archived finding is matched against all three reports of a run together, so one pass's report can recover another pass's finding.
| | Separate, run 1 | Separate, run 2 | Merged |
|---|---|---|---|
@@ -131,13 +94,11 @@ The 6 chunks hold 77 archived findings: stability 27, product-owner 16, prose 34
| Cost | $17.03 | $17.14 | **$9.77** |
| Tokens | 15.8M | 17.5M | 12.9M |
- The merged reviewer missed 14 of run 1's recoveries, 12 of them prose.
- It missed 16 of run 2's recoveries, 11 of them prose.
- Stability and product-owner stayed inside the noise.
- The merged total, 45, sits just below the separate runs' 47 and 52. The loss is in prose.
- The merged reviewer missed 14 of run 1's recoveries, 12 of them prose, and 16 of run 2's, 11 of them prose.
- Stability and product-owner stayed within the separate runs' range, or above it.
**Verdict: unsafe.** The totals sit inside the noise, but the prose loss is systematic. The merged reviewer wrote about a third as many prose findings.
The recovery figures here match each archived finding against all three reports together. A report of one pass can recover another pass's finding.
**Verdict: unsafe.** The merged reviewer wrote about a third as many prose findings.
## Candidate 2: systems-architect + principles-auditor
@@ -159,7 +120,7 @@ The 5 release reviews hold 64 archived findings: architecture 47, principles 17.
- Two rounds' file lists were regenerated from the clone. Their places lists were not archived.
- There is no noise run, because the budget did not allow one.
**Verdict: unsafe.** The merged reviewer acts like an architect that audits a little. The drop has the same shape and size as candidate 1's prose drop.
**Verdict: likely unsafe.** The merged reviewer acts like an architect that audits a little: principles findings halve.
## Candidate 3: override-reviewer folded into stability
@@ -167,6 +128,38 @@ The archive holds 38 override-reviewer runs. None has a 🔴 or 🟡, so there i
**Verdict: no data.**
## Candidate 4: stability + product-owner, prose kept separate
Candidate 4 merges only the two passes that held up inside candidate 1's merge.
It uses candidate 1's 6 chunks, without their prose rounds.
They hold 43 archived findings: stability 27, product-owner 16. Two are 🔴, one per pass.
The separate figures reuse candidate 1's runs 1 and 2. Nothing was replayed again for them.
Their stability and product-owner reports were matched again, against these 43 findings only.
So no prose report can recover a finding here, and these figures sit below candidate 1's.
The merged pair ran twice, so it has its own noise figure.
| | Separate, run 1 | Separate, run 2 | Merged, run 1 | Merged, run 2 |
|---|---|---|---|---|
| Recovered, both passes (of 43) | 20 | 28 | 20 | 17 |
| Stability (of 27) | 13 | 18 | 15 | 13 |
| **Product-owner (of 16)** | **7** | **10** | **5** | **4** |
| 🔴 recovered (of 2) | 0 | 2 | 2 | 2 |
| 🔴/🟡 written under stability | 19 | 19 | 19 | 16 |
| **🔴/🟡 written under product-owner** | **8** | **9** | **2** | **3** |
| New 🔴/🟡 | 6 | 6 | 4 | 4 |
| Cost | $12.28 | $12.76 | **$8.49** | **$8.04** |
| Tokens | 13.0M | 14.7M | 11.0M | 9.7M |
- Stability stays inside the noise: 13 to 18 separate, 13 to 15 merged.
- Product-owner does not: 7 to 10 separate, 4 to 5 merged.
- The merged reviewer writes about a third as many product-owner findings.
- Merged run 1 recovered 9 findings that merged run 2 missed, and run 2 recovered 6 that run 1 missed. The two separate runs differ by 3 and 11 the same way.
- Both merged runs recovered both 🔴. Separate run 1 recovered neither.
- The separate runs' new findings are candidate 1's counts for their stability and product-owner reports. They were counted against all 77 archived findings, not these 43.
**Verdict: unsafe.** The sample is small, 16 product-owner findings, but both merged runs fall below both separate runs.
## Cost
All costs are at list prices, as Claude Code reports them.
@@ -176,20 +169,25 @@ All costs are at list prices, as Claude Code reports them.
| Candidates 1 and 2, with candidate 1's noise run | about $89 | about $5 |
| Candidate 4, two merged runs | $16.53 | about $6 |
Candidate 4 reused candidate 1's separate replays, so its separate figures cost only their matching.
Each match and new-finding count costs about $0.30 on Opus 5.5 at medium.
## How to rerun
All scripts sit in this folder. They import review-replay from `~/code/claude-config/bin`, and run on the host's Python.
Run every script from this folder, on the host's Python. They import review-replay from `~/code/claude-config/bin`; the trials ran against commit 82eb8ff, and candidate 4 against 0442074, where review-replay was unchanged.
The scripts name the candidates `wave3` (candidate 1), `release` (candidate 2) and `pair` (candidate 4).
1. `collect.py` reads loop-claude's archive and writes every archived reviewer round to `runs.pkl`.
To replay on another model, change two places: `collect.py` keeps only projects on `claude-opus-5-5`, and `driver.py`'s `RUNG` is Opus 5.5 at medium.
The tracked `*-reviewer.md` files are the merged definitions the figures came from. `build_agents.py` and `build_pair.py` write them again from the current `~/.claude/agents/`, beside the `agents-*.json` the driver passes to claude, which wrap the same text. To replay the exact tracked definitions, restore them with `git checkout` after building, and rebuild the JSON from them in the shape `build_agents.py` writes.
1. `collect.py` reads loop-claude's archive and writes the archived reviewer rounds of Opus 5.5 projects to `runs.pkl`.
2. `mkcases1.py` writes `cases-wave3.json`, candidate 1's cases. `cases-release.json` holds candidate 2's.
3. `build_agents.py` writes candidate 1's and 2's merged agents, `agents-wave3.json` and `agents-release.json`, from `~/.claude/agents/`.
4. `build_pair.py` writes candidate 4's merged agent, `agents-pair.json`, from candidate 1's brief texts. It also writes `cases-pair.json`: candidate 1's cases without the prose round.
5. `driver.py <candidate> <sep|merged> <run>` replays one variant. Candidates are `wave3`, `release` and `pair`. `AT_ONCE` sets how many replays run at once, 4 by default. Results land in `results/<candidate>/<variant>-<run>/`.
6. `analyze.py <candidate> <variant>-<run>...` matches each report and counts new findings. `analyze_metered.py` does the same and prints what the matching cost. With `BASELINE=wave3` it reuses that candidate's new-finding counts instead of counting again.
3. `build_agents.py` writes candidate 1's and 2's merged agents. `build_pair.py` writes candidate 4's, and `cases-pair.json`: candidate 1's cases without the prose round.
4. `driver.py <candidate> <sep|merged> <run>` replays one variant. `AT_ONCE` sets how many replays run at once, 4 by default. Results land in `results/<candidate>/<variant>-<run>/`.
5. For candidate 4's separate baseline, copy each `*--stability.json` and `*--product-owner.json` from `results/wave3/sep-<run>/` into `results/pair/sep-<run>/`. `driver.py pair sep <run>` would instead replay them afresh, at about $12 a run.
6. `analyze.py <candidate> <variant>-<run>...` matches each report and counts new findings. `analyze_metered.py` does the same and prints what the matching cost. With `BASELINE=wave3` it reuses candidate 1's new-finding counts.
7. `summarize.py <candidate> <variant>-<run>...` prints the tables' figures. `counts.py` counts the findings each report wrote, per pass.
8. `shas.py <candidate>...` prints each case's reviewed commit and base.
A replay or match that exists on disk is not rerun. Delete its file to run it again.
A replay or match that exists on disk is not run again. Delete its file to run it again.
`runs.pkl`, `results/` and `agents-*.json` stay out of git: they hold transcript and report text from other projects.