Reviewer merge trials
These trials ask one question: can larv-review run fewer, merged reviewers without losing findings?
A merged reviewer is one agent that does the work of two or three separate reviewers in one run.
It costs less, because it reads the change once.
They ran on 2026-11-09. docs/decisions.md → review: each pass keeps a reviewer of its own records what was decided from them.
Terms
- Pass: one kind of review, run by one reviewer agent: stability, product-owner, prose, architecture or principles.
- Round: one run of one pass on one change, archived with its report.
- Archived finding: a 🔴 (blocker) or 🟡 (should fix) that an archived round reported.
- Recovered: a replay's report raises an archived finding again, as 🔴 or 🟡.
- New finding: a 🔴 or 🟡 in a replay's report that matches no archived finding. New findings are counted only; nobody checked whether they are real.
- Noise: how much two runs of the same reviewer on the same rounds differ.
Verdicts
No merge tested is safe. In each, one absorbed pass held and another lost findings.
| Candidate | Reviewers merged | Verdict |
|---|---|---|
| 1 | stability + product-owner + prose | Unsafe. Prose fell to 15 of 34, below both separate runs (22 and 23). The merged reviewer ran once. |
| 2 | systems-architect + principles-auditor | Likely unsafe. Principles fell to 5 of 17, against 10, and both principles 🔴 were lost. No noise run. |
| 3 | override-reviewer folded into stability | No data. No archived override round has a 🔴 or 🟡. |
| 4 | stability + product-owner, prose kept separate | Unsafe. Product-owner fell to 4–5 of 16, against 7–10; the ranges don't overlap. Stability held. Each variant ran twice. |
Candidates 1 and 2 cut cost about in half. Candidate 4 cut it by about a third.
Method
- Pick archived first rounds where every pass under test ran on the same chunk.
- Replay each round with review-replay's own functions, from
bin/review_replay/andbin/rung_replay/. - Each replay runs on Opus 5.5 at medium effort, in a throwaway clone of the project at the reviewed commit.
- Each replay gets a copy of the review folder holding only the reports written before the case's first round, so no variant sees another pass's findings.
- The separate variant runs each reviewer unchanged, with the worker's archived brief.
- The merged variant runs one agent, defined for the run through claude's
--agentsflag. Its definition holds the full text of every reviewer it replaces, under a short preamble asking for one report with one heading per pass. Its prompt joins the worker's archived briefs to each pass. - review-replay's matcher, on Opus 5.5 at medium, counts which archived findings each report recovers.
- A second call on the same rung counts the report's new findings.
The noise lesson
Two runs of the same separate reviewers on the same rounds differ a lot. In candidate 1, run 1 recovered 47 of 77 findings and run 2 recovered 52. Run 1 found 8 that run 2 missed, and run 2 found 13 that run 1 missed. So one run per variant cannot tell a small loss from chance. A firm verdict needs two runs of each variant, and a loss larger than the spread between them. Candidate 4 has that. Candidate 1's merged reviewer ran once, but its prose loss is far below both separate runs. Candidate 2 has no noise run at all.
The reviewed-commit fix
Every trial took the commit with the newest author time, log order breaking ties, as review-replay does from 2026-11-10. Before that it took the first commit in log order authored by then, and a trial's driver patched that in. Without the patch, two plainpages rounds would have replayed at a commit older than their own base.
Samples
The reviewed commit is the one each replay read; the base is the default branch as the chunk began.
Candidates 1 and 4 use these 6 chunks.
| Project | Chunk | Reviewed commit | Base |
|---|---|---|---|
| adf-codec | item-45 | ca6016b4b6627159f369f3ce047ebb4d1727a95a | 9dfa14a095222c5b8fd4d4d435e2940720483feb |
| claude-config | item-230 | b37d03056c |
2a07feebec |
| fejkdata | item-223 | a3967a01956bb67d25a83caf8f9be45426cec837 | f40bdd0dcefc01b821feae72d86e70ad606d4c99 |
| log | item-7 | 08fe3562230affe87ca5aa4661f199831189de91 | 08e8945eaa182fcf06689337d8140ebf91b2a2e7 |
| plainpages | item-2 | dd39c7a07ad3e497792a645a0c94c4eb496db9e4 | 34dfbea156649fd54ff4ea17221d98d19d0d53df |
| plainpages | item-23 | 80070762e7337fbdd21a59ebce2cb75d98dcd2f6 | 01f368d663e9b66e2ba488de4b75d64559106b63 |
Candidate 2 uses round 1 of the 5 release reviews where both the architect and the auditor ran.
| Project | Chunk | Reviewed commit | Base |
|---|---|---|---|
| adf-codec | release-review-0.3.0 | 8e6d7efff16f0df15892916549a7515d25b97886 | 27e8f33f30a46bfa7dc1bc52f6ca85d342ec4d2c |
| fejkdata | release-review-0.3.0 | 5e1c3464c2b8d39d7f989323b287b6a8eda0f02e | 453a020864f3e5aa5aca728a48faac9b553bea9c |
| fejkdata | release-review-0.4.0 | a5dd82ca691d9b4135f1c7fed8e97efb4076459b | 4cd2f6d8693091e6e710db9bf5ef7b58f83b89aa |
| plainpages-plugin-fastjira | release-review-0.2.0 | 3ffedfee6203a95053f12ef674b0ae69d903555b | 6345fc75fb24562e8a975eccae05a865d33fc003 |
| plainpages-plugin-fastjira | release-review-0.3.0 | 30a4a61b5bde493ce0d76d10dda281b94a14aad2 | 468d89c9186a6b4bad3bbd372e2e9035e47fc7b6 |
Candidate 1: stability + product-owner + prose
The 6 chunks hold 77 archived findings: stability 27, product-owner 16, prose 34. Four are 🔴. Here each archived finding is matched against all three reports of a run together, so one pass's report can recover another pass's finding.
| Separate, run 1 | Separate, run 2 | Merged | |
|---|---|---|---|
| Recovered, all (of 77) | 47 | 52 | 45 |
| Stability (of 27) | 15 | 19 | 20 |
| Product-owner (of 16) | 9 | 11 | 10 |
| Prose (of 34) | 23 | 22 | 15 |
| 🔴 recovered (of 4) | 2 | 4 | 4 |
| 🔴/🟡 written under prose | 29 | 24 | 9 |
| New 🔴/🟡 | 11 | 12 | 11 |
| Cost | $17.03 | $17.14 | $9.77 |
| Tokens | 15.8M | 17.5M | 12.9M |
- The merged total, 45, sits just below the separate runs' 47 and 52. The loss is in prose.
- The merged reviewer missed 14 of run 1's recoveries, 12 of them prose, and 16 of run 2's, 11 of them prose.
- Stability and product-owner stayed within the separate runs' range, or above it.
Verdict: unsafe. The merged reviewer wrote about a third as many prose findings.
Candidate 2: systems-architect + principles-auditor
The 5 release reviews hold 64 archived findings: architecture 47, principles 17. Four are 🔴, two per pass.
| Separate | Merged | |
|---|---|---|
| Recovered, all (of 64) | 36 | 31 |
| Architecture (of 47) | 26 | 26 |
| Principles (of 17) | 10 | 5 |
| 🔴 recovered (of 4) | 3 | 1 |
| 🔴/🟡 written under principles | 34 | 10 |
| New 🔴/🟡 | 23 (4 🔴) | 20 (1 🔴) |
| Cost | $29.40 | $15.62 |
| Tokens | 41.6M | 20.7M |
- The merged reviewer lost both principles 🔴.
- Every variant reviewed the architect's round-1 commit. The archived auditor read later code, after the architect's fixes. This affects both variants equally.
- Two rounds' file lists were regenerated from the clone. Their places lists were not archived.
- There is no noise run, because the budget did not allow one.
Verdict: likely unsafe. The merged reviewer acts like an architect that audits a little: principles findings halve.
Candidate 3: override-reviewer folded into stability
The archive holds 38 override-reviewer runs. None has a 🔴 or 🟡, so there isn't anything to recover. Nothing was replayed.
Verdict: no data.
Candidate 4: stability + product-owner, prose kept separate
Candidate 4 merges only the two passes that held up inside candidate 1's merge. It uses candidate 1's 6 chunks, without their prose rounds. They hold 43 archived findings: stability 27, product-owner 16. Two are 🔴, one per pass.
The separate figures reuse candidate 1's runs 1 and 2. Nothing was replayed again for them. Their stability and product-owner reports were matched again, against these 43 findings only. So no prose report can recover a finding here, and these figures sit below candidate 1's. The merged pair ran twice, so it has its own noise figure.
| Separate, run 1 | Separate, run 2 | Merged, run 1 | Merged, run 2 | |
|---|---|---|---|---|
| Recovered, both passes (of 43) | 20 | 28 | 20 | 17 |
| Stability (of 27) | 13 | 18 | 15 | 13 |
| Product-owner (of 16) | 7 | 10 | 5 | 4 |
| 🔴 recovered (of 2) | 0 | 2 | 2 | 2 |
| 🔴/🟡 written under stability | 19 | 19 | 19 | 16 |
| 🔴/🟡 written under product-owner | 8 | 9 | 2 | 3 |
| New 🔴/🟡 | 6 | 6 | 4 | 4 |
| Cost | $12.28 | $12.76 | $8.49 | $8.04 |
| Tokens | 13.0M | 14.7M | 11.0M | 9.7M |
- Stability stays inside the noise: 13 to 18 separate, 13 to 15 merged.
- Product-owner does not: 7 to 10 separate, 4 to 5 merged.
- The merged reviewer writes about a third as many product-owner findings.
- Merged run 1 recovered 9 findings that merged run 2 missed, and run 2 recovered 6 that run 1 missed. The two separate runs differ by 3 and 11 the same way.
- Both merged runs recovered both 🔴. Separate run 1 recovered neither.
- The separate runs' new findings are candidate 1's counts for their stability and product-owner reports. They were counted against all 77 archived findings, not these 43.
Verdict: unsafe. The sample is small, 16 product-owner findings, but both merged runs fall below both separate runs.
Cost
All costs are at list prices, as Claude Code reports them.
| Trial | Reviewer replays | Matching |
|---|---|---|
| Candidates 1 and 2, with candidate 1's noise run | about $89 | about $5 |
| Candidate 4, two merged runs | $16.53 | about $6 |
Each match and new-finding count costs about $0.30 on Opus 5.5 at medium.
How to rerun
Run every script from this folder, on the host's Python. They import from this checkout's bin/, and bin/repo_checks/test_trials_imports.py checks that every name they import from it still exists. The trials ran against commit 82eb8ff, and candidate 4 against 0442074, where review-replay was unchanged.
The scripts name the candidates wave3 (candidate 1), release (candidate 2) and pair (candidate 4).
To replay on another model, change two places: collect.py keeps only projects on claude-opus-5-5, and driver.py's RUNG is Opus 5.5 at medium.
The tracked *-reviewer.md files are the merged definitions the figures came from. build_agents.py and build_pair.py write them again from the current ~/.claude/agents/, beside the agents-*.json the driver passes to claude, which wrap the same text. To replay the exact tracked definitions, restore them with git checkout after building, and rebuild the JSON from them in the shape build_agents.py writes.
collect.pyreads loop-claude's archive and writes the archived reviewer rounds of Opus 5.5 projects toruns.pkl.mkcases1.pywritescases-wave3.json, candidate 1's cases.cases-release.jsonholds candidate 2's.build_agents.pywrites candidate 1's and 2's merged agents.build_pair.pywrites candidate 4's, andcases-pair.json: candidate 1's cases without the prose round.driver.py <candidate> <sep|merged> <run>replays one variant.AT_ONCEsets how many replays run at once, 4 by default. Results land inresults/<candidate>/<variant>-<run>/.- For candidate 4's separate baseline, copy each
*--stability.jsonand*--product-owner.jsonfromresults/wave3/sep-<run>/intoresults/pair/sep-<run>/.driver.py pair sep <run>would instead replay them afresh, at about $12 a run. analyze.py <candidate> <variant>-<run>...matches each report and counts new findings.analyze_metered.pydoes the same and prints what the matching cost. WithBASELINE=wave3it reuses candidate 1's new-finding counts.summarize.py <candidate> <variant>-<run>...prints the tables' figures.counts.pycounts the findings each report wrote, per pass.shas.py <candidate>...prints each case's reviewed commit and base.
A replay or match that exists on disk is not run again. Delete its file to run it again.
runs.pkl, results/ and agents-*.json stay out of git: they hold transcript and report text from other projects.