Files
2026-11-20 18:02:12 +01:00
..

Reviewer merge trials

These trials ask one question: can larv-review run fewer, merged reviewers without losing findings? A merged reviewer is one agent that does the work of two or three separate reviewers in one run. It costs less, because it reads the change once. They ran on 2026-11-09. docs/decisions.md → review: each pass keeps a reviewer of its own records what was decided from them.

Terms

  • Pass: one kind of review, run by one reviewer agent: stability, product-owner, prose, architecture or principles.
  • Round: one run of one pass on one change, archived with its report.
  • Archived finding: a 🔴 (blocker) or 🟡 (should fix) that an archived round reported.
  • Recovered: a replay's report raises an archived finding again, as 🔴 or 🟡.
  • New finding: a 🔴 or 🟡 in a replay's report that matches no archived finding. New findings are counted only; nobody checked whether they are real.
  • Noise: how much two runs of the same reviewer on the same rounds differ.

Verdicts

No merge tested is safe. In each, one absorbed pass held and another lost findings.

Candidate Reviewers merged Verdict
1 stability + product-owner + prose Unsafe. Prose fell to 15 of 34, below both separate runs (22 and 23). The merged reviewer ran once.
2 systems-architect + principles-auditor Likely unsafe. Principles fell to 5 of 17, against 10, and both principles 🔴 were lost. No noise run.
3 override-reviewer folded into stability No data. No archived override round has a 🔴 or 🟡.
4 stability + product-owner, prose kept separate Unsafe. Product-owner fell to 4–5 of 16, against 7–10; the ranges don't overlap. Stability held. Each variant ran twice.

Candidates 1 and 2 cut cost about in half. Candidate 4 cut it by about a third.

Method

  1. Pick archived first rounds where every pass under test ran on the same chunk.
  2. Replay each round with review-replay's own functions, from bin/review_replay/ and bin/rung_replay/.
  3. Each replay runs on Opus 5.5 at medium effort, in a throwaway clone of the project at the reviewed commit.
  4. Each replay gets a copy of the review folder holding only the reports written before the case's first round, so no variant sees another pass's findings.
  5. The separate variant runs each reviewer unchanged, with the worker's archived brief.
  6. The merged variant runs one agent, defined for the run through claude's --agents flag. Its definition holds the full text of every reviewer it replaces, under a short preamble asking for one report with one heading per pass. Its prompt joins the worker's archived briefs to each pass.
  7. review-replay's matcher, on Opus 5.5 at medium, counts which archived findings each report recovers.
  8. A second call on the same rung counts the report's new findings.

The noise lesson

Two runs of the same separate reviewers on the same rounds differ a lot. In candidate 1, run 1 recovered 47 of 77 findings and run 2 recovered 52. Run 1 found 8 that run 2 missed, and run 2 found 13 that run 1 missed. So one run per variant cannot tell a small loss from chance. A firm verdict needs two runs of each variant, and a loss larger than the spread between them. Candidate 4 has that. Candidate 1's merged reviewer ran once, but its prose loss is far below both separate runs. Candidate 2 has no noise run at all.

The reviewed-commit fix

Every trial took the commit with the newest author time, log order breaking ties, as review-replay does from 2026-11-10. Before that it took the first commit in log order authored by then, and a trial's driver patched that in. Without the patch, two plainpages rounds would have replayed at a commit older than their own base.

Samples

The reviewed commit is the one each replay read; the base is the default branch as the chunk began.

Candidates 1 and 4 use these 6 chunks.

Project Chunk Reviewed commit Base
adf-codec item-45 ca6016b4b6627159f369f3ce047ebb4d1727a95a 9dfa14a095222c5b8fd4d4d435e2940720483feb
claude-config item-230 b37d03056c 2a07feebec
fejkdata item-223 a3967a01956bb67d25a83caf8f9be45426cec837 f40bdd0dcefc01b821feae72d86e70ad606d4c99
log item-7 08fe3562230affe87ca5aa4661f199831189de91 08e8945eaa182fcf06689337d8140ebf91b2a2e7
plainpages item-2 dd39c7a07ad3e497792a645a0c94c4eb496db9e4 34dfbea156649fd54ff4ea17221d98d19d0d53df
plainpages item-23 80070762e7337fbdd21a59ebce2cb75d98dcd2f6 01f368d663e9b66e2ba488de4b75d64559106b63

Candidate 2 uses round 1 of the 5 release reviews where both the architect and the auditor ran.

Project Chunk Reviewed commit Base
adf-codec release-review-0.3.0 8e6d7efff16f0df15892916549a7515d25b97886 27e8f33f30a46bfa7dc1bc52f6ca85d342ec4d2c
fejkdata release-review-0.3.0 5e1c3464c2b8d39d7f989323b287b6a8eda0f02e 453a020864f3e5aa5aca728a48faac9b553bea9c
fejkdata release-review-0.4.0 a5dd82ca691d9b4135f1c7fed8e97efb4076459b 4cd2f6d8693091e6e710db9bf5ef7b58f83b89aa
plainpages-plugin-fastjira release-review-0.2.0 3ffedfee6203a95053f12ef674b0ae69d903555b 6345fc75fb24562e8a975eccae05a865d33fc003
plainpages-plugin-fastjira release-review-0.3.0 30a4a61b5bde493ce0d76d10dda281b94a14aad2 468d89c9186a6b4bad3bbd372e2e9035e47fc7b6

Candidate 1: stability + product-owner + prose

The 6 chunks hold 77 archived findings: stability 27, product-owner 16, prose 34. Four are 🔴. Here each archived finding is matched against all three reports of a run together, so one pass's report can recover another pass's finding.

Separate, run 1 Separate, run 2 Merged
Recovered, all (of 77) 47 52 45
Stability (of 27) 15 19 20
Product-owner (of 16) 9 11 10
Prose (of 34) 23 22 15
🔴 recovered (of 4) 2 4 4
🔴/🟡 written under prose 29 24 9
New 🔴/🟡 11 12 11
Cost $17.03 $17.14 $9.77
Tokens 15.8M 17.5M 12.9M
  • The merged total, 45, sits just below the separate runs' 47 and 52. The loss is in prose.
  • The merged reviewer missed 14 of run 1's recoveries, 12 of them prose, and 16 of run 2's, 11 of them prose.
  • Stability and product-owner stayed within the separate runs' range, or above it.

Verdict: unsafe. The merged reviewer wrote about a third as many prose findings.

Candidate 2: systems-architect + principles-auditor

The 5 release reviews hold 64 archived findings: architecture 47, principles 17. Four are 🔴, two per pass.

Separate Merged
Recovered, all (of 64) 36 31
Architecture (of 47) 26 26
Principles (of 17) 10 5
🔴 recovered (of 4) 3 1
🔴/🟡 written under principles 34 10
New 🔴/🟡 23 (4 🔴) 20 (1 🔴)
Cost $29.40 $15.62
Tokens 41.6M 20.7M
  • The merged reviewer lost both principles 🔴.
  • Every variant reviewed the architect's round-1 commit. The archived auditor read later code, after the architect's fixes. This affects both variants equally.
  • Two rounds' file lists were regenerated from the clone. Their places lists were not archived.
  • There is no noise run, because the budget did not allow one.

Verdict: likely unsafe. The merged reviewer acts like an architect that audits a little: principles findings halve.

Candidate 3: override-reviewer folded into stability

The archive holds 38 override-reviewer runs. None has a 🔴 or 🟡, so there isn't anything to recover. Nothing was replayed.

Verdict: no data.

Candidate 4: stability + product-owner, prose kept separate

Candidate 4 merges only the two passes that held up inside candidate 1's merge. It uses candidate 1's 6 chunks, without their prose rounds. They hold 43 archived findings: stability 27, product-owner 16. Two are 🔴, one per pass.

The separate figures reuse candidate 1's runs 1 and 2. Nothing was replayed again for them. Their stability and product-owner reports were matched again, against these 43 findings only. So no prose report can recover a finding here, and these figures sit below candidate 1's. The merged pair ran twice, so it has its own noise figure.

Separate, run 1 Separate, run 2 Merged, run 1 Merged, run 2
Recovered, both passes (of 43) 20 28 20 17
Stability (of 27) 13 18 15 13
Product-owner (of 16) 7 10 5 4
🔴 recovered (of 2) 0 2 2 2
🔴/🟡 written under stability 19 19 19 16
🔴/🟡 written under product-owner 8 9 2 3
New 🔴/🟡 6 6 4 4
Cost $12.28 $12.76 $8.49 $8.04
Tokens 13.0M 14.7M 11.0M 9.7M
  • Stability stays inside the noise: 13 to 18 separate, 13 to 15 merged.
  • Product-owner does not: 7 to 10 separate, 4 to 5 merged.
  • The merged reviewer writes about a third as many product-owner findings.
  • Merged run 1 recovered 9 findings that merged run 2 missed, and run 2 recovered 6 that run 1 missed. The two separate runs differ by 3 and 11 the same way.
  • Both merged runs recovered both 🔴. Separate run 1 recovered neither.
  • The separate runs' new findings are candidate 1's counts for their stability and product-owner reports. They were counted against all 77 archived findings, not these 43.

Verdict: unsafe. The sample is small, 16 product-owner findings, but both merged runs fall below both separate runs.

Cost

All costs are at list prices, as Claude Code reports them.

Trial Reviewer replays Matching
Candidates 1 and 2, with candidate 1's noise run about $89 about $5
Candidate 4, two merged runs $16.53 about $6

Each match and new-finding count costs about $0.30 on Opus 5.5 at medium.

How to rerun

Run every script from this folder, on the host's Python. They import from this checkout's bin/, and bin/repo_checks/test_trials_imports.py checks that every name they import from it still exists. The trials ran against commit 82eb8ff, and candidate 4 against 0442074, where review-replay was unchanged. The scripts name the candidates wave3 (candidate 1), release (candidate 2) and pair (candidate 4).

To replay on another model, change two places: collect.py keeps only projects on claude-opus-5-5, and driver.py's RUNG is Opus 5.5 at medium.

The tracked *-reviewer.md files are the merged definitions the figures came from. build_agents.py and build_pair.py write them again from the current ~/.claude/agents/, beside the agents-*.json the driver passes to claude, which wrap the same text. To replay the exact tracked definitions, restore them with git checkout after building, and rebuild the JSON from them in the shape build_agents.py writes.

  1. collect.py reads loop-claude's archive and writes the archived reviewer rounds of Opus 5.5 projects to runs.pkl.
  2. mkcases1.py writes cases-wave3.json, candidate 1's cases. cases-release.json holds candidate 2's.
  3. build_agents.py writes candidate 1's and 2's merged agents. build_pair.py writes candidate 4's, and cases-pair.json: candidate 1's cases without the prose round.
  4. driver.py <candidate> <sep|merged> <run> replays one variant. AT_ONCE sets how many replays run at once, 4 by default. Results land in results/<candidate>/<variant>-<run>/.
  5. For candidate 4's separate baseline, copy each *--stability.json and *--product-owner.json from results/wave3/sep-<run>/ into results/pair/sep-<run>/. driver.py pair sep <run> would instead replay them afresh, at about $12 a run.
  6. analyze.py <candidate> <variant>-<run>... matches each report and counts new findings. analyze_metered.py does the same and prints what the matching cost. With BASELINE=wave3 it reuses candidate 1's new-finding counts.
  7. summarize.py <candidate> <variant>-<run>... prints the tables' figures. counts.py counts the findings each report wrote, per pass.
  8. shas.py <candidate>... prints each case's reviewed commit and base.

A replay or match that exists on disk is not run again. Delete its file to run it again. runs.pkl, results/ and agents-*.json stay out of git: they hold transcript and report text from other projects.