Answer product-owner round 1: 0 🔴, 5 🟡 (0 misread); hedge the verdicts that lack a noise run, and complete the rerun steps: run from the folder, reuse candidate 1's separate reports, change the model in two places, replay the tracked definitions, name the commit; prose's order and term fixes ride along
This commit is contained in:
@@ -3,40 +3,40 @@
|
||||
These trials ask one question: can larv-review run fewer, merged reviewers without losing findings?
|
||||
A merged reviewer is one agent that does the work of two or three separate reviewers in one run.
|
||||
It costs less, because it reads the change once.
|
||||
|
||||
## Verdicts
|
||||
|
||||
No merge tested is safe. Each one lost the findings of one pass it absorbed, while the other pass held.
|
||||
|
||||
| Candidate | Reviewers merged | Verdict |
|
||||
|---|---|---|
|
||||
| 1 | stability + product-owner + prose | Unsafe. Prose findings drop well outside the noise. |
|
||||
| 2 | systems-architect + principles-auditor | Unsafe. Principles findings halve, and both principles 🔴 are lost. |
|
||||
| 3 | override-reviewer folded into stability | No data. No archived override round has a 🔴 or 🟡. |
|
||||
| 4 | stability + product-owner, prose kept separate | Unsafe. Product-owner findings drop outside the noise; stability holds. |
|
||||
|
||||
The three-way and architect merges cut cost about in half. The stability and product-owner merge cuts it by about a third.
|
||||
They ran on 2026-11-09. `docs/decisions.md` → `review: each pass keeps a reviewer of its own` records what was decided from them.
|
||||
|
||||
## Terms
|
||||
|
||||
- **Pass:** one kind of review, run by one reviewer agent. The passes here are stability, product-owner, prose, architecture and principles.
|
||||
- **Pass:** one kind of review, run by one reviewer agent: stability, product-owner, prose, architecture or principles.
|
||||
- **Round:** one run of one pass on one change, archived with its report.
|
||||
- **Archived finding:** a 🔴 (blocker) or 🟡 (should fix) that an archived round reported.
|
||||
- **Recovered:** a replay's report raises an archived finding again, as 🔴 or 🟡.
|
||||
- **New finding:** a 🔴 or 🟡 in a replay's report that matches no archived finding. New findings are counted only; nobody checked whether they are real.
|
||||
- **Noise:** how much two runs of the same reviewer on the same round differ.
|
||||
- **Noise:** how much two runs of the same reviewer on the same rounds differ.
|
||||
|
||||
## Verdicts
|
||||
|
||||
No merge tested is safe. In each, one absorbed pass held and another lost findings.
|
||||
|
||||
| Candidate | Reviewers merged | Verdict |
|
||||
|---|---|---|
|
||||
| 1 | stability + product-owner + prose | Unsafe. Prose fell to 15 of 34, below both separate runs (22 and 23). The merged reviewer ran once. |
|
||||
| 2 | systems-architect + principles-auditor | Likely unsafe. Principles fell to 5 of 17, against 10, and both principles 🔴 were lost. No noise run. |
|
||||
| 3 | override-reviewer folded into stability | No data. No archived override round has a 🔴 or 🟡. |
|
||||
| 4 | stability + product-owner, prose kept separate | Unsafe. Product-owner fell to 4–5 of 16, against 7–10; the ranges don't overlap. Stability held. Each variant ran twice. |
|
||||
|
||||
Candidates 1 and 2 cut cost about in half. Candidate 4 cut it by about a third.
|
||||
|
||||
## Method
|
||||
|
||||
1. Pick archived round-1 review rounds where every pass under test ran on the same chunk.
|
||||
2. Replay each round with review-replay's own functions, from `bin/review_replay/` and `bin/rung_replay/` in claude-config.
|
||||
3. Each replay runs on Opus 5.5 at medium effort.
|
||||
4. Each replay runs in a throwaway clone of the project at the reviewed commit.
|
||||
5. Each replay gets a copy of the review folder holding only the reports written before the case's first round. So no variant sees another pass's findings.
|
||||
6. The separate variant runs each reviewer unchanged, with the worker's archived brief.
|
||||
7. The merged variant passes one agent through `--agents`. Its definition holds the full text of every reviewer it replaces, under a short preamble. The preamble asks for one report with one heading per pass. Its prompt joins the worker's archived briefs to each pass.
|
||||
8. review-replay's matcher, on Opus 5.5 at medium, counts which archived findings each report recovers.
|
||||
9. A second call on the same rung lists the report's new findings.
|
||||
1. Pick archived first rounds where every pass under test ran on the same chunk.
|
||||
2. Replay each round with review-replay's own functions, from `bin/review_replay/` and `bin/rung_replay/`.
|
||||
3. Each replay runs on Opus 5.5 at medium effort, in a throwaway clone of the project at the reviewed commit.
|
||||
4. Each replay gets a copy of the review folder holding only the reports written before the case's first round, so no variant sees another pass's findings.
|
||||
5. The separate variant runs each reviewer unchanged, with the worker's archived brief.
|
||||
6. The merged variant runs one agent, defined for the run through claude's `--agents` flag. Its definition holds the full text of every reviewer it replaces, under a short preamble asking for one report with one heading per pass. Its prompt joins the worker's archived briefs to each pass.
|
||||
7. review-replay's matcher, on Opus 5.5 at medium, counts which archived findings each report recovers.
|
||||
8. A second call on the same rung counts the report's new findings.
|
||||
|
||||
### The noise lesson
|
||||
|
||||
@@ -44,32 +44,28 @@ Two runs of the same separate reviewers on the same rounds differ a lot.
|
||||
In candidate 1, run 1 recovered 47 of 77 findings and run 2 recovered 52.
|
||||
Run 1 found 8 that run 2 missed, and run 2 found 13 that run 1 missed.
|
||||
So one run per variant cannot tell a small loss from chance.
|
||||
A verdict needs a noise figure: at least two runs of each variant.
|
||||
Only a loss larger than the spread between runs counts.
|
||||
A firm verdict needs two runs of each variant, and a loss larger than the spread between them.
|
||||
Candidate 4 has that. Candidate 1's merged reviewer ran once, but its prose loss is far below both separate runs. Candidate 2 has no noise run at all.
|
||||
|
||||
### The reviewed-commit bug
|
||||
### The reviewed-commit fix
|
||||
|
||||
review-replay must find the commit the archived reviewer read.
|
||||
It took the first commit in `git log` order authored by the round's time.
|
||||
A commit authored early but merged late, such as a dependency bump, then wins.
|
||||
Two plainpages rounds would have been replayed at a commit older than their own base.
|
||||
|
||||
The fix, in `driver.py` → `code_of`, takes the commit with the newest author time by then, with log order breaking ties.
|
||||
Every trial here used the fix.
|
||||
Archived replays made with the unfixed review-replay may have reviewed the wrong code.
|
||||
Every trial used a fix for a review-replay defect, filed as `todo.md` item 289: `driver.py` → `code_of` takes the commit with the newest author time, log order breaking ties.
|
||||
Without it, two plainpages rounds would have replayed at a commit older than their own base.
|
||||
|
||||
## Samples
|
||||
|
||||
Candidates 1 and 4 use the same 6 chunks. The commit is the one each replay reviewed; the base is the default branch as the chunk began.
|
||||
The reviewed commit is the one each replay read; the base is the default branch as the chunk began.
|
||||
|
||||
Candidates 1 and 4 use these 6 chunks.
|
||||
|
||||
| Project | Chunk | Reviewed commit | Base |
|
||||
|---|---|---|---|
|
||||
| adf-codec | item-45 | ca6016b4b6627159f369f3ce047ebb4d1727a95a | 9dfa14a095222c5b8fd4d4d435e2940720483feb |
|
||||
| claude-config | item-230 | b37d03056c3277cd4a460c91cbbbaa025e7fd672 | 2a07feebecbb7b73ff692bfad9f6f5519423fb84 |
|
||||
| plainpages | item-2 | dd39c7a07ad3e497792a645a0c94c4eb496db9e4 | 34dfbea156649fd54ff4ea17221d98d19d0d53df |
|
||||
| plainpages | item-23 | 80070762e7337fbdd21a59ebce2cb75d98dcd2f6 | 01f368d663e9b66e2ba488de4b75d64559106b63 |
|
||||
| fejkdata | item-223 | a3967a01956bb67d25a83caf8f9be45426cec837 | f40bdd0dcefc01b821feae72d86e70ad606d4c99 |
|
||||
| log | item-7 | 08fe3562230affe87ca5aa4661f199831189de91 | 08e8945eaa182fcf06689337d8140ebf91b2a2e7 |
|
||||
| adf-codec | item-45 | ca6016b4b6627159f369f3ce047ebb4d1727a95a | 9dfa14a095222c5b8fd4d4d435e2940720483feb |
|
||||
| plainpages | item-2 | dd39c7a07ad3e497792a645a0c94c4eb496db9e4 | 34dfbea156649fd54ff4ea17221d98d19d0d53df |
|
||||
| plainpages | item-23 | 80070762e7337fbdd21a59ebce2cb75d98dcd2f6 | 01f368d663e9b66e2ba488de4b75d64559106b63 |
|
||||
|
||||
Candidate 2 uses round 1 of the 5 release reviews where both the architect and the auditor ran.
|
||||
|
||||
@@ -81,43 +77,10 @@ Candidate 2 uses round 1 of the 5 release reviews where both the architect and t
|
||||
| plainpages-plugin-fastjira | release-review-0.2.0 | 3ffedfee6203a95053f12ef674b0ae69d903555b | 6345fc75fb24562e8a975eccae05a865d33fc003 |
|
||||
| plainpages-plugin-fastjira | release-review-0.3.0 | 30a4a61b5bde493ce0d76d10dda281b94a14aad2 | 468d89c9186a6b4bad3bbd372e2e9035e47fc7b6 |
|
||||
|
||||
## Candidate 4: stability + product-owner, prose kept separate
|
||||
|
||||
Candidate 4 merges only the two passes that held up inside candidate 1's merge.
|
||||
It uses candidate 1's 6 chunks, without their prose rounds.
|
||||
They hold 43 archived findings: stability 27, product-owner 16. Two are 🔴, one per pass.
|
||||
|
||||
The separate figures reuse candidate 1's runs 1 and 2. Nothing was rerun for them.
|
||||
Their stability and product-owner reports were matched again, against these 43 findings only.
|
||||
So no prose report can recover a finding here, unlike candidate 1's figures.
|
||||
The merged pair ran twice, so it has its own noise figure.
|
||||
|
||||
| | Separate, run 1 | Separate, run 2 | Merged, run 1 | Merged, run 2 |
|
||||
|---|---|---|---|---|
|
||||
| Recovered, both passes (of 43) | 20 | 28 | 20 | 17 |
|
||||
| Stability (of 27) | 13 | 18 | 15 | 13 |
|
||||
| **Product-owner (of 16)** | **7** | **10** | **5** | **4** |
|
||||
| 🔴 recovered (of 2) | 0 | 2 | 2 | 2 |
|
||||
| 🔴/🟡 written under stability | 19 | 19 | 19 | 16 |
|
||||
| **🔴/🟡 written under product-owner** | **8** | **9** | **2** | **3** |
|
||||
| New 🔴/🟡 | 6 | 6 | 4 | 4 |
|
||||
| Cost | $12.28 | $12.76 | **$8.49** | **$8.04** |
|
||||
| Tokens | 13.0M | 14.7M | 11.0M | 9.7M |
|
||||
|
||||
- Separate runs recover 20 to 28. Merged runs recover 17 to 20.
|
||||
- Stability stays inside the noise: 13 to 18 separate, 13 to 15 merged.
|
||||
- Product-owner does not: 7 to 10 separate, 4 to 5 merged. The ranges don't overlap.
|
||||
- The merged reviewer writes about a third as many product-owner findings.
|
||||
- The merged runs missed 12 to 14 of separate run 2's recoveries, half of them product-owner.
|
||||
- Both merged runs recovered both 🔴. Separate run 1 recovered neither.
|
||||
- The two merged runs differ by 9 and 6 findings each way, about as much as the separate runs.
|
||||
- The separate runs' new-finding counts are candidate 1's, kept where they sit under stability or product-owner. They were counted against all 77 archived findings, not these 43.
|
||||
|
||||
**Verdict: unsafe.** Product-owner findings fall outside the noise in both merged runs. The sample is small, 16 product-owner findings. But the drop has the same shape as in candidates 1 and 2: the merged reviewer does the first brief's job and writes little for the second. The merge saves about a third of the cost, less than the half the larger merges saved.
|
||||
|
||||
## Candidate 1: stability + product-owner + prose
|
||||
|
||||
The 6 chunks hold 77 archived findings: stability 27, product-owner 16, prose 34. Four are 🔴.
|
||||
Here each archived finding is matched against all three reports of a run together, so one pass's report can recover another pass's finding.
|
||||
|
||||
| | Separate, run 1 | Separate, run 2 | Merged |
|
||||
|---|---|---|---|
|
||||
@@ -131,13 +94,11 @@ The 6 chunks hold 77 archived findings: stability 27, product-owner 16, prose 34
|
||||
| Cost | $17.03 | $17.14 | **$9.77** |
|
||||
| Tokens | 15.8M | 17.5M | 12.9M |
|
||||
|
||||
- The merged reviewer missed 14 of run 1's recoveries, 12 of them prose.
|
||||
- It missed 16 of run 2's recoveries, 11 of them prose.
|
||||
- Stability and product-owner stayed inside the noise.
|
||||
- The merged total, 45, sits just below the separate runs' 47 and 52. The loss is in prose.
|
||||
- The merged reviewer missed 14 of run 1's recoveries, 12 of them prose, and 16 of run 2's, 11 of them prose.
|
||||
- Stability and product-owner stayed within the separate runs' range, or above it.
|
||||
|
||||
**Verdict: unsafe.** The totals sit inside the noise, but the prose loss is systematic. The merged reviewer wrote about a third as many prose findings.
|
||||
|
||||
The recovery figures here match each archived finding against all three reports together. A report of one pass can recover another pass's finding.
|
||||
**Verdict: unsafe.** The merged reviewer wrote about a third as many prose findings.
|
||||
|
||||
## Candidate 2: systems-architect + principles-auditor
|
||||
|
||||
@@ -159,7 +120,7 @@ The 5 release reviews hold 64 archived findings: architecture 47, principles 17.
|
||||
- Two rounds' file lists were regenerated from the clone. Their places lists were not archived.
|
||||
- There is no noise run, because the budget did not allow one.
|
||||
|
||||
**Verdict: unsafe.** The merged reviewer acts like an architect that audits a little. The drop has the same shape and size as candidate 1's prose drop.
|
||||
**Verdict: likely unsafe.** The merged reviewer acts like an architect that audits a little: principles findings halve.
|
||||
|
||||
## Candidate 3: override-reviewer folded into stability
|
||||
|
||||
@@ -167,6 +128,38 @@ The archive holds 38 override-reviewer runs. None has a 🔴 or 🟡, so there i
|
||||
|
||||
**Verdict: no data.**
|
||||
|
||||
## Candidate 4: stability + product-owner, prose kept separate
|
||||
|
||||
Candidate 4 merges only the two passes that held up inside candidate 1's merge.
|
||||
It uses candidate 1's 6 chunks, without their prose rounds.
|
||||
They hold 43 archived findings: stability 27, product-owner 16. Two are 🔴, one per pass.
|
||||
|
||||
The separate figures reuse candidate 1's runs 1 and 2. Nothing was replayed again for them.
|
||||
Their stability and product-owner reports were matched again, against these 43 findings only.
|
||||
So no prose report can recover a finding here, and these figures sit below candidate 1's.
|
||||
The merged pair ran twice, so it has its own noise figure.
|
||||
|
||||
| | Separate, run 1 | Separate, run 2 | Merged, run 1 | Merged, run 2 |
|
||||
|---|---|---|---|---|
|
||||
| Recovered, both passes (of 43) | 20 | 28 | 20 | 17 |
|
||||
| Stability (of 27) | 13 | 18 | 15 | 13 |
|
||||
| **Product-owner (of 16)** | **7** | **10** | **5** | **4** |
|
||||
| 🔴 recovered (of 2) | 0 | 2 | 2 | 2 |
|
||||
| 🔴/🟡 written under stability | 19 | 19 | 19 | 16 |
|
||||
| **🔴/🟡 written under product-owner** | **8** | **9** | **2** | **3** |
|
||||
| New 🔴/🟡 | 6 | 6 | 4 | 4 |
|
||||
| Cost | $12.28 | $12.76 | **$8.49** | **$8.04** |
|
||||
| Tokens | 13.0M | 14.7M | 11.0M | 9.7M |
|
||||
|
||||
- Stability stays inside the noise: 13 to 18 separate, 13 to 15 merged.
|
||||
- Product-owner does not: 7 to 10 separate, 4 to 5 merged.
|
||||
- The merged reviewer writes about a third as many product-owner findings.
|
||||
- Merged run 1 recovered 9 findings that merged run 2 missed, and run 2 recovered 6 that run 1 missed. The two separate runs differ by 3 and 11 the same way.
|
||||
- Both merged runs recovered both 🔴. Separate run 1 recovered neither.
|
||||
- The separate runs' new findings are candidate 1's counts for their stability and product-owner reports. They were counted against all 77 archived findings, not these 43.
|
||||
|
||||
**Verdict: unsafe.** The sample is small, 16 product-owner findings, but both merged runs fall below both separate runs.
|
||||
|
||||
## Cost
|
||||
|
||||
All costs are at list prices, as Claude Code reports them.
|
||||
@@ -176,20 +169,25 @@ All costs are at list prices, as Claude Code reports them.
|
||||
| Candidates 1 and 2, with candidate 1's noise run | about $89 | about $5 |
|
||||
| Candidate 4, two merged runs | $16.53 | about $6 |
|
||||
|
||||
Candidate 4 reused candidate 1's separate replays, so its separate figures cost only their matching.
|
||||
Each match and new-finding count costs about $0.30 on Opus 5.5 at medium.
|
||||
|
||||
## How to rerun
|
||||
|
||||
All scripts sit in this folder. They import review-replay from `~/code/claude-config/bin`, and run on the host's Python.
|
||||
Run every script from this folder, on the host's Python. They import review-replay from `~/code/claude-config/bin`; the trials ran against commit 82eb8ff, and candidate 4 against 0442074, where review-replay was unchanged.
|
||||
The scripts name the candidates `wave3` (candidate 1), `release` (candidate 2) and `pair` (candidate 4).
|
||||
|
||||
1. `collect.py` reads loop-claude's archive and writes every archived reviewer round to `runs.pkl`.
|
||||
To replay on another model, change two places: `collect.py` keeps only projects on `claude-opus-5-5`, and `driver.py`'s `RUNG` is Opus 5.5 at medium.
|
||||
|
||||
The tracked `*-reviewer.md` files are the merged definitions the figures came from. `build_agents.py` and `build_pair.py` write them again from the current `~/.claude/agents/`, beside the `agents-*.json` the driver passes to claude, which wrap the same text. To replay the exact tracked definitions, restore them with `git checkout` after building, and rebuild the JSON from them in the shape `build_agents.py` writes.
|
||||
|
||||
1. `collect.py` reads loop-claude's archive and writes the archived reviewer rounds of Opus 5.5 projects to `runs.pkl`.
|
||||
2. `mkcases1.py` writes `cases-wave3.json`, candidate 1's cases. `cases-release.json` holds candidate 2's.
|
||||
3. `build_agents.py` writes candidate 1's and 2's merged agents, `agents-wave3.json` and `agents-release.json`, from `~/.claude/agents/`.
|
||||
4. `build_pair.py` writes candidate 4's merged agent, `agents-pair.json`, from candidate 1's brief texts. It also writes `cases-pair.json`: candidate 1's cases without the prose round.
|
||||
5. `driver.py <candidate> <sep|merged> <run>` replays one variant. Candidates are `wave3`, `release` and `pair`. `AT_ONCE` sets how many replays run at once, 4 by default. Results land in `results/<candidate>/<variant>-<run>/`.
|
||||
6. `analyze.py <candidate> <variant>-<run>...` matches each report and counts new findings. `analyze_metered.py` does the same and prints what the matching cost. With `BASELINE=wave3` it reuses that candidate's new-finding counts instead of counting again.
|
||||
3. `build_agents.py` writes candidate 1's and 2's merged agents. `build_pair.py` writes candidate 4's, and `cases-pair.json`: candidate 1's cases without the prose round.
|
||||
4. `driver.py <candidate> <sep|merged> <run>` replays one variant. `AT_ONCE` sets how many replays run at once, 4 by default. Results land in `results/<candidate>/<variant>-<run>/`.
|
||||
5. For candidate 4's separate baseline, copy each `*--stability.json` and `*--product-owner.json` from `results/wave3/sep-<run>/` into `results/pair/sep-<run>/`. `driver.py pair sep <run>` would instead replay them afresh, at about $12 a run.
|
||||
6. `analyze.py <candidate> <variant>-<run>...` matches each report and counts new findings. `analyze_metered.py` does the same and prints what the matching cost. With `BASELINE=wave3` it reuses candidate 1's new-finding counts.
|
||||
7. `summarize.py <candidate> <variant>-<run>...` prints the tables' figures. `counts.py` counts the findings each report wrote, per pass.
|
||||
8. `shas.py <candidate>...` prints each case's reviewed commit and base.
|
||||
|
||||
A replay or match that exists on disk is not rerun. Delete its file to run it again.
|
||||
A replay or match that exists on disk is not run again. Delete its file to run it again.
|
||||
`runs.pkl`, `results/` and `agents-*.json` stay out of git: they hold transcript and report text from other projects.
|
||||
|
||||
Reference in New Issue
Block a user