1.7 KiB
1.7 KiB
You compare two code-review processes, A and B, on one real chunk of work. Each process reviewed the same chunk three times, independently: runs A1–A3 and B1–B3. Rule whether B reviews at least as well as A, and whether A reviews at least as well as B.
Everything is under {dir}; don't read anything outside it:
truth-issues.md: what the chunk's real review found and fixed, one row per issue.truth.diff: the real review's fixes.<run>/report.md: the run's own account of its passes, findings and outcomes. Its claims are not evidence.<run>/fixes.diff: what the run actually changed. This is the evidence.
Rate each run on:
- Truth coverage: for each truth issue, caught and fixed ✔, partly ◐, or missed ✘, checked against
fixes.diff. A filed or declined issue with a sound reason counts ◐. - Other real defects: a finding the truth lacks that is a real defect in the chunk, fixed or filed.
- Damage: a change that makes the chunk worse — a regression, a false claim written, a needless churn of lines outside the chunk.
- Cost: agents launched and rounds, from the report.
Then, per process, weigh the three runs together: what it catches reliably (two or more runs), what it catches by luck (one run), and its spread. A difference one run makes is variance, not the process.
Write your ruling to {ruling}:
- A table per run: each truth issue's mark, other real defects, damage, cost.
- Per process: reliable catches, lucky catches, damage, cost range.
- Ruling: one line each —
B at least as good as A: yes|noandA at least as good as B: yes|no— each followed by the decisive evidence, citing runs and diff lines. "No" needs a difference that holds across runs, not one run's miss.