Files
claude-config/workflow-bench/compare-prompt.md
T

1.7 KiB
Raw Blame History

You compare two code-review processes, A and B, on one real chunk of work. Each process reviewed the same chunk three times, independently: runs A1–A3 and B1–B3. Rule whether B reviews at least as well as A, and whether A reviews at least as well as B.

Everything is under {dir}; don't read anything outside it:

  • truth-issues.md: what the chunk's real review found and fixed, one row per issue.
  • truth.diff: the real review's fixes.
  • <run>/report.md: the run's own account of its passes, findings and outcomes. Its claims are not evidence.
  • <run>/fixes.diff: what the run actually changed. This is the evidence.

Rate each run on:

  1. Truth coverage: for each truth issue, caught and fixed ✔, partly ◐, or missed ✘, checked against fixes.diff. A filed or declined issue with a sound reason counts ◐.
  2. Other real defects: a finding the truth lacks that is a real defect in the chunk, fixed or filed.
  3. Damage: a change that makes the chunk worse — a regression, a false claim written, a needless churn of lines outside the chunk.
  4. Cost: agents launched and rounds, from the report.

Then, per process, weigh the three runs together: what it catches reliably (two or more runs), what it catches by luck (one run), and its spread. A difference one run makes is variance, not the process.

Write your ruling to {ruling}:

  1. A table per run: each truth issue's mark, other real defects, damage, cost.
  2. Per process: reliable catches, lucky catches, damage, cost range.
  3. Ruling: one line each — B at least as good as A: yes|no and A at least as good as B: yes|no — each followed by the decisive evidence, citing runs and diff lines. "No" needs a difference that holds across runs, not one run's miss.