Answer stability round 1: 0 🔴, 2 🟡 (0 misread); shorten the decision's title to the decision, and drop the log example, where both rules pick the same commit
This commit is contained in:
@@ -36,7 +36,7 @@ Decisions live in `docs/decisions.md`:
|
||||
- review: a reviewer or comprehension seat that rules the work below standard may ask to rerun once, one rung up, in the same round; the ruling that stands is an escalation trigger for the worker unless the fix changes only prose, and never for the judge
|
||||
- review: a reviewer under `agents/` marks a 🔴 or 🟡 `[misread]` where the implementer misread code that had already shipped, each round title counts the marked findings, and `score-outcomes` lists them per chunk
|
||||
- review: an ask no user answers leaves the PR unmerged
|
||||
- review: each pass keeps a reviewer of its own, since a reviewer merged from two passes' briefs dropped most of the narrower pass's findings in every trial
|
||||
- review: each pass keeps a reviewer of its own
|
||||
- review: each reviewer under `agents/` writes its full report into the chunk's review folder under `~/.local/state/larv-review/` and returns the worker its verdict and one line per finding; rule-hook refuses its Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`
|
||||
- review: stability and product-owner report only a defect, a broken requirement of the plan or the goals, a breach of a technical principle, or for stability a hard place the change touches; each architect finding names a later change the PR makes harder; prose runs only on a chunk that changes prose
|
||||
- review: the architecture and principles passes run only in the release review, a release's last chunk, which must land before the release is ready to cut; between releases the product-owner pass reviews changes to what consumers rely on
|
||||
|
||||
+1
-1
@@ -228,7 +228,7 @@ Valid while the ladder orders its rungs by competence.
|
||||
|
||||
2026-09-30, claude. Serves goal 1: a BLOCK's restructure, Stage 0's questions and a goal conflict are the human's call, and an unmerged PR is what brings it to them; merging would make the call for them. Where an option reviews more — run the pass, fix a finding no goal lets the run decline — the run takes it; a ❓, or a fix changing a goal or a decision, is filed in `todo.md`, which reaches the human without holding the chunk. Valid while a human reads every unmerged PR an unattended run leaves.
|
||||
|
||||
## review: each pass keeps a reviewer of its own, since a reviewer merged from two passes' briefs dropped most of the narrower pass's findings in every trial
|
||||
## review: each pass keeps a reviewer of its own
|
||||
|
||||
2026-11-09, lilleman asked for the trials. Serves goals 1.1 and 1.4. Four merges were replayed on archived first rounds, beside the separate reviewers run twice to measure the noise between runs (`reviewer-merge-trials/README.md`). Stability, product-owner and prose as one reviewer cost $9.77 against $17, and recovered 15 of 34 prose findings against 22 and 23. The architect and the principles auditor as one cost $15.62 against $29.40, and recovered 5 of 17 principles findings against 10, losing both 🔴. Stability and product-owner as one cost a third less, and recovered 4 and 5 of 16 product-owner findings against 7 and 10. Each time, the merged reviewer did the first brief's work and wrote little for the second. Merging auditors that check the same kind of thing lost no findings: `review: the principles audit runs only in the release review, …` records that trial. Valid until a new model joins the ladder or a reviewer's definition changes: replay the merge then, with the scripts in `reviewer-merge-trials/`.
|
||||
|
||||
|
||||
@@ -53,7 +53,6 @@ review-replay must find the commit the archived reviewer read.
|
||||
It took the first commit in `git log` order authored by the round's time.
|
||||
A commit authored early but merged late, such as a dependency bump, then wins.
|
||||
Two plainpages rounds would have been replayed at a commit older than their own base.
|
||||
A log round would have been replayed at the wrong one of two commits sharing an author time.
|
||||
|
||||
The fix, in `driver.py` → `code_of`, takes the commit with the newest author time by then, with log order breaking ties.
|
||||
Every trial here used the fix.
|
||||
|
||||
@@ -74,7 +74,7 @@ The README line covers the bench's transcripts in `workflow-bench/logs/` too (go
|
||||
|
||||
### 289. Replay each archived round on the commit with the newest author time by the round's start, in place of the first such commit in git log order.
|
||||
|
||||
`authored_by` in `bin/review_replay/rounds.py` takes the first commit in log order authored at or before a time. A commit authored early but merged late, such as a dependency bump, sits high in log order, so it wins over the commits the round actually reviewed. `code_of` in `bin/review_replay/main.py` uses it for both the reviewed commit and the clone's base. Found on 2026-11-09 by the reviewer-merge trials (`reviewer-merge-trials/README.md`): two plainpages rounds would have replayed at a commit older than their own base, and a log round at the wrong one of two commits sharing an author time. Fix as the trials' driver did: take the commit with the newest author time at or before the time, log order breaking ties. Archived replays, and the routing decisions that cite them, may have reviewed the wrong code; once fixed, say in the PR which of those decisions' replays would change.
|
||||
`authored_by` in `bin/review_replay/rounds.py` takes the first commit in log order authored at or before a time. A commit authored early but merged late, such as a dependency bump, sits high in log order, so it wins over the commits the round actually reviewed. `code_of` in `bin/review_replay/main.py` uses it for both the reviewed commit and the clone's base. Found on 2026-11-09 by the reviewer-merge trials (`reviewer-merge-trials/README.md`): two plainpages rounds would have replayed at a commit older than their own base. Fix as the trials' driver did: take the commit with the newest author time at or before the time, log order breaking ties. Archived replays, and the routing decisions that cite them, may have reviewed the wrong code; once fixed, say in the PR which of those decisions' replays would change.
|
||||
|
||||
### 281. Remove a merged chunk's worktree and branch and fast-forward the main checkout with one command.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user