Answer stability round 1: 0 🔴, 7 🟡 (0 misread); hold a rung to a whole count of findings, tell a 🔴 by its opening mark, test the second reference replay failing alone, and share the candidate rungs
This commit is contained in:
+13
-12
@@ -13,19 +13,20 @@ shows the chunk's change. The prompt names the clone in place of the worktree it
|
||||
chunk has since removed. A round whose code it cannot find in this way is not replayed; stderr
|
||||
names it and why. The prompt also names a copy of the chunk's review folder in place of that
|
||||
folder, holding only the reports written after the chunk began and before the round did. A replay
|
||||
finds nothing where its prompt still names the review folder, or names a file there, other than a
|
||||
round's report, that the folder did not hold from the chunk's start until the round's.
|
||||
fails where its prompt still names the review folder, or names a file there, other than a round's
|
||||
report, that the folder did not hold from the chunk's start until the round's.
|
||||
|
||||
It replays every round twice on the project's rung. Even that rung finds only part of its own
|
||||
archived findings again, so a rung below it is held to the same share, less a margin of 10
|
||||
percentage points: the mean share of those two replays. Rounds that either replay failed on are
|
||||
left out, and where fewer than 8 remain, or the project's rung found none, no rung is tried. It
|
||||
archived findings again, so the bar for a rung below is the mean share of those two replays, less
|
||||
a margin of 10 percentage points. Rounds that either replay failed on are left out, and where
|
||||
fewer than 8 remain, or the bar is zero or below, no rung is tried. It
|
||||
then replays the rounds on each rung below, highest first: the project's model at each lower
|
||||
effort, which the ladder leaves out, then each ladder rung down to the ladder's lowest. It stops at
|
||||
the first rung that doesn't qualify. A rung qualifies where its share reaches the bar and it misses
|
||||
no 🔴 that both replays on the project's rung found; a replay that failed on it finds nothing. For
|
||||
the first rung that doesn't qualify. A rung qualifies where it finds as many findings as reach the
|
||||
bar and misses no 🔴 that both replays on the project's rung found; a replay that failed on it
|
||||
finds nothing. For
|
||||
each replay, claude on the project's rung matches the report to the archived findings, and counts
|
||||
a finding only where the report raises it as 🔴 or 🟡. The tool prints each rung's share, each 🔴
|
||||
a finding only where the report raises it as 🔴 or 🟡. The tool prints each rung's count, each 🔴
|
||||
it missed, and the lowest rung that, like each rung above it, qualifies. That rung is the evidence
|
||||
for running the pass's reviewer on it; the decision above records each run.
|
||||
|
||||
@@ -59,7 +60,7 @@ import typing
|
||||
from transcript_archive.location import archive_root
|
||||
from rung_replay.command import arguments_or_exit, failed, limited_exit, projects_on
|
||||
from review_replay.rounds import authored_by, comparison, lost_inputs, reviewed_path, Round, round_from, sample, stamp_seconds
|
||||
from review_replay.recall import candidates, enough, found_any, Recall, recall_report, Reference, RungRecall, tried
|
||||
from review_replay.recall import candidates, enough, needed, Recall, recall_report, Reference, RungRecall, tried
|
||||
from transcript_archive.project_key import checkout_of
|
||||
from claude_code.reply import claude_reply, Limited
|
||||
from loop_effects.effects import Effects
|
||||
@@ -135,11 +136,11 @@ def recalled(chosen: list[Round], project: Rung, replay_one: typing.Callable[[Ro
|
||||
reference[1].append(second)
|
||||
recalls: list[RungRecall] = []
|
||||
lowest = None
|
||||
for rung in candidates(project) if enough(rounds) and found_any(reference) else []:
|
||||
for rung in candidates(project) if enough(rounds) and needed(rounds, reference) > 0 else []:
|
||||
on_rung: list[set[int] | Failure] | Limited = replayed_on(rounds, rung, replay_one, what)
|
||||
if isinstance(on_rung, Limited):
|
||||
return on_rung
|
||||
# A failed replay on a candidate found nothing, so it lowers that rung's share and never lifts it over the bar.
|
||||
# A failed replay on a candidate found nothing, so it lowers that rung's count and never lifts it to what a rung needs.
|
||||
one = tried(rung, rounds, reference, [set() if isinstance(found, Failure) else found for found in on_rung])
|
||||
recalls.append(one)
|
||||
if not one.qualifies:
|
||||
@@ -222,7 +223,7 @@ def chunk_rounds(reviewer: str, checkout: str, chunk: pathlib.Path) -> list[Roun
|
||||
def code_of(found: Round, effects: Effects) -> Code | Failure:
|
||||
"""Where, and at which commits, found's reviewer read the code, as the archive lets it be found again: the folder rounds.reviewed_path guesses from the prompt, and the commits authored by then.
|
||||
|
||||
Recording the reviewed commit at review time would need no such reading, but then no round archived before that change could be replayed. A wrong guess costs no rung: the reference reviewer, replayed on that code, finds fewer of the archived findings, and no rung is held to those.
|
||||
Recording the reviewed commit at review time would need no such reading, but then no round archived before that change could be replayed. A wrong guess shows every rung the same wrong code: the project's rung finds fewer of that round's archived findings, which lowers the bar, and the rungs below find fewer too.
|
||||
"""
|
||||
if not os.path.isdir(found.root):
|
||||
return Failure("the project has no checkout")
|
||||
|
||||
+31
-26
@@ -1,25 +1,26 @@
|
||||
"""The bar review-replay holds a rung to, and its report: docs/decisions.md → `routing: a review pass runs below the project's rung only on a rung whose replay of its rounds finds a share of their 🔴 and 🟡 findings no more than a margin below the project's rung's own replay`."""
|
||||
import dataclasses
|
||||
from rung_replay.command import down_from
|
||||
from fractions import Fraction
|
||||
|
||||
from review_replay.rounds import is_red, Round
|
||||
from model_routing.ladder import LADDER, Rung
|
||||
from review_replay.rounds import Round
|
||||
import math
|
||||
|
||||
# A rung qualifies with a share of the archived findings up to this far below the project's rung's. The project's rung found 69 to 73% of its own findings again, so about 3 of the 30 to 40 that 10 rounds hold go either way between two replays.
|
||||
# A rung qualifies with a share of the archived findings up to this far below the project's rung's. The project's rung found 69 to 78% of its own findings again, and its two replays of the same rounds differed by up to 4 points.
|
||||
MARGIN = Fraction(1, 10)
|
||||
# The rounds a rung is tried on: 8 of the 10 the tool samples, so up to two may fail on the project's rung.
|
||||
# The fewest rounds a rung is tried on: 8 of the 10 the tool samples, so up to two may fail on the project's rung.
|
||||
ENOUGH_ROUNDS = 8
|
||||
# Lowest first, as Claude Code's --effort takes them.
|
||||
EFFORTS = ("low", "medium", "high", "xhigh", "max")
|
||||
# The two replays on the project's rung: each round's indexes of its findings that each found.
|
||||
# The two replays on the project's rung. Each lists, round by round, the indexes of the findings it found.
|
||||
Reference = tuple[list[set[int]], list[set[int]]]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class RungRecall:
|
||||
"""A rung's replay: its share of the archived findings, and each 🔴 it missed that the project's rung found both times, as a round and the finding's index."""
|
||||
"""A rung's replay: how many of the archived findings it found, and each 🔴 it missed that both of the project's rung's replays found, as a round and the finding's index."""
|
||||
rung: Rung
|
||||
share: Fraction
|
||||
found: int
|
||||
missed_red: list[tuple[Round, int]]
|
||||
qualifies: bool
|
||||
|
||||
@@ -37,48 +38,52 @@ class Recall:
|
||||
def candidates(project: Rung) -> list[Rung]:
|
||||
"""The rungs tried below the project's, highest first: its model at each lower effort, which the ladder leaves out, then each ladder rung below it, down to the ladder's lowest. That reaches below the lowest a worker routes to, since a pass may run below the worker's rung."""
|
||||
lower_efforts = [Rung(project.model, effort) for effort in reversed(EFFORTS[:EFFORTS.index(project.effort)])]
|
||||
return lower_efforts + list(LADDER[:LADDER.index(project)])[::-1]
|
||||
return lower_efforts + down_from(project, LADDER[0])[1:]
|
||||
|
||||
|
||||
def enough(rounds: list[Round]) -> bool:
|
||||
return len(rounds) >= ENOUGH_ROUNDS
|
||||
|
||||
|
||||
def found_any(reference: Reference) -> bool:
|
||||
return any(map(any, reference))
|
||||
def count(found: list[set[int]]) -> int:
|
||||
return sum(map(len, found))
|
||||
|
||||
|
||||
def share(rounds: list[Round], found: list[set[int]]) -> Fraction:
|
||||
"""The share of rounds' findings that found holds; rounds is never empty, and each holds a finding."""
|
||||
return Fraction(sum(map(len, found)), sum(len(one.findings) for one in rounds))
|
||||
def held(rounds: list[Round]) -> int:
|
||||
"""How many findings rounds hold."""
|
||||
return sum(len(one.findings) for one in rounds)
|
||||
|
||||
|
||||
def bar(rounds: list[Round], reference: Reference) -> Fraction:
|
||||
"""The share a rung needs: the mean of the project's rung's two shares, less MARGIN."""
|
||||
return (share(rounds, reference[0]) + share(rounds, reference[1])) / 2 - MARGIN
|
||||
def mean_share(rounds: list[Round], reference: Reference) -> Fraction:
|
||||
"""The mean share of rounds' findings that the project's rung's two replays found; rounds is never empty, and each holds a finding."""
|
||||
return Fraction(count(reference[0]) + count(reference[1]), 2 * held(rounds))
|
||||
|
||||
|
||||
def needed(rounds: list[Round], reference: Reference) -> int:
|
||||
"""How many of rounds' findings a rung needs to find: the fewest that reach the project's rung's mean share less MARGIN. Zero or below, no rung is tried, since a rung that found nothing would qualify."""
|
||||
return math.ceil((mean_share(rounds, reference) - MARGIN) * held(rounds))
|
||||
|
||||
|
||||
def tried(rung: Rung, rounds: list[Round], reference: Reference, found: list[set[int]]) -> RungRecall:
|
||||
"""rung's recall, from the indexes of each round's findings it found. It qualifies at the bar or above, missing no 🔴 the project's rung found both times; where the project's rung found nothing, no rung qualifies."""
|
||||
missed_red = [(one, index) for one, first, second, got in zip(rounds, *reference, found) for index in sorted((first & second) - got) if "🔴" in one.findings[index].splitlines()[0]]
|
||||
got = share(rounds, found)
|
||||
return RungRecall(rung, got, missed_red, found_any(reference) and got >= bar(rounds, reference) and not missed_red)
|
||||
"""rung's recall, from the indexes of each round's findings it found. It qualifies finding as many as needed, missing no 🔴 that both of the project's rung's replays found."""
|
||||
missed_red = [(one, index) for one, first, second, got in zip(rounds, *reference, found) for index in sorted((first & second) - got) if is_red(one.findings[index])]
|
||||
return RungRecall(rung, count(found), missed_red, 0 < needed(rounds, reference) <= count(found) and not missed_red)
|
||||
|
||||
|
||||
def recall_report(reviewed: str, project: Rung, recall: Recall) -> str:
|
||||
rounds, (first, second) = recall.rounds, recall.reference
|
||||
lines = [
|
||||
f"Replayed {len(rounds)} {reviewed} rounds of {project.model} projects, which hold {sum(len(one.findings) for one in rounds)} 🔴 and 🟡 findings, twice on the project's rung, {project}.",
|
||||
f"Replayed {len(rounds)} {reviewed} rounds of {project.model} projects, which hold {held(rounds)} 🔴 and 🟡 findings, twice on the project's rung, {project}.",
|
||||
*([f"Left out {recall.left_out} more, which a replay on the project's rung failed on; stderr says why."] if recall.left_out else []),
|
||||
]
|
||||
if not enough(rounds):
|
||||
return "\n".join([*lines, f"No rung tried: fewer than {ENOUGH_ROUNDS} rounds were replayed on the project's rung without a failure."])
|
||||
if not found_any(recall.reference):
|
||||
return "\n".join([*lines, "No rung tried: the project's rung found none of them."])
|
||||
lines.append(f"The project's rung found {sum(map(len, first))} and then {sum(map(len, second))} of them, {percent(bar(rounds, recall.reference) + MARGIN)}, so a rung qualifies at {percent(bar(rounds, recall.reference))} or more and missing no 🔴 it found both times.")
|
||||
if needed(rounds, recall.reference) <= 0:
|
||||
return "\n".join([*lines, "No rung tried: the project's rung found too few of them to hold a rung to."])
|
||||
lines.append(f"The project's rung found {count(first)} and then {count(second)} of the {held(rounds)}, {percent(mean_share(rounds, recall.reference))} on average, so a rung qualifies finding {needed(rounds, recall.reference)} or more of them, missing no 🔴 that both of its replays found.")
|
||||
for one in recall.tried:
|
||||
verdict = ": it qualifies." if one.qualifies else ":" if one.missed_red else "."
|
||||
lines += [f"{one.rung} found {percent(one.share)} and missed {len(one.missed_red) or 'no'} such 🔴{verdict}", *(f" {at.root} {at.at}: {at.findings[index].splitlines()[0]}" for at, index in one.missed_red)]
|
||||
verdict = "it qualifies" if one.qualifies else "it doesn't qualify"
|
||||
lines += [f"{one.rung} found {one.found}, {percent(Fraction(one.found, held(rounds)))}, and missed {len(one.missed_red) or 'no'} such 🔴: {verdict}.", *(f" {at.root} {at.at}: {at.findings[index].splitlines()[0]}" for at, index in one.missed_red)]
|
||||
lowest = recall.lowest or "none below the project's rung"
|
||||
lines.append(f"Lowest rung that qualifies: {lowest}")
|
||||
return "\n".join(lines)
|
||||
|
||||
@@ -33,6 +33,12 @@ class Round:
|
||||
return stamp_seconds(self.at)
|
||||
|
||||
|
||||
def is_red(finding: str) -> bool:
|
||||
"""Whether finding, which opens with a line FINDING matches, is raised as 🔴: by the mark it opens with, whatever marks its text names."""
|
||||
mark = FINDING.match(finding)
|
||||
return mark is not None and mark[0].endswith("🔴")
|
||||
|
||||
|
||||
def round_from(reviewer: str, root: str, began: float, entries: list[Entry]) -> Round | None:
|
||||
"""The round a reviewer's transcript holds: its first entry's prompt, and its last text. None without a 🔴 or 🟡 finding to hold a rung to."""
|
||||
reports = [part.text for entry in entries if isinstance(entry, AssistantTurn) for part in entry.parts if isinstance(part, Text) and part.text]
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
from fractions import Fraction
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from review_replay.recall import candidates, Recall, recall_report, tried
|
||||
from review_replay.rounds import Round
|
||||
from review_replay.recall import candidates, needed, Recall, recall_report, tried
|
||||
from review_replay.rounds import is_red, Round
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
REFERENCE = Rung("claude-opus-5-5", "medium")
|
||||
@@ -18,10 +17,12 @@ def a_round(at, findings):
|
||||
return Round("prose-reviewer", "/code/p", "x", f"2026-10-14T10:00:0{at}Z", 0, "Review it.", tuple(findings))
|
||||
|
||||
|
||||
# Ten findings over two rounds; the project's rung found 8 and then 6, so its share is 70% and the bar 60%. Both found the first round's 🔴, and only the first found the second's.
|
||||
# Ten findings over two rounds; the project's rung found 8 and then 6, so its share is 70%, the bar 60% and a rung needs 6. Both found the first round's 🔴, and only the first found the second's.
|
||||
ROUNDS = [a_round(1, ("🔴 a", "🟡 b", "🟡 c", "🟡 d", "🟡 e")), a_round(2, ("🔴 f\nmore", "🟡 g", "🟡 h", "🟡 i", "🟡 j"))]
|
||||
FIRST = [{0, 1, 2, 3}, {0, 1, 2, 3}]
|
||||
SECOND = [{0, 1, 2}, {1, 2, 3}]
|
||||
# The project's rung found 1 and then 1 of the 10, so the bar is zero.
|
||||
TENTH = ([{0}, set()], [{0}, set()])
|
||||
|
||||
|
||||
class Candidates(unittest.TestCase):
|
||||
@@ -38,23 +39,40 @@ class Candidates(unittest.TestCase):
|
||||
class Tried(unittest.TestCase):
|
||||
def test_qualifies_a_rung_no_more_than_the_margin_below_the_project_rungs_share_that_misses_no_red_found_both_times(self):
|
||||
found = tried(LOW, ROUNDS, (FIRST, SECOND), [{0, 1, 2}, {0, 1, 2}])
|
||||
self.assertEqual((found.share, found.missed_red, found.qualifies), (Fraction(3, 5), [], True))
|
||||
self.assertEqual((found.found, found.missed_red, found.qualifies), (6, [], True))
|
||||
|
||||
def test_fails_a_rung_below_the_bar(self):
|
||||
found = tried(LOW, ROUNDS, (FIRST, SECOND), [{0, 1}, {0, 3}])
|
||||
self.assertEqual((found.share, found.missed_red, found.qualifies), (Fraction(2, 5), [], False))
|
||||
self.assertEqual((found.found, found.missed_red, found.qualifies), (4, [], False))
|
||||
|
||||
def test_fails_a_rung_that_misses_a_red_found_both_times_and_only_such_a_red(self):
|
||||
above_the_bar = [{1, 2, 3, 4}, {0, 1, 2, 3}]
|
||||
found = tried(LOW, ROUNDS, (FIRST, SECOND), above_the_bar)
|
||||
self.assertEqual((found.share, found.missed_red, found.qualifies), (Fraction(4, 5), [(ROUNDS[0], 0)], False))
|
||||
self.assertEqual((found.found, found.missed_red, found.qualifies), (8, [(ROUNDS[0], 0)], False))
|
||||
# The first round's 🔴 is found both times; a yellow found both times, such as its b, may be missed.
|
||||
self.assertTrue(tried(LOW, ROUNDS, (FIRST, SECOND), [{0, 2, 3, 4}, {0, 2, 3, 4}]).qualifies)
|
||||
# The second round's 🔴, which only the first replay found, may be missed too.
|
||||
self.assertTrue(tried(LOW, ROUNDS, (FIRST, SECOND), [{0, 1, 2, 3}, {1, 2, 3}]).qualifies)
|
||||
|
||||
def test_fails_every_rung_where_the_project_rung_found_nothing(self):
|
||||
def test_fails_every_rung_where_the_bar_is_zero_or_below(self):
|
||||
self.assertFalse(tried(LOW, ROUNDS, ([set(), set()], [set(), set()]), [{0}, {0}]).qualifies)
|
||||
self.assertFalse(tried(LOW, ROUNDS, TENTH, [{0}, {0}]).qualifies)
|
||||
|
||||
def test_rounds_the_findings_a_rung_needs_up(self):
|
||||
# 33 findings, of which the project's rung found 24 and then 23: the bar is 61.2%, which 20 findings, 60.6%, don't reach.
|
||||
rounds = [a_round(1, ["🟡 a"] * 33)]
|
||||
reference = ([set(range(24))], [set(range(23))])
|
||||
self.assertEqual(needed(rounds, reference), 21)
|
||||
self.assertFalse(tried(LOW, rounds, reference, [set(range(20))]).qualifies)
|
||||
self.assertTrue(tried(LOW, rounds, reference, [set(range(21))]).qualifies)
|
||||
|
||||
def test_lets_a_rung_miss_a_yellow_that_mentions_a_red(self):
|
||||
rounds = [a_round(1, ("### 🟡 `SKILL.md:80`: the rule says a 🔴 in round 2 escalates", "🟡 b"))]
|
||||
self.assertFalse(is_red(rounds[0].findings[0]))
|
||||
self.assertTrue(is_red("### 1. **🔴 a\nwhich is no 🟡"))
|
||||
found = tried(LOW, rounds, ([{0, 1}], [{0, 1}]), [{1}])
|
||||
self.assertEqual((found.missed_red, found.qualifies), ([], False))
|
||||
self.assertEqual(tried(LOW, rounds, ([{0, 1}], [{0}]), [{1}]).missed_red, [])
|
||||
|
||||
|
||||
class RecallReport(unittest.TestCase):
|
||||
@@ -69,9 +87,9 @@ class RecallReport(unittest.TestCase):
|
||||
self.assertEqual(lines, [
|
||||
"Replayed 8 prose rounds of claude-opus-5-5 projects, which hold 40 🔴 and 🟡 findings, twice on the project's rung, claude-opus-5-5 at medium effort.",
|
||||
"Left out 1 more, which a replay on the project's rung failed on; stderr says why.",
|
||||
"The project's rung found 32 and then 24 of them, 70%, so a rung qualifies at 60% or more and missing no 🔴 it found both times.",
|
||||
"claude-opus-5-5 at low effort found 60% and missed no such 🔴: it qualifies.",
|
||||
"claude-sonnet-5-5 at high effort found 50% and missed 4 such 🔴:",
|
||||
"The project's rung found 32 and then 24 of the 40, 70% on average, so a rung qualifies finding 24 or more of them, missing no 🔴 that both of its replays found.",
|
||||
"claude-opus-5-5 at low effort found 24, 60%, and missed no such 🔴: it qualifies.",
|
||||
"claude-sonnet-5-5 at high effort found 20, 50%, and missed 4 such 🔴: it doesn't qualify.",
|
||||
*[" /code/p 2026-10-14T10:00:01Z: 🔴 a"] * 4,
|
||||
"Lowest rung that qualifies: claude-opus-5-5 at low effort",
|
||||
])
|
||||
@@ -80,14 +98,15 @@ class RecallReport(unittest.TestCase):
|
||||
few = recall_report("prose", REFERENCE, Recall(ROUNDS, (FIRST, SECOND), 0, [], None)).splitlines()
|
||||
self.assertEqual(few[1:], ["No rung tried: fewer than 8 rounds were replayed on the project's rung without a failure."])
|
||||
rounds, _ = self.EIGHT
|
||||
none_found = recall_report("prose", REFERENCE, Recall(rounds, ([set()] * 8, [set()] * 8), 0, [], None)).splitlines()
|
||||
self.assertEqual(none_found[1:], ["No rung tried: the project's rung found none of them."])
|
||||
for reference in (([set()] * 8, [set()] * 8), (TENTH[0] * 4, TENTH[1] * 4)):
|
||||
too_few = recall_report("prose", REFERENCE, Recall(rounds, reference, 0, [], None)).splitlines()
|
||||
self.assertEqual(too_few[1:], ["No rung tried: the project's rung found too few of them to hold a rung to."])
|
||||
|
||||
def test_names_no_rung_where_none_qualifies(self):
|
||||
rounds, reference = self.EIGHT
|
||||
low = tried(LOW, rounds, reference, [{0}, {0}] * 4)
|
||||
lines = recall_report("prose", REFERENCE, Recall(rounds, reference, 0, [low], None)).splitlines()
|
||||
self.assertEqual(lines[-2:], ["claude-opus-5-5 at low effort found 20% and missed no such 🔴.", "Lowest rung that qualifies: none below the project's rung"])
|
||||
self.assertEqual(lines[-2:], ["claude-opus-5-5 at low effort found 8, 20%, and missed no such 🔴: it doesn't qualify.", "Lowest rung that qualifies: none below the project's rung"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -305,8 +305,8 @@ class Main(unittest.TestCase):
|
||||
self.assertEqual((reviewed, branch, base), ("1\n", "x\n", "0\n"))
|
||||
self.assertEqual(prompt, f"Review branch x in {cwd}.")
|
||||
self.assertFalse(pathlib.Path(cwd).exists())
|
||||
self.assertIn("The project's rung found 2 and then 2 of them, 100%, so a rung qualifies at 90% or more", out)
|
||||
self.assertIn(f"claude-opus-5-5 at low effort found 50% and missed 1 such 🔴:\n {self.checkout} 2026-10-14T10:00:00Z: 🔴 a.txt:1 wrong\n", out)
|
||||
self.assertIn("The project's rung found 2 and then 2 of the 2, 100% on average, so a rung qualifies finding 2 or more of them", out)
|
||||
self.assertIn(f"claude-opus-5-5 at low effort found 1, 50%, and missed 1 such 🔴: it doesn't qualify.\n {self.checkout} 2026-10-14T10:00:00Z: 🔴 a.txt:1 wrong\n", out)
|
||||
self.assertTrue(out.endswith("qualifies: none below the project's rung\n"), out)
|
||||
self.assertEqual(set(self.matched_in), {str(self.checkout)})
|
||||
self.assertTrue(out.startswith("Replayed 1 stability rounds of claude-opus-5-5 projects,"), out)
|
||||
@@ -329,7 +329,7 @@ class Main(unittest.TestCase):
|
||||
self.assertIn(f"replaying 1 prose rounds on {REFERENCE}\n", err)
|
||||
self.assertIn(f"replaying 1 prose rounds on {LOW}\n", err)
|
||||
self.assertIn(f"on {LOW} failed: the reviewer replied with no report", err)
|
||||
self.assertIn("claude-opus-5-5 at low effort found 0% and missed 1 such 🔴:", out)
|
||||
self.assertIn("claude-opus-5-5 at low effort found 0, 0%, and missed 1 such 🔴: it doesn't qualify.", out)
|
||||
|
||||
def review_folder_with_round_1(self):
|
||||
"""The chunk's review folder, holding round 1's report from before the replayed round began at 10:00, that round's own and a later pass's reports, written after, and an earlier attempt's report, from before the chunk began at 09:15."""
|
||||
@@ -405,7 +405,7 @@ class Main(unittest.TestCase):
|
||||
self.assertEqual(self.reviews, [])
|
||||
self.assertIn(f"on {REFERENCE} failed: the prompt names branch.diff, which the review folder did not hold from the chunk's start until the round's", err)
|
||||
|
||||
def test_finds_nothing_and_leaves_no_folder_behind_when_it_cannot_copy_the_reports(self):
|
||||
def test_fails_the_replay_and_leaves_no_folder_behind_when_it_cannot_copy_the_reports(self):
|
||||
folder = self.review_folder_with_round_1()
|
||||
(folder / "stability-round-1.md").chmod(0)
|
||||
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to ~/.local/state/larv-review/code/proj-x/stability-round-2.md.")
|
||||
@@ -427,18 +427,20 @@ class Main(unittest.TestCase):
|
||||
def test_leaves_out_a_round_that_either_replay_on_the_project_rung_failed_on(self):
|
||||
self.write_chunk("proj", "c2", "2026-10-14T10:30:00Z")
|
||||
self.enterContext(unittest.mock.patch.object(search, "AT_ONCE", 1))
|
||||
calls = iter(range(100))
|
||||
# One round at a time, the first replay's three tries on the first round reply with no report; every other review replies.
|
||||
code, out, err = self.run_main(["prose", "claude-opus-5-5"], lambda rung: {"found": [0, 1]}, review=lambda rung: {"type": "result"} if next(calls) < 3 else {"type": "result", "result": f"review on {rung.model} {rung.effort}"})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertTrue(out.startswith("Replayed 1 prose rounds"), out)
|
||||
self.assertIn("Left out 1 more", out)
|
||||
# One round at a time, three tries on a round reply with no report: the first replay's on its first round, then the second replay's on its first, after the first replay's two reviews. Every other review replies.
|
||||
for failing in (range(3), range(2, 5)):
|
||||
with self.subTest(failing=failing):
|
||||
calls = iter(range(100))
|
||||
code, out, err = self.run_main(["prose", "claude-opus-5-5"], lambda rung: {"found": [0, 1]}, review=lambda rung: {"type": "result"} if next(calls) in failing else {"type": "result", "result": f"review on {rung.model} {rung.effort}"})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertTrue(out.startswith("Replayed 1 prose rounds"), out)
|
||||
self.assertIn("Left out 1 more", out)
|
||||
|
||||
def test_tries_no_lower_rung_where_the_project_rung_found_nothing(self):
|
||||
code, out, err = self.run_main(["prose", "claude-opus-5-5"], lambda rung: {"found": []})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual([rung for rung, *_ in self.reviews], [REFERENCE, REFERENCE])
|
||||
self.assertTrue(out.endswith("No rung tried: the project's rung found none of them.\n"), out)
|
||||
self.assertTrue(out.endswith("No rung tried: the project's rung found too few of them to hold a rung to.\n"), out)
|
||||
|
||||
def test_checks_out_the_commit_detached_when_the_reviewer_started_on_the_default_branch(self):
|
||||
self.set_branch("main")
|
||||
|
||||
+5
-5
@@ -391,15 +391,15 @@ Valid until the trial on real chunks shows whether a worker below the project's
|
||||
|
||||
2026-10-29, lilleman. Serves goals 1.4 and 1.5. A branch that changes only Markdown no agent follows reviews on the project's rung, whatever its items' E + I. E + I scores an item before its chunk starts, by the change the item asks for, so one project's item, filed to build a control on a CI runner, kept its E + I of 7 when the chunk found no control the repo could build and shipped a README paragraph: five reviewers read prose on Opus 5.5 at xhigh. rule-hook's `reviewer-rung` reads the branch's files, committed or not, against the default branch as last fetched. Markdown an agent follows counts as config: `CLAUDE.md`, `AGENTS.md`, `workflow-bench/*-prompt.md` and files under `agents/`, `principles/`, `rules/` or `skills/`. On 2026-11-04 lilleman added the workflow bench's prompts, which an agent follows like a skill. `bin/review_rules/review_passes.py` holds the lists, and the worker's prompt renders them. Valid while a README or docs defect costs a reader no more than review on the project's rung lets through.
|
||||
|
||||
## routing: a review pass runs below the project's rung only on a rung whose replay of its rounds finds a share of their 🔴 and 🟡 findings no more than a margin below the project's rung's own replay, and misses no 🔴 the project's rung found both times; and only where the E + I steps no review up and the worker runs no higher than the project's rung, then even below the worker's; no pass has such a rung
|
||||
## routing: a review pass runs below the project's rung only on a rung whose replay of its rounds finds a share of their 🔴 and 🟡 findings no more than a margin below the project's rung's own replay, and misses no 🔴 that both of the project's rung's replays found; and only where the E + I steps no review up and the worker runs no higher than the project's rung, then even below the worker's; no pass has such a rung
|
||||
|
||||
2026-10-15, lilleman proposed running the stability reviewer on the worker's rung, and the agent proposed the replay that gates it. On 2026-10-16 lilleman asked for the prose pass to run on the worker's rung or lower, and, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. On 2026-11-11 lilleman approved the bar below, as somewhat risky, with a close evaluation a release later: `todo.md` item 321. Serves goal 1.4 without giving up goal 1.1: a cheaper reviewer saves money only where it finds as much.
|
||||
2026-10-15, lilleman proposed running the stability reviewer on the worker's rung, and the agent proposed the replay that gates it. On 2026-10-16 lilleman asked for the prose pass to run on the worker's rung or lower, and, answering PR #296, for every pass to drop below the project's rung only where a replay shows it may. On 2026-11-11 lilleman approved the bar below, as somewhat risky, with a close evaluation a release later: `todo.md` item 321. Serves goal 1.4 without giving up goal 1.1: a cheaper reviewer saves money only where it finds nearly as much.
|
||||
|
||||
`bin/review-replay <pass> <model>` replays a pass's archived rounds on the code each reviewed, twice on the project's rung. A rung below qualifies where its replay finds at least the mean share of the archived findings that those two replays found, less 10 percentage points, over at least 8 rounds, and misses no 🔴 that both found. The tool tries the project's model at each lower effort first, since the ladder leaves those out, then each ladder rung below, highest first. It stops at the first rung that doesn't qualify.
|
||||
|
||||
The bar used to be every finding the project's rung found again. But the project's rung itself found only 69 to 73% of its own findings again, so a reviewer exactly as good failed that bar, and so would the project's rung on a second replay. The margin of 10 points allows for that spread: 10 rounds hold 30 to 40 findings, of which about 3 go either way between two replays. A missed 🔴 still fails a rung, since a 🔴 that both replays found is no matter of chance.
|
||||
The bar used to be every finding the project's rung found again. But the project's rung itself found only 69 to 73% of its own findings again, so a reviewer exactly as good failed that bar, and so would the project's rung on a second replay. The margin of 10 points allows for that spread: in the latest run below, the project's rung's two replays found 41 and 43 of the same 55 findings, 4 points apart. A missed 🔴 still fails a rung, since a 🔴 that both replays found is no matter of chance.
|
||||
|
||||
A qualifying rung applies only where the chunk's E + I steps no review up and the worker runs no higher than the project's rung; there the pass runs on that rung, even below the worker's. Otherwise the pass runs where every other pass does: the replay compared reviews on the project's rung only, never reviews on a rung an E + I stepped up, nor of a chunk whose worker climbed above the project's rung. No code lowers a pass while no rung qualifies: on 2026-11-04 lilleman had the release review of 0.6.0 delete that code, which no chunk could reach. The chunk that first qualifies a rung writes that code again in `bin/model_routing/`. It records the qualifying rung for each pass and project model, and names the lowered pass in the worker's prompt and in rule-hook's `reviewer-rung`. A rung off the ladder, such as the project's model at a lower effort, would also need that reviewer's `effort` frontmatter key, which sets the effort per reviewer.
|
||||
A qualifying rung applies only where the chunk's E + I steps no review up and the worker runs no higher than the project's rung; there the pass runs on that rung, even below the worker's. Otherwise the pass runs where every other pass does: the replay compared reviews on the project's rung only, never reviews on a rung an E + I stepped up, nor of a chunk whose worker climbed above the project's rung. No code lowers a pass while no rung qualifies: on 2026-11-04 lilleman had the release review of 0.6.0 delete that code, which no chunk could reach. The chunk that first qualifies a rung writes that code again in `bin/model_routing/`. It records the qualifying rung for each pass and project model, and names the lowered pass in the worker's prompt and in rule-hook's `reviewer-rung`. For a rung off the ladder, such as the project's model at a lower effort, that chunk also sets the `effort` key in the reviewer's frontmatter under `agents/`, which sets the effort per reviewer.
|
||||
|
||||
The runs, each on the latest 10 rounds of Opus 5.5 projects that hold a 🔴 or 🟡:
|
||||
|
||||
@@ -408,7 +408,7 @@ The runs, each on the latest 10 rounds of Opus 5.5 projects that hold a 🔴 or
|
||||
- 2026-10-18, stability again, under the old bar, since that day's change to the reviewer prompts expired the first run: 29 findings. The project's rung found 20 of them again. Sonnet 5.5 at high missed 11 of those 20, 1 of them 🔴. No rung qualifies.
|
||||
- 2026-11-12, prose, under the new bar: 8 rounds, 55 findings, after 2 more rounds failed on the project's rung. The project's rung found 41 and then 43 of them, 76%, so the bar was 66%. Opus 5.5 at low effort found 27, 49%, and missed no 🔴 that both replays found. No rung qualifies, and Sonnet 5.5 at high was not tried.
|
||||
|
||||
The prose pass went first under each bar: a missed prose finding breaks no behaviour, and a later change fixes it cheaply. Sonnet 5.5 missed 44 to 55% of what the project's rung found again, in the prose pass as in both stability runs, so it is unlikely to reach the new bar either; and Opus 5.5 at low, the likeliest rung, fell 17 points short on the pass whose misses cost least. The architecture, principles, overrides and product-owner passes are not replayed yet: `todo.md` item 196 keeps their replays in the plan.
|
||||
The prose pass went first under the new bar: a missed prose finding breaks no behaviour, and a later change fixes it cheaply. Sonnet 5.5 missed 44 to 55% of what the project's rung found again, in the prose pass as in both stability runs, so it is unlikely to reach the new bar either; and Opus 5.5 at low, the likeliest rung, fell 17 points short on the pass whose misses cost least. The architecture, principles, overrides and product-owner passes are not replayed yet: `todo.md` item 196 keeps their replays in the plan.
|
||||
|
||||
Valid until a new model joins the ladder, which calls for a replay of every pass, or a pass's reviewer prompt changes, which calls for a replay of that pass.
|
||||
|
||||
|
||||
@@ -242,7 +242,7 @@ Item 159's chunk, PR #322, replayed only the prose and stability passes. Sonnet
|
||||
|
||||
### 321. Evaluate the review passes that run on a rung `review-replay` qualified, against the chunks before them, and revert any pass whose findings or later fixes worsened.
|
||||
|
||||
Item 318's replay of the prose pass, on 2026-11-12, qualified no rung, so this item waits for item 196's replays of the other passes. lilleman, 2026-11-11: item 318's new bar is a risk, so check it closely a release later. Compare the first 20 chunks reviewed on each newly qualified rung with the 20 before: review rounds past the first, 🔴 and 🟡 per chunk, misread findings, and later fixes that `score-outcomes` counts (defects that escaped review). Revert a pass to the project's rung where any of those worsened beyond the spread of the 20 before.
|
||||
The prose pass's replay under item 318's new bar, on 2026-11-12, qualified no rung, so this item waits for item 196's replays of the other passes. lilleman, 2026-11-11: that bar is a risk, so check it closely a release later. Compare the first 20 chunks reviewed on each newly qualified rung with the 20 before: review rounds past the first, 🔴 and 🟡 per chunk, misread findings, and later fixes that `score-outcomes` counts (defects that escaped review). Revert a pass to the project's rung where any of those worsened beyond the spread of the 20 before.
|
||||
|
||||
### 273. Re-tune the E + I steps of `REVIEW_STEPS` for 6 to 8 once 10 finished chunks have scored an E + I of 6 or more, from how well E + I orders their review rounds and 🔴 findings.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user