Answer prose round 1: 0 🔴, 6 🟡 (0 misread); the decision's judge rule stated once, in the present tense
CI / tests (pull_request) Successful in 2m34s
CI / tests (pull_request) Successful in 2m34s
This commit was merged in pull request #499.
This commit is contained in:
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- loop-claude's judge on an Opus 5.5 project now runs on Opus 5.5 at xhigh in every chunk, not at medium below an E + I of 5 and never on Fable 5.2: a replay found medium returning another kind on quoted answers that xhigh returns, and each Fable 5.2 rung it tried, at low, medium and high, missing an interrupt. `judge-replay` now also replays each rung above the project's that a review climbs to, and reports what each misses.
|
||||
- loop-claude's judge on an Opus 5.5 project now runs on Opus 5.5 at xhigh in every chunk, never at medium: in replays medium sometimes closed a chunk whose worker had asked a question the README or records answer, where xhigh quoted the answer. The judge never runs on Fable 5.2 either, since each Fable 5.2 rung replayed missed an interrupt. `judge-replay` now also replays each rung above the project's that a review climbs to, and reports what each misses.
|
||||
|
||||
## 0.8.0
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ The worker, and the session /talk opens, run on a Route (model_routing/route.py)
|
||||
- Start. routed_item finds the plan item that routes a fresh chunk, if there is one with L, E and I scored; first_route picks the worker's rung from its L, or the project's rung without an item.
|
||||
- Rerun. kept decides whether a rerun resumes the route its archive last names.
|
||||
- Step-up. next_route raises the route after an unfinished verdict (unfinished_route), or an interrupt or a quoted answer that carries an escalation (escalated_route); escalation_cap bounds an escalation, and loop_claude/escalation_mark.py states the escalation mark's rule. step_up names the change where it changes the worker line, which the archive records.
|
||||
- Judge. judge_rung picks the rung of the judge of a JUDGE_REPLAYED model: its model's highest.
|
||||
- Judge. judge_rung picks the rung of the judge of a JUDGE_REPLAYED model.
|
||||
"""
|
||||
import dataclasses
|
||||
|
||||
@@ -20,7 +20,7 @@ from model_routing.route import Route
|
||||
UNFINISHED_PER_STEP = 2
|
||||
# How many rungs above item_rung an escalation trigger takes the worker or a task without the human's yes, as escalation_cap counts them.
|
||||
ESCALATION_CAP = 2
|
||||
# The models whose cheapest rung, in a `judge-replay` run, returned every interrupt the reference judge reproduced, and whose judge runs on their highest rung: docs/decisions.md → `routing: on a model judge-replay covered`.
|
||||
# The models whose cheapest rung, in a `judge-replay` run, returned every interrupt the reference judge reproduced: docs/decisions.md → `routing: on a model judge-replay covered`.
|
||||
JUDGE_REPLAYED = ("claude-opus-5-5",)
|
||||
|
||||
|
||||
@@ -138,7 +138,7 @@ def one_rung_up(route: Route, rung: Rung, escalated_at: int) -> Route:
|
||||
|
||||
|
||||
def judge_rung(judge_model: str) -> Rung:
|
||||
"""The highest of the judge model's own rungs, whatever the chunk's items score: docs/decisions.md → `routing: on a model judge-replay covered`. A model off the ladder, or not in JUDGE_REPLAYED, gets no effort flag, so claude runs it at the effort its settings give."""
|
||||
"""The highest of the judge model's own rungs: docs/decisions.md → `routing: on a model judge-replay covered`. A model off the ladder, or not in JUDGE_REPLAYED, gets no effort flag, so claude runs it at the effort its settings give."""
|
||||
if project_rung(judge_model) is None or judge_model not in JUDGE_REPLAYED:
|
||||
return Rung(judge_model)
|
||||
return highest(*(rung for rung in LADDER if rung.model == judge_model))
|
||||
|
||||
@@ -1134,7 +1134,6 @@ class Routing(unittest.TestCase):
|
||||
self.assertEqual(routing.next_route(self.OPUS, Route(Rung("claude-opus-5-5", "low")), unfinished(2)).rung, self.OPUS)
|
||||
|
||||
def test_runs_the_judge_of_a_replayed_model_on_its_models_highest_rung(self):
|
||||
# Opus 5.5 at medium returned another kind on quoted answers that xhigh returns, so no chunk's judge runs below xhigh.
|
||||
self.assertEqual(routing.judge_rung("claude-opus-5-5"), self.THREE_UP)
|
||||
# A model off the ladder, or not replayed, gets no effort flag.
|
||||
self.assertEqual(routing.judge_rung("judge-model"), Rung("judge-model"))
|
||||
|
||||
+6
-6
@@ -442,7 +442,7 @@ Since 2026-10-16 the reviewers may run above the worker's rung, on the one the h
|
||||
|
||||
## routing: loop-claude reads the items a chunk takes from the worker's latest `Chunk items:` line and scores them by `todo.md` as the default branch held it when the cycle started; the judge reports the escalation and whether a release is ready to cut
|
||||
|
||||
2026-11-11, claude. Serves goal 1.4 and code health. The judge's reply held three facts beside its verdict: the items the worker took into the chunk, with their highest L and E + I; whether a release was ready to cut; and any escalation the worker asked for. Routing lifts the worker's, the reviewers' and the judge's rungs by the items' levels. Code now reads the items: each chunk's first prompt, and each prompt resuming it, asks the worker for a line opening `Chunk items:`, `bin/loop_claude/cycle/chunk_items.py` reads the latest such line in the worker's own text, and `release_plan.todo_table` scores the items. So the judge's reply is smaller, and every L and E + I that routing reads comes from `todo.md`, through one module.
|
||||
2026-11-11, claude. Serves goal 1.4 and code health. The judge's reply held three facts beside its verdict: the items the worker took into the chunk, with their highest L and E + I; whether a release was ready to cut; and any escalation the worker asked for. Routing lifts the worker's and the reviewers' rungs by the items' levels. Code now reads the items: each chunk's first prompt, and each prompt resuming it, asks the worker for a line opening `Chunk items:`, `bin/loop_claude/cycle/chunk_items.py` reads the latest such line in the worker's own text, and `release_plan.todo_table` scores the items. So the judge's reply is smaller, and every L and E + I that routing reads comes from `todo.md`, through one module.
|
||||
|
||||
The judge field trials (`judge-field-trials/`) read each fact by code over the 709 archived cases. Code was wrong in each of the 39 cases where it and the judge disagreed on an escalation, and read a release ready to cut in 3 of the 31 interrupts that asked about a cut, so the judge keeps both. Workers stated an item with its L and E + I in 413 cases, but also stated items they filed, finished or only mentioned, so the worker writes the items on a line that holds only them, which code reads. Three fresh readers picked its label `Chunk items:`, 3 to 0, because it says the items are the chunk's.
|
||||
|
||||
@@ -452,17 +452,17 @@ Valid while workers write the line when asked, and while `todo.md` on the defaul
|
||||
|
||||
## routing: on a model judge-replay covered, the judge runs on its model's highest rung
|
||||
|
||||
2026-10-15, lilleman's plan, settled by the replays below. Serves goals 1.2 and 1.4, under the ladder and steps of `routing: one ladder`.
|
||||
2026-10-15, lilleman's plan, settled by the replays below. Serves goals 1.1, 1.2 and 1.4, under the ladder and steps of `routing: one ladder`.
|
||||
|
||||
On a model listed in `JUDGE_REPLAYED` in `bin/loop_claude/routing.py`, the judge runs on its model's highest rung, whatever the chunk's items score: on an Opus 5.5 project, Opus 5.5 at xhigh. It keeps its rung when the worker steps up. Until 2026-11-12 it ran at medium below an E + I of 5 and at xhigh from 5, climbing with the highest E + I among the routed item and those the worker took in, as a review does. Each archived verdict names the judge's rung on a `Judge:` line.
|
||||
On a model listed in `JUDGE_REPLAYED` in `bin/loop_claude/routing.py`, the judge runs on its model's highest rung, whatever the chunk's items score: on an Opus 5.5 project, Opus 5.5 at xhigh. It keeps its rung when the worker steps up. Each archived verdict names the judge's rung on a `Judge:` line.
|
||||
|
||||
On 2026-10-15, `bin/judge-replay claude-opus-5-5` replayed the latest 15 archived verdicts of each kind from the Opus 5.5 projects, 39 in all. The reference judge, Opus 5.5 at the xhigh effort `settings.json` gave every judge before, reproduced 31 of them. The other 8 may rest on records or judge rules that have changed since; either way no rung is held to them. On the 31, Haiku 5.5 missed 7 at xhigh and 8 at max. That run tried each judge once. 4 of Haiku's misses, 1 at xhigh and 3 at max, misquoted the transcript, which the loop and `judge-replay` now retry; without them it missed 6 and 5. Sonnet 5.5 at high returned another kind on 3 interrupts, and Opus 5.5 at medium missed none. A missed interrupt lets the worker settle a decision that is lilleman's (goal 1.2), so the judge never runs below the project's rung.
|
||||
|
||||
Only Opus 5.5 was replayed, so the judge of a project on another model keeps the effort its settings give until `judge-replay` covers that model (goal 1.1). The judge climbed with the item's E + I from 2026-10-16, lilleman's call: the judge checks the worker's decisions as the reviewers check its code, and a decision it lets through reaches lilleman, as a defect that review lets through reaches the users (goal 1.2).
|
||||
Only Opus 5.5 was replayed, so the judge of a project on another model keeps the effort its settings give until `judge-replay` covers that model (goal 1.1). From 2026-10-16 to 2026-11-12 the judge ran at medium below an E + I of 5 and at xhigh from 5, climbing with the highest E + I among the routed item and those the worker took in, as a review does: lilleman's call, since the judge checks the worker's decisions as the reviewers check its code, and a decision it lets through reaches lilleman, as a defect that review lets through reaches the users (goal 1.2).
|
||||
|
||||
On 2026-11-12, `bin/judge-replay claude-opus-5-5` replayed 60 verdicts, 15 of each kind, and also tried each rung above the project's that a review climbs to by E + I. The reference judge reproduced 46. On those 46, Fable 5.2 missed 2 interrupts and 6 other verdicts at low, 1 interrupt and 4 others at medium, and 1 interrupt and 2 others at high. Every Fable 5.2 rung it tried misses an interrupt, so the judge keeps to its own model's rungs (goal 1.2). That rule also keeps it off Fable 5.2 at max, which no replay tried. Between Opus 5.5's two rungs it climbs to xhigh, not back to medium, since a chunk of E + I 5 or more needs a judge above the project's rung, as it needs reviewers above it. Opus 5.5 at medium missed no interrupt, but returned another kind on 4 quoted answers, 3 of them as finished.
|
||||
On 2026-11-12, `bin/judge-replay claude-opus-5-5` replayed 60 verdicts, 15 of each kind, and also tried each rung above the project's that a review climbs to by E + I. The reference judge reproduced 46. On those 46, Fable 5.2 missed 2 interrupts and 6 other verdicts at low, 1 interrupt and 4 others at medium, and 1 interrupt and 2 others at high. Every Fable 5.2 rung it tried misses an interrupt, so the judge keeps to its own model's rungs (goal 1.2). That rule also keeps it off Fable 5.2 at max, which no replay tried. Opus 5.5 at medium missed no interrupt, but returned another kind on 4 quoted answers, 3 of them as finished.
|
||||
|
||||
2026-11-12, the floor raised, settled by the reruns below. Each of the 4 archived quotes still stands in today's records. Whether the fejkdata quote, `Split a milestone into items in todo.md before starting it`, answers a question about what release 0.5.0 holds, a choice that is lilleman's, is open, so that case is not dropped. Each case was replayed on today's records and the archived transcript copy. Opus 5.5 at medium returned the quoted answer 7 times in 20: adf-codec 4 of 5 (the fifth as finished), claude-config 3 of 5 (2 finished), plainpages 0 of 5 (5 finished) and fejkdata 0 of 5 (5 unfinished). At xhigh it returned it 9 times in 12: adf-codec 1 of 3 (2 interrupts), claude-config 3 of 3, plainpages 2 of 3 (1 finished) and fejkdata 3 of 3. So the misses persist at medium, and xhigh returns the quote more than twice as often. A quoted answer returned as finished closes a chunk whose worker was asking a question, a quality miss (goal 1.1). The judge costs more on every cycle (goal 1.4), which the archive does not yet measure. xhigh is not free of misses: it returned a quoted answer as finished once, and as an interrupt twice, which only asks lilleman more.
|
||||
2026-11-12, claude, raising the judge's floor to xhigh, settled by the reruns below. Serves goal 1.1, at a cost to goal 1.4. Today's records still hold each of the 4 archived quotes, so no case drops out. For fejkdata the quote, `Split a milestone into items in todo.md before starting it`, doesn't plainly answer whether a split changes what release 0.5.0 holds, which is lilleman's choice. That case stays, since xhigh quotes it in 3 of 3 replays. Each case was replayed on today's records and the archived transcript copy. Opus 5.5 at medium returned the quoted answer 7 times in 20: adf-codec 4 of 5 (the fifth as finished), claude-config 3 of 5 (2 finished), plainpages 0 of 5 (5 finished) and fejkdata 0 of 5 (5 unfinished). At xhigh it returned it 9 times in 12: adf-codec 1 of 3 (2 interrupts), claude-config 3 of 3, plainpages 2 of 3 (1 finished) and fejkdata 3 of 3. So the misses persist at medium, and xhigh returns the quote more than twice as often. A quoted answer returned as finished closes a chunk whose worker was asking a question, a quality miss (goal 1.1). xhigh costs more per judge run (goal 1.4); the `Judge cost:` lines of three of the four cases, run at medium, read 0.57 to 0.79. xhigh is not free of misses: it returned a quoted answer as finished once, and as an interrupt twice, which only asks lilleman more.
|
||||
|
||||
Valid while the archived verdicts show what a judge should return, and until a new model joins the ladder: replay then.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user