Answer architecture round 1: 0 🔴, 4 🟡 (0 misread); item 330 replays only the judge, on the Fable 5.2 rungs, and the four decisions' rewording is filed as question item 331
CI / tests (pull_request) Successful in 2m22s
CI / tests (pull_request) Successful in 2m22s
This commit was merged in pull request #493.
This commit is contained in:
@@ -6,13 +6,14 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 331`
|
||||
`Next ID = 332`
|
||||
|
||||
## Items
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 330 | 0.8.0 | | | **Replay the judge and every review pass on the Fable 5.2 ladder, and record what each replay shows.** | 2 | 2 | 2 | 4 | 3 | 1.2 by 4, 1.1 by 3 | 2.9 |
|
||||
| 331 | 0.8.0 | | question | **Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.** | 1 | 1 | 1 | 1 | 2 | 1.4 by 2 | 0.9 |
|
||||
| 330 | 0.8.0 | | | **Replay the judge on each Fable 5.2 rung an Opus 5.5 project's E + I climbs it to, and keep it off any rung that misses an interrupt.** | 3 | 3 | 2 | 6 | 3 | 1.2 by 4, 1.1 by 3 | 2.9 |
|
||||
| 320 | 0.8.0 | | | **Bench a worker below the project's rung with the advisor tool against the same worker alone and the project's rung, and file the switch if it wins.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3, 1.1 by 2 | 2.0 |
|
||||
| 315 | 0.8.0 | | | **Start each reviewer under `agents/` without `CLAUDE.md`, and hand it the vocabulary, the prose rules and the project's decision titles.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 6 | 2.0 |
|
||||
| 312 | 0.8.0 | | | **Research the tooling agents read, search and run commands with, and file an item for each tool that cuts tokens a replay shows keep quality.** | 1 | 2 | 1 | 2 | 3 | 1.4 by 3, 1.5 by 2 | 2.0 |
|
||||
@@ -73,11 +74,24 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 330. Replay the judge and every review pass on the Fable 5.2 ladder, and record what each replay shows.
|
||||
### 331. Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.
|
||||
|
||||
Item 292 put Fable 5.2 on the routing ladder. Two decisions expire when a new model joins it: `routing: on a model judge-replay covered, the judge runs …` ("replay then") and `routing: a review pass runs below the project's rung only on a rung that a replay …` ("a replay of every pass"). On an Opus 5.5 project the judge now climbs onto Fable 5.2 at low, medium and high at E + I 5, 6 and 8, rungs `judge-replay` never covered.
|
||||
The architect's 🔀 answer on PR #493 (item 292): four decisions expire "when a new model joins the ladder". Fable 5.2 joined above every project model's rung, and none of these replays tested a rung above the project's, so the event fired with nothing new to test. Every later ladder rebuild would fire all four again.
|
||||
|
||||
Run `bin/judge-replay claude-opus-5-5` on the new ladder, and `bin/review-replay` for each pass the 2026-10-15 to 2026-10-18 runs covered. Record each result in its decision, and renew or change each "Valid" line. Where a judge rung on Fable 5.2 misses an interrupt the project's rung returns, keep the judge off it until a decision says otherwise.
|
||||
- `review: each pass keeps a reviewer of its own`: the merge trials ran on Opus 5.5 at medium. Reword the "Valid" line to name that rung.
|
||||
- `review: the principles audit runs only in the release review…`: its trial ran on the project's rung. Reword the line to name that rung.
|
||||
- `routing: a review pass runs below the project's rung only on a rung that a replay…`: `review-replay` tests only rungs below the project's rung. Have the line expire when those rungs change, the event item 196 names ("until a cheaper model joins the ladder").
|
||||
- `routing: on a model judge-replay covered, the judge runs…`: the replay tested the project's rung and below, while the climb above rests on the ladder's order alone. Have the line name both, and point the climb at item 330.
|
||||
|
||||
Recommendation: reword all four as above. They are decisions the repo held before item 292, so the rewording is lilleman's call.
|
||||
|
||||
### 330. Replay the judge on each Fable 5.2 rung an Opus 5.5 project's E + I climbs it to, and keep it off any rung that misses an interrupt.
|
||||
|
||||
Item 292 put Fable 5.2 on the routing ladder. On an Opus 5.5 project the judge now climbs with its chunk's E + I onto Fable 5.2 at low, medium and high, at E + I 5, 6 and 8. Fable 5.2 has never returned a judge verdict, and the 2026-10-15 replay showed that another model can miss an interrupt in kind, not only in strength. `routing: on a model judge-replay covered, the judge runs …` rests on that replay.
|
||||
|
||||
`bin/judge-replay` replays only the project's rung and the rungs below it (`down_from` in `judge_replay/main.py`), so as it stands it never runs a Fable 5.2 rung. Give it the rungs the judge climbs to as well, run `bin/judge-replay claude-opus-5-5`, and record the result in that decision. Where a Fable 5.2 rung misses an interrupt the reference judge returns, keep the judge off it until a decision says otherwise.
|
||||
|
||||
The review passes need no replay: `review-replay` tests only rungs below the project's rung, which item 292 left as they were.
|
||||
|
||||
### 320. Bench a worker below the project's rung with the advisor tool against the same worker alone and the project's rung, and file the switch if it wins.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user