Answer stability round 2: 0 🔴, 1 🟡 (0 misread); record the classifier's pass on the 1,116 verdicts beside its fail on the first 1,104, and file the question of its miss rate
This commit is contained in:
+9
-7
@@ -650,17 +650,19 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
|
||||
|
||||
Valid while the index ranks these models, and while no newer model joins the catalog.
|
||||
|
||||
## routing: the judge's model keeps the verdict kind, since no code rule and no classifier trained on the archive decided finished from the worker's final message with no missed interrupt
|
||||
## routing: the judge's model keeps the verdict kind until lilleman rules on the offline classifier, which decided finished with no missed interrupt on 1,116 archived verdicts but missed one on the first 1,104
|
||||
|
||||
2026-11-17, claude. Serves goals 1.2 and 1.4, and applies project principle 1: a cost cut waits for a measurement that shows quality holds.
|
||||
2026-11-18, claude. Serves goals 1.2 and 1.4, and applies project principle 1: a cost cut waits for a measurement that shows quality holds.
|
||||
|
||||
On 2026-11-17, `bin/judge-replay claude-opus-5-5 --offline` replayed all 1,104 archived verdicts of the Opus 5.5 projects (660 finished, 333 interrupt, 33 quoted answer, 78 unfinished) through three deciders. Each reads only the worker's final message, doesn't send anything to an outside service, and decides a kind or leaves the verdict to the judge's model. A missed interrupt is an archived interrupt decided as another kind, which lets the worker settle a decision that is lilleman's. Deciding interrupt can't miss one: a wrong interrupt only asks lilleman more.
|
||||
`bin/judge-replay claude-opus-5-5 --offline` replays all archived verdicts of the Opus 5.5 projects through three deciders. Each reads only the worker's final message, doesn't send anything to an outside service, and decides a kind or leaves the verdict to the judge's model. A missed interrupt is an archived interrupt decided as another kind, which lets the worker settle a decision that is lilleman's. Deciding interrupt can't miss one: a wrong interrupt only asks lilleman more. On 2026-11-17 the archive held 1,104 verdicts (660 finished, 333 interrupt, 33 quoted answer, 78 unfinished); on 2026-11-18 it held 1,116 (668, 336, 34, 78).
|
||||
|
||||
- **A "?" rule** (interrupt where the message holds a "?", finished where it holds none) decided every verdict and missed 197 interrupts.
|
||||
- **A decision table**, the rows of `DECISION_TABLE` in `bin/judge_replay/offline.py`. Its rows were read from this archive's interrupts, so its run isn't a held-out test. It decided 974 and missed 55. Most of its misses are a worker reporting a merged chunk while a release is ready to cut, which depends on the plan; the rest ask for an answer in words no row names.
|
||||
- **A classifier**, a logistic regression over the message's words, trained on the archive. It decides finished only where its chance of an interrupt is below that of every interrupt among its training folds when held out, and interrupt only where it is above that of every other kind. It averages five regressions that differ only in the order they shuffle the samples, since a single shuffle moved its missed interrupts between 0 and 2 on the same archive. On held-out folds it decided 103 (92 finished, 11 interrupt) and missed none. On the latest fifth, learning from the rest, it decided 29, all as finished, and missed 1 interrupt.
|
||||
- **A "?" rule** (interrupt where the message holds a "?", finished where it holds none) decided every one of the 1,104 and missed 197 interrupts.
|
||||
- **A decision table**, the rows of `DECISION_TABLE` in `bin/judge_replay/offline.py`. Its rows were read from this archive's interrupts, so its run isn't a held-out test. It decided 974 of the 1,104 and missed 55. Most of its misses are a worker reporting a merged chunk while a release is ready to cut, which depends on the plan; the rest ask for an answer in words no row names.
|
||||
- **A classifier**, a logistic regression over the message's words, trained on the archive. It decides finished only where its chance of an interrupt is below that of every interrupt among its training folds when held out, and interrupt only where it is above that of every other kind. It averages five regressions that differ only in the order they shuffle the samples, since a single shuffle moved its missed interrupts between 0 and 2 on the same archive. Its bar is no missed interrupt on both tests: held-out folds, each tested by a model trained on the other folds, and the latest fifth of the verdicts, learning from the rest.
|
||||
- On the 1,104 verdicts, held-out folds: 103 decided, none missed. Latest fifth: 29 decided, all as finished, and 1 interrupt missed. That fails the bar, and shuffles 5 to 9 and 10 to 14 failed it alike.
|
||||
- On the 1,116 verdicts, held-out folds: 95 decided (84 finished, 11 interrupt), none missed. Latest fifth: 20 decided, all as finished, none missed. That meets the bar for finished.
|
||||
|
||||
No method meets the bar of no missed interrupt on every test: the classifier met it on held-out folds and missed one interrupt on the latest fifth, the newest verdicts. So no item is filed for a verdict kind, and the judge's model keeps the verdict kind. A classifier's pass counts only where it holds on both tests, since one test alone moved with the shuffle order and with a single added verdict. It also keeps the escalation it quotes and whether a release is ready to cut, which this replay didn't test.
|
||||
The classifier decides about 1 in 12 verdicts, and about 1 in 100 of what it decides were an interrupt on the 1,104. A latest fifth that starts a few verdicts later no longer holds the one it missed, so the bar flips with the archive's size and doesn't separate a classifier that never misses from one that rarely does. The judge's model keeps the verdict kind, the escalation it quotes and whether a release is ready to cut until lilleman rules whether a miss rate of about 1 in 100 decided is close enough to none. The plan holds that question as an item marked `question`.
|
||||
|
||||
Valid while the archived verdicts hold interrupts that read like a finished chunk. Replay with `judge-replay <model> --offline` when the archive has doubled, or when a decider reads more than the final message.
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 398`
|
||||
`Next ID = 399`
|
||||
|
||||
## Items
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
| 341 | 0.12.0 | | | **Hand each worker's claude a lower Read limit through `CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS`, kept only where replays of the archived review rounds find what two unchanged replays find.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.0 |
|
||||
| 310 | 0.12.0 | | | **Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 0.9 |
|
||||
| 298 | 0.12.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 0.9 |
|
||||
| 398 | 0.13.0 | | question | **Skip the judge's model call on a worker's final message that the offline classifier decides is finished, once lilleman rules that its miss rate of about 1 in 100 decided is close enough to no missed interrupt.** | 3 | 3 | 3 | 7 | 5 | 1.4 by 2 | 0.4 |
|
||||
| 365 | 0.13.0 | | | **Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
|
||||
| 368 | 0.13.0 | | | **Call a release review due, call a release ready to cut, and tell whether a chunk owes a scoring run, for a plan outside the checkout, from the project repo's history or from Jira.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
|
||||
| 309 | 0.13.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.2 |
|
||||
@@ -182,6 +183,21 @@ Touches:
|
||||
- `bin/loop_claude/test_loop_claude.py`: `JudgePrompt`, `Records`
|
||||
- `bin/judge_replay/test_judge_replay.py`
|
||||
|
||||
### 398. Skip the judge's model call on a worker's final message that the offline classifier decides is finished, once lilleman rules that its miss rate of about 1 in 100 decided is close enough to no missed interrupt.
|
||||
|
||||
Question for lilleman: is a miss rate of about 1 in 100 decided close enough to none? A missed interrupt lets a worker settle a decision that is yours, unasked. Until you rule, the judge's model keeps the verdict kind (`docs/decisions.md` → `routing: the judge's model keeps the verdict kind until lilleman rules on the offline classifier …`).
|
||||
|
||||
Taking the item builds this: before the judge call, loop-claude runs the classifier trained on its archive over the worker's final message. Where the classifier decides finished, loop-claude skips the call. The judge's model keeps every verdict the classifier leaves, and the escalation it quotes and whether a release is ready to cut. The classifier is re-trained as the archive grows. The judge call sits in `bin/loop_claude/cycle/run.py` → `rule`, a place the comprehension seats found hard, since the order of stamping, archiving and stepping matters across three functions: make it easier to read.
|
||||
|
||||
Evidence, from `judge-replay claude-opus-5-5 --offline`: the classifier reads only the final message, averages five seeded logistic regressions, and decides finished only where its chance of an interrupt is below that of every interrupt among its held-out training folds.
|
||||
- On the 1,116 archived verdicts it decided finished with no missed interrupt on both tests: held-out folds, 95 decided and none missed; latest fifth, 20 decided and none missed.
|
||||
- On the first 1,104 it missed one interrupt on the latest fifth, while the held-out folds decided 103 and missed none.
|
||||
- It decides about 1 in 12 verdicts. About 1 in 100 of those it decided were an interrupt, on the 1,104.
|
||||
|
||||
Savings: one judge call per about 12 verdicts. That is small, and nobody has measured it.
|
||||
|
||||
Fallbacks: decline, and close this item. Or first tighten the bar in `judge-replay`, so it separates a classifier that never misses from one that rarely does, and replay again.
|
||||
|
||||
### 365. Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.
|
||||
|
||||
lilleman's call, 2026-11-12: 0.10.0 and 0.11.0 switch the history tools off for a plan outside the checkout, and 0.13.0 holds only their ports. In Jira a dropped item, which `Drop item` commit titles mark today, is an issue resolved by a resolution the config names.
|
||||
|
||||
Reference in New Issue
Block a user