Record lilleman's ruling that the offline classifier may decide finished #603

Merged
lilleman merged 2 commits from item-423 into main 2026-11-18 14:39:36 +01:00
3 changed files with 19 additions and 19 deletions
+1 -1
View File
@@ -1,4 +1,4 @@
"""Deciders of a verdict's kind that run offline on the worker's final message, replayed on the archive: docs/decisions.md → `routing: the judge's model keeps the verdict kind, …`."""
"""Deciders of a verdict's kind that run offline on the worker's final message, replayed on the archive: docs/decisions.md → `routing: the offline classifier may decide finished, …`."""
import collections
import math
import random
+2 -2
View File
@@ -674,7 +674,7 @@ A project's rung is its model's cheapest rung, which for Opus 5.5 is its default
Valid while the index ranks these models, and while no newer model joins the catalog.
## routing: the judge's model keeps the verdict kind until lilleman rules on the offline classifier, which decided finished with no missed interrupt on 1,116 archived verdicts but missed one on the first 1,104
## routing: the offline classifier may decide finished, since lilleman ruled a miss rate of about 1 in 100 decided close enough to none; the judge's model keeps every other verdict kind
2026-11-18, claude. Serves goals 1.2 and 1.4, and applies project principle 1: a cost cut waits for a measurement that shows quality holds.
@@ -686,7 +686,7 @@ Valid while the index ranks these models, and while no newer model joins the cat
- On the 1,104 verdicts, held-out folds: 103 decided, none missed. Latest fifth: 29 decided, all as finished, and 1 interrupt missed. That fails the bar, and shuffles 5 to 9 and 10 to 14 failed it alike.
- On the 1,116 verdicts, held-out folds: 95 decided (84 finished, 11 interrupt), none missed. Latest fifth: 20 decided, all as finished, none missed. That meets the bar for finished.
The classifier decides about 1 in 12 verdicts, and about 1 in 100 of what it decides were an interrupt on the 1,104. A latest fifth that starts a few verdicts later no longer holds the one it missed, so the bar flips with the archive's size and doesn't separate a classifier that never misses from one that rarely does. The judge's model keeps the verdict kind, the escalation it quotes and whether a release is ready to cut until lilleman rules whether a miss rate of about 1 in 100 decided is close enough to none. The plan holds that question as an item marked `question`.
The classifier decides about 1 in 12 verdicts, and about 1 in 100 of what it decides were an interrupt on the 1,104. A latest fifth that starts a few verdicts later no longer holds the one it missed, so the bar flips with the archive's size and doesn't separate a classifier that never misses from one that rarely does. On 2026-11-18 lilleman ruled, at loop-claude's terminal, that a miss rate of about 1 in 100 decided is close enough to none. So the classifier may decide finished and skip the judge's call; the plan holds that change as an item. The judge's model keeps every verdict the classifier leaves, the escalation it quotes and whether a release is ready to cut, since the replay didn't test the classifier on those.
Valid while the archived verdicts hold interrupts that read like a finished chunk. Replay with `judge-replay <model> --offline` when the archive has doubled, or when a decider reads more than the final message.
+16 -16
View File
@@ -32,7 +32,6 @@
| 316 | 0.12.0 | | | **Launch comprehension seats as an agent type of their own under `agents/`, without `CLAUDE.md` and the skill and agent listings a general-purpose subagent carries.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
| 310 | 0.12.0 | | | **Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 0.9 |
| 422 | 0.12.0 | 401, 420 | | **Check every 5 minutes whether a tmux window of the loop looks stuck by other signs than a lingering worker, investigate it with a headless claude, and kill only the process the investigation names.** | 3 | 2 | 3 | 6 | 5 | 1.5 by 4 | 0.8 |
| 423 | 0.13.0 | | question | **Skip the judge's model call on a worker's final message that the offline classifier decides is finished, once lilleman rules that its miss rate of about 1 in 100 decided is close enough to no missed interrupt.** | 3 | 3 | 3 | 7 | 5 | 1.4 by 2 | 0.4 |
| 365 | 0.13.0 | | | **Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
| 368 | 0.13.0 | | | **Call a release review due, call a release ready to cut, and tell whether a chunk owes a scoring run, for a plan outside the checkout, from the project repo's history or from Jira.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
| 309 | 0.13.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.2 |
@@ -43,6 +42,7 @@
| 298 | 0.13.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 0.9 |
| 395 | 0.13.0 | | decision | **Keep each block a save didn't change, with its localId, and the spacing paragraphs beside it as Jira or Confluence holds them.** | 3 | 2 | 2 | 5 | 5 | 1.6 by 4 | 0.8 |
| 367 | 0.13.0 | | | **Check that every write to a Jira plan leaves it ranked, as rule-hook's `todo-rank-check` checks `todo.md` today.** | 2 | 2 | 3 | 5 | 3 | 1.6 by 2 | 0.7 |
| 423 | 0.13.0 | | | **Skip the judge's model call on a worker's final message that the offline classifier decides is finished.** | 3 | 3 | 3 | 7 | 5 | 1.4 by 2 | 0.4 |
| 77 | 0.14.0 | | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 4 | 2 | 1 | 5 | 6 | 1.1 by 7, 1.4 by 7 | 2.1 |
| 196 | 0.14.0 | | decision | **Replay the architecture, principles, overrides and product-owner passes with `review-replay` on Opus 5.5 projects, record each run in the review-replay routing decision, and, where a run qualifies a rung, add back the code that lowers a pass to it.** | 2 | 2 | 3 | 5 | 2 | 1.4 by 3 | 1.2 |
| 321 | 0.14.0 | 196 | decision | **Evaluate the review passes that run on a rung `review-replay` qualified, against the chunks before them, and revert any pass whose findings or later fixes worsened.** | 1 | 2 | 1 | 2 | 2 | 1.1 by 3, 1.4 by 2 | 2.0 |
@@ -340,21 +340,6 @@ Touches:
- `bin/loop_claude/tmux.py`: `Listed`, `listed`
- `README.md`: loop-claude's section (small)
### 423. Skip the judge's model call on a worker's final message that the offline classifier decides is finished, once lilleman rules that its miss rate of about 1 in 100 decided is close enough to no missed interrupt.
Question for lilleman: is a miss rate of about 1 in 100 decided close enough to none? A missed interrupt lets a worker settle a decision that is yours, unasked. Until you rule, the judge's model keeps the verdict kind (`docs/decisions.md` → `routing: the judge's model keeps the verdict kind until lilleman rules on the offline classifier …`).
Taking the item builds this: before the judge call, loop-claude runs the classifier trained on its archive over the worker's final message. Where the classifier decides finished, loop-claude skips the call. The judge's model keeps every verdict the classifier leaves, and the escalation it quotes and whether a release is ready to cut. The classifier is re-trained as the archive grows. The judge call sits in `bin/loop_claude/cycle/run.py` → `rule`, a place the comprehension seats found hard, since the order of stamping, archiving and stepping matters across three functions: make it easier to read.
Evidence, from `judge-replay claude-opus-5-5 --offline`: the classifier reads only the final message, averages five seeded logistic regressions, and decides finished only where its chance of an interrupt is below that of every interrupt among its held-out training folds.
- On the 1,116 archived verdicts it decided finished with no missed interrupt on both tests: held-out folds, 95 decided and none missed; latest fifth, 20 decided and none missed.
- On the first 1,104 it missed one interrupt on the latest fifth, while the held-out folds decided 103 and missed none.
- It decides about 1 in 12 verdicts. About 1 in 100 of those it decided were an interrupt, on the 1,104.
Savings: one judge call per about 12 verdicts. That is small, and nobody has measured it.
Fallbacks: decline, and close this item. Or first tighten the bar in `judge-replay`, so it separates a classifier that never misses from one that rarely does, and replay again.
### 365. Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.
lilleman's call, 2026-11-12: 0.10.0 and 0.11.0 switch the history tools off for a plan outside the checkout, and 0.13.0 holds only their ports. In Jira a dropped item, which `Drop item` commit titles mark today, is an issue resolved by a resolution the config names.
@@ -459,6 +444,21 @@ Today a Jira detail edited in one line, or a Confluence record with one entry ap
One of 0.13.0's history ports, per lilleman's call of 2026-11-12.
### 423. Skip the judge's model call on a worker's final message that the offline classifier decides is finished.
On 2026-11-18 lilleman ruled that the classifier's miss rate of about 1 in 100 decided is close enough to no missed interrupt (`docs/decisions.md` → `routing: the offline classifier may decide finished, since lilleman ruled a miss rate of about 1 in 100 decided close enough to none; the judge's model keeps every other verdict kind`).
Taking the item builds this: before the judge call, loop-claude runs the classifier trained on its archive over the worker's final message. Where the classifier decides finished, loop-claude skips the call. The classifier is re-trained as the archive grows. The judge call sits in `bin/loop_claude/cycle/run.py` → `rule`, a place the comprehension seats found hard, since the order of stamping, archiving and stepping matters across three functions: make it easier to read.
The judge's model still decides every verdict the classifier leaves, the escalation a worker quotes, and whether a release is ready to cut, since the replay didn't test the classifier on those.
Evidence, from `judge-replay claude-opus-5-5 --offline`: the classifier reads only the final message, averages five seeded logistic regressions, and decides finished only where its chance of an interrupt is below that of every interrupt among its held-out training folds.
- On the 1,116 archived verdicts it decided finished with no missed interrupt on both tests: held-out folds, 95 decided and none missed; latest fifth, 20 decided and none missed.
- On the first 1,104 it missed one interrupt on the latest fifth, while the held-out folds decided 103 and missed none.
- It decides about 1 in 12 verdicts. About 1 in 100 of those it decided were an interrupt, on the 1,104.
Savings: one judge call per about 12 verdicts. That is small, and nobody has measured it.
### 77. Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.
2026-10-08, lilleman: no metric enters review until a rerun on more data shows it tracking the panel.