Record that a stepped worker keeps resuming its session, after the handoff bench
CI / tests (pull_request) Failing after 2m47s
CI / tests (pull_request) Failing after 2m47s
This commit is contained in:
@@ -557,6 +557,16 @@ The step is one rung up from the rung the worker would otherwise hold: its own,
|
||||
|
||||
The archive quotes the judge's passage under `Escalation:` in the unfinished verdict, the interrupt or the quoted answer. The step an unfinished verdict causes is archived after it. The step an interrupt or a quoted answer causes is archived with it, at once: the interrupt or answer names the rung the worker ran on until then, and a step-up archived beside it names the rung it runs on from then. So the routing figures charge each stretch to its own rung, and a loop that dies while the human answers resumes on the stepped rung. Each archived worker line names the mark once there is one, so a resumed loop keeps it. `bin/routing_figures/` counts none of these steps as an escalation by trigger, which it counts only from subagent prompts.
|
||||
|
||||
A stepped worker resumes its session, also when the step changes its model. Item 319 proposed a fresh session with a brief of the chunk in its place, after arXiv 2608.24358 measured a fresh strong model ahead of one handed the weak model's transcript. `workflow-bench/handoff_bench.py` benched it on 2026-11-17, on five L 3 items this repo shipped. Sonnet 5.5 at high effort started each one and stopped at $0.36, after 16 to 20 turns. Opus 5.5 at medium effort then finished it three ways: resuming the session, resuming it after `/compact`, and in a fresh session with a brief of the item, the branch and its diff. A judge that did not know the arm graded each change against the tests the item shipped with.
|
||||
|
||||
| arm | runs | pass | partial | fail | mean cost $ | mean seconds |
|
||||
|---|---|---|---|---|---|---|
|
||||
| resume | 5 | 2 | 3 | 0 | 0.68 | 235 |
|
||||
| compact | 5 | 1 | 3 | 1 | 0.56 | 198 |
|
||||
| brief | 5 | 1 | 4 | 0 | 0.78 | 310 |
|
||||
|
||||
Resume's verdict matched or beat the brief's on every item, and the brief cost less on two of the five. So the bench gives no reason to pay for the change, which is larger than the worker's start: the archive files a chunk under its session's first line, so a chunk that spans two sessions would need the archive, and each figure read from it, to join them. The bench is small: each arm ran once per item, the judge is the strong arm's model, and no start was longer than 20 turns. Run it again before changing this, with longer starts, since the paper's gap may open only on a long transcript.
|
||||
|
||||
Valid while the worker reports in its transcript the triggers that fired, and while a chunk's session transcript only grows.
|
||||
|
||||
## routing: loop-claude measures review rounds and unfinished verdicts per rung every 20 cycles, and calls for a routing change when, of two rungs with 5 or more chunks each, the lower took more of either than the one above, or when 5 or more subagents of one type had reports longer than the tool results they read
|
||||
|
||||
Reference in New Issue
Block a user