Drop item 319: its bench did not favour a fresh session over resuming, and lilleman dropped it
CI / tests (pull_request) Successful in 4m6s

This commit is contained in:
2026-11-17 07:13:20 +01:00
parent e6d4321f6c
commit b1c31f93bc
3 changed files with 2 additions and 18 deletions
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- Add `workflow-bench/handoff_bench.py`, a bench of how a worker's step to a stronger model hands over the chunk, on five archived L 3 chunks. Opus 5.5 resuming Sonnet 5.5's session passed two and was partial on three, for a mean $0.68; a fresh session with a brief passed one and was partial on four, for $0.78. So a stepped worker still resumes its session.
- todo-item edit --blocked-by sets the "is blocked by" links of a Jira issue the loop filed.
- A `Touches:` list in the description of a Jira issue is read as one in `todo.md` is, a path or name that holds a backtick included: the Jira description spells it in backticks with no padding spaces, and the prediction reader takes that spelling too.
- `todo-item edit --detail` and `project-record save` leave a Jira description or a Confluence page exactly as it is, with its block ids and spacing paragraphs, where the text saved is the text a read showed. An unedited save over a page a teammate saved since still refuses.
+1 -1
View File
@@ -578,7 +578,7 @@ The step is one rung up from the rung the worker would otherwise hold: its own,
The archive quotes the judge's passage under `Escalation:` in the unfinished verdict, the interrupt or the quoted answer. The step an unfinished verdict causes is archived after it. The step an interrupt or a quoted answer causes is archived with it, at once: the interrupt or answer names the rung the worker ran on until then, and a step-up archived beside it names the rung it runs on from then. So the routing figures charge each stretch to its own rung, and a loop that dies while the human answers resumes on the stepped rung. Each archived worker line names the mark once there is one, so a resumed loop keeps it. `bin/routing_figures/` counts none of these steps as an escalation by trigger, which it counts only from subagent prompts.
A stepped worker resumes its session, also when the step changes its model. Item 319 proposed a fresh session with a brief of the chunk in its place, after arXiv 2608.24358 measured a fresh strong model ahead of one handed the weak model's transcript. `workflow-bench/handoff_bench.py` benched it on 2026-11-17, on L 3 items this repo shipped. Sonnet 5.5 at high effort started each of six and was stopped at $0.36. It stopped itself on item 163 as it reached the budget, which the bench reads as a finished run, so five items went on, each stopped after 9 to 13 turns. Opus 5.5 at medium effort then finished it three ways: resuming the session, resuming it after `/compact`, and in a fresh session with the item's text and a brief, which says that the unfinished change lies uncommitted in the clone. A judge that did not know the arm graded each change. It saw the item, the change that shipped for it, and the result of the tests run on the worker's tree.
A stepped worker resumes its session, also when the step changes its model. A plan item, since dropped, proposed a fresh session with a brief of the chunk in its place, after arXiv 2608.24358 measured a fresh strong model ahead of one handed the weak model's transcript. `workflow-bench/handoff_bench.py` benched it on 2026-11-17, on L 3 items this repo shipped. Sonnet 5.5 at high effort started each of six and was stopped at $0.36. It stopped itself on item 163 as it reached the budget, which the bench reads as a finished run, so five items went on, each stopped after 9 to 13 turns. Opus 5.5 at medium effort then finished it three ways: resuming the session, resuming it after `/compact`, and in a fresh session with the item's text and a brief, which says that the unfinished change lies uncommitted in the clone. A judge that did not know the arm graded each change. It saw the item, the change that shipped for it, and the result of the tests run on the worker's tree.
| arm | runs | pass | partial | fail | mean cost $ | mean seconds |
|---|---|---|---|---|---|---|
-17
View File
@@ -12,7 +12,6 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 319 | 0.12.0 | | question | **Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.** | 4 | 2 | 3 | 7 | 5 | 1.1 by 2, 1.4 by 2 | 0.9 |
| 387 | 0.12.0 | | | **Bind a project's rung to its ladder in one value, so the routing functions and `Cycle` take one argument for both.** | 3 | 1 | 3 | 5 | 5 | code health by 3 | 2.9 |
| 338 | 0.12.0 | | owner | **Script each step that 0.9.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.3 |
| 309 | 0.12.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.2 |
@@ -69,22 +68,6 @@
## Details
### 319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
On SWE-bench Verified with Haiku 4.5 and Opus 4.7, handing the strong model the weak model's transcript solved 69.2% at $1.61 a task; starting the strong model fresh solved 79.2% at $0.90 (arXiv 2608.24358, The Handoff Tax). A model switch also rereads the whole context uncached (Claude Code prompt caching docs). A worker that steps from Sonnet 5.5 to Opus 5.5 by `--resume` is the raw handoff; an effort-only step within one model keeps the cache and isn't. The escalated subagent, which starts fresh with the trigger's evidence, already follows the pattern that won.
Bench it first on `workflow-bench`: resume against a fresh start with a brief. The brief is the chunk's item, its branch and diff, open findings and the trigger. lilleman approved on 2026-11-11. Revise the decision `routing: a worker whose run ends asking for a higher rung…`.
Touches:
- `bin/loop_claude/cycle/run.py`: `Cycle`, `run_cycle`, `worker_start`
- `bin/loop_claude/cycle/prompts.py`: `resumed`, `stepped_to`
- `bin/loop_claude/cycle/worker.py`: `worker_args`
- `bin/loop_claude/cycle/states.py`
- `bin/loop_claude/test_loop_claude.py`: `Cycles`, `WorkerArgs`
- `workflow-bench/bench.py`: `run`, `main`
**Question for lilleman, 2026-11-17: drop this item, or build it despite the bench?** `workflow-bench/handoff_bench.py` benched it on five L 3 items. Resume passed 2 and was partial on 3, at a mean $0.68. The fresh session with a brief passed 1 and was partial on 4, at $0.78. Resume's verdict matched or beat the brief's on every item. The decision `routing: a worker whose run ends asking for a higher rung…` holds the table and the bench's limits. Building it also costs more than the files listed above: the archive files a chunk under its session's first line, so `bin/transcript_archive` and each figure read from it would have to join a chunk's two sessions. Recommendation: drop it with `Drop item 319`, and bench again with longer first runs before filing it anew. Whoever closes the item adds the bench to `CHANGELOG.md`.
### 387. Bind a project's rung to its ladder in one value, so the routing functions and `Cycle` take one argument for both.
Filed by PR #553, which made the routing ladder a file a project may replace. Its stability round 1 found two places the comprehension seats name as hard, touched and no easier to read: `bin/loop_claude/routing.py` `unfinished_route` took `ladder` as a fourth parameter, and `Cycle` in `bin/loop_claude/cycle/run.py` gained the field `ladder`. The patch threaded the ladder as an argument, since a wider change did not fit a patch cut during an outage.