Ask whether item 319 is dropped after its bench

This commit is contained in:
2026-11-17 04:01:29 +01:00
parent 07674b2c20
commit 376a5a8323
+17 -15
View File
@@ -12,6 +12,7 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 319 | 0.12.0 | | question | **Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.** | 4 | 2 | 3 | 7 | 5 | 1.1 by 2, 1.4 by 2 | 0.9 |
| 306 | 0.12.0 | | | **Set a strategy that keeps loop-claude's archive and larv-review's review folders backed up and small over the years, compressing older entries without losing any.** | 3 | 2 | 4 | 7 | 5 | 3 by 5 | 3.9 |
| 387 | 0.12.0 | | | **Bind a project's rung to its ladder in one value, so the routing functions and `Cycle` take one argument for both.** | 3 | 1 | 3 | 5 | 5 | code health by 3 | 2.9 |
| 338 | 0.12.0 | | owner | **Script each step that 0.9.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.3 |
@@ -23,7 +24,6 @@
| 316 | 0.12.0 | | | **Launch comprehension seats as an agent type of their own under `agents/`, without `CLAUDE.md` and the skill and agent listings a general-purpose subagent carries.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
| 339 | 0.12.0 | | | **Lower the reading cost of the files a chunk reads most before its first commit.** | 2 | 2 | 1 | 3 | 3 | 1.4 by 3 | 1.0 |
| 341 | 0.12.0 | | | **Hand each worker's claude a lower Read limit through `CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS`, kept only where replays of the archived review rounds find what two unchanged replays find.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.0 |
| 319 | 0.12.0 | | | **Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.** | 4 | 2 | 3 | 7 | 5 | 1.1 by 2, 1.4 by 2 | 0.9 |
| 310 | 0.12.0 | | | **Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 0.9 |
| 393 | 0.12.0 | | decision | **Step an L 3 worker one rung up and an L 4 worker two up from the project's rung, as lilleman ruled on 2026-11-14.** | 2 | 1 | 3 | 4 | 4 | 1.4 by 3 | 0.9 |
| 298 | 0.12.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 0.9 |
@@ -74,6 +74,22 @@
## Details
### 319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
On SWE-bench Verified with Haiku 4.5 and Opus 4.7, handing the strong model the weak model's transcript solved 69.2% at $1.61 a task; starting the strong model fresh solved 79.2% at $0.90 (arXiv 2608.24358, The Handoff Tax). A model switch also rereads the whole context uncached (Claude Code prompt caching docs). A worker that steps from Sonnet 5.5 to Opus 5.5 by `--resume` is the raw handoff; an effort-only step within one model keeps the cache and isn't. The escalated subagent, which starts fresh with the trigger's evidence, already follows the pattern that won.
Bench it first on `workflow-bench`: resume against a fresh start with a brief. The brief is the chunk's item, its branch and diff, open findings and the trigger. lilleman approved on 2026-11-11. Revise the decision `routing: a worker whose run ends asking for a higher rung…`.
Touches:
- `bin/loop_claude/cycle/run.py`: `Cycle`, `run_cycle`, `worker_start`
- `bin/loop_claude/cycle/prompts.py`: `resumed`, `stepped_to`
- `bin/loop_claude/cycle/worker.py`: `worker_args`
- `bin/loop_claude/cycle/states.py`
- `bin/loop_claude/test_loop_claude.py`: `Cycles`, `WorkerArgs`
- `workflow-bench/bench.py`: `run`, `main`
**Question for lilleman, 2026-11-17: drop this item, or build it despite the bench?** `workflow-bench/handoff_bench.py` benched it on five L 3 items. Resume passed 2 and was partial on 3, at a mean $0.68. The fresh session with a brief passed 1 and was partial on 4, at $0.78. Resume's verdict matched or beat the brief's on every item. The decision `routing: a worker whose run ends asking for a higher rung…` holds the table and the bench's limits. Building it also costs more than the files listed above: the archive files a chunk under its session's first line, so `bin/transcript_archive` and each figure read from it would have to join a chunk's two sessions. Recommendation: drop it with `Drop item 319`, and bench again with longer first runs before filing it anew. Whoever closes the item adds the bench to `CHANGELOG.md`.
### 306. Set a strategy that keeps loop-claude's archive and larv-review's review folders backed up and small over the years, compressing older entries without losing any.
Measured on 2026-11-11: loop-claude's archive is 6.3 GB (12,721 files from October alone, 0.84 GB older than 30 days), and `~/.local/state/larv-review/` is 997 MB, 984 MB of it under `code/`, with no pruning: `skills/larv-review/SKILL.md` → "Reports" keeps every folder after the merge, because `review-replay` replays rounds from them. The largest archived worker transcript, 23.4 MB, gzips to 9.1 MB at level 6. The disk holds 65 GB free today.
@@ -228,20 +244,6 @@ Touches:
- `bin/review_replay/test_review_replay.py`
- `bin/loop_claude/test_loop_claude.py`: `Cycles` (small), `HelpLimits` (small)
### 319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
On SWE-bench Verified with Haiku 4.5 and Opus 4.7, handing the strong model the weak model's transcript solved 69.2% at $1.61 a task; starting the strong model fresh solved 79.2% at $0.90 (arXiv 2608.24358, The Handoff Tax). A model switch also rereads the whole context uncached (Claude Code prompt caching docs). A worker that steps from Sonnet 5.5 to Opus 5.5 by `--resume` is the raw handoff; an effort-only step within one model keeps the cache and isn't. The escalated subagent, which starts fresh with the trigger's evidence, already follows the pattern that won.
Bench it first on `workflow-bench`: resume against a fresh start with a brief. The brief is the chunk's item, its branch and diff, open findings and the trigger. lilleman approved on 2026-11-11. Revise the decision `routing: a worker whose run ends asking for a higher rung…`.
Touches:
- `bin/loop_claude/cycle/run.py`: `Cycle`, `run_cycle`, `worker_start`
- `bin/loop_claude/cycle/prompts.py`: `resumed`, `stepped_to`
- `bin/loop_claude/cycle/worker.py`: `worker_args`
- `bin/loop_claude/cycle/states.py`
- `bin/loop_claude/test_loop_claude.py`: `Cycles`, `WorkerArgs`
- `workflow-bench/bench.py`: `run`, `main`
### 310. Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.
281 worker resumes after the 1-hour cache expired rewrote a whole context, $379 of the $9,154 measured from 2026-09-30 to 2026-11-11 (4.1%). A compaction while the cache is warm reads the context at a twentieth of the input price and leaves about 20k tokens to rewrite later.