Bench a model step's handoff, and keep resuming the session #575

Merged
lilleman merged 12 commits from item-319 into main 2026-11-17 04:06:55 +01:00
8 changed files with 509 additions and 24 deletions
+4
View File
@@ -187,9 +187,13 @@
!workflow-bench/bench.py
!workflow-bench/compare-prompt.md
!workflow-bench/compare.py
!workflow-bench/handoff-brief-prompt.md
!workflow-bench/handoff-resume-prompt.md
!workflow-bench/handoff_bench.py
!workflow-bench/review_git.py
!workflow-bench/run-prompt.md
!workflow-bench/setup.cfg
!workflow-bench/test_advisor_bench.py
!workflow-bench/test_bench.py
!workflow-bench/test_handoff_bench.py
!workflow-bench/test_review_git.py
+10
View File
@@ -578,6 +578,16 @@ The step is one rung up from the rung the worker would otherwise hold: its own,
The archive quotes the judge's passage under `Escalation:` in the unfinished verdict, the interrupt or the quoted answer. The step an unfinished verdict causes is archived after it. The step an interrupt or a quoted answer causes is archived with it, at once: the interrupt or answer names the rung the worker ran on until then, and a step-up archived beside it names the rung it runs on from then. So the routing figures charge each stretch to its own rung, and a loop that dies while the human answers resumes on the stepped rung. Each archived worker line names the mark once there is one, so a resumed loop keeps it. `bin/routing_figures/` counts none of these steps as an escalation by trigger, which it counts only from subagent prompts.
A stepped worker resumes its session, also when the step changes its model. Item 319 proposed a fresh session with a brief of the chunk in its place, after arXiv 2608.24358 measured a fresh strong model ahead of one handed the weak model's transcript. `workflow-bench/handoff_bench.py` benched it on 2026-11-17, on L 3 items this repo shipped. Sonnet 5.5 at high effort started each of six and was stopped at $0.36. It stopped itself on item 163 as it reached the budget, which the bench reads as a finished run, so five items went on, each stopped after 9 to 13 turns. Opus 5.5 at medium effort then finished it three ways: resuming the session, resuming it after `/compact`, and in a fresh session with the item's text and a brief, which says that the unfinished change lies uncommitted in the clone. A judge that did not know the arm graded each change. It saw the item, the change that shipped for it, and the result of the tests run on the worker's tree.
| arm | runs | pass | partial | fail | mean cost $ | mean seconds |
|---|---|---|---|---|---|---|
| resume | 5 | 2 | 3 | 0 | 0.68 | 235 |
| compact | 5 | 1 | 3 | 1 | 0.56 | 198 |
| brief | 5 | 1 | 4 | 0 | 0.78 | 310 |
Resume's verdict matched or beat the brief's on every item, and the brief cost less on two of the five. So the bench gives no reason to pay for a fresh session. Its price is more than a change to how the worker starts. The archive files a chunk under its session's first line. For a chunk that spans two sessions, the archive, and each figure read from it, would have to join them. The bench is small: each arm ran once per item, the judge is Opus 5.5 too, and no first run was longer than 13 turns. Run it again before changing this, with longer first runs, since the paper's gap may open only on a long transcript.
Valid while the worker reports in its transcript the triggers that fired, and while a chunk's session transcript only grows.
## routing: loop-claude measures review rounds and unfinished verdicts per rung every 20 cycles, and calls for a routing change when, of two rungs with 5 or more chunks each, the lower took more of either than the one above, or when 5 or more subagents of one type had reports longer than the tool results they read
+17 -15
View File
@@ -12,10 +12,10 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 319 | 0.12.0 | | question | **Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.** | 4 | 2 | 3 | 7 | 5 | 1.1 by 2, 1.4 by 2 | 0.9 |
| 387 | 0.12.0 | | | **Bind a project's rung to its ladder in one value, so the routing functions and `Cycle` take one argument for both.** | 3 | 1 | 3 | 5 | 5 | code health by 3 | 2.9 |
| 338 | 0.12.0 | | owner | **Script each step that 0.9.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.3 |
| 288 | 0.12.0 | | | **Point the two reading-cost decisions at code health and goal 1.4's counter-signal, in place of the goal 1.1 they cite.** | 1 | 2 | 1 | 2 | 1 | code health by 1, 1.4 by 1 | 2.2 |
| 319 | 0.12.0 | | | **Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 2, 1.4 by 2 | 1.2 |
| 309 | 0.12.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.2 |
| 299 | 0.12.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
| 311 | 0.12.0 | | | **Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.0 |
@@ -73,6 +73,22 @@
## Details
### 319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
On SWE-bench Verified with Haiku 4.5 and Opus 4.7, handing the strong model the weak model's transcript solved 69.2% at $1.61 a task; starting the strong model fresh solved 79.2% at $0.90 (arXiv 2608.24358, The Handoff Tax). A model switch also rereads the whole context uncached (Claude Code prompt caching docs). A worker that steps from Sonnet 5.5 to Opus 5.5 by `--resume` is the raw handoff; an effort-only step within one model keeps the cache and isn't. The escalated subagent, which starts fresh with the trigger's evidence, already follows the pattern that won.
Bench it first on `workflow-bench`: resume against a fresh start with a brief. The brief is the chunk's item, its branch and diff, open findings and the trigger. lilleman approved on 2026-11-11. Revise the decision `routing: a worker whose run ends asking for a higher rung…`.
Touches:
- `bin/loop_claude/cycle/run.py`: `Cycle`, `run_cycle`, `worker_start`
- `bin/loop_claude/cycle/prompts.py`: `resumed`, `stepped_to`
- `bin/loop_claude/cycle/worker.py`: `worker_args`
- `bin/loop_claude/cycle/states.py`
- `bin/loop_claude/test_loop_claude.py`: `Cycles`, `WorkerArgs`
- `workflow-bench/bench.py`: `run`, `main`
**Question for lilleman, 2026-11-17: drop this item, or build it despite the bench?** `workflow-bench/handoff_bench.py` benched it on five L 3 items. Resume passed 2 and was partial on 3, at a mean $0.68. The fresh session with a brief passed 1 and was partial on 4, at $0.78. Resume's verdict matched or beat the brief's on every item. The decision `routing: a worker whose run ends asking for a higher rung…` holds the table and the bench's limits. Building it also costs more than the files listed above: the archive files a chunk under its session's first line, so `bin/transcript_archive` and each figure read from it would have to join a chunk's two sessions. Recommendation: drop it with `Drop item 319`, and bench again with longer first runs before filing it anew. Whoever closes the item adds the bench to `CHANGELOG.md`.
### 387. Bind a project's rung to its ladder in one value, so the routing functions and `Cycle` take one argument for both.
Filed by PR #553, which made the routing ladder a file a project may replace. Its stability round 1 found two places the comprehension seats name as hard, touched and no easier to read: `bin/loop_claude/routing.py` `unfinished_route` took `ladder` as a fourth parameter, and `Cycle` in `bin/loop_claude/cycle/run.py` gained the field `ladder`. The patch threaded the ladder as an argument, since a wider change did not fit a patch cut during an outage.
@@ -106,20 +122,6 @@ The chunk that ships this item files a copy of this audit in release 0.15.0, the
Both `docs/decisions.md` entries under `reading cost: a rise is …` and `reading cost: larv-review measures each finished chunk …` open "Serves goal 1.1: comprehension …". No goal says that any more: code health (`CLAUDE.md` → "Shared goal") holds comprehension, and since PR #455 a rising reading cost is goal 1.4's first counter-signal. Raised by the stability review of PR #455 as a 🟢 outside that change.
### 319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
On SWE-bench Verified with Haiku 4.5 and Opus 4.7, handing the strong model the weak model's transcript solved 69.2% at $1.61 a task; starting the strong model fresh solved 79.2% at $0.90 (arXiv 2608.24358, The Handoff Tax). A model switch also rereads the whole context uncached (Claude Code prompt caching docs). A worker that steps from Sonnet 5.5 to Opus 5.5 by `--resume` is the raw handoff; an effort-only step within one model keeps the cache and isn't. The escalated subagent, which starts fresh with the trigger's evidence, already follows the pattern that won.
Bench it first on `workflow-bench`: resume against a fresh start with a brief. The brief is the chunk's item, its branch and diff, open findings and the trigger. lilleman approved on 2026-11-11. Revise the decision `routing: a worker whose run ends asking for a higher rung…`.
Touches:
- `bin/loop_claude/cycle/run.py`: `Cycle`, `run_cycle`, `worker_start`
- `bin/loop_claude/cycle/prompts.py`: `resumed`, `stepped_to`
- `bin/loop_claude/cycle/worker.py`: `worker_args`
- `bin/loop_claude/cycle/states.py`
- `bin/loop_claude/test_loop_claude.py`: `Cycles`, `WorkerArgs`
- `workflow-bench/bench.py`: `run`, `main`
### 309. Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.
Seats cost $1,143 of the $9,154 measured from 2026-09-30 to 2026-11-11 (12.5%), scoring runs $470 of that. A scoring run today is four seats, run twice on one head: eight seat runs. Two seats on the second pass make it six, and two on both passes four.
+15 -9
View File
@@ -146,19 +146,25 @@ def run(work, name):
"--no-session-persistence", "--output-format", "stream-json", "--verbose"],
cwd=clone, stdout=out, stderr=subprocess.STDOUT)
read_result((work / f"{name}.jsonl").read_text().splitlines())
git(clone, "add", "--all")
(work / f"{name}.diff").write_text(git(clone, "diff", "--cached", base))
changed = [*git(clone, "diff", "--cached", "--name-only", base).split(), *git(REPO, "diff", "--name-only", base, CASES[case][1]).split()]
targets = [target for target in suite_targets(changed) if (clone / target).exists()] or ["bin"]
tests = subprocess.run([str(RUN_HEAVY), *SUITE, *targets], cwd=clone, capture_output=True, text=True)
(work / f"{name}.tests").write_text(f"exit {tests.returncode}\n{tests.stdout[-1500:]}")
save_change(work, name, clone, CASES[case])
return f"{name}: ran"
def grade(work, name):
"""Have the judge rule on one run's diff, blind to its arm, and save the verdict beside it."""
def save_change(work, name, clone, commits):
"""Save the clone's change since the case's start commit as <name>.diff, and as <name>.tests the result of the tests of the packages that the clone's change or the merged change touches."""
base, merged = commits
git(clone, "add", "--all")
(work / f"{name}.diff").write_text(git(clone, "diff", "--cached", base))
changed = [*git(clone, "diff", "--cached", "--name-only", base).split(), *git(REPO, "diff", "--name-only", base, merged).split()]
targets = [target for target in suite_targets(changed) if (clone / target).exists()] or ["bin"]
tests = subprocess.run([str(RUN_HEAVY), *SUITE, *targets], cwd=clone, capture_output=True, text=True)
(work / f"{name}.tests").write_text(f"exit {tests.returncode}\n{tests.stdout[-1500:]}")
def grade(work, name, cases=None):
"""Have the judge rule on one run's diff, blind to its arm, and save the verdict beside it. cases stands in for CASES where another bench grades its own."""
case = split_name(name)[0]
base, merged = CASES[case]
base, merged = (cases or CASES)[case]
prompt = (BENCH / "advisor-judge-prompt.md").read_text().format(
text=item_text(git(REPO, "show", f"{base}:todo.md"), case), reference=git(REPO, "diff", base, merged, "--", ".", ":!todo.md", ":!docs", ":!CHANGELOG.md"),
diff=(work / f"{name}.diff").read_text(), tests=(work / f"{name}.tests").read_text())
+4
View File
@@ -0,0 +1,4 @@
## Handoff
A worker on a weaker model began this item and stopped before it was done, asking for a stronger model. You start fresh, without its transcript. Its unfinished change is in the clone, uncommitted: read `git status` and `git diff` first. Keep what holds, redo what doesn't, and finish the item.
+1
View File
@@ -0,0 +1 @@
Continue the task; the previous run ended before the item was done. Your request for a higher rung went through: you now run on {rung}. The bench rules still hold.
+200
View File
@@ -0,0 +1,200 @@
#!/usr/bin/env python3
"""Bench how a worker's step to a stronger model hands the chunk over: handoff_bench.py <work dir> start <case>-<n> ..., then run|grade <case>-<arm>-<n> ..., then summary
The question: when loop-claude steps a worker to a rung on another model, which handoff serves the stronger model
best? Each case is an archived chunk of this repo, as in advisor_bench.py, whose judge and files this bench shares.
- `start` runs WEAK on the item in the clone <work dir>/<case>-<n> until it has spent START_BUDGET, which
stands in for a run that ends asking for a higher rung. A worker that finishes within the budget leaves no
handoff, so `start` fails on it. It saves the change so far (<case>-start-<n>.diff), and a copy of the clone
(.tree), which every arm starts from, and a copy of the session's transcript (.session.jsonl), which the resume
and compact arms continue.
- `run` puts the clone and the session back as `start` left them, then continues on STRONG, one arm of a case at a
time. The resume arm resumes WEAK's session with handoff-resume-prompt.md. The compact arm does the same once WEAK
has compacted the session with `/compact`, whose stream it saves as <name>.compact.jsonl. The brief arm starts a
session that has the item and handoff-brief-prompt.md, and no transcript.
- `grade` is advisor_bench.py's, over these cases. `summary` counts an arm's cost and seconds from its own runs,
the compaction included, and leaves out the start, which the arms share (`outcome`).
Run each arm of each case once; the table shows a direction, not a significance.
"""
import collections
import concurrent.futures
import json
import pathlib
import shutil
import subprocess
import sys
import threading
import uuid
import advisor_bench
from advisor_bench import Arm
BENCH = pathlib.Path(__file__).resolve().parent
# The ladder's first step that changes the model: bin/model_routing/routing-ladder.json.
WEAK = Arm("claude-sonnet-5-5", "high")
STRONG = Arm("claude-opus-5-5", "medium")
ARMS = ("resume", "compact", "brief")
# Claude Code keeps a session's transcript at <CLAUDE_PROJECTS>/<the working directory's slug>/<session id>.jsonl.
CLAUDE_PROJECTS = pathlib.Path.home() / ".claude/projects"
# One per case and run number: its arms share one clone, so they run one at a time.
CLONE_LOCKS: collections.defaultdict = collections.defaultdict(threading.Lock)
# L 3 chunks, by todo item. advisor_bench's L 1 and L 2 chunks are too small: Sonnet 5.5 finished five of the six for
# under $0.20, before any handoff.
CASES = {
"143": ("45cb331a4", "545f3c347"),
"163": ("d4cf6e7fe", "d1b0fe3a4"),
"198": ("4a0ba5921", "fd60b6da"),
"210": ("4764d3d52", "dcbdc970"),
"227": ("eb7b50698", "3bde9cd6d"),
"239": ("d583f8d48", "47a40757"),
}
# Dollars WEAK may spend before the handoff. In a trial, $0.36 stopped Sonnet 5.5 on items 163 and 227 after 11 to 14
# turns, as the result line's `num_turns` counts them, with about 250 lines of diff.
START_BUDGET = 0.36
CLAUDE_FLAGS = ["--setting-sources", "project", "--permission-mode", "bypassPermissions", "--output-format", "stream-json", "--verbose"]
def stopped_at_budget(lines):
"""Whether a claude stream's first result line, the worker's own, says the run stopped at --max-budget-usd."""
for line in lines:
try:
entry = json.loads(line)
except ValueError:
continue
if isinstance(entry, dict) and entry.get("type") == "result":
return entry.get("subtype") == "error_max_budget_usd"
return False
def run_prompt(case):
"""advisor_bench's prompt for the case's item, as the item's text stood at the start commit."""
base = CASES[case][0]
text = advisor_bench.item_text(advisor_bench.git(advisor_bench.REPO, "show", f"{base}:todo.md"), case)
return (BENCH / "advisor-run-prompt.md").read_text().format(base=base, item=case, text=text)
def claude(prompt, flags, clone, stream):
with stream.open("w") as out:
subprocess.run(["claude", "-p", prompt, *flags, *CLAUDE_FLAGS], cwd=clone, stdin=subprocess.DEVNULL, stdout=out, stderr=subprocess.STDOUT)
def start(work, name):
"""WEAK's capped run of <case>-<n>, and the tree and session it leaves the arms; the files are in the module docstring."""
case, number = name.split("-")
base = CASES[case][0]
clone, saved = work / name, work / f"{case}-start-{number}"
if clone.exists():
raise ValueError(f"{name} already started: remove {clone}")
advisor_bench.fresh_clone(base, clone)
session = str(uuid.uuid4())
stream = saved.with_suffix(".jsonl")
claude(run_prompt(case), [*WEAK.flags(), "--session-id", session, "--max-budget-usd", str(START_BUDGET)], clone, stream)
if not stopped_at_budget(stream.read_text().splitlines()):
raise ValueError("the weak worker finished within its budget, or its run failed, so there is no handoff to bench")
transcripts = list(CLAUDE_PROJECTS.glob(f"*/{session}.jsonl"))
if len(transcripts) != 1:
raise ValueError(f"expected one transcript of session {session} under {CLAUDE_PROJECTS}, found {len(transcripts)}")
shutil.copy(transcripts[0], saved.with_suffix(".session.jsonl"))
saved.with_suffix(".session").write_text(str(transcripts[0]))
# The worker may have committed against the bench rules, so the arms get its whole change since base, uncommitted.
advisor_bench.git(clone, "add", "--all")
saved.with_suffix(".diff").write_text(advisor_bench.git(clone, "diff", "--cached", base))
advisor_bench.git(clone, "reset", "-q", "--mixed", base)
shutil.copytree(clone, saved.with_suffix(".tree"), symlinks=True)
return f"{name}: started"
def run(work, name):
"""STRONG's run of one arm, <case>-<arm>-<n>, from the tree and session `start` left."""
case, arm, number = advisor_bench.split_name(name)
if arm not in ARMS:
raise ValueError(f"the arm {arm!r} is none of {ARMS}")
clone, saved = work / f"{case}-{number}", work / f"{case}-start-{number}"
if not saved.with_suffix(".tree").exists():
raise ValueError(f"start {case}-{number} first")
stream = work / f"{name}.jsonl"
if stream.exists():
raise ValueError(f"{name} already ran: remove {stream}")
with CLONE_LOCKS[clone]:
shutil.rmtree(clone)
shutil.copytree(saved.with_suffix(".tree"), clone, symlinks=True)
transcript = pathlib.Path(saved.with_suffix(".session").read_text())
shutil.copy(saved.with_suffix(".session.jsonl"), transcript)
resumed = ["--resume", transcript.stem]
if arm == "compact":
claude("/compact", [*WEAK.flags(), *resumed], clone, work / f"{name}.compact.jsonl")
finished(work / f"{name}.compact.jsonl")
if arm == "brief":
claude(run_prompt(case) + "\n" + (BENCH / "handoff-brief-prompt.md").read_text(), [*STRONG.flags(), "--no-session-persistence"], clone, stream)
else:
prompt = (BENCH / "handoff-resume-prompt.md").read_text().format(rung=f"{STRONG.model} at {STRONG.effort} effort")
claude(prompt, [*STRONG.flags(), *resumed], clone, stream)
outcome(work, name)
advisor_bench.save_change(work, name, clone, CASES[case])
return f"{name}: ran"
def results(stream):
"""The result lines of a claude stream. A resumed run may open with one that holds no work, and a background task's notification adds one."""
found = []
for line in stream.read_text().splitlines():
try:
entry = json.loads(line)
except ValueError:
continue
if isinstance(entry, dict) and entry.get("type") == "result":
found.append(entry)
return found
def finished(stream):
"""The result lines of a run that ended well; a run without one, or whose last one is an error, raises."""
ran = results(stream)
if not ran or ran[-1].get("is_error"):
raise ValueError(f"the run ended in an error: {ran[-1].get('result')}" if ran else "the stream has no result line")
return ran
def outcome(work, name):
"""The cost and seconds of an arm's run, its compaction included. A resumed session's cost counts from the session's start, so the start's cost comes off it."""
case, arm, number = advisor_bench.split_name(name)
ran = finished(work / f"{name}.jsonl")
compacted = results(work / f"{name}.compact.jsonl") if arm == "compact" else []
before = 0 if arm == "brief" else results(work / f"{case}-start-{number}.jsonl")[-1]["total_cost_usd"]
return {"cost": ran[-1]["total_cost_usd"] - before, "seconds": sum(entry["duration_ms"] for entry in [*compacted, *ran]) / 1000}
def rows_of(work):
"""One row per graded run: its arm, verdict, cost and seconds."""
rows = []
for verdict_file in sorted(work.glob("*.verdict")):
name = verdict_file.stem
rows.append({"arm": advisor_bench.split_name(name)[1], "verdict": json.loads(verdict_file.read_text())["verdict"], **outcome(work, name)})
return rows
def summary(rows):
"""A markdown table with one row per arm, in ARMS' order: its runs, passes, partials, mean cost and mean seconds."""
lines = ["| arm | runs | pass | partial | mean cost $ | mean seconds |", "|---|---|---|---|---|---|"]
for name in ARMS:
own = [row for row in rows if row["arm"] == name]
if own:
passes, partials = (sum(row["verdict"] == kind for row in own) for kind in ("pass", "partial"))
cost, seconds = (sum(row[key] for row in own) / len(own) for key in ("cost", "seconds"))
lines.append(f"| {name} | {len(own)} | {passes} | {partials} | {cost:.2f} | {seconds:.0f} |")
return "\n".join(lines)
def main(argv):
work, step, names = pathlib.Path(argv[0]), argv[1], argv[2:]
if step == "summary":
print(summary(rows_of(work)))
return
steps = {"start": start, "run": run, "grade": lambda work, name: advisor_bench.grade(work, name, CASES)}
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
for result in pool.map(lambda name: advisor_bench.attempted(steps[step], work, name), names):
print(result, flush=True)
if __name__ == "__main__":
main(sys.argv[1:])
+258
View File
@@ -0,0 +1,258 @@
import json
import pathlib
import subprocess
import tempfile
import unittest
import unittest.mock
import uuid
import advisor_bench
from test_advisor_bench import commit, TODO
import handoff_bench
SESSION = uuid.UUID("11111111-2222-4333-8444-555555555555")
def result_line(**fields):
return json.dumps({"type": "result", "subtype": "success", "total_cost_usd": 0.5, "duration_ms": 12000, **fields})
class StoppedAtBudget(unittest.TestCase):
def test_holds_where_the_first_result_is_the_budget_stop(self):
self.assertTrue(handoff_bench.stopped_at_budget(["not json", '{"type": "assistant"}', result_line(subtype="error_max_budget_usd", is_error=True)]))
def test_fails_where_the_worker_finished_within_the_budget(self):
self.assertFalse(handoff_bench.stopped_at_budget([result_line()]))
def test_fails_where_the_stream_has_no_result(self):
self.assertFalse(handoff_bench.stopped_at_budget(['{"type": "assistant"}', "[]"]))
class Summary(unittest.TestCase):
def test_prints_one_row_per_arm_with_passes_and_means(self):
rows = [
{"arm": "brief", "verdict": "pass", "cost": 0.5, "seconds": 100.0},
{"arm": "brief", "verdict": "partial", "cost": 1.5, "seconds": 300.0},
{"arm": "resume", "verdict": "fail", "cost": 2.0, "seconds": 50.0},
{"arm": "compact", "verdict": "pass", "cost": 0.25, "seconds": 9.0},
]
self.assertEqual(handoff_bench.summary(rows), "\n".join([
"| arm | runs | pass | partial | mean cost $ | mean seconds |",
"|---|---|---|---|---|---|",
"| resume | 1 | 0 | 0 | 2.00 | 50 |",
"| compact | 1 | 1 | 0 | 0.25 | 9 |",
"| brief | 2 | 1 | 1 | 1.00 | 200 |",
]))
class Orchestration(unittest.TestCase):
"""start, run and main on a throwaway repo, with claude and run-heavy faked. Case 7 starts at a commit holding bin/pkg/mod.py and merges a change to it. The fake claude writes a session's transcript as Claude Code does, and each resumed run adds a line to it."""
def setUp(self):
scratch = tempfile.TemporaryDirectory()
self.addCleanup(scratch.cleanup)
root = pathlib.Path(scratch.name)
self.repo, self.work, self.projects = root / "repo", root / "work", root / "projects"
self.work.mkdir()
subprocess.run(["git", "init", "-q", str(self.repo)], check=True)
(self.repo / "link").symlink_to("todo.md")
self.base = self.commit({"todo.md": TODO, "bin/pkg/mod.py": "x = 1\n", "bin/other/o.py": "o = 1\n"})
merged = self.commit({"bin/pkg/mod.py": "x = 2\n"})
self.commit({"bin/other/o.py": "o = 2\n"})
self.calls, self.heavy = [], []
self.start_result = {"subtype": "error_max_budget_usd", "is_error": True, "total_cost_usd": 1.0}
self.compact_result, self.strong_result = {}, {}
real_run = subprocess.run
for patch in (
unittest.mock.patch.object(advisor_bench, "REPO", self.repo),
unittest.mock.patch.object(handoff_bench, "CASES", {"7": (self.base, merged)}),
unittest.mock.patch.object(handoff_bench, "START_BUDGET", 0.25),
unittest.mock.patch.object(handoff_bench, "CLAUDE_PROJECTS", self.projects),
unittest.mock.patch.object(handoff_bench.uuid, "uuid4", return_value=SESSION),
unittest.mock.patch.object(advisor_bench.subprocess, "run", side_effect=lambda args, **kwargs: self.fake(real_run, args, **kwargs)),
):
patch.start()
self.addCleanup(patch.stop)
self.transcript = self.projects / "slug" / f"{SESSION}.jsonl"
def commit(self, files):
for path, text in files.items():
(self.repo / path).parent.mkdir(parents=True, exist_ok=True)
(self.repo / path).write_text(text)
subprocess.run(["git", "-C", str(self.repo), "add", "."], check=True)
commit(self.repo, "a commit")
return advisor_bench.git(self.repo, "rev-parse", "HEAD").strip()
def fake(self, real_run, args, **kwargs):
if args[0] == "claude":
cwd = pathlib.Path(kwargs["cwd"])
# What the run found as it began: the session's transcript and the files under bin/pkg.
found = (self.transcript.read_text() if self.transcript.exists() else None, sorted(path.name for path in (cwd / "bin/pkg").iterdir()))
self.calls.append((args, kwargs, found))
if "--max-budget-usd" in args:
(cwd / "bin/pkg/mod.py").write_text("x = 3\n")
# Against the bench rules, the worker commits a part of its change.
real_run(["git", "-C", str(cwd), "add", "bin/pkg/mod.py"], check=True)
commit(cwd, "against the rules")
(cwd / "bin/pkg/new.py").write_text("n = 1\n")
self.transcript.parent.mkdir(parents=True)
self.transcript.write_text("weak\n")
kwargs["stdout"].write(result_line(**self.start_result) + "\n")
elif args[2] == "/compact":
self.transcript.write_text(self.transcript.read_text() + "compacted\n")
kwargs["stdout"].write(result_line(**self.compact_result, total_cost_usd=1.25, duration_ms=3000) + "\n")
else:
if "--resume" in args:
self.transcript.write_text(self.transcript.read_text() + "strong\n")
# A resumed session counts its cost from the session's start, and may open with a result that holds no work.
kwargs["stdout"].write(result_line(total_cost_usd=1.0, duration_ms=0) + "\n")
(cwd / "bin/pkg" / f"done_{len(self.calls)}.py").write_text("d = 1\n")
kwargs["stdout"].write("a line that is not JSON\n")
kwargs["stdout"].write(result_line(**self.strong_result, total_cost_usd=3.0 if "--resume" in args else 0.5) + "\n")
return subprocess.CompletedProcess(args, 0)
if args[0] == str(advisor_bench.RUN_HEAVY):
self.heavy.append(args)
return subprocess.CompletedProcess(args, 0, stdout="3 passed\n", stderr="")
return real_run(args, **kwargs)
def started(self):
handoff_bench.start(self.work, "7-1")
self.calls.clear()
def test_start_runs_the_weak_worker_to_its_budget_and_saves_the_tree_and_the_session(self):
self.assertEqual(handoff_bench.start(self.work, "7-1"), "7-1: started")
(claude, kwargs, _), = self.calls
self.assertEqual(kwargs["cwd"], self.work / "7-1")
self.assertEqual((kwargs["stdin"], kwargs["stderr"]), (subprocess.DEVNULL, subprocess.STDOUT))
self.assertEqual(claude[:2] + claude[3:], [
"claude", "-p", *handoff_bench.WEAK.flags(), "--session-id", str(SESSION), "--max-budget-usd", "0.25", "--setting-sources", "project",
"--permission-mode", "bypassPermissions", "--output-format", "stream-json", "--verbose"])
self.assertIn("Do the seventh thing.", claude[2])
self.assertIn(f"a throwaway clone at `{self.base}`. Implement todo item 7 below", claude[2])
self.assertEqual((self.work / "7-start-1.session").read_text(), str(self.transcript))
self.assertEqual((self.work / "7-start-1.session.jsonl").read_text(), "weak\n")
self.assertIn("-x = 1\n+x = 3\n", (self.work / "7-start-1.diff").read_text())
self.assertIn("+n = 1\n", (self.work / "7-start-1.diff").read_text())
tree = self.work / "7-start-1.tree"
self.assertEqual(((tree / "bin/pkg/mod.py").read_text(), (tree / "bin/pkg/new.py").read_text()), ("x = 3\n", "n = 1\n"))
self.assertEqual(advisor_bench.git(tree, "status", "--porcelain"), " M bin/pkg/mod.py\n?? bin/pkg/new.py\n")
self.assertEqual(advisor_bench.git(tree, "rev-parse", "HEAD").strip(), self.base)
self.assertTrue((tree / "link").is_symlink())
def test_start_refuses_a_worker_that_finished_within_its_budget(self):
self.start_result = {}
with self.assertRaisesRegex(ValueError, "^the weak worker finished within its budget, or its run failed, so there is no handoff to bench$"):
handoff_bench.start(self.work, "7-1")
def test_start_refuses_a_name_that_already_started(self):
(self.work / "7-1").mkdir()
with self.assertRaisesRegex(ValueError, "already started"):
handoff_bench.start(self.work, "7-1")
def test_start_refuses_a_session_whose_transcript_it_cannot_find(self):
self.transcript = self.work / "elsewhere" / f"{SESSION}.jsonl"
with self.assertRaisesRegex(ValueError, "found 0"):
handoff_bench.start(self.work, "7-1")
def test_the_resume_arm_resumes_the_weak_workers_session_on_the_strong_rung(self):
self.started()
self.assertEqual(handoff_bench.run(self.work, "7-resume-1"), "7-resume-1: ran")
(claude, kwargs, _), = self.calls
self.assertEqual(kwargs["cwd"], self.work / "7-1")
self.assertEqual(claude[:2] + claude[3:], [
"claude", "-p", *handoff_bench.STRONG.flags(), "--resume", str(SESSION), "--setting-sources", "project",
"--permission-mode", "bypassPermissions", "--output-format", "stream-json", "--verbose"])
self.assertEqual(claude[2], (handoff_bench.BENCH / "handoff-resume-prompt.md").read_text().format(rung="claude-opus-5-5 at medium effort"))
diff = (self.work / "7-resume-1.diff").read_text()
self.assertIn("+x = 3\n", diff)
self.assertIn("+++ b/bin/pkg/done_1.py", diff)
self.assertEqual((self.work / "7-resume-1.tests").read_text(), "exit 0\n3 passed\n")
self.assertEqual(self.heavy[-1][-1], "bin/pkg")
self.assertNotIn("bin/other", self.heavy[-1])
self.assertTrue((self.work / "7-1" / "link").is_symlink())
self.assertEqual(handoff_bench.outcome(self.work, "7-resume-1"), {"cost": 2.0, "seconds": 12.0})
def test_the_compact_arm_has_the_weak_model_compact_the_session_before_the_strong_one_resumes_it(self):
self.started()
handoff_bench.run(self.work, "7-compact-1")
(compact, _, _), (claude, _, found) = self.calls
self.assertEqual(compact[:2] + compact[3:], [
"claude", "-p", *handoff_bench.WEAK.flags(), "--resume", str(SESSION), "--setting-sources", "project",
"--permission-mode", "bypassPermissions", "--output-format", "stream-json", "--verbose"])
self.assertEqual(compact[2], "/compact")
self.assertEqual(claude[3:7], [*handoff_bench.STRONG.flags()][:4])
self.assertIn("--resume", claude)
self.assertEqual(found[0], "weak\ncompacted\n")
self.assertEqual(handoff_bench.outcome(self.work, "7-compact-1"), {"cost": 2.0, "seconds": 15.0})
def test_the_brief_arm_starts_a_fresh_session_with_the_item_and_the_brief(self):
self.started()
self.assertEqual(handoff_bench.run(self.work, "7-brief-1"), "7-brief-1: ran")
(claude, kwargs, _), = self.calls
self.assertEqual(kwargs["cwd"], self.work / "7-1")
self.assertEqual(claude[:2] + claude[3:], [
"claude", "-p", *handoff_bench.STRONG.flags(), "--no-session-persistence", "--setting-sources", "project",
"--permission-mode", "bypassPermissions", "--output-format", "stream-json", "--verbose"])
self.assertIn("### 7. Do the seventh thing.", claude[2])
self.assertTrue(claude[2].endswith("- End with a two-line summary of what you changed.\n\n\n## Handoff\n\n" + (handoff_bench.BENCH / "handoff-brief-prompt.md").read_text().split("## Handoff\n\n")[1]), claude[2][-400:])
self.assertIn("+++ b/bin/pkg/done_1.py", (self.work / "7-brief-1.diff").read_text())
def test_each_arm_begins_on_the_tree_and_the_session_that_start_left(self):
self.started()
for arm in ("resume", "compact", "brief"):
handoff_bench.run(self.work, f"7-{arm}-1")
self.assertEqual([found for _, _, found in self.calls if found[0] != "weak\ncompacted\n"], [("weak\n", ["mod.py", "new.py"])] * 3)
self.assertNotIn("done_1.py", (self.work / "7-brief-1.diff").read_text())
def test_run_refuses_an_arm_that_already_ran_or_never_started(self):
with self.assertRaisesRegex(ValueError, "start 7-1 first"):
handoff_bench.run(self.work, "7-brief-1")
self.started()
handoff_bench.run(self.work, "7-brief-1")
with self.assertRaisesRegex(ValueError, "already ran"):
handoff_bench.run(self.work, "7-brief-1")
def test_run_refuses_an_unknown_arm(self):
self.started()
with self.assertRaisesRegex(ValueError, "arm"):
handoff_bench.run(self.work, "7-sonnet-1")
def test_run_refuses_an_arm_whose_run_ended_in_an_error_before_it_tests(self):
self.started()
self.strong_result = {"is_error": True, "result": "overloaded"}
with unittest.mock.patch.object(advisor_bench, "save_change") as save_change:
with self.assertRaisesRegex(ValueError, "ended in an error: overloaded"):
handoff_bench.run(self.work, "7-brief-1")
save_change.assert_not_called()
def test_run_refuses_a_compact_arm_whose_compaction_failed_before_the_strong_model_runs(self):
self.started()
self.compact_result = {"is_error": True, "result": "no conversation to compact"}
with self.assertRaisesRegex(ValueError, "^the run ended in an error: no conversation to compact$"):
handoff_bench.run(self.work, "7-compact-1")
self.assertEqual([args[2] for args, _, _ in self.calls], ["/compact"])
self.assertFalse((self.work / "7-compact-1.jsonl").exists())
def test_outcome_refuses_a_stream_with_no_result_line(self):
self.started()
(self.work / "7-brief-1.jsonl").write_text("not json\n[]\n")
with self.assertRaisesRegex(ValueError, "^the stream has no result line$"):
handoff_bench.outcome(self.work, "7-brief-1")
def test_main_grades_with_the_handoff_cases_and_prints_the_summary(self):
with unittest.mock.patch("builtins.print") as printed:
handoff_bench.main([str(self.work), "start", "7-1"])
handoff_bench.main([str(self.work), "run", "7-brief-1"])
self.assertEqual([call.args[0] for call in printed.call_args_list], ["7-1: started", "7-brief-1: ran"])
with unittest.mock.patch.object(advisor_bench, "grade", return_value="7-brief-1: pass") as grade, unittest.mock.patch("builtins.print") as printed:
handoff_bench.main([str(self.work), "grade", "7-brief-1"])
grade.assert_called_once_with(self.work, "7-brief-1", handoff_bench.CASES)
printed.assert_called_once_with("7-brief-1: pass", flush=True)
(self.work / "7-brief-1.verdict").write_text(json.dumps({"verdict": "pass"}))
with unittest.mock.patch("builtins.print") as printed:
handoff_bench.main([str(self.work), "summary"])
self.assertIn("| brief | 1 | 1 | 0 | 0.50 | 12 |", printed.call_args.args[0])
if __name__ == "__main__":
unittest.main()