Refuse a compact arm whose compaction failed, and state the bench as it ran

This commit is contained in:
2026-11-17 03:56:25 +01:00
parent 24800c156b
commit b4490b4a8c
4 changed files with 30 additions and 14 deletions
+2 -2
View File
@@ -557,7 +557,7 @@ The step is one rung up from the rung the worker would otherwise hold: its own,
The archive quotes the judge's passage under `Escalation:` in the unfinished verdict, the interrupt or the quoted answer. The step an unfinished verdict causes is archived after it. The step an interrupt or a quoted answer causes is archived with it, at once: the interrupt or answer names the rung the worker ran on until then, and a step-up archived beside it names the rung it runs on from then. So the routing figures charge each stretch to its own rung, and a loop that dies while the human answers resumes on the stepped rung. Each archived worker line names the mark once there is one, so a resumed loop keeps it. `bin/routing_figures/` counts none of these steps as an escalation by trigger, which it counts only from subagent prompts.
A stepped worker resumes its session, also when the step changes its model. Item 319 proposed a fresh session with a brief of the chunk in its place, after arXiv 2608.24358 measured a fresh strong model ahead of one handed the weak model's transcript. `workflow-bench/handoff_bench.py` benched it on 2026-11-17, on five L 3 items this repo shipped. Sonnet 5.5 at high effort started each one and stopped at $0.36, after 16 to 20 turns. Opus 5.5 at medium effort then finished it three ways: resuming the session, resuming it after `/compact`, and in a fresh session with a brief of the item, the branch and its diff. A judge that did not know the arm graded each change against the tests the item shipped with.
A stepped worker resumes its session, also when the step changes its model. Item 319 proposed a fresh session with a brief of the chunk in its place, after arXiv 2608.24358 measured a fresh strong model ahead of one handed the weak model's transcript. `workflow-bench/handoff_bench.py` benched it on 2026-11-17, on L 3 items this repo shipped. Sonnet 5.5 at high effort started each of six and was stopped at $0.36. It stopped itself on item 163 as it reached the budget, which the bench reads as a finished run, so five items went on, each stopped after 9 to 13 turns. Opus 5.5 at medium effort then finished it three ways: resuming the session, resuming it after `/compact`, and in a fresh session with the item's text and a brief, which says that the unfinished change lies uncommitted in the clone. A judge that did not know the arm graded each change. It saw the item, the change that shipped for it, and the result of the tests run on the worker's tree.
| arm | runs | pass | partial | fail | mean cost $ | mean seconds |
|---|---|---|---|---|---|---|
@@ -565,7 +565,7 @@ A stepped worker resumes its session, also when the step changes its model. Item
| compact | 5 | 1 | 3 | 1 | 0.56 | 198 |
| brief | 5 | 1 | 4 | 0 | 0.78 | 310 |
Resume's verdict matched or beat the brief's on every item, and the brief cost less on two of the five. So the bench gives no reason to pay for the change, which is larger than the worker's start: the archive files a chunk under its session's first line, so a chunk that spans two sessions would need the archive, and each figure read from it, to join them. The bench is small: each arm ran once per item, the judge is the strong arm's model, and no start was longer than 20 turns. Run it again before changing this, with longer starts, since the paper's gap may open only on a long transcript.
Resume's verdict matched or beat the brief's on every item, and the brief cost less on two of the five. So the bench gives no reason to pay for a fresh session. Its price is more than a change to how the worker starts. The archive files a chunk under its session's first line. For a chunk that spans two sessions, the archive, and each figure read from it, would have to join them. The bench is small: each arm ran once per item, the judge is Opus 5.5 too, and no first run was longer than 13 turns. Run it again before changing this, with longer first runs, since the paper's gap may open only on a long transcript.
Valid while the worker reports in its transcript the triggers that fired, and while a chunk's session transcript only grows.
+1 -1
View File
@@ -151,7 +151,7 @@ def run(work, name):
def save_change(work, name, clone, commits):
"""Save the clone's change since the case's start commit as <name>.diff, and the result of the tests of the packages that it or the merged change touches as <name>.tests."""
"""Save the clone's change since the case's start commit as <name>.diff, and as <name>.tests the result of the tests of the packages that the clone's change or the merged change touches."""
base, merged = commits
git(clone, "add", "--all")
(work / f"{name}.diff").write_text(git(clone, "diff", "--cached", base))
+14 -6
View File
@@ -6,7 +6,8 @@ best? Each case is an archived chunk of this repo, as in advisor_bench.py, whose
- `start` runs WEAK on the item in the clone <work dir>/<case>-<n> until it has spent START_BUDGET, which
stands in for a run that ends asking for a higher rung. A worker that finishes within the budget leaves no
handoff, so `start` fails on it. It saves the change so far (<case>-start-<n>.diff), and a copy of the clone
(.tree) and of the session's transcript (.session.jsonl), which every arm starts from.
(.tree), which every arm starts from, and a copy of the session's transcript (.session.jsonl), which the resume
and compact arms continue.
- `run` puts the clone and the session back as `start` left them, then continues on STRONG, one arm of a case at a
time. The resume arm resumes WEAK's session with handoff-resume-prompt.md. The compact arm does the same once WEAK
has compacted the session with `/compact`, whose stream it saves as <name>.compact.jsonl. The brief arm starts a
@@ -48,7 +49,7 @@ CASES = {
"239": ("d583f8d48", "47a40757"),
}
# Dollars WEAK may spend before the handoff. In a trial, $0.36 stopped Sonnet 5.5 on items 163 and 227 after 11 to 14
# turns, with about 250 lines of diff.
# turns, as the result line's `num_turns` counts them, with about 250 lines of diff.
START_BUDGET = 0.36
CLAUDE_FLAGS = ["--setting-sources", "project", "--permission-mode", "bypassPermissions", "--output-format", "stream-json", "--verbose"]
@@ -60,7 +61,7 @@ def stopped_at_budget(lines):
entry = json.loads(line)
except ValueError:
continue
if entry.get("type") == "result":
if isinstance(entry, dict) and entry.get("type") == "result":
return entry.get("subtype") == "error_max_budget_usd"
return False
@@ -122,6 +123,7 @@ def run(work, name):
resumed = ["--resume", transcript.stem]
if arm == "compact":
claude("/compact", [*WEAK.flags(), *resumed], clone, work / f"{name}.compact.jsonl")
finished(work / f"{name}.compact.jsonl")
if arm == "brief":
claude(run_prompt(case) + "\n" + (BENCH / "handoff-brief-prompt.md").read_text(), [*STRONG.flags(), "--no-session-persistence"], clone, stream)
else:
@@ -145,12 +147,18 @@ def results(stream):
return found
def finished(stream):
"""The result lines of a run that ended well; a run without one, or whose last one is an error, raises."""
ran = results(stream)
if not ran or ran[-1].get("is_error"):
raise ValueError(f"the run ended in an error: {ran[-1].get('result')}" if ran else "the stream has no result line")
return ran
def outcome(work, name):
"""The cost and seconds of an arm's run, its compaction included. A resumed session's cost counts from the session's start, so the start's cost comes off it."""
case, arm, number = advisor_bench.split_name(name)
ran = results(work / f"{name}.jsonl")
if not ran or ran[-1].get("is_error"):
raise ValueError(f"the run ended in an error: {ran[-1].get('result')}" if ran else "the stream has no result line")
ran = finished(work / f"{name}.jsonl")
compacted = results(work / f"{name}.compact.jsonl") if arm == "compact" else []
before = 0 if arm == "brief" else results(work / f"{case}-start-{number}.jsonl")[-1]["total_cost_usd"]
return {"cost": ran[-1]["total_cost_usd"] - before, "seconds": sum(entry["duration_ms"] for entry in [*compacted, *ran]) / 1000}
+13 -5
View File
@@ -25,7 +25,7 @@ class StoppedAtBudget(unittest.TestCase):
self.assertFalse(handoff_bench.stopped_at_budget([result_line()]))
def test_fails_where_the_stream_has_no_result(self):
self.assertFalse(handoff_bench.stopped_at_budget(['{"type": "assistant"}']))
self.assertFalse(handoff_bench.stopped_at_budget(['{"type": "assistant"}', "[]"]))
class Summary(unittest.TestCase):
@@ -61,7 +61,7 @@ class Orchestration(unittest.TestCase):
self.commit({"bin/other/o.py": "o = 2\n"})
self.calls, self.heavy = [], []
self.start_result = {"subtype": "error_max_budget_usd", "is_error": True, "total_cost_usd": 1.0}
self.strong_result = {}
self.compact_result, self.strong_result = {}, {}
real_run = subprocess.run
for patch in (
unittest.mock.patch.object(advisor_bench, "REPO", self.repo),
@@ -100,7 +100,7 @@ class Orchestration(unittest.TestCase):
kwargs["stdout"].write(result_line(**self.start_result) + "\n")
elif args[2] == "/compact":
self.transcript.write_text(self.transcript.read_text() + "compacted\n")
kwargs["stdout"].write(result_line(total_cost_usd=1.25, duration_ms=3000) + "\n")
kwargs["stdout"].write(result_line(**self.compact_result, total_cost_usd=1.25, duration_ms=3000) + "\n")
else:
if "--resume" in args:
self.transcript.write_text(self.transcript.read_text() + "strong\n")
@@ -193,8 +193,8 @@ class Orchestration(unittest.TestCase):
self.assertEqual(claude[:2] + claude[3:], [
"claude", "-p", *handoff_bench.STRONG.flags(), "--no-session-persistence", "--setting-sources", "project",
"--permission-mode", "bypassPermissions", "--output-format", "stream-json", "--verbose"])
self.assertEqual(claude[2], handoff_bench.run_prompt("7") + "\n" + (handoff_bench.BENCH / "handoff-brief-prompt.md").read_text())
self.assertIn("## Handoff", claude[2])
self.assertIn("### 7. Do the seventh thing.", claude[2])
self.assertTrue(claude[2].endswith("- End with a two-line summary of what you changed.\n\n\n## Handoff\n\n" + (handoff_bench.BENCH / "handoff-brief-prompt.md").read_text().split("## Handoff\n\n")[1]), claude[2][-400:])
self.assertIn("+++ b/bin/pkg/done_1.py", (self.work / "7-brief-1.diff").read_text())
def test_each_arm_begins_on_the_tree_and_the_session_that_start_left(self):
@@ -225,6 +225,14 @@ class Orchestration(unittest.TestCase):
handoff_bench.run(self.work, "7-brief-1")
save_change.assert_not_called()
def test_run_refuses_a_compact_arm_whose_compaction_failed_before_the_strong_model_runs(self):
self.started()
self.compact_result = {"is_error": True, "result": "no conversation to compact"}
with self.assertRaisesRegex(ValueError, "^the run ended in an error: no conversation to compact$"):
handoff_bench.run(self.work, "7-compact-1")
self.assertEqual([args[2] for args, _, _ in self.calls], ["/compact"])
self.assertFalse((self.work / "7-compact-1.jsonl").exists())
def test_outcome_refuses_a_stream_with_no_result_line(self):
self.started()
(self.work / "7-brief-1.jsonl").write_text("not json\n[]\n")