diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e184ec6..a17ba2d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Unreleased -- A bench of the advisor tool, `workflow-bench/advisor_bench.py`, found Sonnet 5.5 alone passing six archived L 1 and L 2 chunks as Opus 5.5 at medium did, for 41% less, and Sonnet's advisor called once in six runs, so workers stay without it. +- Add `workflow-bench/advisor_bench.py`, a bench of the advisor tool on six archived L 1 and L 2 chunks. Sonnet 5.5 alone passed all six, as Opus 5.5 at medium did, for 41% less, and its advisor ran in one of six runs, so workers stay without it. - loop-claude's judge on an Opus 5.5 project now runs on Opus 5.5 at xhigh in every chunk, never at medium: in replays medium sometimes closed a chunk whose worker had asked a question the README or records answer, where xhigh quoted the answer. The judge never runs on Fable 5.2 either, since each Fable 5.2 rung replayed missed an interrupt. `judge-replay` now also replays each rung above the project's that a review climbs to, and reports what each misses. ## 0.8.0 diff --git a/docs/decisions.md b/docs/decisions.md index bdc2230d..45c76ae0 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -28,7 +28,6 @@ Each project principle states one rule that the decisions below apply; their ent - review: agents of one type launch in one message, not one a message before the others - routing: a worker delegates a task only where it needs just the task's conclusion, and each subagent runs on the rung its task's L picks from the project's rung, never below the project's rung when it files or scores todo.md items - routing: a worker hands a task one rung up on each escalation trigger, up to two rungs above the rung its item picks unless the human agrees, and opens the subagent's prompt with the trigger so the routing figures count it -- routing: a worker runs without the advisor tool, since six archived L 1 and L 2 chunks showed Sonnet 5.5 alone passing as often as Opus 5.5 at medium for less, and its advisor was called once in six runs - routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, steps one rung up as the judge interrupts or quotes an answer, or next cycle, up to two rungs above the rung its item picks - routing: loop-claude reads the items a chunk takes from the worker's latest `Chunk items:` line and scores them by `todo.md` as the default branch held it when the cycle started; the judge reports the escalation and whether a release is ready to cut - routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds @@ -417,19 +416,19 @@ loop-claude's prompt names the rungs a trigger opens, so the worker computes no Valid while the ladder orders its rungs by competence. -## routing: a worker runs without the advisor tool, since six archived L 1 and L 2 chunks showed Sonnet 5.5 alone passing as often as Opus 5.5 at medium for less, and its advisor was called once in six runs +## routing: a worker runs without the advisor tool 2026-11-12, lilleman, settled by the bench below. Serves goal 1.4: a switch that raises cost without raising quality is no saving. Valid while the ladder's L 1 and L 2 workers run on Sonnet 5.5 and the advisor would be Opus 5.5. -`claude -p` calls the advisor: a probe with `--advisor claude-opus-5-5` made one call, which read 20,187 uncached input tokens, the whole conversation, and cost $0.093 beside the worker's $0.036. `workflow-bench/advisor_bench.py` ran six archived chunks of this repo (todo items 290, 251 and 218 at L 1, and 289, 229 and 214 at L 2) once on each of three arms: Sonnet 5.5 at high with the advisor, Sonnet 5.5 at high alone, and Opus 5.5 at medium. A judge on Opus 5.5 at medium, blind to the arm, ruled each run's change against the change that shipped, with the tests of the packages either change touches. +`claude -p` calls the advisor. A probe with `--advisor claude-opus-5-5` made one call: it read the whole conversation, 20,187 uncached input tokens, and cost $0.093 beside the worker's $0.036. `workflow-bench/advisor_bench.py` ran six archived chunks of this repo (todo items 290, 251 and 218 at L 1, and 289, 229 and 214 at L 2) once on each of three arms: Sonnet 5.5 at high with the advisor, Sonnet 5.5 at high alone, and Opus 5.5 at medium. A judge on Opus 5.5 at medium, blind to the arm, ruled each run's change against the change that shipped, with the tests of the packages either change touches. -| Arm | Passes | Mean cost | Mean seconds | Advisor calls | +| Arm | Passes | Mean cost | Mean seconds | Runs with an advisor call | |---|---|---|---|---| -| Sonnet 5.5 with the advisor | 6 of 6 | $0.37 | 286 | 1 run of 6 | -| Sonnet 5.5 alone | 6 of 6 | $0.27 | 151 | 0 | -| Opus 5.5 at medium | 6 of 6 | $0.46 | 197 | 0 | +| Sonnet 5.5 with the advisor | 6 of 6 | $0.37 | 286 | 1 of 6 | +| Sonnet 5.5 alone | 6 of 6 | $0.27 | 151 | 0 of 6 | +| Opus 5.5 at medium | 6 of 6 | $0.46 | 197 | 0 of 6 | -Every arm passed every case, so the bench found no quality gap for an advisor to close, and Sonnet alone cost 41% less than Opus at medium. The advisor's one call, in item 214's run, cost $0.72 of that run's $1.24, against $0.41 for Sonnet alone. Anthropic's reported gain (https://claude.com/blog/the-advisor-strategy) came on SWE-bench Multilingual, a harder set than L 1 and L 2 chunks. This bench cannot say whether the advisor pays on items above L 2: it ran one run per arm and case, and its judge is of the worker's own model family. +Every arm passed every case, so the bench found no quality gap for an advisor to close, and Sonnet alone cost 41% less than Opus at medium. The advisor's one call, in item 214's run, cost $0.72 of that run's $1.24, against $0.41 for Sonnet alone. Anthropic's reported gain (https://claude.com/blog/the-advisor-strategy) came on SWE-bench Multilingual, a harder set than L 1 and L 2 chunks. The bench ran each arm once per case, and its judge is of the worker's own model family, so it shows a direction only. It tested L 1 and L 2 items alone. Workers on items above L 2 run without the advisor too, since nothing shows it pays there, until a bench of such items does. ## routing: a worker whose run ends asking for a higher rung, citing an escalation trigger the judge finds in the transcript, steps one rung up as the judge interrupts or quotes an answer, or next cycle, up to two rungs above the rung its item picks diff --git a/workflow-bench/advisor-judge-prompt.md b/workflow-bench/advisor-judge-prompt.md index 9d1fa1cb..759709f8 100644 --- a/workflow-bench/advisor-judge-prompt.md +++ b/workflow-bench/advisor-judge-prompt.md @@ -1,6 +1,6 @@ You judge one worker's change to a todo item. You see the item, the change that shipped for it, the worker's change, and the result of the tests of the packages that either change touches, run on the worker's tree. You don't know which model wrote the worker's change. -Rule `pass` where the worker's change delivers what the item asks, with a test that fails without the change, and those tests pass or fail only on tests the change cannot have touched. Rule `partial` where it delivers most of the item or lacks the test, and `fail` where it misses the item, breaks those tests, or changes unrelated code. The shipped change is one correct answer, not the only one. +Rule `pass` where the worker's change does what the item asks, comes with a test that fails without the change, and causes no failing test. Rule `partial` where it does most of the item or lacks that test. Rule `fail` where it misses the item, causes a failing test, or changes unrelated code. The shipped change is one correct answer, not the only one. {text} diff --git a/workflow-bench/advisor-run-prompt.md b/workflow-bench/advisor-run-prompt.md index e6681a82..6d1f0a26 100644 --- a/workflow-bench/advisor-run-prompt.md +++ b/workflow-bench/advisor-run-prompt.md @@ -4,7 +4,7 @@ You are the worker of one chunk of this repository, a throwaway clone at `{base} ## Bench rules -- The clone is yours and has no remote: make no worktree and no branch, open no pull request, push nothing, run no review pass, and don't edit `todo.md`, `CHANGELOG.md` or `docs/`. Leave your change uncommitted. +- The clone is yours and has no remote: make no worktree and no branch, open no pull request, don't push anything, run no review pass, and don't edit `todo.md`, `CHANGELOG.md` or `docs/`. Leave your change uncommitted. - Write the test first where a test can check the change, and see it fail. - Run the tests as `README.md` → Test says. No human is reachable: where the item leaves a choice open, take your recommendation. - End with a two-line summary of what you changed.