Answer architecture round 3: 0 🔴, 1 🟡 (0 misread); every suite sits beside the module it tests, and test_loop_claude.py holds only loop-claude's own
CI / tests (pull_request) Successful in 5m17s
CI / tests (pull_request) Successful in 5m17s
This commit is contained in:
@@ -222,10 +222,10 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests. Beside the packages, `bin/`'s root holds only the launchers, the tools of one file with their tests, `test_env.py`, `failure.py` and `setup.cfg`. `failure.py`'s `Failure`, which modules across `bin/` return for a failed step, belongs to no single package. `agent_cycle/`, `atlassian/`, `branch_chunks/`, `chunk_driver/`, `claude_code/`, `command_lines/`, `forge/`, `git_cli/`, `loop_effects/`, `loop_state/`, `machine_load/`, `model_routing/`, `notifier/`, `plan/`, `plan_store/`, `record_store/`, `release_plan/`, `review_rules/`, `rung_replay/`, `text_files/` and `transcript_archive/` serve more than one tool. No package imports one that imports it back, which `repo_checks/test_package_cycles.py` checks.
|
||||
|
||||
- `adf_codec/` converts markdown to ADF, the rich text of Jira Cloud and Confluence, and back, in a throwaway container running a pinned `@larvit/adf-codec`. It builds the container's image inside a heavy slot.
|
||||
- `agent_cycle/` runs one cycle (`run.py`): a headless claude worker, then the judge of its transcript. It also holds the interrupt, the tmux window and the routing of the worker's rung. The tool that drives the cycles supplies the prompts and file names of its own workflow (`worker_prompts.py`), and reads which items a worker took in. `loop_claude` and `do_epic` drive it; `agent_cycle` doesn't import either.
|
||||
- `agent_cycle/` runs one cycle (`run.py`): a headless claude worker, then the judge of its transcript. It also holds the interrupt, the tmux window and the routing of the worker's rung. The tool that drives the cycles supplies the prompts and file names of its own workflow (`worker_prompts.py`), and reads which items a worker took in. `loop_claude` and `do_epic` drive it; `agent_cycle` doesn't import either. `doubles.py` holds the doubles its tests share with `chunk_driver`'s.
|
||||
- `atlassian/` holds what the Jira and Confluence backends share (`client.py`): one Atlassian account's credentials, the request both backends send, and each service's error messages. `jira.py` holds what every reader of Jira shares: a search read page by page, the head of each entry it answers, an issue's links and the transition that finishes it, for `plan_store` and `do_epic`.
|
||||
- `branch_chunks/` cuts a default branch into chunks and finds, for each, the items it shipped and the later chunks that fixed its code, for `score_outcomes` and `routing_figures`. It imports `reading_cost`'s record path.
|
||||
- `chunk_driver/` holds what `loop_claude` and `do_epic` share as they drive chunks through `agent_cycle`: how a cycle starts (`cycle_start.py`) and closes (`chunk_run.py`), the claim on a checkout before a chunk starts in it (`claims.py`), the prompts every worker gets (`prompts.py`), and a checkout's premises, records and saved models (`project/`). It imports neither tool, and `repo_checks` fails any package that imports one of the two. `agent_cycle` is the layer below it, and `repo_checks` fails `agent_cycle` where it imports `chunk_driver`, tests included.
|
||||
- `chunk_driver/` holds what `loop_claude` and `do_epic` share as they drive chunks through `agent_cycle`: how a cycle starts (`cycle_start.py`) and closes (`chunk_run.py`), the claim on a checkout before a chunk starts in it (`claims.py`), the prompts every worker gets (`prompts.py`), and a checkout's premises, records and saved models (`project/`). `doubles.py` holds the loop its tests share with `loop_claude`'s. It imports neither tool, and `repo_checks` fails any package that imports one of the two. `agent_cycle` is the layer below it, and `repo_checks` fails `agent_cycle` where it imports `chunk_driver`, tests included.
|
||||
- `command_lines/` reads a shell command: its simple commands, and a docker call's subcommand and memory limits, for `command_patterns`, `reading_cost`, `routing_figures`, `rule_hook` and `run_heavy`.
|
||||
- `deploy_release` imports `release_plan`'s `release_standing`.
|
||||
- `do_epic/` holds `do-epic`, the tool that drives one Jira epic. `main.py` starts the epic's chunks and runs one chunk's cycles in its window, through `agent_cycle`. `planning.py` settles, before the chunks start, where each open issue is worked and the order of the issues do-epic filed, through `split.py`, `repo_pick.py` and `ranking.py`. `scheduler.py` says which issues run now: as many as the epic's cap and each repo's `parallel-chunks` allow, and none in a repo where loop-claude runs chunks. `gates.py` reads once whether a chunk's pull request waits on its checks, an approval or a new review, and `parked.py` keeps each chunk that does in a state file, so it takes no running slot while it waits and goes on waiting after a reboot. `chunk_prompt.py` words what a chunk's worker reads. Through `findings.py` that worker files a finding as an issue of the epic, with `--file`, and appends a decision to its description, with `--record-decision`. `finish.py` ends an epic that holds no open issue: its finishing chunk files a release review of each repo the epic changed, has the human confirm each goal, and with `--close-epic` posts the summary on the epic and closes it. `config.py` reads its machine-wide settings, importing `project_config`'s settings and `model_routing`'s ladder file, and `epic_record.py` reads and writes an epic's description by section, language line and cap line, through `record_store`'s `EpicDescription`. `repos.py` finds a repo's checkout and project config by its name, and clones a missing checkout from the host and owner of the epic's other repos, through `forge`'s `url_beside`. `repo_pick.py` asks claude, by `repo-rule.md`, where an epic item is worked, and records the answer as a label on the issue. `split.py` has claude predict from the code which repos an item changes, and files a sub-task per repo under it where there are several. `ranking.py` orders an epic's items by score over one chain of goals, the epic's, then code health, then the item's repo's, through `plan`'s ranking, and re-ranks the tracker's backlog to that order through `backend/`. `backend/` reads, files and ranks the epic's issues, adds the labels `repo_pick.py` and `split.py` record, posts comments, moves a finished chunk's issue to done, and closes the epic. `e2e/` holds `do-epic-e2e`.
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
"""Test doubles for the suites that run a cycle: the tmux window it runs in, and the prompts of an entry point that exists only in tests."""
|
||||
from agent_cycle import tmux
|
||||
from agent_cycle.worker_prompts import WorkerPrompts
|
||||
|
||||
WINDOW = tmux.Window("loop-proj", "item-7", "%4")
|
||||
# The prompts of an entry point that exists only in tests, so a cycle's tests read no real entry point's wording.
|
||||
PROMPTS = WorkerPrompts(
|
||||
driver="a-driver",
|
||||
fresh=lambda task, premises, route, project, facts: f"Fresh: {task}\n\n{premises}\n\n{facts}",
|
||||
resumed=lambda evidence, route, project: f"Resumed: {evidence}",
|
||||
rungs_held=lambda route, project: f" You hold {route.rung}.",
|
||||
quote_sources="the project's records",
|
||||
answer_destinations="the plan or a comment",
|
||||
records_named="the project's decisions",
|
||||
)
|
||||
@@ -21,7 +21,7 @@ from model_routing.ladder import Rung
|
||||
from agent_cycle.stop import Stop
|
||||
from agent_cycle import tmux
|
||||
from text_files.utc_stamp import utc
|
||||
from agent_cycle.test_tmux import WINDOW
|
||||
from agent_cycle.doubles import WINDOW
|
||||
|
||||
|
||||
class Interrupts(unittest.TestCase):
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import dataclasses
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
@@ -10,6 +11,8 @@ from loop_effects.doubles import assistant, FakeEffects, human, judged, kinds
|
||||
from failure import Failure
|
||||
from agent_cycle.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished
|
||||
from agent_cycle import judge
|
||||
from agent_cycle.doubles import PROMPTS
|
||||
from model_routing.route import Route
|
||||
from model_routing.ladder import Rung
|
||||
from agent_cycle.judge import verdict_from
|
||||
|
||||
@@ -123,5 +126,52 @@ class Verdict(unittest.TestCase):
|
||||
self.assertEqual(verdict, Unfinished("round 2: 🔴", entries_read=2))
|
||||
|
||||
|
||||
class JudgePrompt(unittest.TestCase):
|
||||
def brief(self, entries, escalated_at, answer_destinations="the plan or a comment"):
|
||||
"""What the judge of a-driver reads of a transcript holding entries."""
|
||||
return judge.Brief("Do it.", "## Goals", "", "the project's decisions", answer_destinations, kinds(*entries), escalated_at, "a-driver")
|
||||
|
||||
def test_shows_the_transcripts_tail_and_returns_what_it_showed(self):
|
||||
entries = [assistant({"type": "text", "text": "old " + "x" * judge.JUDGE_PROMPT_CHARS}), assistant({"type": "text", "text": "new"})]
|
||||
prompt, grounds = judge.judge_prompt(self.brief(entries, 0))
|
||||
self.assertTrue(grounds.shown.endswith("\n\nnew"))
|
||||
self.assertNotIn("old", grounds.shown)
|
||||
self.assertIn(grounds.shown, prompt)
|
||||
self.assertLessEqual(len(prompt), judge.JUDGE_PROMPT_CHARS)
|
||||
self.assertEqual(grounds.after_mark, "old " + "x" * judge.JUDGE_PROMPT_CHARS + "\n\nnew")
|
||||
self.assertEqual(grounds.quotable, "## Goals\n\n")
|
||||
|
||||
def test_the_judge_prompt_and_schema_call_the_worker_the_worker(self):
|
||||
prompt, _ = judge.judge_prompt(self.brief([assistant({"type": "text", "text": "done"})], 1))
|
||||
self.assertNotRegex(prompt + json.dumps(judge.judge_schema("a-driver")), r"\b[Aa]gent")
|
||||
|
||||
def test_the_judge_prompt_names_the_entry_points_answer_destinations_in_both_rules_on_a_recorded_answer(self):
|
||||
prompt, _ = judge.judge_prompt(self.brief([assistant({"type": "text", "text": "done"})], 0, "the ledger"))
|
||||
self.assertIn("is recorded in the ledger at the one place it governs", prompt)
|
||||
self.assertIn("the worker writing it into the ledger, but not into one of the human's rules", prompt)
|
||||
|
||||
def test_shows_the_judge_where_the_worker_last_stepped_up_and_checks_an_escalation_only_after_it(self):
|
||||
entries = [assistant({"type": "text", "text": "round 2: 🔴"}), assistant({"type": "text", "text": "round 3: 🔴"})]
|
||||
prompt, grounds = judge.judge_prompt(self.brief(entries, 1))
|
||||
self.assertRegex(prompt, "round 2: 🔴\n\n\\[step\\] [^\n]+\n\nround 3: 🔴\n\n")
|
||||
self.assertEqual(grounds.after_mark, "round 3: 🔴")
|
||||
prompt, grounds = judge.judge_prompt(self.brief(entries, 0))
|
||||
self.assertNotIn("stepped the worker up", prompt)
|
||||
self.assertEqual(grounds.shown, "round 2: 🔴\n\nround 3: 🔴")
|
||||
self.assertEqual(grounds.after_mark, "round 2: 🔴\n\nround 3: 🔴")
|
||||
|
||||
def test_drops_an_escalation_quote_from_after_the_step_up_that_the_tail_it_shows_cuts_off(self):
|
||||
entries = [assistant({"type": "text", "text": "round 2: 🔴"}), assistant({"type": "text", "text": "old " + "x" * judge.JUDGE_PROMPT_CHARS}), assistant({"type": "text", "text": "new"})]
|
||||
_, grounds = judge.judge_prompt(self.brief(entries, 1))
|
||||
self.assertEqual(judge.quotes_checked(Unfinished("e", "old x", entries_read=3), grounds), Unfinished("e", entries_read=3))
|
||||
self.assertEqual(judge.quotes_checked(Unfinished("e", "new", entries_read=3), grounds), Unfinished("e", "new", entries_read=3))
|
||||
|
||||
def test_shows_a_typed_answer_before_the_step_up_as_the_humans(self):
|
||||
typed = PROMPTS.typed("Which one?", "Use B.", Route(Rung("worker-model")), None, False)
|
||||
entries = [{**human(typed), "promptSource": "sdk"}, assistant({"type": "text", "text": "round 3: 🔴"})]
|
||||
_, grounds = judge.judge_prompt(self.brief(entries, 1))
|
||||
self.assertIn("\n\n[human] Use B.\n\n[step] ", grounds.shown)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -1,19 +1,109 @@
|
||||
import dataclasses
|
||||
import pathlib
|
||||
import sys
|
||||
import types
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from agent_cycle.verdicts import Interrupt, TakenIn, Unfinished
|
||||
from model_routing.ladder import ProjectRung, Rung
|
||||
from model_routing.ladder_file import read_ladder, SHIPPED
|
||||
from model_routing.route import Route
|
||||
from agent_cycle import routing
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
LADDER = read_ladder(SHIPPED)
|
||||
OPUS_PROJECT = ProjectRung(LADDER, Rung("claude-opus-5-5", "medium"))
|
||||
# The ladder's rungs around the project's: the one below it, its own, and those one to four steps up.
|
||||
SONNET = Rung("claude-sonnet-5-5", "high")
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
ONE_UP = Rung("claude-fable-5-2", "low")
|
||||
TWO_UP = Rung("claude-fable-5-2", "medium")
|
||||
THREE_UP = Rung("claude-opus-5-5", "xhigh")
|
||||
FOUR_UP = Rung("claude-fable-5-2", "high")
|
||||
|
||||
|
||||
def scored(likelihood, escape_and_impact):
|
||||
"""Item 3 with that L and E + I, as routing.Scored reads an item."""
|
||||
return types.SimpleNamespace(id=3, likelihood=likelihood, escape_and_impact=escape_and_impact)
|
||||
|
||||
|
||||
def unfinished(likelihood=0, escalated_at=None):
|
||||
"""An unfinished verdict: the worker took in an item of that L, 0 for none, and the judge found an escalation after reading escalated_at transcript entries, None for none."""
|
||||
return Unfinished("e", escalation="round 2: 🔴" if escalated_at is not None else "", taken_in=TakenIn(highest_likelihood=likelihood), entries_read=escalated_at)
|
||||
|
||||
|
||||
def interrupted(likelihood=0, escalated_at=None):
|
||||
"""An interrupt, its arguments as unfinished's."""
|
||||
return Interrupt("e", "q", escalation="round 2: 🔴" if escalated_at is not None else "", taken_in=TakenIn(highest_likelihood=likelihood), entries_read=escalated_at)
|
||||
|
||||
|
||||
class FirstRoute(unittest.TestCase):
|
||||
def test_starts_a_chunk_on_the_rung_its_items_l_picks_below_the_projects_too(self):
|
||||
self.assertEqual(routing.first_route(OPUS_PROJECT, scored(4, 5)), Route(TWO_UP, 3, 4, 5))
|
||||
self.assertEqual(routing.first_route(OPUS_PROJECT, scored(1, 5)), Route(SONNET, 3, 1, 5))
|
||||
self.assertEqual(routing.first_route(OPUS_PROJECT, None), Route(OPUS))
|
||||
|
||||
|
||||
class NextRoute(unittest.TestCase):
|
||||
def test_steps_up_a_rung_after_every_second_unfinished_verdict(self):
|
||||
once = routing.next_route(OPUS_PROJECT, Route(ONE_UP, 3, 3, 4), unfinished())
|
||||
self.assertEqual(once, Route(ONE_UP, 3, 3, 4, unfinished=1))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, once, unfinished()), Route(TWO_UP, 3, 3, 4))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(LADDER.rungs[-1], unfinished=1), unfinished()).rung, LADDER.rungs[-1])
|
||||
|
||||
def test_steps_up_to_the_rung_a_taken_in_items_l_picks_and_never_down(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(OPUS), unfinished(4)).rung, TWO_UP)
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(THREE_UP), unfinished(4)).rung, THREE_UP)
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(OPUS, unfinished=1), unfinished(4)).rung, THREE_UP)
|
||||
|
||||
def test_steps_up_from_a_rung_below_the_projects_one_rung_at_a_time(self):
|
||||
low = Route(SONNET, 3, 1, 3, unfinished=1)
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, low, unfinished()).rung, OPUS)
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(low.rung, 3, 1, 3), unfinished()).rung, low.rung)
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(low.rung, 3, 1, 3), unfinished(3)).rung, ONE_UP)
|
||||
|
||||
def test_steps_up_a_rung_on_an_escalation_the_judge_found_up_to_two_above_the_items_rung_marking_the_transcript_entries_it_read(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(TWO_UP, 3, 4, 5), unfinished(escalated_at=7)), Route(THREE_UP, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(THREE_UP, 3, 4, 5, escalated_at=7), unfinished(escalated_at=9)), Route(FOUR_UP, 3, 4, 5, escalated_at=9))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(FOUR_UP, 3, 4, 5, escalated_at=9), unfinished(escalated_at=12)), Route(FOUR_UP, 3, 4, 5, unfinished=1, escalated_at=9))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(TWO_UP, 3, 4, 5, unfinished=1), unfinished(escalated_at=7)), Route(THREE_UP, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(TWO_UP), unfinished(escalated_at=7)), Route(TWO_UP, unfinished=1))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(SONNET, 3, 2, 4), unfinished(4, escalated_at=7)), Route(TWO_UP, 3, 2, 4, unfinished=1))
|
||||
|
||||
def test_keeps_the_mark_through_a_step_no_escalation_took(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(ONE_UP, 3, 3, 4, unfinished=1, escalated_at=5), unfinished()), Route(TWO_UP, 3, 3, 4, escalated_at=5))
|
||||
|
||||
def test_steps_up_a_rung_as_the_judge_interrupts_up_to_two_above_the_items_rung_without_counting_an_unfinished_verdict(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(TWO_UP, 3, 4, 5, unfinished=1), interrupted(escalated_at=7)), Route(THREE_UP, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(THREE_UP, 3, 4, 5, escalated_at=7), interrupted(escalated_at=9)), Route(FOUR_UP, 3, 4, 5, escalated_at=9))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(FOUR_UP, 3, 4, 5, unfinished=1, escalated_at=9), interrupted(escalated_at=12)), Route(FOUR_UP, 3, 4, 5, unfinished=1, escalated_at=9))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(SONNET, 3, 2, 4), interrupted(escalated_at=7)), Route(OPUS, 3, 2, 4, escalated_at=7))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(TWO_UP, 3, 4, 5, unfinished=1, escalated_at=2), interrupted(4)), Route(TWO_UP, 3, 4, 5, unfinished=1, escalated_at=2))
|
||||
|
||||
def test_raises_the_worker_to_a_taken_in_items_rung_above_the_escalation_cap_without_a_step_as_the_judge_interrupts(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(OPUS, 3, 1, 3, escalated_at=2), interrupted(4, escalated_at=7)), Route(TWO_UP, 3, 1, 3, escalated_at=2))
|
||||
|
||||
def test_steps_up_from_the_rung_a_taken_in_item_picks_as_the_judge_interrupts(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(ONE_UP, 3, 3, 4), interrupted(4, escalated_at=7)), Route(THREE_UP, 3, 3, 4, escalated_at=7))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(TWO_UP, 3, 4, 5), interrupted(4, escalated_at=7)), Route(THREE_UP, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(FOUR_UP, 3, 4, 5, unfinished=1), interrupted(4, escalated_at=7)), Route(FOUR_UP, 3, 4, 5, unfinished=1))
|
||||
|
||||
def test_a_rung_an_edit_took_off_the_ladder_counts_as_the_projects(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(Rung("claude-opus-5-5", "low")), unfinished()).rung, OPUS)
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(Rung("claude-opus-5-5", "low")), unfinished(2)).rung, OPUS)
|
||||
|
||||
|
||||
class StepUp(unittest.TestCase):
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
|
||||
def test_names_no_step_up_where_only_the_count_of_unfinished_verdicts_changed(self):
|
||||
route = Route(self.OPUS, 3, 3, 4)
|
||||
route = Route(OPUS, 3, 3, 4)
|
||||
self.assertIsNone(routing.step_up(route, dataclasses.replace(route, unfinished=1)))
|
||||
self.assertEqual(routing.step_up(route, route.taking_in(6)), routing.StepUp(route, route.taking_in(6)))
|
||||
|
||||
|
||||
class JudgeRung(unittest.TestCase):
|
||||
def test_runs_the_judge_of_a_replayed_model_on_its_models_highest_rung(self):
|
||||
self.assertEqual(routing.judge_rung(LADDER, "claude-opus-5-5"), THREE_UP)
|
||||
# A model off the ladder, or not replayed, gets no effort flag.
|
||||
self.assertEqual(routing.judge_rung(LADDER, "judge-model"), Rung("judge-model"))
|
||||
self.assertEqual(routing.judge_rung(LADDER, "claude-sonnet-5-5"), Rung("claude-sonnet-5-5"))
|
||||
|
||||
@@ -23,6 +23,7 @@ from transcript_archive import outcomes, write
|
||||
from loop_state import paths
|
||||
from transcript_archive.location import project_archive
|
||||
from model_routing.ladder import ProjectRung, Rung
|
||||
from agent_cycle.doubles import PROMPTS, WINDOW
|
||||
from model_routing.ladder_file import read_ladder, SHIPPED
|
||||
from model_routing.route import Route
|
||||
from agent_cycle import routing
|
||||
@@ -31,8 +32,6 @@ from claude_code import session
|
||||
from loop_effects.child_shells import Shell
|
||||
from agent_cycle.stop import Stop
|
||||
from text_files.utc_stamp import utc
|
||||
from agent_cycle.test_tmux import WINDOW
|
||||
from agent_cycle.worker_prompts import WorkerPrompts
|
||||
|
||||
|
||||
LADDER = read_ladder(SHIPPED)
|
||||
@@ -51,16 +50,6 @@ SONNET_179 = '{"model": "claude-sonnet-5-5", "effort": "high", "item": 179, "lik
|
||||
OPUS_179_AT_3 = '{"model": "claude-opus-5-5", "effort": "medium", "item": 179, "likelihood": 2, "escape_and_impact": 5, "escalated_at": 3}'
|
||||
OPUS_179_AT_6 = '{"model": "claude-opus-5-5", "effort": "medium", "item": 179, "likelihood": 2, "escape_and_impact": 5, "escalated_at": 6}'
|
||||
JUDGE_RUN_LINE = re.compile(r"^Judge run: judge/[0-9a-f]{64}\.json\n", re.MULTILINE)
|
||||
# The prompts of an entry point that exists only here, so a cycle's tests read no real entry point's wording.
|
||||
PROMPTS = WorkerPrompts(
|
||||
driver="a-driver",
|
||||
fresh=lambda task, premises, route, project, facts: f"Fresh: {task}\n\n{premises}\n\n{facts}",
|
||||
resumed=lambda evidence, route, project: f"Resumed: {evidence}",
|
||||
rungs_held=lambda route, project: f" You hold {route.rung}.",
|
||||
quote_sources="the project's records",
|
||||
answer_destinations="the plan or a comment",
|
||||
records_named="the project's decisions",
|
||||
)
|
||||
|
||||
|
||||
class QuestionFor(unittest.TestCase):
|
||||
|
||||
@@ -9,9 +9,7 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
from failure import Failure
|
||||
from loop_effects.doubles import FakeEffects
|
||||
from agent_cycle import tmux
|
||||
|
||||
|
||||
WINDOW = tmux.Window("loop-proj", "item-7", "%4")
|
||||
from agent_cycle.doubles import WINDOW
|
||||
|
||||
|
||||
class TmuxCommands(unittest.TestCase):
|
||||
|
||||
@@ -10,7 +10,7 @@ import unittest.mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from claude_code import entries
|
||||
from claude_code import entries, session
|
||||
from failure import Failure
|
||||
from loop_effects.doubles import FakeEffects, NOW, ran, Streamed
|
||||
from agent_cycle import judge, lingering_workers, worker
|
||||
@@ -131,5 +131,24 @@ class MemoryWait(unittest.TestCase):
|
||||
self.assertEqual([seam for seam, _ in fake.calls], ["memory", "stream_claude"])
|
||||
|
||||
|
||||
class WorkerArgs(unittest.TestCase):
|
||||
OPENED = session.session_named("A proj a-driver", pathlib.Path("/h/.claude/projects"))
|
||||
|
||||
def test_opens_a_fresh_session_by_name_and_resumes_an_open_one(self):
|
||||
fresh = worker.worker_args(self.OPENED, Rung("m"), "Do it.", resume=False)
|
||||
self.assertEqual(fresh[fresh.index("--session-id") + 1], str(self.OPENED.id))
|
||||
self.assertEqual(fresh[fresh.index("--name") + 1], "A proj a-driver")
|
||||
resumed = worker.worker_args(self.OPENED, Rung("m"), "Continue the task.", resume=True)
|
||||
self.assertEqual(resumed[resumed.index("--resume") + 1], str(self.OPENED.id))
|
||||
self.assertNotIn("--session-id", resumed)
|
||||
self.assertEqual(resumed[resumed.index("-p") + 1], "Continue the task.")
|
||||
|
||||
def test_disallows_a_wakeup_so_a_worker_that_ends_its_turn_exits(self):
|
||||
for resume in (False, True):
|
||||
with self.subTest(resume=resume):
|
||||
args = worker.worker_args(self.OPENED, Rung("m"), "Do it.", resume=resume)
|
||||
self.assertEqual(args[args.index("--disallowedTools") + 1], "ScheduleWakeup")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
import dataclasses
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from agent_cycle.doubles import PROMPTS
|
||||
from model_routing.route import Route
|
||||
from model_routing.ladder import Rung
|
||||
from agent_cycle import worker_prompts
|
||||
|
||||
|
||||
class TypedAnswers(unittest.TestCase):
|
||||
def test_splits_the_prompt_handing_over_the_answer_into_the_answer_on_the_line_after_its_marker_and_the_rest_and_no_other_prompt(self):
|
||||
route = Route(Rung("worker-model"))
|
||||
marked = "The human answered your question at a-driver's terminal:\nB"
|
||||
typed = worker_prompts.typed_answer(PROMPTS.typed("Which one?", "Use B.", route, None, False))
|
||||
self.assertEqual(typed.answer, "Use B.")
|
||||
self.assertTrue(typed.loop_words.startswith("Your question was: Which one?\n\nContinue the task."))
|
||||
self.assertEqual(worker_prompts.typed_answer(PROMPTS.typed(f"Which one?\n{marked}", "Use B.", route, None, False)).answer, "Use B.")
|
||||
# A ruling that stepped the worker up adds the sentences naming its new rung.
|
||||
self.assertEqual(worker_prompts.typed_answer(PROMPTS.typed("Which one?", "Use B.", Route(Rung("claude-sonnet-5-5", "high"), 179, 2, 5), None, True)).answer, "Use B.")
|
||||
self.assertIsNone(worker_prompts.typed_answer(PROMPTS.answered("Which one?", marked, route, None, False)))
|
||||
self.assertIsNone(worker_prompts.typed_answer(PROMPTS.resumed(marked, route, None)))
|
||||
|
||||
def test_the_typed_answers_prompt_names_the_entry_points_answer_destinations(self):
|
||||
ledger = dataclasses.replace(PROMPTS, answer_destinations="the ledger")
|
||||
self.assertIn("is recorded in the ledger at the one place it governs.", ledger.typed("Which one?", "Use B.", Route(Rung("worker-model")), None, False))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Test doubles for the suites that drive a chunk: the loop of an entry point that exists only in tests."""
|
||||
import pathlib
|
||||
import typing
|
||||
|
||||
from loop_effects.effects import Effects
|
||||
from chunk_driver.this_loop import Loop
|
||||
from loop_state import paths
|
||||
from claude_code import session
|
||||
from agent_cycle.doubles import WINDOW
|
||||
|
||||
|
||||
def a_loop(effects: Effects, **fields: typing.Any) -> Loop:
|
||||
"""A loop in /r on effects, with fields in place of its defaults."""
|
||||
home = pathlib.Path("/h")
|
||||
defaults = {"root": "/r", "session": session.session_named("A proj a-driver", home / ".claude" / "projects"), "paths": paths.paths_from(home), "env": {}, "launcher": pathlib.Path("/b/a-driver"), "window": WINDOW}
|
||||
return Loop(**{**defaults, **fields}, effects=effects)
|
||||
@@ -8,6 +8,7 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2]))
|
||||
|
||||
from project_config.test_project_config import configured
|
||||
from failure import Failure
|
||||
from agent_cycle import judge
|
||||
from chunk_driver.project import records
|
||||
|
||||
|
||||
@@ -86,5 +87,10 @@ class Records(unittest.TestCase):
|
||||
self.assertEqual(records.read_records(root, claude_md.parents[1], claude_md.parent), Failure(f"todo-items.md in {rules} is not UTF-8 on line 1 - the judge cannot read the records."))
|
||||
|
||||
|
||||
class JudgeRules(unittest.TestCase):
|
||||
def test_the_judge_prompt_names_every_rule_the_judge_reads_from_the_human(self):
|
||||
self.assertEqual([name for name in records.HUMAN_RULE_NAMES if name not in judge.judge_rules("the ledger", "a-driver")], [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -9,28 +9,20 @@ import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from chunk_driver import chunk_run, cycle_start, this_loop
|
||||
from chunk_driver.doubles import a_loop
|
||||
from chunk_driver import chunk_run, cycle_start
|
||||
from loop_effects.doubles import FakeEffects, NOW
|
||||
from agent_cycle.verdicts import Finished, TakenIn, Unfinished
|
||||
from transcript_archive import outcomes, write
|
||||
from loop_state import paths
|
||||
from model_routing.ladder import ProjectRung, Rung
|
||||
from model_routing.ladder_file import read_ladder, SHIPPED
|
||||
from model_routing.route import Route
|
||||
from agent_cycle import run
|
||||
from claude_code import session
|
||||
from agent_cycle.test_tmux import WINDOW
|
||||
|
||||
OPUS_PROJECT = ProjectRung(read_ladder(SHIPPED), Rung("claude-opus-5-5", "medium"))
|
||||
|
||||
|
||||
def a_loop(effects, **fields):
|
||||
"""A loop in /r on effects, with fields in place of its defaults."""
|
||||
home = pathlib.Path("/h")
|
||||
defaults = {"root": "/r", "session": session.session_named("A proj a-driver", home / ".claude" / "projects"), "paths": paths.paths_from(home), "env": {}, "launcher": pathlib.Path("/b/a-driver"), "window": WINDOW}
|
||||
return this_loop.Loop(**{**defaults, **fields}, effects=effects)
|
||||
|
||||
|
||||
class Steps(unittest.TestCase):
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
# The rungs an L 3 and an L 4 pick from OPUS.
|
||||
|
||||
@@ -1,15 +1,28 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
import unittest.mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from chunk_driver.doubles import a_loop
|
||||
from chunk_driver import cycle_start
|
||||
from failure import Failure
|
||||
from loop_effects.doubles import FakeEffects, NOW, STAMP
|
||||
from git_cli.git_repo import git
|
||||
from transcript_archive import outcomes, write
|
||||
from model_routing.ladder import ProjectRung, Rung
|
||||
from model_routing.ladder_file import read_ladder, SHIPPED
|
||||
from model_routing.route import Route, WorkerLine
|
||||
from claude_code import session
|
||||
|
||||
OPUS_PROJECT = ProjectRung(read_ladder(SHIPPED), Rung("claude-opus-5-5", "medium"))
|
||||
|
||||
|
||||
class SetupVersion(unittest.TestCase):
|
||||
@@ -36,5 +49,54 @@ class SetupVersion(unittest.TestCase):
|
||||
self.assertEqual(cycle_start.setup_version(pathlib.Path("/r"), fake_git), "unknown: fatal: bad HEAD hint: repair it")
|
||||
|
||||
|
||||
class Routes(unittest.TestCase):
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
SONNET = Rung("claude-sonnet-5-5", "high")
|
||||
|
||||
def test_keeps_a_chunks_route_or_its_archived_one_at_or_above_the_rung_its_item_picks_and_else_routes_afresh(self):
|
||||
item = types.SimpleNamespace(id=3, likelihood=4, escape_and_impact=5)
|
||||
archived = Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 5)
|
||||
def never():
|
||||
raise AssertionError("not asked")
|
||||
self.assertEqual(cycle_start.routed(archived, None, "worker-model", never, never), Route(Rung("worker-model")))
|
||||
self.assertEqual(cycle_start.routed(archived, OPUS_PROJECT, "claude-opus-5-5", never, never), archived)
|
||||
self.assertEqual(cycle_start.routed(None, OPUS_PROJECT, "claude-opus-5-5", lambda: archived, never), archived)
|
||||
self.assertEqual(cycle_start.routed(None, OPUS_PROJECT, "claude-opus-5-5", lambda: None, lambda: item), Route(Rung("claude-fable-5-2", "medium"), 3, 4, 5))
|
||||
self.assertEqual(cycle_start.routed(Route(self.SONNET), OPUS_PROJECT, "claude-opus-5-5", never, lambda: None), Route(self.OPUS))
|
||||
cheap = Route(self.SONNET, 4, 2, 3)
|
||||
self.assertEqual(cycle_start.routed(cheap, OPUS_PROJECT, "claude-opus-5-5", never, never), cheap)
|
||||
self.assertEqual(cycle_start.routed(Route(Rung("claude-haiku-5-5", "max"), 4, 2, 3), OPUS_PROJECT, "claude-opus-5-5", never, lambda: None), Route(self.OPUS))
|
||||
|
||||
def test_resumes_on_the_archived_route_and_says_why_it_routes_afresh(self):
|
||||
with tempfile.TemporaryDirectory() as home:
|
||||
opened = session.session_named("A proj a-driver", pathlib.Path(home, "projects"))
|
||||
archive = pathlib.Path(home, "archive")
|
||||
transcript = pathlib.Path(home, "projects", "-fake", f"{opened.id}.jsonl")
|
||||
route = Route(Rung("claude-fable-5-2", "low"), 3, 4, 5)
|
||||
with contextlib.redirect_stderr(io.StringIO()) as stderr:
|
||||
self.assertIsNone(cycle_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)))
|
||||
transcript.parent.mkdir(parents=True)
|
||||
transcript.write_text(json.dumps({"type": "queue-operation", "timestamp": "1"}) + "\n")
|
||||
self.assertIsNone(cycle_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)))
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(route.worker_line(), "working", "j", ""), NOW))
|
||||
with contextlib.redirect_stderr(stderr):
|
||||
self.assertEqual(cycle_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)), route)
|
||||
self.assertEqual(stderr.getvalue(), "")
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.stepped_up(WorkerLine('{"model": "claude-opus-5-5", "rung": 3}')), NOW + 1))
|
||||
with contextlib.redirect_stderr(io.StringIO()) as stderr:
|
||||
self.assertIsNone(cycle_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)))
|
||||
self.assertEqual(stderr.getvalue(), f'{STAMP} routing the chunk afresh: the archive\'s latest Worker: line names no route that can be read: {{"model": "claude-opus-5-5", "rung": 3}}\n')
|
||||
(chunk,) = archive.iterdir()
|
||||
chunk.chmod(0)
|
||||
try:
|
||||
with contextlib.redirect_stderr(io.StringIO()) as stderr:
|
||||
self.assertIsNone(cycle_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)))
|
||||
finally:
|
||||
chunk.chmod(0o700)
|
||||
self.assertTrue(stderr.getvalue().startswith(f"{STAMP} routing the chunk afresh: could not read the archive"), stderr.getvalue())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from model_routing.ladder import ProjectRung, Rung
|
||||
from chunk_driver import prompts
|
||||
from model_routing.ladder_file import read_ladder, SHIPPED
|
||||
from model_routing.route import Route
|
||||
|
||||
LADDER = read_ladder(SHIPPED)
|
||||
# The ladder's rungs around the project's: the one below it, its own, and those one to three steps up.
|
||||
SONNET = Rung("claude-sonnet-5-5", "high")
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
FABLE_LOW = Rung("claude-fable-5-2", "low")
|
||||
FABLE_MEDIUM = Rung("claude-fable-5-2", "medium")
|
||||
OPUS_XHIGH = Rung("claude-opus-5-5", "xhigh")
|
||||
OPUS_PROJECT = ProjectRung(LADDER, OPUS)
|
||||
# The sentence with which an entry point that exists only here asks its worker to name the chunk's items.
|
||||
NAMING = "Name the items."
|
||||
SCAN = " Run a full scan only from such a subagent, since the scan's agents run on the model that starts them."
|
||||
SEATS = " Launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung."
|
||||
MARKDOWN = " Where your branch changes only Markdown, none of it `CLAUDE.md`, `AGENTS.md`, `workflow-bench/*-prompt.md` or under `agents/`, `principles/`, `rules/` or `skills/`, launch each reviewer on the project's rung instead, with the Agent tool set to model opus, effort medium."
|
||||
|
||||
|
||||
def held(route, project=OPUS_PROJECT):
|
||||
"""The rung sentences a-driver's worker reads on route, each after a space."""
|
||||
return prompts.rungs_held("a-driver", route, project)
|
||||
|
||||
|
||||
def resumed(route, project=OPUS_PROJECT):
|
||||
"""The prompt that resumes a-driver's worker on route, with no evidence of work remaining."""
|
||||
return prompts.resumed("a-driver", NAMING, "", route, project)
|
||||
|
||||
|
||||
class Resumed(unittest.TestCase):
|
||||
def test_resumes_the_worker_saying_when_the_chunk_is_finished(self):
|
||||
self.assertTrue(resumed(Route(Rung("m")), None).startswith("Continue the task; the previous run ended before the chunk was finished. The chunk is finished once"))
|
||||
|
||||
|
||||
class ReviewLaunches(unittest.TestCase):
|
||||
def test_keeps_every_review_on_the_projects_rung_while_the_worker_runs_below_it(self):
|
||||
below = held(Route(SONNET, 3, 2, 4))
|
||||
self.assertTrue(below.startswith(f" Launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.{SCAN} Your rung is below the project's: launch each subagent that files or scores plan items on the rung `~/.claude/bin/subagent-rung --plan claude-opus-5-5 <L>` prints. Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model fable, effort low` from 5, `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8. When an escalation trigger"), below)
|
||||
self.assertNotIn("Launch each", held(Route(OPUS, 3, 2, 4)))
|
||||
|
||||
def test_launches_reviewers_on_the_rung_the_items_e_plus_i_picks_above_the_workers(self):
|
||||
level = held(Route(OPUS, 3, 2, 6))
|
||||
self.assertTrue(level.startswith(f" Launch each reviewer with the Agent tool set to model fable, effort medium, the rung item 3's E + I of 6 picks. Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.{MARKDOWN} When an escalation trigger"), level)
|
||||
below = held(Route(SONNET, 3, 2, 5))
|
||||
self.assertTrue(below.startswith(f" Launch each reviewer with the Agent tool set to model fable, effort low, the rung item 3's E + I of 5 picks.{SEATS}{SCAN} Your rung is below the project's:"), below)
|
||||
self.assertNotIn("Launch each reviewer", held(Route(FABLE_MEDIUM, 3, 4, 6)))
|
||||
|
||||
def test_names_the_reviewers_rung_each_higher_e_plus_i_an_item_taken_in_picks(self):
|
||||
taking_in = " Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model fable, effort low` from 5, `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8."
|
||||
self.assertTrue(held(Route(OPUS, 3, 2, 4)).startswith(f"{taking_in} When an escalation trigger"))
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.", held(Route(OPUS, 3, 2, 6)))
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.", held(Route(FABLE_MEDIUM, 3, 4, 4)))
|
||||
self.assertNotIn("Where an item you take into", held(Route(OPUS, 3, 2, 8)))
|
||||
self.assertNotIn("Where an item you take into", held(Route(Rung("m")), None))
|
||||
|
||||
def test_launches_reviewers_on_the_rung_the_highest_e_plus_i_taken_in_picks(self):
|
||||
self.assertIn(f"a-driver runs you on claude-opus-5-5 at medium effort. Launch each reviewer with the Agent tool set to model opus, effort xhigh, the rung picked by an item of E + I 7 taken into the chunk. Where an item you take into the chunk has an E + I of 8 or more, launch each reviewer with the Agent tool set to `model fable, effort high` from 8.{MARKDOWN} When an escalation trigger", resumed(Route(OPUS, 3, 2, 5, escape_and_impact_taken_in=7)))
|
||||
self.assertIn(" Launch each reviewer with the Agent tool set to model fable, effort low, the rung picked by an item of E + I 5 taken into the chunk.", held(Route(OPUS, escape_and_impact_taken_in=5)))
|
||||
|
||||
def test_keeps_seats_and_the_security_pass_on_the_projects_rung_while_the_worker_runs_above_it(self):
|
||||
above = Route(FABLE_LOW, 3, 3, 4)
|
||||
sentences = f"{SEATS}{SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.{MARKDOWN} When an escalation trigger"
|
||||
self.assertIn(f"{prompts.ANSWERS_RECORDED}. {NAMING} a-driver runs you on claude-fable-5-2 at low effort.{sentences}", resumed(above))
|
||||
self.assertNotIn("the project's", resumed(Route(OPUS, 3, 2, 4)))
|
||||
self.assertNotIn("runs you on", resumed(Route(Rung("m")), None))
|
||||
self.assertTrue(held(above).startswith(sentences), held(above))
|
||||
|
||||
def test_lets_a_markdown_only_branch_review_on_the_projects_rung_where_the_reviewers_run_above_it(self):
|
||||
self.assertIn(MARKDOWN, held(Route(OPUS, 3, 2, 7)))
|
||||
self.assertIn(MARKDOWN, held(Route(FABLE_LOW, 3, 3, 4)))
|
||||
self.assertNotIn("only Markdown", held(Route(OPUS, 3, 2, 4)))
|
||||
|
||||
|
||||
class Escalation(unittest.TestCase):
|
||||
def test_names_the_rungs_an_escalation_trigger_lets_the_worker_hand_a_task_to_at_most_two_above_its_items(self):
|
||||
routed = held(Route(OPUS, 3, 2, 4))
|
||||
self.assertTrue(routed.startswith(" Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model fable, effort low` from 5, `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort low`, one rung above yours; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number."), routed)
|
||||
self.assertIn(" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort medium`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", held(Route(FABLE_LOW, 3, 3, 4)))
|
||||
self.assertIn(" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort xhigh`, one rung above yours; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", held(Route(FABLE_MEDIUM, 3, 3, 4)))
|
||||
self.assertNotIn("escalation", held(Route(Rung("claude-fable-5-2", "max"), 3, 4, 8)))
|
||||
self.assertIn(" An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `model fable, effort high`, one rung above yours, only once the human agrees; a-driver will not resume you higher for a trigger either, so ask the human for that yes.", held(Route(OPUS_XHIGH, 3, 3, 4)))
|
||||
self.assertIn("set to `model fable, effort max`, one rung above yours, only once the human agrees;", held(Route(Rung("claude-fable-5-2", "high"), 3, 4, 5)), "one rung below the top still names the rung above")
|
||||
self.assertNotIn("rerun", held(Route(Rung("claude-opus-5-5", "low"), 3, 3, 4)), "a worker off the ladder is told no rerun rung")
|
||||
self.assertIn("hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort high`, one rung above yours;", held(Route(OPUS_XHIGH, 3, 4, 5)))
|
||||
self.assertNotIn("escalation", held(Route(Rung("m")), None))
|
||||
self.assertNotIn("escalation", held(Route(Rung("claude-opus-5-5", "low"), 3, 3, 4)))
|
||||
unrouted = held(Route(OPUS))
|
||||
self.assertTrue(unrouted.startswith(" Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model fable, effort low` from 5, `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort low`, one rung above yours, and on each further trigger one rung higher, up to `model fable, effort medium`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number. Where the hard part is the rest of the chunk, end your run instead, asking a-driver for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model fable, effort medium`. A reviewer"), unrouted)
|
||||
|
||||
def test_offers_the_worker_a_higher_rung_for_the_rest_of_the_chunk_up_to_the_escalation_cap(self):
|
||||
offer = lambda cap: f" Where the hard part is the rest of the chunk, end your run instead, asking a-driver for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `{cap}`."
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer('model fable, effort low')}", held(Route(OPUS, 3, 2, 4)))
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer('model opus, effort xhigh')}", held(Route(FABLE_LOW, 3, 3, 4)))
|
||||
self.assertNotIn("Where the hard part", held(Route(OPUS_XHIGH, 3, 3, 4)))
|
||||
|
||||
|
||||
class Reruns(unittest.TestCase):
|
||||
def test_names_the_rung_a_reviewer_or_seat_reruns_on_one_above_its_own(self):
|
||||
both = " A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort low`."
|
||||
self.assertTrue(resumed(Route(OPUS, 3, 2, 4)).endswith(both))
|
||||
self.assertTrue(resumed(Route(SONNET, 3, 2, 4)).endswith(both))
|
||||
self.assertTrue(held(Route(OPUS)).endswith(both))
|
||||
self.assertTrue(resumed(Route(FABLE_LOW, 3, 3, 4)).endswith(" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort medium`, and a comprehension seat with `model fable, effort low`."))
|
||||
self.assertTrue(resumed(Route(OPUS, 3, 2, 7)).endswith(" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort high`, and a comprehension seat with `model fable, effort low`."))
|
||||
self.assertTrue(resumed(Route(Rung("claude-fable-5-2", "max"), 3, 4, 8)).endswith(" A comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort low`."))
|
||||
self.assertNotIn("rerun", resumed(Route(Rung("m")), None))
|
||||
self.assertNotIn("rerun", held(Route(Rung("m", "high"), 3, 4, 5), None))
|
||||
fable = Rung("claude-fable-5-2", "max")
|
||||
self.assertNotIn("rerun", resumed(Route(fable), ProjectRung(LADDER, fable)))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -13,38 +13,37 @@ import unittest.mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from chunk_driver.doubles import a_loop
|
||||
from routing_figures import archive, state
|
||||
from test_env import assert_not_root
|
||||
from loop_effects.doubles import assistant, CWD, FakeEffects, human, kinds, NOW, ran, said, STAMP
|
||||
from loop_effects.doubles import assistant, CWD, FakeEffects, NOW, ran, said, STAMP
|
||||
from claude_code.output_limits import BASH_CEILING_CHARACTERS
|
||||
from routing_figures.calls import Call
|
||||
from loop_claude.chunk import chunk_items
|
||||
from chunk_driver import chunk_run, cycle_start, prompts, this_loop
|
||||
from chunk_driver import chunk_run, prompts, this_loop
|
||||
from loop_claude.loop import chunk_start, chunk_windows
|
||||
from project_config.test_project_config import configured
|
||||
from release_plan.release_standing import DefaultPlan, LastCut, UNSEEN_RELEASES
|
||||
from claude_code import entries, reply, session
|
||||
from failure import Failure
|
||||
from agent_cycle.verdicts import Finished, Interrupt, TakenIn, Unfinished
|
||||
from agent_cycle.verdicts import Finished, TakenIn, Unfinished
|
||||
from loop_claude.chunk import fresh_prompt
|
||||
from agent_cycle import judge, retry, states, worker, worker_prompts
|
||||
from agent_cycle import judge, retry, states, worker
|
||||
import loop_claude.routed_item
|
||||
from loop_claude.loop import main, reload, routing_check, usage
|
||||
from release_plan import next_chunk
|
||||
from transcript_archive import outcomes, write
|
||||
from transcript_archive import outcomes
|
||||
from loop_state import paths
|
||||
from plan_store.todo_md import plan_of
|
||||
from model_routing.ladder import ProjectRung, Rung
|
||||
from loop_effects.effects import PYTHON_TIMEOUT
|
||||
from model_routing.ladder_file import read_ladder, SHIPPED
|
||||
from chunk_driver.project import records
|
||||
from model_routing.route import Route, route_from, WorkerLine
|
||||
from agent_cycle import routing
|
||||
from model_routing.route import Route
|
||||
from agent_cycle.stop import Stop
|
||||
from agent_cycle import tmux
|
||||
from touch_predictions.test_touch_predictions import plan as touch_plan
|
||||
from text_files.utc_stamp import utc
|
||||
from agent_cycle.test_tmux import WINDOW
|
||||
|
||||
LADDER = read_ladder(SHIPPED)
|
||||
OPUS_PROJECT = ProjectRung(LADDER, Rung("claude-opus-5-5", "medium"))
|
||||
@@ -53,13 +52,6 @@ OPUS_PROJECT = ProjectRung(LADDER, Rung("claude-opus-5-5", "medium"))
|
||||
UNREAD = Failure("not a git repository")
|
||||
|
||||
|
||||
def a_loop(effects, **fields):
|
||||
"""A loop in /r on effects, with fields in place of its defaults."""
|
||||
home = pathlib.Path("/h")
|
||||
defaults = {"root": "/r", "session": session.session_named("A proj loop-claude", home / ".claude" / "projects"), "paths": paths.paths_from(home), "env": {}, "launcher": pathlib.Path("/b/loop-claude"), "window": WINDOW}
|
||||
return this_loop.Loop(**{**defaults, **fields}, effects=effects)
|
||||
|
||||
|
||||
class RoutingCheck(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.home = pathlib.Path(tempfile.mkdtemp())
|
||||
@@ -200,24 +192,7 @@ TODO = """# todo
|
||||
"""
|
||||
|
||||
|
||||
def unfinished(likelihood=0, escalated_at=None):
|
||||
"""An unfinished verdict: the worker took in an item of that L, 0 for none, and the judge found an escalation after reading escalated_at transcript entries, None for none."""
|
||||
return Unfinished("e", escalation="round 2: 🔴" if escalated_at is not None else "", taken_in=TakenIn(highest_likelihood=likelihood), entries_read=escalated_at)
|
||||
|
||||
|
||||
def interrupted(likelihood=0, escalated_at=None):
|
||||
"""An interrupt, its arguments as unfinished's."""
|
||||
return Interrupt("e", "q", escalation="round 2: 🔴" if escalated_at is not None else "", taken_in=TakenIn(highest_likelihood=likelihood), entries_read=escalated_at)
|
||||
|
||||
|
||||
class Routing(unittest.TestCase):
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
ONE_UP = Rung("claude-fable-5-2", "low")
|
||||
TWO_UP = Rung("claude-fable-5-2", "medium")
|
||||
THREE_UP = Rung("claude-opus-5-5", "xhigh")
|
||||
FOUR_UP = Rung("claude-fable-5-2", "high")
|
||||
SONNET = Rung("claude-sonnet-5-5", "high")
|
||||
|
||||
class RoutedItem(unittest.TestCase):
|
||||
def test_routes_on_the_next_releases_top_item_that_waits_on_no_question(self):
|
||||
self.assertEqual(routed_item(TODO, NO_CUT).id, 3)
|
||||
self.assertIsNone(routed_item(TODO.replace("| | **Do", "| question | **Do"), NO_CUT))
|
||||
@@ -244,129 +219,10 @@ class Routing(unittest.TestCase):
|
||||
first = "| 7 | 0.2.0 | | question | **Ask first.** | 4 | 3 | 5 | 10 | 1 | 1 by 9 | 30 |\n"
|
||||
self.assertEqual(routed_item(todo.replace(first, "") + first, NO_CUT).id, 3)
|
||||
|
||||
def test_starts_a_chunk_on_the_rung_its_items_l_picks_below_the_projects_too(self):
|
||||
self.assertEqual(routing.first_route(OPUS_PROJECT, routed_item(TODO, NO_CUT)), Route(self.TWO_UP, 3, 4, 5))
|
||||
cheap = routed_item(TODO.replace("| 4 | 2 | 3 |", "| 1 | 2 | 3 |"), NO_CUT)
|
||||
self.assertEqual(routing.first_route(OPUS_PROJECT, cheap), Route(Rung("claude-sonnet-5-5", "high"), 3, 1, 5))
|
||||
self.assertEqual(routing.first_route(OPUS_PROJECT, None), Route(self.OPUS))
|
||||
|
||||
def test_steps_up_a_rung_after_every_second_unfinished_verdict(self):
|
||||
once = routing.next_route(OPUS_PROJECT, Route(self.ONE_UP, 3, 3, 4), unfinished())
|
||||
self.assertEqual(once, Route(self.ONE_UP, 3, 3, 4, unfinished=1))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, once, unfinished()), Route(self.TWO_UP, 3, 3, 4))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(LADDER.rungs[-1], unfinished=1), unfinished()).rung, LADDER.rungs[-1])
|
||||
|
||||
def test_steps_up_to_the_rung_a_taken_in_items_l_picks_and_never_down(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.OPUS), unfinished(4)).rung, self.TWO_UP)
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.THREE_UP), unfinished(4)).rung, self.THREE_UP)
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.OPUS, unfinished=1), unfinished(4)).rung, self.THREE_UP)
|
||||
|
||||
def test_steps_up_from_a_rung_below_the_projects_one_rung_at_a_time(self):
|
||||
low = Route(self.SONNET, 3, 1, 3, unfinished=1)
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, low, unfinished()).rung, self.OPUS)
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(low.rung, 3, 1, 3), unfinished()).rung, low.rung)
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(low.rung, 3, 1, 3), unfinished(3)).rung, self.ONE_UP)
|
||||
|
||||
def test_steps_up_a_rung_on_an_escalation_the_judge_found_up_to_two_above_the_items_rung_marking_the_transcript_entries_it_read(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.TWO_UP, 3, 4, 5), unfinished(escalated_at=7)), Route(self.THREE_UP, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.THREE_UP, 3, 4, 5, escalated_at=7), unfinished(escalated_at=9)), Route(self.FOUR_UP, 3, 4, 5, escalated_at=9))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.FOUR_UP, 3, 4, 5, escalated_at=9), unfinished(escalated_at=12)), Route(self.FOUR_UP, 3, 4, 5, unfinished=1, escalated_at=9))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.TWO_UP, 3, 4, 5, unfinished=1), unfinished(escalated_at=7)), Route(self.THREE_UP, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.TWO_UP), unfinished(escalated_at=7)), Route(self.TWO_UP, unfinished=1))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.SONNET, 3, 2, 4), unfinished(4, escalated_at=7)), Route(self.TWO_UP, 3, 2, 4, unfinished=1))
|
||||
|
||||
def test_keeps_the_mark_through_a_step_no_escalation_took(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.ONE_UP, 3, 3, 4, unfinished=1, escalated_at=5), unfinished()), Route(self.TWO_UP, 3, 3, 4, escalated_at=5))
|
||||
|
||||
def test_steps_up_a_rung_as_the_judge_interrupts_up_to_two_above_the_items_rung_without_counting_an_unfinished_verdict(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.TWO_UP, 3, 4, 5, unfinished=1), interrupted(escalated_at=7)), Route(self.THREE_UP, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.THREE_UP, 3, 4, 5, escalated_at=7), interrupted(escalated_at=9)), Route(self.FOUR_UP, 3, 4, 5, escalated_at=9))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.FOUR_UP, 3, 4, 5, unfinished=1, escalated_at=9), interrupted(escalated_at=12)), Route(self.FOUR_UP, 3, 4, 5, unfinished=1, escalated_at=9))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.SONNET, 3, 2, 4), interrupted(escalated_at=7)), Route(self.OPUS, 3, 2, 4, escalated_at=7))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.TWO_UP, 3, 4, 5, unfinished=1, escalated_at=2), interrupted(4)), Route(self.TWO_UP, 3, 4, 5, unfinished=1, escalated_at=2))
|
||||
|
||||
def test_raises_the_worker_to_a_taken_in_items_rung_above_the_escalation_cap_without_a_step_as_the_judge_interrupts(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.OPUS, 3, 1, 3, escalated_at=2), interrupted(4, escalated_at=7)), Route(self.TWO_UP, 3, 1, 3, escalated_at=2))
|
||||
|
||||
def test_steps_up_from_the_rung_a_taken_in_item_picks_as_the_judge_interrupts(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.ONE_UP, 3, 3, 4), interrupted(4, escalated_at=7)), Route(self.THREE_UP, 3, 3, 4, escalated_at=7))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.TWO_UP, 3, 4, 5), interrupted(4, escalated_at=7)), Route(self.THREE_UP, 3, 4, 5, escalated_at=7))
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(self.FOUR_UP, 3, 4, 5, unfinished=1), interrupted(4, escalated_at=7)), Route(self.FOUR_UP, 3, 4, 5, unfinished=1))
|
||||
|
||||
def test_keeps_an_e_plus_i_taken_in_only_where_it_picks_a_higher_review_rung(self):
|
||||
taken = Route(self.OPUS, 3, 3, 4).taking_in(6)
|
||||
self.assertEqual(taken, Route(self.OPUS, 3, 3, 4, escape_and_impact_taken_in=6))
|
||||
self.assertEqual(taken.taking_in(7), Route(self.OPUS, 3, 3, 4, escape_and_impact_taken_in=7))
|
||||
self.assertEqual(taken.taking_in(5), taken)
|
||||
self.assertEqual(taken.taking_in(6), taken)
|
||||
self.assertEqual(Route(self.OPUS, 3, 3, 4).taking_in(4), Route(self.OPUS, 3, 3, 4))
|
||||
self.assertEqual(Route(self.OPUS, 3, 2, 2).taking_in(4), Route(self.OPUS, 3, 2, 2))
|
||||
self.assertEqual(Route(self.OPUS).taking_in(5), Route(self.OPUS, escape_and_impact_taken_in=5))
|
||||
self.assertEqual(Route(self.OPUS).taking_in(3), Route(self.OPUS))
|
||||
self.assertEqual(Route(self.OPUS).taking_in(0), Route(self.OPUS))
|
||||
|
||||
def test_a_rung_an_edit_took_off_the_ladder_counts_as_the_projects(self):
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(Rung("claude-opus-5-5", "low")), unfinished()).rung, self.OPUS)
|
||||
self.assertEqual(routing.next_route(OPUS_PROJECT, Route(Rung("claude-opus-5-5", "low")), unfinished(2)).rung, self.OPUS)
|
||||
|
||||
def test_runs_the_judge_of_a_replayed_model_on_its_models_highest_rung(self):
|
||||
self.assertEqual(routing.judge_rung(LADDER, "claude-opus-5-5"), self.THREE_UP)
|
||||
# A model off the ladder, or not replayed, gets no effort flag.
|
||||
self.assertEqual(routing.judge_rung(LADDER, "judge-model"), Rung("judge-model"))
|
||||
self.assertEqual(routing.judge_rung(LADDER, "claude-sonnet-5-5"), Rung("claude-sonnet-5-5"))
|
||||
|
||||
def test_refuses_a_route_whose_item_has_levels_off_their_scales_or_levels_without_an_item(self):
|
||||
for item, likelihood, escape_and_impact in ((3, 0, 5), (3, 5, 5), (3, 4, 1), (3, 4, 9), (0, 4, 5), (0, 4, 0), (0, 0, 5)):
|
||||
with self.subTest(item=item, likelihood=likelihood, escape_and_impact=escape_and_impact):
|
||||
with self.assertRaises(ValueError):
|
||||
Route(self.ONE_UP, item, likelihood, escape_and_impact)
|
||||
|
||||
def test_refuses_a_route_taking_in_an_e_plus_i_off_its_scale_or_not_above_its_items(self):
|
||||
for taken_in in (1, 9, -1, 5, 4):
|
||||
with self.subTest(taken_in=taken_in):
|
||||
with self.assertRaises(ValueError):
|
||||
Route(self.ONE_UP, 3, 4, 5, escape_and_impact_taken_in=taken_in)
|
||||
|
||||
def test_takes_a_route_whose_item_holds_each_levels_lowest_value(self):
|
||||
self.assertEqual(Route(self.ONE_UP, 3, 1, 2).escape_and_impact, 2)
|
||||
|
||||
def test_routes_nothing_on_an_item_without_e_or_i(self):
|
||||
without_i = TODO.replace("| E | I | R |", "| E | R |").replace("|---|---|---|---|---|---|---|---|---|---|---|", "|---|---|---|---|---|---|---|---|---|---|").replace("| 4 | 3 | 5 | 10 |", "| 4 | 3 | 10 |").replace("| 4 | 2 | 3 | 7 |", "| 4 | 2 | 7 |").replace("| 1 | 1 | 1 | 1 |", "| 1 | 1 | 1 |")
|
||||
self.assertIsNone(routed_item(without_i, NO_CUT))
|
||||
|
||||
def test_archives_the_route_as_a_json_worker_line_it_reads_back_field_by_field(self):
|
||||
route = Route(self.ONE_UP, 3, 4, 5)
|
||||
self.assertEqual(route.worker_line(), '{"model": "claude-fable-5-2", "effort": "low", "item": 3, "likelihood": 4, "escape_and_impact": 5}')
|
||||
self.assertEqual(route_from(route.worker_line()), route)
|
||||
self.assertEqual(Route(Rung("worker-model")).worker_line(), '{"model": "worker-model"}')
|
||||
self.assertEqual(route_from('{"model": "worker-model"}'), Route(Rung("worker-model")))
|
||||
marked = Route(self.ONE_UP, 3, 4, 5, escape_and_impact_taken_in=7, escalated_at=12)
|
||||
self.assertEqual(marked.worker_line(), '{"model": "claude-fable-5-2", "effort": "low", "item": 3, "likelihood": 4, "escape_and_impact": 5, "escape_and_impact_taken_in": 7, "escalated_at": 12}')
|
||||
self.assertEqual(route_from(marked.worker_line()), marked)
|
||||
self.assertEqual(route_from(Route(self.ONE_UP, escalated_at=3).worker_line()), Route(self.ONE_UP, escalated_at=3))
|
||||
|
||||
def test_leaves_the_count_of_unfinished_verdicts_out_of_the_worker_line_so_a_rerun_restarts_it(self):
|
||||
self.assertEqual(route_from(Route(self.ONE_UP, 3, 4, 5, unfinished=1).worker_line()), Route(self.ONE_UP, 3, 4, 5))
|
||||
|
||||
def test_reads_no_route_from_a_worker_line_that_route_worker_line_could_not_have_written(self):
|
||||
for line in ('{"effort": "high"}', '{"model": "claude-opus-5-5", "effort": "high", "unfinished": 1}', '{"model": "claude-opus-5-5", "item": "3"}', '{"model": "claude-opus-5-5", "effort": 3}', '{"model": 5}', '{"model": "claude-opus-5-5", "effort": "high", "escalated_at": -3}', '{"model": "claude-opus-5-5", "item": true, "likelihood": 4, "escape_and_impact": 5}', '{"model": "claude-opus-5-5", "effort": "high", "item": 3, "likelihood": 5, "escape_and_impact": 5}', '{"model": "claude-opus-5-5", "effort": "high", "escape_and_impact_taken_in": 9}', '["claude-opus-5-5"]', '{"model": ', "worker-model", "claude-fable-5-2 at low effort, routed by item 3's L of 4 and E + I of 5"):
|
||||
with self.subTest(line=line):
|
||||
self.assertIsNone(route_from(line))
|
||||
|
||||
def test_keeps_a_chunks_route_or_its_archived_one_at_or_above_the_rung_its_item_picks_and_else_routes_afresh(self):
|
||||
item = routed_item(TODO, NO_CUT)
|
||||
archived = Route(self.THREE_UP, 3, 4, 5)
|
||||
def never():
|
||||
raise AssertionError("not asked")
|
||||
self.assertEqual(cycle_start.routed(archived, None, "worker-model", never, never), Route(Rung("worker-model")))
|
||||
self.assertEqual(cycle_start.routed(archived, OPUS_PROJECT, "claude-opus-5-5", never, never), archived)
|
||||
self.assertEqual(cycle_start.routed(None, OPUS_PROJECT, "claude-opus-5-5", lambda: archived, never), archived)
|
||||
self.assertEqual(cycle_start.routed(None, OPUS_PROJECT, "claude-opus-5-5", lambda: None, lambda: item), Route(self.TWO_UP, 3, 4, 5))
|
||||
self.assertEqual(cycle_start.routed(Route(self.SONNET), OPUS_PROJECT, "claude-opus-5-5", never, lambda: None), Route(self.OPUS))
|
||||
cheap = Route(self.SONNET, 4, 2, 3)
|
||||
self.assertEqual(cycle_start.routed(cheap, OPUS_PROJECT, "claude-opus-5-5", never, never), cheap)
|
||||
self.assertEqual(cycle_start.routed(Route(Rung("claude-haiku-5-5", "max"), 4, 2, 3), OPUS_PROJECT, "claude-opus-5-5", never, lambda: None), Route(self.OPUS))
|
||||
|
||||
|
||||
class RouteSources(unittest.TestCase):
|
||||
def test_routes_on_the_plan_item_and_says_why_it_routes_on_the_projects_rung(self):
|
||||
@@ -440,36 +296,6 @@ class RouteSources(unittest.TestCase):
|
||||
self.assertIn("the items the worker names lift no rung: no config names the checkout", stderr.getvalue())
|
||||
self.assertIn("no config names the checkout", chunk_start.fetched_plan(loop).reason)
|
||||
|
||||
def test_resumes_on_the_archived_route_and_says_why_it_routes_afresh(self):
|
||||
with tempfile.TemporaryDirectory() as home:
|
||||
opened = session.session_named("A proj loop-claude", pathlib.Path(home, "projects"))
|
||||
archive = pathlib.Path(home, "archive")
|
||||
transcript = pathlib.Path(home, "projects", "-fake", f"{opened.id}.jsonl")
|
||||
route = Route(Rung("claude-fable-5-2", "low"), 3, 4, 5)
|
||||
with contextlib.redirect_stderr(io.StringIO()) as stderr:
|
||||
self.assertIsNone(cycle_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)))
|
||||
transcript.parent.mkdir(parents=True)
|
||||
transcript.write_text(json.dumps({"type": "queue-operation", "timestamp": "1"}) + "\n")
|
||||
self.assertIsNone(cycle_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)))
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.unfinished(route.worker_line(), "working", "j", ""), NOW))
|
||||
with contextlib.redirect_stderr(stderr):
|
||||
self.assertEqual(cycle_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)), route)
|
||||
self.assertEqual(stderr.getvalue(), "")
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(archive, str(transcript), "Do it.", outcomes.stepped_up(WorkerLine('{"model": "claude-opus-5-5", "rung": 3}')), NOW + 1))
|
||||
with contextlib.redirect_stderr(io.StringIO()) as stderr:
|
||||
self.assertIsNone(cycle_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)))
|
||||
self.assertEqual(stderr.getvalue(), f'{STAMP} routing the chunk afresh: the archive\'s latest Worker: line names no route that can be read: {{"model": "claude-opus-5-5", "rung": 3}}\n')
|
||||
(chunk,) = archive.iterdir()
|
||||
chunk.chmod(0)
|
||||
try:
|
||||
with contextlib.redirect_stderr(io.StringIO()) as stderr:
|
||||
self.assertIsNone(cycle_start.archived_route(archive, a_loop(FakeEffects().effects(), session=opened)))
|
||||
finally:
|
||||
chunk.chmod(0o700)
|
||||
self.assertTrue(stderr.getvalue().startswith(f"{STAMP} routing the chunk afresh: could not read the archive"), stderr.getvalue())
|
||||
|
||||
|
||||
class ChunkWindows(unittest.TestCase):
|
||||
def plan(self, *touched):
|
||||
@@ -748,23 +574,8 @@ class ChunkFacts(unittest.TestCase):
|
||||
self.assertEqual((resumed.made("python"), facts), ([], ""))
|
||||
|
||||
|
||||
class TypedAnswers(unittest.TestCase):
|
||||
def test_splits_the_prompt_handing_over_the_answer_into_the_answer_on_the_line_after_its_marker_and_the_rest_and_no_other_prompt(self):
|
||||
route = Route(Rung("worker-model"))
|
||||
marked = "The human answered your question at loop-claude's terminal:\nB"
|
||||
typed = worker_prompts.typed_answer(fresh_prompt.WORKER.typed("Which one?", "Use B.", route, None, False))
|
||||
self.assertEqual(typed.answer, "Use B.")
|
||||
self.assertTrue(typed.loop_words.startswith("Your question was: Which one?\n\nContinue the task."))
|
||||
self.assertEqual(worker_prompts.typed_answer(fresh_prompt.WORKER.typed(f"Which one?\n{marked}", "Use B.", route, None, False)).answer, "Use B.")
|
||||
self.assertEqual(worker_prompts.typed_answer(fresh_prompt.WORKER.typed("Which one?", "Use B.", Route(Routing.SONNET, 179, 2, 5), OPUS_PROJECT, True)).answer, "Use B.")
|
||||
self.assertIsNone(worker_prompts.typed_answer(fresh_prompt.WORKER.answered("Which one?", marked, route, None, False)))
|
||||
self.assertIsNone(worker_prompts.typed_answer(fresh_prompt.WORKER.resumed(marked, route, None)))
|
||||
|
||||
|
||||
class WorkerArgs(unittest.TestCase):
|
||||
SCAN = " Run a full scan only from such a subagent, since the scan's agents run on the model that starts them."
|
||||
SEATS = " Launch each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung."
|
||||
MARKDOWN = " Where your branch changes only Markdown, none of it `CLAUDE.md`, `AGENTS.md`, `workflow-bench/*-prompt.md` or under `agents/`, `principles/`, `rules/` or `skills/`, launch each reviewer on the project's rung instead, with the Agent tool set to model opus, effort medium."
|
||||
class FreshPrompt(unittest.TestCase):
|
||||
"""loop-claude's first prompt of a chunk."""
|
||||
# What the preflight found, as the fresh prompt quotes it.
|
||||
FACTS = "- A preflight fact."
|
||||
|
||||
@@ -799,153 +610,10 @@ class WorkerArgs(unittest.TestCase):
|
||||
with self.subTest(facts=facts):
|
||||
self.assertEqual(fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(Rung("m")), None, facts).count("Continue any open draft PR or worktree"), 1)
|
||||
|
||||
def test_keeps_every_review_on_the_projects_rung_while_the_worker_runs_below_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
below = fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 4), OPUS_PROJECT, self.FACTS)
|
||||
self.assertIn(f"item 3's L of 2. Launch each reviewer, each comprehension seat and the security pass's subagent with the Agent tool set to model opus, effort medium, the project's rung.{self.SCAN} Your rung is below the project's: launch each subagent that files or scores plan items on the rung `~/.claude/bin/subagent-rung --plan claude-opus-5-5 <L>` prints. Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model fable, effort low` from 5, `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8. When an escalation trigger", below)
|
||||
self.assertNotIn("Launch each", fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(opus, 3, 2, 4), OPUS_PROJECT, self.FACTS))
|
||||
|
||||
def test_launches_reviewers_on_the_rung_the_items_e_plus_i_picks_above_the_workers(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
level = fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(opus, 3, 2, 6), OPUS_PROJECT, self.FACTS)
|
||||
self.assertIn(f"item 3's L of 2. Launch each reviewer with the Agent tool set to model fable, effort medium, the rung item 3's E + I of 6 picks. Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.{self.MARKDOWN} When an escalation trigger", level)
|
||||
below = fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 5), OPUS_PROJECT, self.FACTS)
|
||||
self.assertIn(f"item 3's L of 2. Launch each reviewer with the Agent tool set to model fable, effort low, the rung item 3's E + I of 5 picks.{self.SEATS}{self.SCAN} Your rung is below the project's:", below)
|
||||
self.assertNotIn("Launch each reviewer", fresh_prompt.WORKER.resumed("", Route(Rung("claude-fable-5-2", "medium"), 3, 4, 6), OPUS_PROJECT))
|
||||
|
||||
def test_names_the_reviewers_rung_each_higher_e_plus_i_an_item_taken_in_picks(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
taking_in = " Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model fable, effort low` from 5, `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8."
|
||||
self.assertIn(f"item 3's L of 2.{taking_in} When an escalation trigger", fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(opus, 3, 2, 4), OPUS_PROJECT, self.FACTS))
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.", fresh_prompt.WORKER.resumed("", Route(opus, 3, 2, 6), OPUS_PROJECT))
|
||||
self.assertIn(" Where an item you take into the chunk has an E + I of 7 or more, launch each reviewer with the Agent tool set to `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.", fresh_prompt.WORKER.resumed("", Route(Rung("claude-fable-5-2", "medium"), 3, 4, 4), OPUS_PROJECT))
|
||||
self.assertNotIn("Where an item you take into", fresh_prompt.WORKER.resumed("", Route(opus, 3, 2, 8), OPUS_PROJECT))
|
||||
self.assertNotIn("Where an item you take into", fresh_prompt.WORKER.resumed("", Route(Rung("m")), None))
|
||||
|
||||
def test_launches_reviewers_on_the_rung_the_highest_e_plus_i_taken_in_picks(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
self.assertIn(f"loop-claude runs you on claude-opus-5-5 at medium effort. Launch each reviewer with the Agent tool set to model opus, effort xhigh, the rung picked by an item of E + I 7 taken into the chunk. Where an item you take into the chunk has an E + I of 8 or more, launch each reviewer with the Agent tool set to `model fable, effort high` from 8.{self.MARKDOWN} When an escalation trigger", fresh_prompt.WORKER.resumed("", Route(opus, 3, 2, 5, escape_and_impact_taken_in=7), OPUS_PROJECT))
|
||||
self.assertIn(" Launch each reviewer with the Agent tool set to model fable, effort low, the rung picked by an item of E + I 5 taken into the chunk.", fresh_prompt.WORKER.resumed("", Route(opus, escape_and_impact_taken_in=5), OPUS_PROJECT))
|
||||
|
||||
def test_keeps_seats_and_the_security_pass_on_the_projects_rung_while_the_worker_runs_above_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
above = fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(Rung("claude-fable-5-2", "low"), 3, 3, 4), OPUS_PROJECT, self.FACTS)
|
||||
stepped_up = fresh_prompt.WORKER.resumed("", Route(Rung("claude-fable-5-2", "low"), 3, 3, 4), OPUS_PROJECT)
|
||||
self.assertIn(f"{prompts.ANSWERS_RECORDED}. {fresh_prompt.NAME_THE_ITEMS} loop-claude runs you on claude-fable-5-2 at low effort.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.{self.MARKDOWN} When an escalation trigger", stepped_up)
|
||||
self.assertNotIn("the project's", fresh_prompt.WORKER.resumed("", Route(opus, 3, 2, 4), OPUS_PROJECT))
|
||||
self.assertNotIn("runs you on", fresh_prompt.WORKER.resumed("", Route(Rung("m")), None))
|
||||
self.assertIn(f"item 3's L of 3.{self.SEATS}{self.SCAN} Where an item you take into the chunk has an E + I of 6 or more, launch each reviewer with the Agent tool set to `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8.{self.MARKDOWN} When an escalation trigger", above)
|
||||
|
||||
def test_names_the_rungs_an_escalation_trigger_lets_the_worker_hand_a_task_to_at_most_two_above_its_items(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
fresh = fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(opus, 3, 2, 4), OPUS_PROJECT, self.FACTS)
|
||||
self.assertIn("item 3's L of 2. Where an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model fable, effort low` from 5, `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort low`, one rung above yours; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", fresh)
|
||||
self.assertIn(" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort medium`, one rung above yours, and on each further trigger one rung higher, up to `model opus, effort xhigh`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", fresh_prompt.WORKER.resumed("", Route(Rung("claude-fable-5-2", "low"), 3, 3, 4), OPUS_PROJECT))
|
||||
self.assertIn(" When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model opus, effort xhigh`, one rung above yours; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number.", fresh_prompt.WORKER.resumed("", Route(Rung("claude-fable-5-2", "medium"), 3, 3, 4), OPUS_PROJECT))
|
||||
self.assertNotIn("escalation", fresh_prompt.WORKER.resumed("", Route(Rung("claude-fable-5-2", "max"), 3, 4, 8), OPUS_PROJECT))
|
||||
self.assertIn(" An escalation trigger in `~/.claude/CLAUDE.md` lets you hand the hard part of a task to a subagent with the Agent tool set to `model fable, effort high`, one rung above yours, only once the human agrees;", fresh_prompt.WORKER.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 3, 4), OPUS_PROJECT))
|
||||
self.assertIn("set to `model fable, effort max`, one rung above yours, only once the human agrees;", fresh_prompt.WORKER.resumed("", Route(Rung("claude-fable-5-2", "high"), 3, 4, 5), OPUS_PROJECT), "one rung below the top still names the rung above")
|
||||
self.assertNotIn("rerun", fresh_prompt.WORKER.resumed("", Route(Rung("claude-opus-5-5", "low"), 3, 3, 4), OPUS_PROJECT), "a worker off the ladder is told no rerun rung")
|
||||
self.assertIn("hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort high`, one rung above yours;", fresh_prompt.WORKER.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 4, 5), OPUS_PROJECT))
|
||||
self.assertNotIn("escalation", fresh_prompt.WORKER.resumed("", Route(Rung("m")), None))
|
||||
self.assertNotIn("escalation", fresh_prompt.WORKER.resumed("", Route(Rung("claude-opus-5-5", "low"), 3, 3, 4), OPUS_PROJECT))
|
||||
self.assertIn(f"{prompts.READING_RULE} {fresh_prompt.NAME_THE_ITEMS}\n\nWhere an item you take into the chunk has an E + I of 5 or more, launch each reviewer with the Agent tool set to `model fable, effort low` from 5, `model fable, effort medium` from 6, `model opus, effort xhigh` from 7 and `model fable, effort high` from 8. When an escalation trigger in `~/.claude/CLAUDE.md` fires, hand the hard part of the task to a subagent with the Agent tool set to `model fable, effort low`, one rung above yours, and on each further trigger one rung higher, up to `model fable, effort medium`; above that, ask the human first. Open the subagent's prompt with `Escalation trigger <n>:`, n the trigger's number. Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `model fable, effort medium`. A reviewer", fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(opus), OPUS_PROJECT, self.FACTS))
|
||||
|
||||
def test_offers_the_worker_a_higher_rung_for_the_rest_of_the_chunk_up_to_the_escalation_cap(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
offer = lambda cap: f" Where the hard part is the rest of the chunk, end your run instead, asking loop-claude for a higher rung and naming a trigger that fired since your last step-up, if any: once the judge finds it in your transcript, the next cycle resumes you one rung up, up to `{cap}`."
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer('model fable, effort low')}", fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(opus, 3, 2, 4), OPUS_PROJECT, self.FACTS))
|
||||
self.assertIn(f"`Escalation trigger <n>:`, n the trigger's number.{offer('model opus, effort xhigh')}", fresh_prompt.WORKER.resumed("", Route(Rung("claude-fable-5-2", "low"), 3, 3, 4), OPUS_PROJECT))
|
||||
self.assertNotIn("Where the hard part", fresh_prompt.WORKER.resumed("", Route(Rung("claude-opus-5-5", "xhigh"), 3, 3, 4), OPUS_PROJECT))
|
||||
|
||||
def test_lets_a_markdown_only_branch_review_on_the_projects_rung_where_the_reviewers_run_above_it(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
markdown = " Where your branch changes only Markdown, none of it `CLAUDE.md`, `AGENTS.md`, `workflow-bench/*-prompt.md` or under `agents/`, `principles/`, `rules/` or `skills/`, launch each reviewer on the project's rung instead, with the Agent tool set to model opus, effort medium."
|
||||
self.assertIn(markdown, fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(opus, 3, 2, 7), OPUS_PROJECT, self.FACTS))
|
||||
self.assertIn(markdown, fresh_prompt.WORKER.resumed("", Route(Rung("claude-fable-5-2", "low"), 3, 3, 4), OPUS_PROJECT))
|
||||
self.assertNotIn("only Markdown", fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(opus, 3, 2, 4), OPUS_PROJECT, self.FACTS))
|
||||
|
||||
def test_names_the_rung_a_reviewer_or_seat_reruns_on_one_above_its_own(self):
|
||||
opus = Rung("claude-opus-5-5", "medium")
|
||||
both = " A reviewer or comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort low`."
|
||||
self.assertTrue(fresh_prompt.WORKER.resumed("", Route(opus, 3, 2, 4), OPUS_PROJECT).endswith(both))
|
||||
self.assertTrue(fresh_prompt.WORKER.resumed("", Route(Rung("claude-sonnet-5-5", "high"), 3, 2, 4), OPUS_PROJECT).endswith(both))
|
||||
self.assertTrue(fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(opus), OPUS_PROJECT, self.FACTS).split(f"\n\n{self.FACTS}")[0].endswith(both))
|
||||
self.assertTrue(fresh_prompt.WORKER.resumed("", Route(Rung("claude-fable-5-2", "low"), 3, 3, 4), OPUS_PROJECT).endswith(" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort medium`, and a comprehension seat with `model fable, effort low`."))
|
||||
self.assertTrue(fresh_prompt.WORKER.resumed("", Route(opus, 3, 2, 7), OPUS_PROJECT).endswith(" A reviewer that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort high`, and a comprehension seat with `model fable, effort low`."))
|
||||
self.assertTrue(fresh_prompt.WORKER.resumed("", Route(Rung("claude-fable-5-2", "max"), 3, 4, 8), OPUS_PROJECT).endswith(" A comprehension seat that asks for a rerun one rung up reruns with the Agent tool set to `model fable, effort low`."))
|
||||
self.assertNotIn("rerun", fresh_prompt.WORKER.resumed("", Route(Rung("m")), None))
|
||||
self.assertNotIn("rerun", fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(Rung("m", "high"), 3, 4, 5), None, self.FACTS))
|
||||
self.assertIn(f"{both} After your first read", fresh_prompt.WORKER.fresh("Do it.", "## Goals", Route(opus, 3, 2, 4), OPUS_PROJECT, self.FACTS))
|
||||
fable = Rung("claude-fable-5-2", "max")
|
||||
self.assertNotIn("rerun", fresh_prompt.WORKER.resumed("", Route(fable), ProjectRung(LADDER, fable)))
|
||||
|
||||
def test_opens_a_fresh_session_by_name_and_resumes_an_open_one(self):
|
||||
opened = session.session_named("A proj loop-claude", pathlib.Path("/h/.claude/projects"))
|
||||
fresh = worker.worker_args(opened, Rung("m"), fresh_prompt.WORKER.fresh("Do it.", "## Goals\n1. g\n\n## Audience\nme", Route(Rung("m")), None, self.FACTS), resume=False)
|
||||
self.assertEqual(fresh[fresh.index("--session-id") + 1], str(opened.id))
|
||||
self.assertEqual(fresh[fresh.index("--name") + 1], "A proj loop-claude")
|
||||
resumed = worker.worker_args(opened, Rung("m"), fresh_prompt.WORKER.resumed("", Route(Rung("m")), None), resume=True)
|
||||
self.assertEqual(resumed[resumed.index("--resume") + 1], str(opened.id))
|
||||
self.assertNotIn("--session-id", resumed)
|
||||
self.assertTrue(resumed[resumed.index("-p") + 1].startswith("Continue the task; the previous run ended before the chunk was finished. The chunk is finished once"))
|
||||
|
||||
def test_disallows_a_wakeup_so_a_worker_that_ends_its_turn_exits(self):
|
||||
opened = session.session_named("A proj loop-claude", pathlib.Path("/h/.claude/projects"))
|
||||
for resume in (False, True):
|
||||
with self.subTest(resume=resume):
|
||||
args = worker.worker_args(opened, Rung("m"), "Do it.", resume=resume)
|
||||
self.assertEqual(args[args.index("--disallowedTools") + 1], "ScheduleWakeup")
|
||||
|
||||
|
||||
class JudgePrompt(unittest.TestCase):
|
||||
def test_shows_the_transcripts_tail_and_returns_what_it_showed(self):
|
||||
entries = [assistant({"type": "text", "text": "old " + "x" * judge.JUDGE_PROMPT_CHARS}), assistant({"type": "text", "text": "new"})]
|
||||
prompt, grounds = judge.judge_prompt(judge.Brief("Do it.", "## Goals", "", records.RECORDS_NAMED, prompts.ANSWER_DESTINATIONS, kinds(*entries), 0, fresh_prompt.DRIVER))
|
||||
self.assertTrue(grounds.shown.endswith("\n\nnew"))
|
||||
self.assertNotIn("old", grounds.shown)
|
||||
self.assertIn(grounds.shown, prompt)
|
||||
self.assertLessEqual(len(prompt), judge.JUDGE_PROMPT_CHARS)
|
||||
self.assertEqual(grounds.after_mark, "old " + "x" * judge.JUDGE_PROMPT_CHARS + "\n\nnew")
|
||||
self.assertEqual(grounds.quotable, "## Goals\n\n")
|
||||
|
||||
def test_the_judge_prompt_and_schema_call_the_worker_the_worker(self):
|
||||
prompt, _ = judge.judge_prompt(judge.Brief("Do it.", "## Goals", "", records.RECORDS_NAMED, prompts.ANSWER_DESTINATIONS, kinds(assistant({"type": "text", "text": "done"})), 1, fresh_prompt.DRIVER))
|
||||
self.assertNotRegex(prompt + json.dumps(judge.judge_schema(fresh_prompt.DRIVER)), r"\b[Aa]gent")
|
||||
|
||||
def test_the_judge_prompt_names_the_entry_points_answer_destinations_in_both_rules_on_a_recorded_answer(self):
|
||||
prompt, _ = judge.judge_prompt(judge.Brief("Do it.", "## Goals", "", records.RECORDS_NAMED, "the ledger", kinds(assistant({"type": "text", "text": "done"})), 0, fresh_prompt.DRIVER))
|
||||
self.assertIn("is recorded in the ledger at the one place it governs", prompt)
|
||||
self.assertIn("the worker writing it into the ledger, but not into one of the human's rules", prompt)
|
||||
|
||||
def test_the_typed_answers_prompt_names_the_entry_points_answer_destinations(self):
|
||||
ledger = dataclasses.replace(fresh_prompt.WORKER, answer_destinations="the ledger")
|
||||
self.assertIn("is recorded in the ledger at the one place it governs.", ledger.typed("Which one?", "Use B.", Route(Rung("worker-model")), None, False))
|
||||
|
||||
def test_the_judge_prompt_names_every_rule_the_judge_reads_from_the_human(self):
|
||||
self.assertEqual([name for name in records.HUMAN_RULE_NAMES if name not in judge.judge_rules(prompts.ANSWER_DESTINATIONS, fresh_prompt.DRIVER)], [])
|
||||
|
||||
def test_shows_the_judge_where_the_worker_last_stepped_up_and_checks_an_escalation_only_after_it(self):
|
||||
entries = [assistant({"type": "text", "text": "round 2: 🔴"}), assistant({"type": "text", "text": "round 3: 🔴"})]
|
||||
prompt, grounds = judge.judge_prompt(judge.Brief("Do it.", "## Goals", "", records.RECORDS_NAMED, prompts.ANSWER_DESTINATIONS, kinds(*entries), 1, fresh_prompt.DRIVER))
|
||||
self.assertRegex(prompt, "round 2: 🔴\n\n\\[step\\] [^\n]+\n\nround 3: 🔴\n\n")
|
||||
self.assertEqual(grounds.after_mark, "round 3: 🔴")
|
||||
prompt, grounds = judge.judge_prompt(judge.Brief("Do it.", "## Goals", "", records.RECORDS_NAMED, prompts.ANSWER_DESTINATIONS, kinds(*entries), 0, fresh_prompt.DRIVER))
|
||||
self.assertNotIn("stepped the worker up", prompt)
|
||||
self.assertEqual(grounds.shown, "round 2: 🔴\n\nround 3: 🔴")
|
||||
self.assertEqual(grounds.after_mark, "round 2: 🔴\n\nround 3: 🔴")
|
||||
|
||||
def test_drops_an_escalation_quote_from_after_the_step_up_that_the_tail_it_shows_cuts_off(self):
|
||||
entries = [assistant({"type": "text", "text": "round 2: 🔴"}), assistant({"type": "text", "text": "old " + "x" * judge.JUDGE_PROMPT_CHARS}), assistant({"type": "text", "text": "new"})]
|
||||
_, grounds = judge.judge_prompt(judge.Brief("Do it.", "## Goals", "", records.RECORDS_NAMED, prompts.ANSWER_DESTINATIONS, kinds(*entries), 1, fresh_prompt.DRIVER))
|
||||
self.assertEqual(judge.quotes_checked(Unfinished("e", "old x", entries_read=3), grounds), Unfinished("e", entries_read=3))
|
||||
self.assertEqual(judge.quotes_checked(Unfinished("e", "new", entries_read=3), grounds), Unfinished("e", "new", entries_read=3))
|
||||
|
||||
def test_shows_a_typed_answer_before_the_step_up_as_the_humans(self):
|
||||
typed = fresh_prompt.WORKER.typed("Which one?", "Use B.", Route(Rung("worker-model")), None, False)
|
||||
entries = [{**human(typed), "promptSource": "sdk"}, assistant({"type": "text", "text": "round 3: 🔴"})]
|
||||
_, grounds = judge.judge_prompt(judge.Brief("Do it.", "## Goals", "", records.RECORDS_NAMED, prompts.ANSWER_DESTINATIONS, kinds(*entries), 1, fresh_prompt.DRIVER))
|
||||
self.assertIn("\n\n[human] Use B.\n\n[step] ", grounds.shown)
|
||||
def test_puts_the_rung_sentences_after_the_route_or_else_before_the_facts(self):
|
||||
routed, unrouted = Route(OPUS_PROJECT.rung, 3, 2, 4), Route(OPUS_PROJECT.rung)
|
||||
self.assertIn(f"routed by item 3's L of 2.{prompts.rungs_held(fresh_prompt.DRIVER, routed, OPUS_PROJECT)} After your first read", fresh_prompt.WORKER.fresh("Do it.", "## Goals", routed, OPUS_PROJECT, self.FACTS))
|
||||
self.assertIn(f"{fresh_prompt.NAME_THE_ITEMS}\n\n{prompts.rungs_held(fresh_prompt.DRIVER, unrouted, OPUS_PROJECT).lstrip()}\n\n{self.FACTS}", fresh_prompt.WORKER.fresh("Do it.", "## Goals", unrouted, OPUS_PROJECT, self.FACTS))
|
||||
|
||||
|
||||
# Item 297 scores the higher L, item 298 the higher E + I.
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from model_routing.route import Route, route_from
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
OPUS = Rung("claude-opus-5-5", "medium")
|
||||
FABLE = Rung("claude-fable-5-2", "low")
|
||||
|
||||
|
||||
class Levels(unittest.TestCase):
|
||||
def test_refuses_a_route_whose_item_has_levels_off_their_scales_or_levels_without_an_item(self):
|
||||
for item, likelihood, escape_and_impact in ((3, 0, 5), (3, 5, 5), (3, 4, 1), (3, 4, 9), (0, 4, 5), (0, 4, 0), (0, 0, 5)):
|
||||
with self.subTest(item=item, likelihood=likelihood, escape_and_impact=escape_and_impact):
|
||||
with self.assertRaises(ValueError):
|
||||
Route(FABLE, item, likelihood, escape_and_impact)
|
||||
|
||||
def test_refuses_a_route_taking_in_an_e_plus_i_off_its_scale_or_not_above_its_items(self):
|
||||
for taken_in in (1, 9, -1, 5, 4):
|
||||
with self.subTest(taken_in=taken_in):
|
||||
with self.assertRaises(ValueError):
|
||||
Route(FABLE, 3, 4, 5, escape_and_impact_taken_in=taken_in)
|
||||
|
||||
def test_takes_a_route_whose_item_holds_each_levels_lowest_value(self):
|
||||
self.assertEqual(Route(FABLE, 3, 1, 2).escape_and_impact, 2)
|
||||
|
||||
def test_keeps_an_e_plus_i_taken_in_only_where_it_picks_a_higher_review_rung(self):
|
||||
taken = Route(OPUS, 3, 3, 4).taking_in(6)
|
||||
self.assertEqual(taken, Route(OPUS, 3, 3, 4, escape_and_impact_taken_in=6))
|
||||
self.assertEqual(taken.taking_in(7), Route(OPUS, 3, 3, 4, escape_and_impact_taken_in=7))
|
||||
self.assertEqual(taken.taking_in(5), taken)
|
||||
self.assertEqual(taken.taking_in(6), taken)
|
||||
self.assertEqual(Route(OPUS, 3, 3, 4).taking_in(4), Route(OPUS, 3, 3, 4))
|
||||
self.assertEqual(Route(OPUS, 3, 2, 2).taking_in(4), Route(OPUS, 3, 2, 2))
|
||||
self.assertEqual(Route(OPUS).taking_in(5), Route(OPUS, escape_and_impact_taken_in=5))
|
||||
self.assertEqual(Route(OPUS).taking_in(3), Route(OPUS))
|
||||
self.assertEqual(Route(OPUS).taking_in(0), Route(OPUS))
|
||||
|
||||
|
||||
class WorkerLines(unittest.TestCase):
|
||||
def test_archives_the_route_as_a_json_worker_line_it_reads_back_field_by_field(self):
|
||||
route = Route(FABLE, 3, 4, 5)
|
||||
self.assertEqual(route.worker_line(), '{"model": "claude-fable-5-2", "effort": "low", "item": 3, "likelihood": 4, "escape_and_impact": 5}')
|
||||
self.assertEqual(route_from(route.worker_line()), route)
|
||||
self.assertEqual(Route(Rung("worker-model")).worker_line(), '{"model": "worker-model"}')
|
||||
self.assertEqual(route_from('{"model": "worker-model"}'), Route(Rung("worker-model")))
|
||||
marked = Route(FABLE, 3, 4, 5, escape_and_impact_taken_in=7, escalated_at=12)
|
||||
self.assertEqual(marked.worker_line(), '{"model": "claude-fable-5-2", "effort": "low", "item": 3, "likelihood": 4, "escape_and_impact": 5, "escape_and_impact_taken_in": 7, "escalated_at": 12}')
|
||||
self.assertEqual(route_from(marked.worker_line()), marked)
|
||||
self.assertEqual(route_from(Route(FABLE, escalated_at=3).worker_line()), Route(FABLE, escalated_at=3))
|
||||
|
||||
def test_leaves_the_count_of_unfinished_verdicts_out_of_the_worker_line_so_a_rerun_restarts_it(self):
|
||||
self.assertEqual(route_from(Route(FABLE, 3, 4, 5, unfinished=1).worker_line()), Route(FABLE, 3, 4, 5))
|
||||
|
||||
def test_reads_no_route_from_a_worker_line_that_route_worker_line_could_not_have_written(self):
|
||||
for line in ('{"effort": "high"}', '{"model": "claude-opus-5-5", "effort": "high", "unfinished": 1}', '{"model": "claude-opus-5-5", "item": "3"}', '{"model": "claude-opus-5-5", "effort": 3}', '{"model": 5}', '{"model": "claude-opus-5-5", "effort": "high", "escalated_at": -3}', '{"model": "claude-opus-5-5", "item": true, "likelihood": 4, "escape_and_impact": 5}', '{"model": "claude-opus-5-5", "effort": "high", "item": 3, "likelihood": 5, "escape_and_impact": 5}', '{"model": "claude-opus-5-5", "effort": "high", "escape_and_impact_taken_in": 9}', '["claude-opus-5-5"]', '{"model": ', "worker-model", "claude-fable-5-2 at low effort, routed by item 3's L of 4 and E + I of 5"):
|
||||
with self.subTest(line=line):
|
||||
self.assertIsNone(route_from(line))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -183,7 +183,7 @@ Touches:
|
||||
- `bin/chunk_driver/project/records.py`: `read_records`
|
||||
- `bin/judge_replay/main.py`: `rule` (small)
|
||||
- `bin/judge_replay/replay.py`: `case_from` (small)
|
||||
- `bin/loop_claude/test_loop_claude.py`: `JudgePrompt`
|
||||
- `bin/agent_cycle/test_judge.py`: `JudgePrompt`
|
||||
- `bin/chunk_driver/project/test_records.py`: `Records`
|
||||
- `bin/judge_replay/test_judge_replay.py`
|
||||
|
||||
|
||||
Reference in New Issue
Block a user