Refuse a review launch below the rung loop-claude gives its worker
This commit is contained in:
@@ -8,6 +8,7 @@
|
||||
- An item that restores a maintain goal the project breaks today is exempt as `defect`, and its Moves may name that maintain goal, weighed as the heaviest achieve goal. A defect ranks by its score like any item. Only two kinds rank ahead of the rest of their release: questions first, since an answer may change other items, then security items.
|
||||
- The stability reviewer applies `principles/technical-principles.md` as written, in place of its own copies of it. Its test-data check now asks for UUID v7, where it asked for randomised UUIDs.
|
||||
- larv-review's principles audit is a pass of its own, and runs on every chunk that changes code, config or tests, not only on one with an architectural decision. A chunk of prose or planning alone skips it.
|
||||
- `rule-hook` refuses an Agent call that launches a reviewer, a comprehension seat or the security pass on a rung below the one `loop-claude` gives its worker, and takes no override: pass the model and effort the refusal names. A reviewer is held to the higher of the worker's rung and the project's; a seat or the security pass to the project's.
|
||||
|
||||
## 0.1.0
|
||||
|
||||
|
||||
@@ -88,7 +88,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
|
||||
- `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format.
|
||||
- `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure.
|
||||
- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit and Write call, and `rule-hook --help` states the rules and the override.
|
||||
- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` → `hooks` runs it before each Bash, Edit, MultiEdit, Write and Agent call, and `rule-hook --help` states the rules and the override.
|
||||
- `score-outcomes [R S C]` lists the finished items whose guessed R, S and C sit nearest the given scores, with what their chunks measured, and each score's median outcome; `score-outcomes --help` states the measures.
|
||||
- `stability-replay <model>` replays archived stability reviews of the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, finds every 🔴 and 🟡 the project's rung finds again; `stability-replay --help` states the replay. Run it only in a sandbox, like loop-claude: each replayed reviewer has Bash and skips every permission prompt, and an archived prompt may name paths on the host.
|
||||
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
|
||||
|
||||
@@ -19,6 +19,7 @@ from model_routing.route import Route
|
||||
from loop_claude.cycle.worker import run_worker, worker_args
|
||||
from model_routing.ladder import Rung
|
||||
from loop_claude.stop import Stop
|
||||
from model_routing import worker_rungs
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -84,12 +85,17 @@ def work(cycle: Cycle, answered: QuotedAnswer | None) -> WorkerRan | Stop:
|
||||
if stop := keep_answer(cycle, answered):
|
||||
return stop
|
||||
# worker_start runs inside the lambda: a failed attempt may have opened the session.
|
||||
if stop := retried("worker", lambda: run_worker(cycle.root, worker_start(cycle, answered), cycle.effects), cycle.effects):
|
||||
if stop := retried("worker", lambda: run_worker(cycle.root, worker_start(cycle, answered), worker_environment(cycle), cycle.effects), cycle.effects):
|
||||
return stop
|
||||
path = transcript_file(cycle.session)
|
||||
return WorkerRan(path) if path is not None else Stop(1, f"claude left no transcript for session {cycle.session.id}")
|
||||
|
||||
|
||||
def worker_environment(cycle: Cycle) -> dict[str, str]:
|
||||
"""The rungs the worker's claude is handed, which rule-hook holds its review launches to."""
|
||||
return worker_rungs.environment(cycle.anchor, cycle.route.rung)
|
||||
|
||||
|
||||
def worker_start(cycle: Cycle, answered: QuotedAnswer | None) -> list[str]:
|
||||
if answered is not None:
|
||||
return worker_args(cycle.session, cycle.route.rung, prompts.answered(answered.question, answered.answer), resume=True)
|
||||
|
||||
@@ -9,9 +9,9 @@ from claude_code.session import Session
|
||||
PRINTED_TOOL_INPUT = 160
|
||||
|
||||
|
||||
def run_worker(root: str, args: list[str], effects: Effects) -> Failure | Limited | None:
|
||||
def run_worker(root: str, args: list[str], env: dict[str, str], effects: Effects) -> Failure | Limited | None:
|
||||
try:
|
||||
proc = effects.stream_claude(args, root)
|
||||
proc = effects.stream_claude(args, root, env)
|
||||
except OSError as failure:
|
||||
return Failure(f"could not start claude: {failure.strerror}")
|
||||
result: dict | None = None
|
||||
|
||||
@@ -26,12 +26,12 @@ class Effects:
|
||||
on_terminal: typing.Callable[[], bool]
|
||||
python: typing.Callable[[list[str]], subprocess.CompletedProcess[str]]
|
||||
sleep: typing.Callable[[float], None]
|
||||
stream_claude: typing.Callable[[list[str], str], Streaming]
|
||||
stream_claude: typing.Callable[[list[str], str, dict[str, str]], Streaming]
|
||||
|
||||
|
||||
class StreamingClaude:
|
||||
def __init__(self, args: list[str], cwd: str) -> None:
|
||||
self.proc = subprocess.Popen(["claude", *args], cwd=cwd, encoding="utf-8", errors="replace", stdin=subprocess.DEVNULL, stdout=subprocess.PIPE)
|
||||
def __init__(self, args: list[str], cwd: str, env: dict[str, str]) -> None:
|
||||
self.proc = subprocess.Popen(["claude", *args], cwd=cwd, env={**os.environ, **env}, encoding="utf-8", errors="replace", stdin=subprocess.DEVNULL, stdout=subprocess.PIPE)
|
||||
|
||||
def lines(self) -> typing.Iterable[str]:
|
||||
with self.proc:
|
||||
|
||||
@@ -52,6 +52,7 @@ class FakeEffects:
|
||||
self.calls = []
|
||||
self.scripts = {"claude": list(claude), "claude_on_terminal": list(on_terminal_claude), "git": list(git), "input": list(inputs), "python": list(python), "stream_claude": list(streams)}
|
||||
self.terminal = on_terminal
|
||||
self.worker_environments = []
|
||||
|
||||
def effects(self):
|
||||
return Effects(
|
||||
@@ -65,9 +66,13 @@ class FakeEffects:
|
||||
on_terminal=lambda: self.terminal,
|
||||
python=lambda args: self.answer("python", args),
|
||||
sleep=lambda seconds: self.calls.append(("sleep", seconds)),
|
||||
stream_claude=lambda args, cwd: self.answer("stream_claude", args),
|
||||
stream_claude=self.stream,
|
||||
)
|
||||
|
||||
def stream(self, args, cwd, env):
|
||||
self.worker_environments.append(env)
|
||||
return self.answer("stream_claude", args)
|
||||
|
||||
def answer(self, seam, what):
|
||||
self.calls.append((seam, what))
|
||||
answer = self.scripts[seam].pop(0)
|
||||
@@ -186,6 +191,12 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: finished\n\nEvidence:\n> working\n\nJudge: judge-model\nWorker: worker-model\n"])
|
||||
self.assertIsNone(session.transcript_file(self.session))
|
||||
|
||||
def test_a_worker_is_handed_the_project_and_its_own_rung_in_its_environment(self):
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("finished")])
|
||||
cycle = dataclasses.replace(self.cycle(fake), anchor=Rung("claude-opus-5-5", "medium"), route=Route(Rung("claude-haiku-5-5", "max"), 163, 5))
|
||||
self.assertEqual(run.run_cycle(cycle), Finished("working"))
|
||||
self.assertEqual(fake.worker_environments, [{"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 medium", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 max"}])
|
||||
|
||||
def test_an_unfinished_chunks_transcript_is_archived_and_the_next_cycle_resumes_its_session_with_the_judges_evidence(self):
|
||||
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)] * 2, claude=[judged("unfinished")] * 2)
|
||||
self.assertEqual(run.run_cycle(self.cycle(fake)), Unfinished("working"))
|
||||
@@ -333,7 +344,7 @@ class Limits(unittest.TestCase):
|
||||
allowed = {"type": "rate_limit_event", "rate_limit_info": {"status": "allowed", "resetsAt": NOW + 9000}}
|
||||
limited = {"type": "result", "api_error_status": 429, "result": "You've hit your session limit"}
|
||||
fake = FakeEffects(streams=[Streamed(1, rejected[1], allowed, rejected[0], limited)])
|
||||
self.assertEqual(worker.run_worker("/r", [], fake.effects()), Limited(NOW + 600, "You've hit your session limit"))
|
||||
self.assertEqual(worker.run_worker("/r", [], {}, fake.effects()), Limited(NOW + 600, "You've hit your session limit"))
|
||||
|
||||
def test_a_limit_outranks_the_exit_code_and_its_message_fits_one_line(self):
|
||||
def limited(**said):
|
||||
|
||||
@@ -11,9 +11,13 @@ class Rung:
|
||||
def flags(self) -> list[str]:
|
||||
return ["--model", self.model, *(["--effort", self.effort] if self.effort else [])]
|
||||
|
||||
def family(self) -> str:
|
||||
"""The Agent tool's name for the model, such as opus for claude-opus-5-5."""
|
||||
return self.model.split("-")[1]
|
||||
|
||||
def agent_args(self) -> str:
|
||||
"""The rung in the Agent tool's words. The tool takes a family name, such as opus, and runs that family's latest model, which is the model each rung names."""
|
||||
return f"model {self.model.split('-')[1]}, effort {self.effort}"
|
||||
return f"model {self.family()}, effort {self.effort}"
|
||||
|
||||
# The archive records this wording, and model_routing.route.ROUTED reads it back to resume a chunk on its rung.
|
||||
def __str__(self) -> str:
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
import unittest
|
||||
|
||||
from model_routing.worker_rungs import environment, handed
|
||||
from model_routing.ladder import Rung
|
||||
|
||||
|
||||
class Environment(unittest.TestCase):
|
||||
PROJECT = Rung("claude-opus-5-5", "medium")
|
||||
WORKER = Rung("claude-haiku-5-5", "max")
|
||||
|
||||
def test_the_project_and_the_worker_rung_are_named_in_the_environment(self):
|
||||
self.assertEqual(environment(self.PROJECT, self.WORKER), {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 medium", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 max"})
|
||||
|
||||
def test_a_project_off_the_ladder_leaves_its_rung_unset(self):
|
||||
self.assertEqual(environment(None, self.WORKER), {"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 max"})
|
||||
|
||||
def test_handed_reads_back_what_environment_named(self):
|
||||
self.assertEqual(handed(environment(self.PROJECT, self.WORKER)), (self.PROJECT, self.WORKER))
|
||||
self.assertEqual(handed(environment(None, self.WORKER)), (None, self.WORKER))
|
||||
|
||||
def test_a_missing_or_unknown_rung_reads_as_none(self):
|
||||
self.assertEqual(handed({}), (None, None))
|
||||
self.assertEqual(handed({"LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 bogus"}), (None, None))
|
||||
@@ -0,0 +1,27 @@
|
||||
"""The rungs loop-claude hands a worker's claude in its environment, so rule-hook holds each review launch to them: docs/decisions.md → `routing:`."""
|
||||
import typing
|
||||
|
||||
from model_routing.ladder import LADDER, Rung
|
||||
|
||||
PROJECT_RUNG = "LOOP_CLAUDE_PROJECT_RUNG"
|
||||
WORKER_RUNG = "LOOP_CLAUDE_WORKER_RUNG"
|
||||
|
||||
|
||||
def environment(project: Rung | None, worker: Rung) -> dict[str, str]:
|
||||
"""The variables a worker's claude starts with; a project off the ladder has no rung to hand."""
|
||||
return ({PROJECT_RUNG: encoded(project)} if project else {}) | {WORKER_RUNG: encoded(worker)}
|
||||
|
||||
|
||||
def handed(environ: typing.Mapping[str, str]) -> tuple[Rung | None, Rung | None]:
|
||||
"""The project's and the worker's rungs, each None when unset or not on the ladder."""
|
||||
return decoded(environ.get(PROJECT_RUNG)), decoded(environ.get(WORKER_RUNG))
|
||||
|
||||
|
||||
def encoded(rung: Rung) -> str:
|
||||
return f"{rung.model} {rung.effort}"
|
||||
|
||||
|
||||
def decoded(value: str | None) -> Rung | None:
|
||||
model, _, effort = (value or "").partition(" ")
|
||||
rung = Rung(model, effort)
|
||||
return rung if rung in LADDER else None
|
||||
@@ -3,12 +3,12 @@
|
||||
usage: rule-hook (a PreToolUse hook: reads the call as JSON on stdin)
|
||||
rule-hook overrides (prints this chunk's overrides)
|
||||
|
||||
settings.json runs it before each Bash, Edit, MultiEdit and Write call, in every session and
|
||||
settings.json runs it before each Bash, Edit, MultiEdit, Write and Agent call, in every session and
|
||||
subagent. The rules:
|
||||
|
||||
{rules}
|
||||
|
||||
A file a Bash command writes, such as through `cat > Dockerfile`, is not checked.
|
||||
A file a Bash command writes, such as through `cat > Dockerfile`, is not checked. An Agent call takes no override.
|
||||
|
||||
A call overrides a rule with `# rule-override <rule>: <reason>`:
|
||||
|
||||
@@ -34,12 +34,14 @@ import os
|
||||
import pathlib
|
||||
import sys
|
||||
import time
|
||||
import typing
|
||||
|
||||
from rule_hook.call import applied, Call, FILE_TOOLS, parsed_call, text
|
||||
from rule_hook.rules import Breach, command_breaches, file_breaches, Outcome, rules_help, verdict
|
||||
from git_command import commits
|
||||
from failure import Failure
|
||||
from rule_hook.override_log import Log, log_path, logged, print_overrides, write_override
|
||||
from rule_hook.reviewer_rung import reviewer_breaches
|
||||
from shell_command import simple_commands
|
||||
from rule_hook.todo_check import todo_breaches
|
||||
|
||||
@@ -60,15 +62,17 @@ def main(argv: list[str], stdin: str) -> int:
|
||||
if isinstance(call, Failure):
|
||||
print(f"rule-hook cannot read the hook input: {call.reason}", file=sys.stderr)
|
||||
return 1
|
||||
found = breaches_of(call, os.path.expanduser("~"))
|
||||
found = breaches_of(call, os.path.expanduser("~"), os.environ)
|
||||
if found is None:
|
||||
return 0
|
||||
written, what, breaches = found
|
||||
return answer(verdict(breaches, written, text(call.given, "file_path").endswith(".md"), call.tool == "Bash"), what, time.time())
|
||||
|
||||
|
||||
def breaches_of(call: Call, home: str) -> tuple[str, str, list[Breach]] | None:
|
||||
def breaches_of(call: Call, home: str, environ: typing.Mapping[str, str]) -> tuple[str, str, list[Breach]] | None:
|
||||
"""The text that may carry an override, what the log names the call by, and its breaches; None for a call no rule checks."""
|
||||
if call.tool == "Agent":
|
||||
return "", text(call.given, "subagent_type"), reviewer_breaches(call, environ)
|
||||
if call.tool == "Bash":
|
||||
command = text(call.given, "command")
|
||||
return command, command.split("\n", 1)[0], bash_breaches(command, call.cwd, home)
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
"""Hold each Agent call that launches a review to the rung loop-claude hands its worker: docs/decisions.md → `routing:`."""
|
||||
import re
|
||||
import typing
|
||||
|
||||
from rule_hook.rules import Breach
|
||||
from rule_hook.call import Call, text
|
||||
from model_routing.worker_rungs import handed
|
||||
from model_routing.ladder import highest, LADDER, position, Rung
|
||||
|
||||
# larv-review §3's reviewers, by the subagent type its passes launch.
|
||||
REVIEWERS = frozenset(("override-reviewer", "principles-auditor", "product-owner-reviewer", "prose-reviewer", "stability-reviewer", "systems-architect"))
|
||||
# The opening of every comprehension seat's prompt: skills/comprehension-panel/seat-prompts.md → "Opening".
|
||||
SEAT = "You are one seat of a comprehension panel."
|
||||
# The security pass's subagent invokes one of these skills.
|
||||
SECURITY = re.compile(r"/security-review\b|/claude-security\b")
|
||||
SUBJECTS = {"seat": "a comprehension seat", "security": "the security pass's subagent"}
|
||||
|
||||
|
||||
def reviewer_breaches(call: Call, environ: typing.Mapping[str, str]) -> list[Breach]:
|
||||
"""The breach of an Agent call that launches a review below its rung; none where loop-claude handed no rungs."""
|
||||
project, worker = handed(environ)
|
||||
kind = review_kind(call.given)
|
||||
if kind is None or project is None or worker is None:
|
||||
return []
|
||||
# A reviewer runs on the higher of the worker's rung and the project's; a seat or the security pass on the project's.
|
||||
required = highest(project, worker) if kind == "reviewer" else project
|
||||
named = named_rung(call.given, worker)
|
||||
if position(named) >= position(required):
|
||||
return []
|
||||
subject = text(call.given, "subagent_type") if kind == "reviewer" else SUBJECTS[kind]
|
||||
return [Breach("reviewer-rung", f"{subject} would run on {named}, below {required}: launch it with {required.agent_args()}", call.cwd)]
|
||||
|
||||
|
||||
def review_kind(given: dict[str, typing.Any]) -> str | None:
|
||||
"""What a launch reviews, a reviewer, a comprehension seat or the security pass; None for any other subagent."""
|
||||
if text(given, "subagent_type") in REVIEWERS:
|
||||
return "reviewer"
|
||||
prompt = text(given, "prompt")
|
||||
if SEAT in prompt:
|
||||
return "seat"
|
||||
if SECURITY.search(prompt):
|
||||
return "security"
|
||||
return None
|
||||
|
||||
|
||||
def named_rung(given: dict[str, typing.Any], worker: Rung) -> Rung:
|
||||
"""The rung an Agent call's model and effort name. A call naming no model runs on the worker's model, at the effort it names or else the worker's; one naming a model but no effort runs on that model's cheapest rung."""
|
||||
model, effort = text(given, "model"), text(given, "effort")
|
||||
if not model:
|
||||
return Rung(worker.model, effort or worker.effort)
|
||||
family = [rung for rung in LADDER if rung.family() == model]
|
||||
if not family:
|
||||
return Rung(model, effort)
|
||||
return Rung(family[0].model, effort) if effort else family[0]
|
||||
+11
-1
@@ -7,13 +7,15 @@ from git_command import commits
|
||||
from round_titles import lacks_counts
|
||||
from shell_command import Simple
|
||||
|
||||
RuleName = typing.Literal["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "round-title-counts", "todo-rank-check"]
|
||||
RuleName = typing.Literal["action-pin", "commit-one-line", "ignorefile-sorted", "image-pin", "no-ai-attribution", "reviewer-rung", "round-title-counts", "todo-rank-check"]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Rule:
|
||||
holds: str
|
||||
checked: str
|
||||
# False where no marker can waive the rule, since its call's text is not scanned for one.
|
||||
overridable: bool = True
|
||||
|
||||
|
||||
RULES: dict[RuleName, Rule] = {
|
||||
@@ -37,6 +39,11 @@ RULES: dict[RuleName, Rule] = {
|
||||
"no commit, PR, issue comment or file carries a Claude or Anthropic attribution, such as a co-author trailer or a generated-with line",
|
||||
"a git, gh or tea command, and any file written",
|
||||
),
|
||||
"reviewer-rung": Rule(
|
||||
"a reviewer, a comprehension seat or the security pass's subagent runs on the rung loop-claude names for it: the project's rung, or for a reviewer the worker's rung where that is higher, so a worker below the project's rung cannot hand a review to its own cheaper model",
|
||||
"an Agent call that launches a reviewer type, a comprehension seat, or a /security-review or /claude-security launch, in a worker loop-claude started",
|
||||
overridable=False,
|
||||
),
|
||||
"round-title-counts": Rule(
|
||||
"a larv-review round's commit title gives the counts of 🔴 and 🟡 findings the round reported, such as `Answer stability round 1: 1 🔴, 2 🟡; flag a deleted test`, since score-outcomes counts them",
|
||||
"a git commit whose message opens `Answer <pass> round <n>:`",
|
||||
@@ -88,6 +95,9 @@ def verdict(breaches: list[Breach], written: str, markdown: bool, bash: bool) ->
|
||||
where = "the command" if bash else "the text you write, as a comment line"
|
||||
for rule in sorted({breach.rule for breach in blocking}):
|
||||
details = "; ".join(breach.detail for breach in blocking if breach.rule == rule)
|
||||
if not RULES[rule].overridable:
|
||||
messages.append(f"Blocked by rule {rule}: {details}. The rule: {RULES[rule].holds}.")
|
||||
continue
|
||||
marker = f"<!-- rule-override {rule}: <reason> -->" if markdown else f"# rule-override {rule}: <reason>"
|
||||
messages.append(f"Blocked by rule {rule}: {details}. The rule: {RULES[rule].holds}. Where it does not fit this case, add `{marker}` to {where}; larv-review rules on the reason.")
|
||||
return Outcome(messages, bool(blocking), [(breach, granted[breach.rule]) for breach in breaches if breach.rule in granted])
|
||||
|
||||
@@ -9,10 +9,12 @@ import unittest
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from rule_hook.rules import blocks, Breach, command_breaches, file_breaches, overrides, verdict
|
||||
from rule_hook.call import Call
|
||||
from git_command import Commit, commit_message, commits
|
||||
from test_rank_todo import DECISIONS, README, TODO
|
||||
from test_env import isolated_env
|
||||
from rule_hook.override_log import LogLine, parsed_entry, this_chunks
|
||||
from rule_hook.reviewer_rung import reviewer_breaches
|
||||
from shell_command import simple_commands
|
||||
|
||||
HOOK = pathlib.Path(__file__).resolve().parents[1] / "rule-hook"
|
||||
@@ -32,8 +34,8 @@ def checked(command: str) -> list[str]:
|
||||
return rules(command_breaches(command, simple_commands(command, "/r", "/h"), "/h"))
|
||||
|
||||
|
||||
def hook(call: dict, cwd: str) -> subprocess.CompletedProcess:
|
||||
return subprocess.run([str(HOOK)], capture_output=True, cwd=cwd, env=isolated_env(), input=json.dumps(call), text=True)
|
||||
def hook(call: dict, cwd: str, env: dict[str, str] | None = None) -> subprocess.CompletedProcess:
|
||||
return subprocess.run([str(HOOK)], capture_output=True, cwd=cwd, env=isolated_env() | (env or {}), input=json.dumps(call), text=True)
|
||||
|
||||
|
||||
def git(cwd: str, *args: str) -> str:
|
||||
@@ -158,6 +160,56 @@ class OverrideLog(unittest.TestCase):
|
||||
self.assertEqual(this_chunks(LogLine("t", branch, commit, "image-pin", "r", "w"), {"base", "c1"}, "chunk", on_default), expected)
|
||||
|
||||
|
||||
class ReviewerRung(unittest.TestCase):
|
||||
# A worker routed to Haiku 5.5 at max, on a project whose rung is Opus 5.5 at medium.
|
||||
ENVIRON = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 medium", "LOOP_CLAUDE_WORKER_RUNG": "claude-haiku-5-5 max"}
|
||||
# The same project, with a worker routed above it to Fable 5.1 at max.
|
||||
ABOVE = {"LOOP_CLAUDE_PROJECT_RUNG": "claude-opus-5-5 medium", "LOOP_CLAUDE_WORKER_RUNG": "claude-fable-5-1 max"}
|
||||
|
||||
def refused(self, given: dict, environ: dict[str, str] = ENVIRON) -> list[str]:
|
||||
return rules(reviewer_breaches(Call("Agent", given, "/r"), environ))
|
||||
|
||||
def test_a_reviewer_below_the_project_rung_is_refused_naming_the_arguments_to_pass(self):
|
||||
breaches = reviewer_breaches(Call("Agent", {"subagent_type": "stability-reviewer", "model": "sonnet", "effort": "high", "prompt": "Review."}, "/r"), self.ENVIRON)
|
||||
self.assertEqual(rules(breaches), ["reviewer-rung"])
|
||||
self.assertEqual(breaches[0].detail, "stability-reviewer would run on claude-sonnet-5-5 at high effort, below claude-opus-5-5 at medium effort: launch it with model opus, effort medium")
|
||||
|
||||
def test_a_reviewer_on_the_project_rung_or_a_rerun_above_it_passes(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "opus", "effort": "medium"}), [])
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "opus", "effort": "high"}), [])
|
||||
|
||||
def test_a_call_naming_no_model_runs_on_the_workers_rung(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "prompt": "Review."}), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "effort": "medium"}), ["reviewer-rung"])
|
||||
|
||||
def test_a_model_named_without_an_effort_runs_on_its_cheapest_rung(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "opus"}), [])
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "sonnet"}), ["reviewer-rung"])
|
||||
|
||||
def test_an_effort_off_the_ladder_for_its_model_is_refused(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "prose-reviewer", "model": "opus", "effort": "low"}), ["reviewer-rung"])
|
||||
|
||||
def test_a_reviewer_is_held_to_the_workers_rung_where_that_is_higher(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium"}, self.ABOVE), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "fable", "effort": "max"}, self.ABOVE), [])
|
||||
|
||||
def test_a_seat_or_a_security_launch_is_held_to_the_project_rung_alone(self):
|
||||
seat = {"prompt": "You are one seat of a comprehension panel. Tools: Read, Grep, Glob.", "model": "opus", "effort": "medium"}
|
||||
self.assertEqual(self.refused(seat, self.ABOVE), [])
|
||||
self.assertEqual(self.refused({"prompt": "Invoke /security-review over the branch diff.", "model": "sonnet", "effort": "high"}), ["reviewer-rung"])
|
||||
self.assertEqual(self.refused({"prompt": "Run /claude-security scan this branch's changes at high.", "model": "opus", "effort": "medium"}), [])
|
||||
|
||||
def test_a_seat_named_below_the_project_rung_is_refused(self):
|
||||
self.assertEqual(self.refused({"prompt": "You are one seat of a comprehension panel.", "model": "sonnet", "effort": "high"}), ["reviewer-rung"])
|
||||
|
||||
def test_any_other_subagent_keeps_its_own_routing(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "Explore", "model": "haiku", "effort": "low", "prompt": "Find the callers of run_worker."}), [])
|
||||
self.assertEqual(self.refused({"prompt": "Note that security-review runs in CI.", "model": "haiku", "effort": "low"}), [])
|
||||
|
||||
def test_a_claude_loop_claude_did_not_start_is_not_checked(self):
|
||||
self.assertEqual(self.refused({"subagent_type": "stability-reviewer", "model": "haiku", "effort": "low"}, {}), [])
|
||||
|
||||
|
||||
class Hook(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.repo = tempfile.mkdtemp()
|
||||
@@ -218,6 +270,17 @@ class Hook(unittest.TestCase):
|
||||
self.assertEqual(hook({"cwd": self.repo, "tool_input": {"command": command}, "tool_name": "Bash"}, self.repo).returncode, 2)
|
||||
self.assertEqual(self.listed(), "")
|
||||
|
||||
def test_an_agent_call_below_its_rung_is_refused_and_offers_no_override(self):
|
||||
call = {"cwd": self.repo, "tool_input": {"subagent_type": "stability-reviewer", "model": "sonnet", "effort": "high", "prompt": "Review. # rule-override reviewer-rung: trust me"}, "tool_name": "Agent"}
|
||||
ran = hook(call, self.repo, ReviewerRung.ENVIRON)
|
||||
self.assertEqual(ran.returncode, 2)
|
||||
self.assertIn("Blocked by rule reviewer-rung: stability-reviewer would run on claude-sonnet-5-5 at high effort", ran.stderr)
|
||||
self.assertNotIn("rule-override", ran.stderr)
|
||||
|
||||
def test_an_agent_call_at_its_rung_passes(self):
|
||||
call = {"cwd": self.repo, "tool_input": {"subagent_type": "stability-reviewer", "model": "opus", "effort": "medium", "prompt": "Review."}, "tool_name": "Agent"}
|
||||
self.assertEqual(hook(call, self.repo, ReviewerRung.ENVIRON).returncode, 0)
|
||||
|
||||
def test_a_commit_override_is_logged_in_the_commit_s_checkout(self):
|
||||
outside = tempfile.mkdtemp()
|
||||
self.addCleanup(subprocess.run, ["rm", "-rf", outside])
|
||||
|
||||
+1
-1
@@ -36,7 +36,7 @@
|
||||
"hooks": {
|
||||
"PreToolUse": [
|
||||
{
|
||||
"matcher": "Bash|Edit|MultiEdit|Write",
|
||||
"matcher": "Bash|Edit|MultiEdit|Write|Agent",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | R | S | C | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| 163 | 0.2.0 | | | **Have `rule-hook` block an Agent call that launches a reviewer, a comprehension seat or the security pass's subagent below the rung loop-claude names for them, so a worker below the project's rung cannot hand a review to its own cheaper model.** | 2 | 3 | 3 | 1.1 by 5 | 6.8 |
|
||||
| 146 | 0.2.0 | | decision | **Score R as L + E + I − 2 from anchored levels, at least 7 at I5 and 5 at I4, with L, E and I in their own columns and `rank_todo.py` computing R, drop C, and route a task's model and effort on its R alone.** | 6 | 7 | 7 | 1.1 by 8, 1.2 by 8, 1.4 by 8 | 2.8 |
|
||||
| 152 | 0.2.0 | 146 | decision | **Anchor each score level in `rules/todo-items.md` with words, and each project's with two or three items it shipped, and have a scorer compare a new item against them.** | 2 | 4 | 3 | 1.1 by 7 | 6.4 |
|
||||
| 50 | 0.2.0 | | principle | **Grant the bench's runs only the tools a review needs, in place of skipping every permission.** | 2 | 2 | 3 | 1.1 by 3 | 5.5 |
|
||||
@@ -80,12 +79,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 163. Have `rule-hook` block an Agent call that launches a reviewer, a comprehension seat or the security pass's subagent below the rung loop-claude names for them, so a worker below the project's rung cannot hand a review to its own cheaper model.
|
||||
|
||||
`docs/decisions.md` → `routing: a chunk's worker runs on the rung its item picks…` keeps every reviewer, comprehension seat and the security pass's subagent at or above the project's rung. Only loop-claude's prompt holds that today. No agent under `agents/` sets `model:`, so an Agent call that names no model runs on the caller's: a Haiku worker that leaves out `model` once gets a Haiku review, and nothing reports it. On 2026-10-16, item 67's chunk passed `model=opus, effort=medium` on every such call, so no review has run low yet.
|
||||
|
||||
The hook sees only the tool call and its working directory, so loop-claude must hand it the project's rung and the worker's, such as in environment variables on the worker's `claude`. The hook then refuses an Agent call of a reviewer type in larv-review §3's table, or a seat or security launch, whose model and effort fall below the project's rung, counting a call that names no model as the worker's rung, and names the arguments to pass. A rerun one rung up, per `review: a reviewer or comprehension seat that rules the work below standard…`, passes, since it is higher. A subagent that is not a reviewer keeps routing by its own R + C.
|
||||
|
||||
### 146. Score R as L + E + I − 2 from anchored levels, at least 7 at I5 and 5 at I4, with L, E and I in their own columns and `rank_todo.py` computing R, drop C, and route a task's model and effort on its R alone.
|
||||
|
||||
Each level is about ×10 of a real quantity; `docs/decisions.md` → `todo.md: …` says why. The anchors, for `rules/todo-items.md`:
|
||||
|
||||
Reference in New Issue
Block a user