Grant the bench's runs only the tools a review needs, in place of skipping every permission

This commit is contained in:
2026-10-16 20:17:42 +02:00
parent 11b9ec313d
commit 14c8e7f012
4 changed files with 18 additions and 8 deletions
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- The bench's runs, `workflow-bench/bench.py`, no longer skip permissions: each run gets `Agent`, `Skill(security-review)`, `git`, edits to its clone and report, and reads of its clone, its version's copy and `~/.claude/agents/`, and is denied the rest, `~/.claude/.credentials.json` and any other shell command included. A review that needs to run a project's code or tests now has no way to; `workflow-bench/run-prompt.md` says so.
- Every goal in `README.md` → Goals is marked `**Maintain:**`, a state to hold, or `**Achieve:**`, a direction to pursue. Each achieve goal has a signal, a counter-signal and a one-line reason under it. A change that breaks a maintain goal is redesigned or dropped, whatever it scores: `rules/todo-items.md` says so, and larv-review's goal alignment step no longer merges such a PR. `rules/goals-and-audience.md` states the form for every project's README. Expect larv-review to stop a PR on a maintain goal where it asked you before.
- Reshape every project's `todo.md` to the new shape below: until it is reshaped, that is the project's only work, as it is while README.md lacks Goals and Audience, and `rank_todo.py` says so. First move every machine to this release: an older release rejects the new formula, so it fails on every reshaped `todo.md`. Then mark the README's goals `**Maintain:**` or `**Achieve:**`, score each item's L, E and I, and set a bar on the new scale below the score of every item to keep, since the old bar would drop most items. Until then, an old-shape `todo.md` still ranks by its own formula and weights.
- `todo.md` ranks items by value per cost: this setup's own plan uses `Score = 100 * V / (2**S + 8)`, where 8 is a chunk's fixed cost, about one size-3 change, and `rank_todo.py --help` shows it. A Moves column replaces A, G and Goals, naming each achieve goal an item moves and how far, such as `1.1 by 8, 1.2 by 5`. V sums each named goal's weight times its move. `rank_todo.py` weighs each achieve goal by its position in README.md, the first heaviest, so the goal weights table goes. An item names the most specific goal it moves, never a goal beside its own sub-goal. `rules/todo-items.md` and `rank_todo.py --help` show the new shape.
-5
View File
@@ -12,7 +12,6 @@
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 50 | 0.2.0 | | principle | **Grant the bench's runs only the tools a review needs, in place of skipping every permission.** | 2 | 2 | 1 | 3 | 2 | 1.1 by 3 | 5.5 |
| 65 | 0.2.0 | | defect | **Archive each judge reply the loop rejects.** | 2 | 1 | 3 | 4 | 4 | 3 by 3 | 5.5 |
| 153 | 0.2.0 | | decision | **Route the judge and the reviewers on the highest E + I in their chunk, an item the worker takes in included.** | 3 | 1 | 3 | 5 | 4 | 1.2 by 5 | 3.0 |
| 161 | 0.2.0 | | | **Have loop-claude run a preflight script before each fresh worker and put its findings in the worker's prompt, so the worker no longer spends tool calls finding the chunk's mechanical facts.** | 4 | 2 | 3 | 7 | 6 | 1.1 by 4, 1.4 by 6, 1.5 by 6 | 2.9 |
@@ -84,10 +83,6 @@
## Details
### 50. Grant the bench's runs only the tools a review needs, in place of skipping every permission.
"Always grant explicit access to data", in `principles/technical-principles.md`: `workflow-bench/bench.py` starts claude with `--dangerously-skip-permissions`, and no rule confines it to a sandbox, so a run on the host reaches the host's credentials and other checkouts.
### 65. Archive each judge reply the loop rejects.
A defect: goal 3, keeping unattended runs' transcripts backed up, does not hold while a rejected judge reply reaches only stderr. Raised by the principles audit of PR #215: the judge runs with `--no-session-persistence`, so the archive cannot answer the 3am question of why the loop rejected a verdict.
+16 -2
View File
@@ -33,6 +33,17 @@ CASES = {
ATTEMPTS = 4
def allowed_tools(clone, version_dir, report):
"""The tools a review needs and the paths each reaches; anything else is denied, not asked."""
agents = pathlib.Path.home() / ".claude" / "agents"
# A rule names an absolute path as "//<path>", and "/**" takes a whole tree.
reads = [f"/{root}/**" for root in (clone, version_dir, agents)]
return [
"Agent", "Skill(security-review)", "Bash(git *)", f"Edit(/{clone}/**)", f"Edit(/{report})",
*(f"{tool}({tree})" for tool in ("Read", "Grep", "Glob") for tree in reads),
]
def git(repo, *args):
return subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True).stdout.strip()
@@ -73,13 +84,16 @@ def run(name):
if report.exists():
return f"{name}: done"
base, head = fresh_clone(case, clone)
version_dir = BENCH / "versions" / version
prompt = (BENCH / "run-prompt.md").read_text().format(
version_dir=BENCH / "versions" / version, base=base, head=head, case=case, report=report)
version_dir=version_dir, base=base, head=head, case=case, report=report)
log = BENCH / "logs" / f"{name}.{attempt}.jsonl"
log.parent.mkdir(exist_ok=True)
with log.open("w") as out:
subprocess.run(
["claude", "-p", prompt, "--dangerously-skip-permissions", "--model", "claude-opus-5-5", "--output-format", "stream-json", "--verbose"],
["claude", "-p", prompt, "--permission-mode", "dontAsk", "--permission-prompts", "none",
"--allowedTools", *allowed_tools(clone, version_dir, report),
"--model", "claude-opus-5-5", "--output-format", "stream-json", "--verbose"],
cwd=clone, stdout=out, stderr=subprocess.STDOUT)
if report.exists():
return f"{name}: done on attempt {attempt}"
+1 -1
View File
@@ -15,7 +15,7 @@ Report: write it to `{report}`.
- Launch every agent in the foreground (`run_in_background: false`): background agents die when this headless turn ends.
- No remote and no forge: skip the PR, CI, the external auto-reviewer and the merge; review locally.
- Do not run the repo's test suites. A targeted probe is fine, in the repo's own container tooling.
- Do not run the repo's code or test suites: the shell is limited to git.
- Skip the comprehension check; say so in the report.
- No human is reachable. Where the process names a no-human branch, take it; where it names none, take your recommendation and list the question in the report.
- Commit each round's fixes on `chunk`, messages prefixed `{case} - review: `. Touch nothing outside this repository except the report.