Grant the bench's runs only the tools a review needs, in place of skipping every permission
This commit is contained in:
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- The bench's runs, `workflow-bench/bench.py`, no longer skip permissions: each run gets `Agent`, `Skill(security-review)`, `git`, edits to its clone and report, and reads of its clone, its version's copy and `~/.claude/agents/`, and is denied the rest, `~/.claude/.credentials.json` and any other shell command included. A review that needs to run a project's code or tests now has no way to; `workflow-bench/run-prompt.md` says so.
|
||||
- Every goal in `README.md` → Goals is marked `**Maintain:**`, a state to hold, or `**Achieve:**`, a direction to pursue. Each achieve goal has a signal, a counter-signal and a one-line reason under it. A change that breaks a maintain goal is redesigned or dropped, whatever it scores: `rules/todo-items.md` says so, and larv-review's goal alignment step no longer merges such a PR. `rules/goals-and-audience.md` states the form for every project's README. Expect larv-review to stop a PR on a maintain goal where it asked you before.
|
||||
- Reshape every project's `todo.md` to the new shape below: until it is reshaped, that is the project's only work, as it is while README.md lacks Goals and Audience, and `rank_todo.py` says so. First move every machine to this release: an older release rejects the new formula, so it fails on every reshaped `todo.md`. Then mark the README's goals `**Maintain:**` or `**Achieve:**`, score each item's L, E and I, and set a bar on the new scale below the score of every item to keep, since the old bar would drop most items. Until then, an old-shape `todo.md` still ranks by its own formula and weights.
|
||||
- `todo.md` ranks items by value per cost: this setup's own plan uses `Score = 100 * V / (2**S + 8)`, where 8 is a chunk's fixed cost, about one size-3 change, and `rank_todo.py --help` shows it. A Moves column replaces A, G and Goals, naming each achieve goal an item moves and how far, such as `1.1 by 8, 1.2 by 5`. V sums each named goal's weight times its move. `rank_todo.py` weighs each achieve goal by its position in README.md, the first heaviest, so the goal weights table goes. An item names the most specific goal it moves, never a goal beside its own sub-goal. `rules/todo-items.md` and `rank_todo.py --help` show the new shape.
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 50 | 0.2.0 | | principle | **Grant the bench's runs only the tools a review needs, in place of skipping every permission.** | 2 | 2 | 1 | 3 | 2 | 1.1 by 3 | 5.5 |
|
||||
| 65 | 0.2.0 | | defect | **Archive each judge reply the loop rejects.** | 2 | 1 | 3 | 4 | 4 | 3 by 3 | 5.5 |
|
||||
| 153 | 0.2.0 | | decision | **Route the judge and the reviewers on the highest E + I in their chunk, an item the worker takes in included.** | 3 | 1 | 3 | 5 | 4 | 1.2 by 5 | 3.0 |
|
||||
| 161 | 0.2.0 | | | **Have loop-claude run a preflight script before each fresh worker and put its findings in the worker's prompt, so the worker no longer spends tool calls finding the chunk's mechanical facts.** | 4 | 2 | 3 | 7 | 6 | 1.1 by 4, 1.4 by 6, 1.5 by 6 | 2.9 |
|
||||
@@ -84,10 +83,6 @@
|
||||
|
||||
## Details
|
||||
|
||||
### 50. Grant the bench's runs only the tools a review needs, in place of skipping every permission.
|
||||
|
||||
"Always grant explicit access to data", in `principles/technical-principles.md`: `workflow-bench/bench.py` starts claude with `--dangerously-skip-permissions`, and no rule confines it to a sandbox, so a run on the host reaches the host's credentials and other checkouts.
|
||||
|
||||
### 65. Archive each judge reply the loop rejects.
|
||||
|
||||
A defect: goal 3, keeping unattended runs' transcripts backed up, does not hold while a rejected judge reply reaches only stderr. Raised by the principles audit of PR #215: the judge runs with `--no-session-persistence`, so the archive cannot answer the 3am question of why the loop rejected a verdict.
|
||||
|
||||
+16
-2
@@ -33,6 +33,17 @@ CASES = {
|
||||
ATTEMPTS = 4
|
||||
|
||||
|
||||
def allowed_tools(clone, version_dir, report):
|
||||
"""The tools a review needs and the paths each reaches; anything else is denied, not asked."""
|
||||
agents = pathlib.Path.home() / ".claude" / "agents"
|
||||
# A rule names an absolute path as "//<path>", and "/**" takes a whole tree.
|
||||
reads = [f"/{root}/**" for root in (clone, version_dir, agents)]
|
||||
return [
|
||||
"Agent", "Skill(security-review)", "Bash(git *)", f"Edit(/{clone}/**)", f"Edit(/{report})",
|
||||
*(f"{tool}({tree})" for tool in ("Read", "Grep", "Glob") for tree in reads),
|
||||
]
|
||||
|
||||
|
||||
def git(repo, *args):
|
||||
return subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True).stdout.strip()
|
||||
|
||||
@@ -73,13 +84,16 @@ def run(name):
|
||||
if report.exists():
|
||||
return f"{name}: done"
|
||||
base, head = fresh_clone(case, clone)
|
||||
version_dir = BENCH / "versions" / version
|
||||
prompt = (BENCH / "run-prompt.md").read_text().format(
|
||||
version_dir=BENCH / "versions" / version, base=base, head=head, case=case, report=report)
|
||||
version_dir=version_dir, base=base, head=head, case=case, report=report)
|
||||
log = BENCH / "logs" / f"{name}.{attempt}.jsonl"
|
||||
log.parent.mkdir(exist_ok=True)
|
||||
with log.open("w") as out:
|
||||
subprocess.run(
|
||||
["claude", "-p", prompt, "--dangerously-skip-permissions", "--model", "claude-opus-5-5", "--output-format", "stream-json", "--verbose"],
|
||||
["claude", "-p", prompt, "--permission-mode", "dontAsk", "--permission-prompts", "none",
|
||||
"--allowedTools", *allowed_tools(clone, version_dir, report),
|
||||
"--model", "claude-opus-5-5", "--output-format", "stream-json", "--verbose"],
|
||||
cwd=clone, stdout=out, stderr=subprocess.STDOUT)
|
||||
if report.exists():
|
||||
return f"{name}: done on attempt {attempt}"
|
||||
|
||||
@@ -15,7 +15,7 @@ Report: write it to `{report}`.
|
||||
|
||||
- Launch every agent in the foreground (`run_in_background: false`): background agents die when this headless turn ends.
|
||||
- No remote and no forge: skip the PR, CI, the external auto-reviewer and the merge; review locally.
|
||||
- Do not run the repo's test suites. A targeted probe is fine, in the repo's own container tooling.
|
||||
- Do not run the repo's code or test suites: the shell is limited to git.
|
||||
- Skip the comprehension check; say so in the report.
|
||||
- No human is reachable. Where the process names a no-human branch, take it; where it names none, take your recommendation and list the question in the report.
|
||||
- Commit each round's fixes on `chunk`, messages prefixed `{case} - review: `. Touch nothing outside this repository except the report.
|
||||
|
||||
Reference in New Issue
Block a user