Answer security round 1: 1 🔴, 1 🟡; allow only the git commands a review needs, deny the options that name a path, and load no user settings

This commit is contained in:
2026-10-16 20:24:02 +02:00
parent a80bb6cc66
commit 58252f8948
4 changed files with 20 additions and 13 deletions
+1 -1
View File
@@ -2,7 +2,7 @@
## Unreleased
- The bench's runs, `workflow-bench/bench.py`, no longer skip permissions: each run gets `Agent`, `Skill(security-review)`, `git`, edits to its clone and report, and reads of its clone, its version's copy and `~/.claude/agents/`, and is denied the rest, `~/.claude/.credentials.json` and any other shell command included. A review that needs to run a project's code or tests now has no way to; `workflow-bench/run-prompt.md` says so.
- `workflow-bench/bench.py` gives each run a fixed list of allowed tools, where it skipped every permission check. A run may launch agents, run `/security-review`, edit its clone (not the clone's `.git/`) and its report, read its clone and the copy of `~/.claude` it benches, and run `git log`, `diff`, `show`, `status`, `ls-files`, `add` and `commit -m`, without options that name a path git reads or writes. It loads no user settings, so `~/.claude/settings.json`'s hooks and extra directories do not apply to it. Every other call is denied. Expect a benched review that needs to run a project's code or tests to say in its report that it could not.
- Every goal in `README.md` → Goals is marked `**Maintain:**`, a state to hold, or `**Achieve:**`, a direction to pursue. Each achieve goal has a signal, a counter-signal and a one-line reason under it. A change that breaks a maintain goal is redesigned or dropped, whatever it scores: `rules/todo-items.md` says so, and larv-review's goal alignment step no longer merges such a PR. `rules/goals-and-audience.md` states the form for every project's README. Expect larv-review to stop a PR on a maintain goal where it asked you before.
- Reshape every project's `todo.md` to the new shape below: until it is reshaped, that is the project's only work, as it is while README.md lacks Goals and Audience, and `rank_todo.py` says so. First move every machine to this release: an older release rejects the new formula, so it fails on every reshaped `todo.md`. Then mark the README's goals `**Maintain:**` or `**Achieve:**`, score each item's L, E and I, and set a bar on the new scale below the score of every item to keep, since the old bar would drop most items. Until then, an old-shape `todo.md` still ranks by its own formula and weights.
- `todo.md` ranks items by value per cost: this setup's own plan uses `Score = 100 * V / (2**S + 8)`, where 8 is a chunk's fixed cost, about one size-3 change, and `rank_todo.py --help` shows it. A Moves column replaces A, G and Goals, naming each achieve goal an item moves and how far, such as `1.1 by 8, 1.2 by 5`. V sums each named goal's weight times its move. `rank_todo.py` weighs each achieve goal by its position in README.md, the first heaviest, so the goal weights table goes. An item names the most specific goal it moves, never a goal beside its own sub-goal. `rules/todo-items.md` and `rank_todo.py --help` show the new shape.
+3 -1
View File
@@ -332,6 +332,8 @@ The maintainability senior and the inherited architect of PR #257's scoring run
"Most, if not all, code should have automated tests", in `principles/technical-principles.md`: `compare.blind()` keeps the comparison blind, and a version name it misses invalidates the bench silently.
The tests also cover `workflow-bench/bench.py`'s `permission_flags`, with a probe: a headless claude given its flags is denied a read of a file outside the granted trees, a `git -c alias.x=… x` and a `git diff --no-index`, and is allowed the review's own git commands.
### 4. Name each chunk's archive folder by the chunk's start time, so `<key>/` lists chunks in the order they ran.
Today the folder is the SHA-256 of the transcript's first timestamped line, which tells the human nothing; that line carries the start time. Drop the fallback to the first line, which every chunk of a project shares, and give the tests' fake transcripts ISO timestamps. `bin/reading-cost` takes a chunk's id from the folder's first 12 characters and skips a chunk whose id `docs/reading-cost.md` holds, so give it a way to match a figure written under the old name.
@@ -360,7 +362,7 @@ While `rank_todo.py` ranks two shapes, projects follow two sets of scoring rules
### 155. Lower the reading cost of the files a chunk reads most before its first commit.
`bin/reading-cost` on 2026-10-16: the last 5 chunks' median is 213946 new input tokens before their first commit, the 5 before them 139738. Read most over the last 5, by opens and searches: `todo.md` 58, `README.md` 13, `CHANGELOG.md` 11, `bin/loop_claude/test_loop_claude.py` 11, `docs/decisions.md` 11, `AGENTS.md` 10, `bin/todo_table.py` 7, `agents/stability-reviewer.md` 6, `bin/loop_claude/cycle/judge.py` 5, `bin/rule_hook/test_rule_hook.py` 5. The chunk that ships this item runs `bin/reading-cost --reset`.
`bin/reading-cost` on 2026-10-16: the median of the last 5 chunks is 213,946 new input tokens before their first commit, up from 139,738 for the 5 before them. Read most over the last 5, by opens and searches: `todo.md` 58, `README.md` 13, `CHANGELOG.md` 11, `bin/loop_claude/test_loop_claude.py` 11, `docs/decisions.md` 11, `AGENTS.md` 10, `bin/todo_table.py` 7, `agents/stability-reviewer.md` 6, `bin/loop_claude/cycle/judge.py` 5, `bin/rule_hook/test_rule_hook.py` 5. The chunk that ships this item runs `bin/reading-cost --reset`.
### 1. Say in README → Install how the transcript archive is backed up, once bender-infra backs up ai-sandbox's data volume.
+15 -10
View File
@@ -33,15 +33,21 @@ CASES = {
ATTEMPTS = 4
def allowed_tools(clone, version_dir, report):
"""The tools a review needs and the paths each reaches; anything else is denied, not asked."""
agents = pathlib.Path.home() / ".claude" / "agents"
# A rule names an absolute path as "//<path>", and "/**" takes a whole tree.
reads = [f"/{root}/**" for root in (clone, version_dir, agents)]
return [
"Agent", "Skill(security-review)", "Bash(git *)", f"Edit(/{clone}/**)", f"Edit(/{report})",
*(f"{tool}({tree})" for tool in ("Read", "Grep", "Glob") for tree in reads),
def permission_flags(clone, version_dir, report):
"""The tools a review needs, each limited to what it may touch; the run denies every other call."""
# A permission rule writes an absolute path as "//<path>"; a trailing "/**" matches everything under it.
trees = (f"/{clone}/**", f"/{version_dir}/**")
git = ("log", "diff", "show", "status", "ls-files", "add", "commit -m", "commit -am")
allowed = [
"Agent", "Skill(security-review)", f"Edit(/{clone}/**)", f"Edit(/{report})",
*(f"Bash(git {command} *)" for command in git),
*(f"{tool}({tree})" for tool in ("Read", "Grep", "Glob") for tree in trees),
]
# git reads or writes any path through these options, and runs any command through a hook or config.
denied = [f"Edit(/{clone}/.git/**)", *(f"Bash(git * {option}*)" for option in ("--no-index", "--output", "-F", "--file", "-t", "--template"))]
# Only the clone's own settings: the user's add every checkout under ~/code to the readable trees.
return ["--setting-sources", "project,local", "--permission-mode", "dontAsk", "--permission-prompts", "none",
"--allowedTools", *allowed, "--disallowedTools", *denied]
def git(repo, *args):
@@ -91,8 +97,7 @@ def run(name):
log.parent.mkdir(exist_ok=True)
with log.open("w") as out:
subprocess.run(
["claude", "-p", prompt, "--permission-mode", "dontAsk", "--permission-prompts", "none",
"--allowedTools", *allowed_tools(clone, version_dir, report),
["claude", "-p", prompt, *permission_flags(clone, version_dir, report),
"--model", "claude-opus-5-5", "--output-format", "stream-json", "--verbose"],
cwd=clone, stdout=out, stderr=subprocess.STDOUT)
if report.exists():
+1 -1
View File
@@ -15,7 +15,7 @@ Report: write it to `{report}`.
- Launch every agent in the foreground (`run_in_background: false`): background agents die when this headless turn ends.
- No remote and no forge: skip the PR, CI, the external auto-reviewer and the merge; review locally.
- Do not run the repo's code or test suites: the shell is limited to git.
- The shell runs only git, so run no code or test suite. Where the process asks for a run, such as the stability pass's mutation test, review by reading and list the skipped run under Ambiguities.
- Skip the comprehension check; say so in the report.
- No human is reachable. Where the process names a no-human branch, take it; where it names none, take your recommendation and list the question in the report.
- Commit each round's fixes on `chunk`, messages prefixed `{case} - review: `. Touch nothing outside this repository except the report.