Answer security round 3: 0 🔴, 0 🟡; the answer holds
This commit is contained in:
@@ -88,4 +88,5 @@
|
||||
!workflow-bench/bench.py
|
||||
!workflow-bench/compare-prompt.md
|
||||
!workflow-bench/compare.py
|
||||
!workflow-bench/review-git.py
|
||||
!workflow-bench/run-prompt.md
|
||||
|
||||
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- `workflow-bench/bench.py` gives each run a fixed list of allowed tools, where it skipped every permission check. A run may launch agents, run `/security-review`, edit its clone (not the clone's `.git/`) and its report, read its clone and the copy of `~/.claude` it benches, and run `git log`, `diff`, `show`, `status`, `ls-files`, `add` and `commit -m` (also as `-am` or `-a -m`). A git argument that names a path outside the clone is denied: an absolute path, one from `~`, one through `..` or one through a shell variable. It loads no user settings, so `~/.claude/settings.json`'s hooks and extra directories do not apply to it. Every other call is denied. Expect a benched review that needs to run a project's code or tests to say in its report that it could not.
|
||||
- `workflow-bench/bench.py` gives each run a fixed list of allowed tools, where it skipped every permission check. A run may launch agents, run `/security-review`, edit its clone (not the clone's `.git/` or `.claude/`) and its report, and read its clone and the copy of `~/.claude` it benches. For git it gets one command, `workflow-bench/review-git.py`, and no bare `git`: the wrapper runs `log`, `diff`, `show`, `status`, `ls-files`, `add` and `commit` under the clone with a scrubbed config, refusing an argument that names a path outside the clone and any option that reads a file, writes a file or runs a program. So no git argument, option, hook, redirect or shell substitution reaches beyond the clone. It loads no user settings, so `~/.claude/settings.json`'s hooks and extra directories do not apply to it. Every other call is denied. Expect a benched review that needs to run a project's code or tests to say in its report that it could not.
|
||||
- Every goal in `README.md` → Goals is marked `**Maintain:**`, a state to hold, or `**Achieve:**`, a direction to pursue. Each achieve goal has a signal, a counter-signal and a one-line reason under it. A change that breaks a maintain goal is redesigned or dropped, whatever it scores: `rules/todo-items.md` says so, and larv-review's goal alignment step no longer merges such a PR. `rules/goals-and-audience.md` states the form for every project's README. Expect larv-review to stop a PR on a maintain goal where it asked you before.
|
||||
- Reshape every project's `todo.md` to the new shape below: until it is reshaped, that is the project's only work, as it is while README.md lacks Goals and Audience, and `rank_todo.py` says so. First move every machine to this release: an older release rejects the new formula, so it fails on every reshaped `todo.md`. Then mark the README's goals `**Maintain:**` or `**Achieve:**`, score each item's L, E and I, and set a bar on the new scale below the score of every item to keep, since the old bar would drop most items. Until then, an old-shape `todo.md` still ranks by its own formula and weights.
|
||||
- `todo.md` ranks items by value per cost: this setup's own plan uses `Score = 100 * V / (2**S + 8)`, where 8 is a chunk's fixed cost, about one size-3 change, and `rank_todo.py --help` shows it. A Moves column replaces A, G and Goals, naming each achieve goal an item moves and how far, such as `1.1 by 8, 1.2 by 5`. V sums each named goal's weight times its move. `rank_todo.py` weighs each achieve goal by its position in README.md, the first heaviest, so the goal weights table goes. An item names the most specific goal it moves, never a goal beside its own sub-goal. `rules/todo-items.md` and `rank_todo.py --help` show the new shape.
|
||||
|
||||
+8
-10
@@ -23,6 +23,7 @@ import sys
|
||||
import time
|
||||
|
||||
BENCH = pathlib.Path(__file__).resolve().parent
|
||||
REVIEW_GIT = BENCH / "review-git.py"
|
||||
SOURCES = pathlib.Path(os.environ["BENCH_SOURCES"])
|
||||
CASES = {
|
||||
"cjk": ("151bf38^", "6c49912"),
|
||||
@@ -37,19 +38,16 @@ def permission_flags(clone, version_dir, report):
|
||||
"""The tools a review needs, each limited to what it may touch; the run denies every other call."""
|
||||
# A permission rule writes an absolute path as "//<path>"; a trailing "/**" matches everything under it.
|
||||
trees = (f"/{clone}/**", f"/{version_dir}/**")
|
||||
commands = ("log", "diff", "show", "status", "ls-files", "add", "commit -m", "commit -am", "commit -a -m")
|
||||
# git can reach outside the clone through an argument, an option or a hook, and a quote or redirect can
|
||||
# smuggle one past a text pattern. So the run gets no bare git: review-git.py is its one shell command
|
||||
# and confines git to the clone (see its docstring). The clone's own .git and .claude stay unwritable,
|
||||
# lest an edit plant a hook or widen the run's tools; the user's settings would add ~/code to the trees.
|
||||
allowed = [
|
||||
"Agent", "Skill(security-review)", f"Edit(/{clone}/**)", f"Edit(/{report})",
|
||||
*(f"Bash(git {command} *)" for command in commands),
|
||||
f"Bash(python3 {REVIEW_GIT} *)",
|
||||
*(f"{tool}({tree})" for tool in ("Read", "Grep", "Glob") for tree in trees),
|
||||
]
|
||||
# git reaches a path outside the clone only through an argument that names one: absolute, from ~, up
|
||||
# through "..", through a variable, or glued to -O or -X. With -m, git refuses -F and ignores -t
|
||||
# before reading either file. An edit to .git/ could add a hook, which runs any command.
|
||||
outside = (" /", "=/", " ~", "=~", "../", " ..", "$", "O/", "X/")
|
||||
denied = [f"Edit(/{clone}/.git/**)", *(f"Bash(git *{shape}*)" for shape in outside)]
|
||||
# Only the clone's own settings: the user's add every checkout under ~/code to the readable trees. A run
|
||||
# cannot widen the clone's settings either: Claude Code refuses its edit to them, as a probe showed.
|
||||
denied = ["Bash(git *)", f"Edit(/{clone}/.git/**)", f"Edit(/{clone}/.claude/**)"]
|
||||
return ["--setting-sources", "project,local", "--permission-mode", "dontAsk", "--permission-prompts", "none",
|
||||
"--allowedTools", *allowed, "--disallowedTools", *denied]
|
||||
|
||||
@@ -96,7 +94,7 @@ def run(name):
|
||||
base, head = fresh_clone(case, clone)
|
||||
version_dir = BENCH / "versions" / version
|
||||
prompt = (BENCH / "run-prompt.md").read_text().format(
|
||||
version_dir=version_dir, base=base, head=head, case=case, report=report)
|
||||
version_dir=version_dir, base=base, head=head, case=case, report=report, review_git=REVIEW_GIT)
|
||||
log = BENCH / "logs" / f"{name}.{attempt}.jsonl"
|
||||
log.parent.mkdir(exist_ok=True)
|
||||
with log.open("w") as out:
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run one git command for a bench review, confined to the clone: review-git.py <subcommand> <args>...
|
||||
|
||||
The bench's review runs no bare git. It calls this wrapper, its one allowed command (bench.py's
|
||||
permission_flags). The wrapper is git with everything that could reach outside the clone removed:
|
||||
it accepts only the review's subcommands, each with no option that reads a file, writes a file or
|
||||
runs a program; it refuses an argument that names a path outside the clone; and it runs git under
|
||||
the clone with a scrubbed config, so no global or system config, hook, pager or external diff fires.
|
||||
Quoting, redirects and `$` are the shell's, never git's, so they cannot smuggle a path past this.
|
||||
The wrapper lives outside the clone, so the clone is not on Python's import path.
|
||||
"""
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
SUBCOMMANDS = {"log", "diff", "show", "status", "ls-files", "add", "commit"}
|
||||
# Options that read a file, write a file or run a program; "--no-index" lets diff read any two paths.
|
||||
DENY_EXACT = {"--no-index", "-t"}
|
||||
DENY_PREFIX = ("--output", "-O", "--pathspec-from-file", "-F", "--file", "--template",
|
||||
"--exclude-from", "-X", "--git-dir", "--work-tree", "--exec-path", "--ext-diff", "--textconv")
|
||||
|
||||
|
||||
def reject(reason):
|
||||
sys.exit(f"review-git.py: refused: {reason}")
|
||||
|
||||
|
||||
def bad_option(token):
|
||||
name = token.split("=", 1)[0]
|
||||
return name in DENY_EXACT or any(name.startswith(prefix) for prefix in DENY_PREFIX)
|
||||
|
||||
|
||||
def bad_path(clone, arg):
|
||||
# A revision such as HEAD~1 or a..b is not a path; a path that escapes the clone is refused.
|
||||
if arg.startswith("/") or arg.startswith("~"):
|
||||
return True
|
||||
if arg == ".." or any(part == ".." for part in arg.split("/")):
|
||||
return True
|
||||
target = os.path.realpath(os.path.join(clone, arg))
|
||||
return os.path.lexists(target) and target != clone and not target.startswith(clone + os.sep)
|
||||
|
||||
|
||||
def main(argv):
|
||||
if len(argv) < 1 or argv[0] not in SUBCOMMANDS:
|
||||
reject(f"subcommand must be one of {' '.join(sorted(SUBCOMMANDS))}")
|
||||
sub, args = argv[0], argv[1:]
|
||||
clone = os.path.realpath(os.getcwd())
|
||||
operands_only = False # Everything after "--" is a path operand.
|
||||
skip_message = False # commit's -m value is free text, never a path.
|
||||
for token in args:
|
||||
if skip_message:
|
||||
skip_message = False
|
||||
continue
|
||||
if not operands_only and token == "--":
|
||||
operands_only = True
|
||||
elif not operands_only and token.startswith("-"):
|
||||
if bad_option(token):
|
||||
reject(f"option not allowed: {token}")
|
||||
short_m = not token.startswith("--") and token.endswith("m") # -m, -am
|
||||
if sub == "commit" and "=" not in token and (token == "--message" or short_m):
|
||||
skip_message = True # the next token is the commit message, free text, never a path.
|
||||
elif bad_path(clone, token):
|
||||
reject(f"argument names a path outside the clone: {token}")
|
||||
env = {**os.environ, "GIT_CONFIG_GLOBAL": "/dev/null", "GIT_CONFIG_SYSTEM": "/dev/null",
|
||||
"GIT_CONFIG_NOSYSTEM": "1", "GIT_PAGER": "cat", "GIT_TERMINAL_PROMPT": "0"}
|
||||
for name in ("GIT_EXTERNAL_DIFF", "GIT_SSH", "GIT_SSH_COMMAND", "GIT_PROXY_COMMAND", "GIT_CONFIG"):
|
||||
env.pop(name, None)
|
||||
git = ["git", "-c", "core.hooksPath=/dev/null", "-c", "core.fsmonitor=false",
|
||||
"-c", "core.pager=cat", "-c", "protocol.ext.allow=never", sub, *args]
|
||||
return subprocess.run(git, cwd=clone, env=env).returncode
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -15,10 +15,10 @@ Report: write it to `{report}`.
|
||||
|
||||
- Launch every agent in the foreground (`run_in_background: false`): background agents die when this headless turn ends.
|
||||
- No remote and no forge: skip the PR, CI, the external auto-reviewer and the merge; review locally.
|
||||
- The shell runs only git, so run no code or test suite. Where the process asks for a run, such as the stability pass's mutation test, review by reading and list the skipped run under Ambiguities.
|
||||
- Run every git command through `python3 {review_git} <subcommand> <args>`, never bare `git`: that wrapper is the only shell command allowed, and it confines git to this clone. Run no other code or test suite. Where the process asks for a run, such as the stability pass's mutation test, review by reading and list the skipped run under Ambiguities.
|
||||
- Skip the comprehension check; say so in the report.
|
||||
- No human is reachable. Where the process names a no-human branch, take it; where it names none, take your recommendation and list the question in the report.
|
||||
- Commit each round's fixes on `chunk`, messages prefixed `{case} - review: `. Touch nothing outside this repository except the report.
|
||||
- Commit each round's fixes on `chunk` with `python3 {review_git} commit`, messages prefixed `{case} - review: `. Touch nothing outside this repository except the report.
|
||||
|
||||
## Report
|
||||
|
||||
|
||||
Reference in New Issue
Block a user