35 Commits

Author SHA1 Message Date
Lilleman a75826d35a Load no MCP server in the bench's headless claude calls that are denied tools
CI / tests (pull_request) Successful in 3m45s
2026-11-20 18:02:12 +01:00
lilleman 762614353c Answer principles round 2: 0 🔴, 2 🟡 (0 misread); workflow-bench hands its effects in as arguments, and no Python file of the repo patches a global
CI / tests (pull_request) Successful in 3m46s
2026-11-19 12:55:35 +01:00
lilleman 4875e07f98 Answer stability round 1: 0 🔴, 2 🟡 (0 misread); refuse a failed compaction and count the starts' turns as claude does 2026-11-17 03:58:52 +01:00
lilleman b4490b4a8c Refuse a compact arm whose compaction failed, and state the bench as it ran 2026-11-17 03:56:25 +01:00
lilleman 24800c156b Assert that the start's saved diff holds an untracked file 2026-11-17 03:51:01 +01:00
lilleman 7da82b4748 Hold the handoff bench's prompts, saved change and steps to their tests 2026-11-17 03:48:46 +01:00
lilleman 1bc2ce89e7 Net a resumed arm's cost of the start it resumes 2026-11-17 03:44:53 +01:00
lilleman e42eb5b288 Bench a model step's handoff: resume, compact or a fresh session with a brief 2026-11-17 02:57:22 +01:00
lilleman 14e6756a53 Answer prose round 1: 0 🔴, 6 🟡 (0 misread); rewrite each <verb> nothing to the negation rule, tighten the new docstrings and comments 2026-11-12 20:33:32 +01:00
lilleman 2556fdace9 Answer prose round 1: 0 🔴, 7 🟡 (0 misread); the decision's scope, the changelog bullet and the prompts read once
CI / tests (pull_request) Successful in 2m30s
2026-11-12 05:30:15 +01:00
lilleman 6817b074bd Answer stability round 1: 0 🔴, 6 🟡 (0 misread); the bench tests only the packages the clone holds, refuses a run that ended in an error, and pins a worker's committed change 2026-11-12 05:30:15 +01:00
lilleman e1e99645be Pin the advisor bench's argv, kwargs, diff exclusions and failures in its tests 2026-11-12 05:20:39 +01:00
lilleman 45fd66d3ab Test the advisor bench's run, grade, main and rows_of against a throwaway repo, and kill the mutants of its pure functions 2026-11-12 05:18:44 +01:00
lilleman 66b9651d68 Bench the advisor tool on six archived L 1 and L 2 chunks: Sonnet alone passes as often as Opus at medium for less, so workers stay without it
CI / tests (pull_request) Successful in 2m18s
2026-11-12 04:27:48 +01:00
lilleman a52ebb94bf Run mutation tests on a chunk's changed lines before review with bin/mutate-changes, and hand stability-reviewer the survivors
CI / tests (pull_request) Successful in 2m14s
2026-11-11 18:58:31 +01:00
lilleman c74ec6fc0a Answer stability round 1: 0 🔴, 4 🟡 (0 misread); test main's continuing past a failed run, name the denied commits in decided's tests, check the replay puts premises before records 2026-11-08 22:58:52 +01:00
lilleman 268200de23 Pass the judge a Brief, decide read_seats from its inputs and return a failed bench run as a line 2026-11-08 22:51:27 +01:00
lilleman 53e87d1051 Answer security round 5: 0 🔴, 1 🟡; load no settings source at all, and answer stability round 5's nit with a test that no config or hook program runs 2026-10-16 22:54:18 +02:00
lilleman 57cd5f76a4 Answer security round 4: 1 🔴, 1 🟡; refuse the format options that start gpg, and load no settings from the case's clone 2026-10-16 22:42:11 +02:00
lilleman b30c91aafc Answer stability round 4: 2 🔴, 1 🟡; the wrapper allows only listed options, takes the clone from the environment, and skips only a message its flag takes whole 2026-10-16 22:30:24 +02:00
lilleman cf4072f69f Answer security round 3: 0 🔴, 0 🟡; the answer holds 2026-10-16 20:53:27 +02:00
lilleman 4625b805eb Record in bench.py why the clone's settings need no deny 2026-10-16 20:32:42 +02:00
lilleman 95aa2b3477 Answer security round 2: 1 🔴, 0 🟡; deny a git argument that names a path outside the clone, in place of listing options 2026-10-16 20:32:36 +02:00
lilleman 58252f8948 Answer security round 1: 1 🔴, 1 🟡; allow only the git commands a review needs, deny the options that name a path, and load no user settings 2026-10-16 20:24:02 +02:00
lilleman 14c8e7f012 Grant the bench's runs only the tools a review needs, in place of skipping every permission 2026-10-16 20:17:42 +02:00
lilleman dbe7bd3abd Answer stability, product-owner, prose and architecture round 3: count only a commit's last history entry toward the base, merge the default branch in before each ruling, judge the scoring run's trigger on the plan as the chunk leaves it, count a vote naming no place as same, say in the changelog what a worse ruling and the floor do to the owner, and correct item 29 2026-10-08 20:50:19 +02:00
lilleman 26826685e2 Answer architecture round 1: rule against the last measured commit, launch the third and fourth seats together, name the floor's chunk by the plan, skip the whole check in the bench, and correct the history's writer and the paired brief 2026-10-08 20:36:25 +02:00
lilleman 7b38aed614 Answer prose round 1: precedent names what a pass skips on, the bench and floor-decision prose reads on one pass 2026-10-03 18:21:33 +02:00
lilleman d3613f45d4 Answer architecture round 1: Failure and the file modules get truthful homes, the interrupt question is a pure function, the judge grounds on what it saw 2026-10-03 18:06:56 +02:00
lilleman 5f7d34b87d Give record, verdict and judge one meaning each 2026-10-03 17:53:25 +02:00
lilleman 31ec74498c Answer round two: an answer quoted without elisions, unambiguous referents in --help, the changelog and the security line 2026-10-03 15:56:54 +02:00
lilleman 61b7b44243 Answer round one: the bench runs the security pass in a subagent, an unavailable skill is reported, the answer's strictness explained where it is checked, prose nits 2026-10-03 15:54:42 +02:00
lilleman 43aa6ec4df Answer the second review round 2026-10-02 08:47:00 +02:00
lilleman c729a130a6 Answer the first review round 2026-10-02 08:45:25 +02:00
lilleman 765705e3ca Keep handoff material in the repo, and move the bench scripts the plan points at into workflow-bench/ 2026-10-02 08:43:20 +02:00