Keep handoff material in the repo, and move the bench scripts the plan points at into workflow-bench/

This commit is contained in:
2026-10-02 08:43:20 +02:00
parent bef4bb0ef8
commit 765705e3ca
8 changed files with 218 additions and 8 deletions
+3
View File
@@ -32,3 +32,6 @@
!skills/comprehension-panel/*.md
!skills/larv-review/
!skills/larv-review/*.md
!workflow-bench/
!workflow-bench/*.md
!workflow-bench/*.py
+2 -3
View File
@@ -129,9 +129,8 @@ The rules every project follows, compose-not-entangle first: `~/.claude/principl
- Use `gh` on GitHub remotes and `tea` on Gitea remotes (e.g. `gitea.larvit.se`) — same command shape: `tea pr list|create`, `tea issue`, `tea release`, `tea api`. Everything in this section applies to both.
## Handoffs
- One document per project: `~/code/claude-handoffs/<project>_<YYYY-MM-DD>.md`, with its supporting files (scripts, probes, saved sessions) under `~/code/claude-handoffs/<project>/<YYYY-MM-DD>/`, dated the day they were written.
- Each document stands alone — projects move in parallel, so never reference another project's handoff. A fact two of them share (a test site, where credentials live) is stated in each, or kept in a memory file and named there.
- Content: only what the repo cannot hold — probe results, where credentials live. What is done and the next step live in the plan; goals, audience and decisions in `README.md`/`AGENTS.md`. A handoff never restates them: the repo's copy travels with the code and this one does not.
- Write no handoff document. The repo holds everything the next session needs: the plan, the decisions, and every script, sketch or measurement they depend on. A file outside the repo is unreachable from another machine's checkout.
- Four kinds stay out of the repo, in a memory file: a secret's value; where this machine keeps its copy of one; a quirk of the agent's own environment that a human contributor does not share; data the repo may not publish, unlicensed or holding PII.
## Web research
- **A list or threshold a consumer relies on is sourced.** Deny-lists, defaults, limits: follow a published standard or a widely deployed tool, and cite it in the decision recording the choice; where none exists, ask. Never assemble one from names that seem plausible.
+1 -1
View File
@@ -1,6 +1,6 @@
# Claude Code configuration
My personal `~/.claude`, version controlled: `CLAUDE.md` and `AGENTS.md` for instructions, the words under `CLAUDE.md` → "Vocabulary"; `agents/` and `skills/`; `principles/` for what the reviewer agents hold every project to; `docs/` for decisions; `settings.json` for the harness and `.claude/settings.json` for the harness inside this repo; `statusline.sh`; `CHANGELOG.md` and `todo.md`; and the tools in `bin/`.
My personal `~/.claude`, version controlled: `CLAUDE.md` and `AGENTS.md` for instructions, the words under `CLAUDE.md` → "Vocabulary"; `agents/` and `skills/`; `principles/` for what the reviewer agents hold every project to; `docs/` for decisions; `settings.json` for the harness and `.claude/settings.json` for the harness inside this repo; `statusline.sh`; `CHANGELOG.md` and `todo.md`; the tools in `bin/`; and `workflow-bench/`, the scripts and prompts of the 2026-09-30 bench of larv-review.
## Goals
+1 -4
View File
@@ -2,8 +2,6 @@
## 0.1.0
The rebuild's design sketches, the blind judges' verdicts on them and what to take from each: `~/code/claude-handoffs/claude-config_2026-09-28.md`.
### Under the comprehension floor
The panels on `rebuild` at 8585257 and on PR #182 named these under Self-sufficiency and Locality, the two dimensions under the floor. Scores: `docs/decisions.md` → "comprehension floor".
@@ -44,13 +42,12 @@ The panels on `rebuild` at 8585257 and on PR #182 named these under Self-suffici
- **Launch a principles auditor only for a principles file the project shows applies, and rerun only the reviewer whose finding a fix answers.** On this repo the frontend, REST and logging auditors each read the whole project to report "does not apply", and one 🟢 fix reruns ten whole-project reads. Goal 4.4.
- **Give `settings.json`'s model one writer.** Claude Code rewrites `"model"` in this tracked file whenever a model is picked, so one `/model` dirties the checkout and overrides the synced choice. Goal 4.3.
- **Rewrite `statusline.sh` in Python, and test it.** It needs `jq` and GNU `date` on every machine, `eval`s jq output, has three near-identical usage blocks and no test. Goal 1.
- **Move what the plan and the decision log point at under `~/code/claude-handoffs/` into the repo.** The design sketches, the judges' verdicts and the bench scripts are unreachable from another machine's checkout. Goal 1.
- **Say in `systems-architect.md` that a REST API's `/v1/` prefix is not versioning machinery to defer.** Its "defer the versioning story" can be read against `principles/rest-apis.md`, which requires the prefix from the start. Goal 4.3.
- **Judge a cycle on more than the transcript's last 40,000 characters.** `judge_prompt` cuts the transcript there. The session spans every cycle of the chunk, so a decision made early in a long run is never judged, and the human is not interrupted for it. Larger chunks make runs longer. Goal 4.2.
- **Record each finished chunk's cost and review numbers on one stats line.** Goal 4.4.
- Have `loop-claude` write the line to one file at the root of the transcript archive. The line names the project, the commit the chunk started from and the archived transcript's hash. It holds the worker's tokens from the transcript, the judge's from its JSON output, and the chunk's cycles and interrupts, which the loop counts.
- End larv-review with a machine-readable summary, and have `loop-claude` copy it from the transcript onto the line: agents launched and rounds per pass; findings per pass by severity, inside the chunk's diff and outside it; and the comprehension scores when the panel ran.
- **Write a `workflow-review` skill that reads the stats and files each change it proposes to the workflow as a plan item, and have `AGENTS.md` run it on the chunk shipping a release's last item.** The workflow is `CLAUDE.md`, `principles/`, the skills, the agents and `loop-claude`. The skill's signals: findings outside the chunk's diff and comprehension scores per release, both from the stats; fixes after merge, from git; and the human's corrections, from the archived transcripts. Goal 4.4.
- **Bench about one chunk in ten against an open PR that changes the workflow, and have a blind claude rule which run better serves this repo's goal 4.** Redo the chunk under the PR, from the commit it started from. The ruling claude is not told which run is which. `loop-claude` picks the chunk at random, small diffs and big ones alike; with no such PR open, no bench runs. The same outcome with less code and prose, in the chunk or in the workflow, also counts as better. The PR merges only once three benches rule it better. A bench that rules it worse closes the PR and records, in `docs/workflow-learnings.md`, what the PR changed and why it lost. The ruling and both runs' numbers go on the chunk's stats line. Start from the 2026-09-30 bench's scripts: `~/code/claude-handoffs/claude-config/2026-09-30/bench/`. Needs the stats line. Goals 4.1 and 4.4.
- **Bench about one chunk in ten against an open PR that changes the workflow, and have a blind claude rule which run better serves this repo's goal 4.** Redo the chunk under the PR, from the commit it started from. The ruling claude is not told which run is which. `loop-claude` picks the chunk at random, small diffs and big ones alike; with no such PR open, no bench runs. The same outcome with less code and prose, in the chunk or in the workflow, also counts as better. The PR merges only once three benches rule it better. A bench that rules it worse closes the PR and records, in `docs/workflow-learnings.md`, what the PR changed and why it lost. The ruling and both runs' numbers go on the chunk's stats line. Start from `workflow-bench/`, written for that bench's four fixed cases. Needs the stats line. Goals 4.1 and 4.4.
- **Bring the project to the comprehension floor on the four-seat panel.** Every dimension and the overall must reach 7.0, and none may fall below its best run on `rebuild`. A dimension under the floor gets its places filed under "Under the comprehension floor".
- **Wait out a usage limit and retry however long it lasts, without counting it as a failed run.** Today a worker or judge that hits a usage limit fails three times a minute apart, and the loop exits 1, so an unattended loop dies at every limit. The transcript line Claude Code writes for it, marked `"error":"rate_limit"`, holds the reset time: `You've hit your session limit · resets 7:40pm (Europe/Stockholm)`. Sleep until 2 minutes past it. The judge runs without a transcript, so find what its JSON reply carries on a limit. Where no reset time can be read, retry every 10 minutes: a retry before the reset fails (https://platform.claude.com/docs/en/api/rate-limits), so polling sooner only spends requests. Print when the next try is due, and update `--help`'s exit 1. Goal 4.
+83
View File
@@ -0,0 +1,83 @@
#!/usr/bin/env python3
"""Run larv-review bench runs: bench.py <case>-<version>-<n> ..."""
import concurrent.futures
import json
import pathlib
import shutil
import subprocess
import sys
import time
BENCH = pathlib.Path(__file__).resolve().parent
SOURCES = BENCH.parent.parent / "2026-09-29" / "bench"
CASES = {
"cjk": ("151bf38^", "6c49912"),
"35b": ("692433b", "1d369ec"),
"brlan": ("59ce55b^", "92e466a"),
"mypy": ("89a66d3", "b32d131"),
}
ATTEMPTS = 4
def git(repo, *args):
return subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True).stdout.strip()
def fresh_clone(case, dest):
if dest.exists():
shutil.rmtree(dest)
source = SOURCES / f"{case}-main"
base_ref, head_ref = CASES[case]
base = git(source, "rev-parse", base_ref)
head = git(source, "rev-parse", head_ref)
subprocess.run(["git", "clone", "-q", "--no-local", str(source), str(dest)], check=True)
git(dest, "checkout", "-q", "-B", "chunk", head)
for branch in git(dest, "for-each-ref", "--format=%(refname:short)", "refs/heads").split():
if branch != "chunk":
git(dest, "branch", "-D", branch)
for tag in git(dest, "tag").split():
git(dest, "tag", "-d", tag)
git(dest, "remote", "remove", "origin")
git(dest, "reflog", "expire", "--expire=now", "--all")
git(dest, "gc", "-q", "--prune=now")
return base, head
def rate_limit_reset(log):
resets = [0]
for line in log.read_text().splitlines():
if '"status":"rejected"' in line:
resets.append(json.loads(line)["rate_limit_info"]["resetsAt"])
return max(resets)
def run(name):
case, version, _ = name.split("-")
clone = BENCH / name
report = BENCH / f"{name}.md"
for attempt in range(1, ATTEMPTS + 1):
if report.exists():
return f"{name}: done"
base, head = fresh_clone(case, clone)
prompt = (BENCH / "run-prompt.md").read_text().format(
version_dir=BENCH / "versions" / version, base=base, head=head, case=case, report=report)
log = BENCH / "logs" / f"{name}.{attempt}.jsonl"
log.parent.mkdir(exist_ok=True)
with log.open("w") as out:
subprocess.run(
["claude", "-p", prompt, "--dangerously-skip-permissions", "--model", "claude-opus-5-5", "--output-format", "stream-json", "--verbose"],
cwd=clone, stdout=out, stderr=subprocess.STDOUT)
if report.exists():
return f"{name}: done on attempt {attempt}"
time.sleep(max(1200, rate_limit_reset(log) + 120 - time.time()))
return f"{name}: FAILED after {ATTEMPTS} attempts"
def main():
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
for result in pool.map(run, sys.argv[1:]):
print(result, flush=True)
if __name__ == "__main__":
main()
+23
View File
@@ -0,0 +1,23 @@
You judge a bench of two code-review processes, A and B, on one real chunk of work. Each process reviewed the same chunk three times, independently: runs A1–A3 and B1–B3. Rule whether B reviews at least as well as A, and whether A reviews at least as well as B.
Everything is under `{dir}`; read nothing outside it:
- `truth-issues.md`: what the chunk's real review found and fixed, one row per issue.
- `truth.diff`: the real review's fixes.
- `<run>/report.md`: the run's own account of its passes, findings and outcomes. Its claims are not evidence.
- `<run>/fixes.diff`: what the run actually changed. This is the evidence.
Judge each run on:
1. **Truth coverage**: for each truth issue, caught and fixed ✔, partly ◐, or missed ✘, checked against `fixes.diff`. A filed or declined issue with a sound reason counts ◐.
2. **Other real defects**: a finding the truth lacks that is a real defect in the chunk, fixed or filed.
3. **Damage**: a change that makes the chunk worse — a regression, a false claim written, a needless churn of lines outside the chunk.
4. **Cost**: agents launched and rounds, from the report.
Then, per process, weigh the three runs together: what it catches reliably (two or more runs), what it catches by luck (one run), and its spread. A difference one run makes is variance, not the process.
Write your ruling to `{ruling}`:
1. A table per run: each truth issue's mark, other real defects, damage, cost.
2. Per process: reliable catches, lucky catches, damage, cost range.
3. **Ruling**: one line each — `B at least as good as A: yes|no` and `A at least as good as B: yes|no` — each followed by the decisive evidence, citing runs and diff lines. "No" needs a difference that holds across runs, not one run's miss.
+74
View File
@@ -0,0 +1,74 @@
#!/usr/bin/env python3
"""Blind-judge each case's six runs: judge.py <case> ..."""
import concurrent.futures
import json
import pathlib
import random
import re
import shutil
import subprocess
import sys
import bench
BENCH = bench.BENCH
TRUTH_END = {"cjk": "160c2a7", "35b": "a4b6b48", "brlan": "8f6e22d", "mypy": "628ee30"}
PREFIX = {"cjk": "C", "35b": "B", "brlan": "R", "mypy": "M"}
def runs(case, version):
"""The version's first three runs as (report, clone), main's 2026-09-29 run last."""
found = []
for n in range(1, 4):
report = BENCH / f"{case}-{version}-{n}.md"
if report.exists():
found.append((report, BENCH / f"{case}-{version}-{n}"))
if version == "main":
found.append((bench.SOURCES / f"{case}-main.md", bench.SOURCES / f"{case}-main"))
return found[:3]
def blind(text):
return re.sub(r"\S*versions/(main|rebuild)\S*|\b(main|rebuild)\b(?=[`\s]*(process|version))", "VERSION_DIR", text)
def prepare(case):
folder = BENCH / "judge" / case
if folder.exists():
shutil.rmtree(folder)
folder.mkdir(parents=True)
source = bench.SOURCES / f"{case}-main"
head = bench.git(source, "rev-parse", bench.CASES[case][1])
(folder / "truth.diff").write_text(bench.git(source, "diff", head, TRUTH_END[case]) + "\n")
rows = [line.split("|")[1:3] for line in (bench.SOURCES / "comparison.md").read_text().splitlines()
if re.match(rf"\| {PREFIX[case]}\d+ \|", line)]
(folder / "truth-issues.md").write_text("".join(f"-{i.strip()}: {t.strip()}\n" for i, t in rows))
groups = ["main", "rebuild"]
random.shuffle(groups)
mapping = {}
for label, version in zip("AB", groups):
for n, (report, clone) in enumerate(runs(case, version), 1):
run = folder / f"{label}{n}"
run.mkdir()
(run / "report.md").write_text(blind(report.read_text()))
(run / "fixes.diff").write_text(bench.git(clone, "diff", head, "chunk") + "\n")
mapping[f"{label}{n}"] = str(report)
(BENCH / "judge" / f"{case}.mapping.json").write_text(json.dumps(mapping, indent="\t") + "\n")
return folder
def judge(case):
folder = prepare(case)
ruling = BENCH / "judge" / f"{case}.ruling.md"
prompt = (BENCH / "judge-prompt.md").read_text().format(dir=folder, ruling=ruling)
with (BENCH / "logs" / f"judge-{case}.jsonl").open("w") as out:
subprocess.run(
["claude", "-p", prompt, "--model", "claude-opus-5-5", "--allowedTools", "Read,Grep,Glob,Write", "--output-format", "stream-json", "--verbose"],
cwd=folder, stdout=out, stderr=subprocess.STDOUT)
return f"{case}: {'ruled' if ruling.exists() else 'FAILED'}"
if __name__ == "__main__":
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
for result in pool.map(judge, sys.argv[1:]):
print(result, flush=True)
+31
View File
@@ -0,0 +1,31 @@
You are one run of a bench comparing two versions of the larv-review process. Review the chunk in this repository (your working directory) exactly as the process below says, then write a report.
VERSION_DIR = {version_dir}
Chunk: commits `{base}..chunk` on branch `chunk` (head at start: `{head}`). Case `{case}`.
Report: write it to `{report}`.
## The process
- `VERSION_DIR/skills/larv-review/SKILL.md` is larv-review. Do not invoke the installed `larv-review` skill.
- Wherever a text cites `~/.claude/<path>`, read `VERSION_DIR/<path>` instead. `VERSION_DIR/CLAUDE.md` replaces `~/.claude/CLAUDE.md`, which you already have loaded: where the two differ, VERSION_DIR's wins.
- Launch every reviewer named `<name>` under `~/.claude/agents/` as a `general-purpose` agent whose prompt opens with the body of `VERSION_DIR/agents/<name>.md` (frontmatter dropped), then the task. A reviewer the process names that has no file there gets the process's own description of it as its prompt.
- `/security-review`: use the Skill tool's `security-review` if it is available; otherwise emulate it with a `general-purpose` agent, told to review the branch diff for exploitable vulnerabilities and report only those at ≥0.8 confidence.
## Bench rules
- Launch every agent in the foreground (`run_in_background: false`): background agents die when this headless turn ends.
- No remote and no forge: skip the PR, CI, the external auto-reviewer and the merge; review locally.
- Do not run the repo's test suites. A targeted probe is fine, in the repo's own container tooling.
- Skip the comprehension floor's scoring run; say so in the report.
- No human is reachable. Where the process names a no-human branch, take it; where it names none, take your recommendation and list the question in the report.
- Commit each round's fixes on `chunk`, messages prefixed `{case} - review: `. Touch nothing outside this repository except the report.
## Report
Sections, in order:
1. **Passes**: a table of each pass, run or skipped, and why.
2. **Findings**: per pass and round, each finding's severity, location, one-line summary and outcome (fixed in `<sha>`, answered, declined, filed, disproved).
3. **Questions for the human**: each ask you could not put, with your recommendation.
4. **Totals**: agents launched per pass, rounds per pass, commits.
5. **Ambiguities**: every place the process text was unclear, contradicted itself, or could not be followed, and how you read it.