Answer prose round 1: 0 🔴, 8 🟡 (0 misread); the help texts say where the runs go, the decision gets its own dated paragraph, and the comments reread

This commit is contained in:
2026-11-17 02:33:40 +01:00
parent 48e69063b8
commit b8a1a13ad0
6 changed files with 24 additions and 10 deletions
+2 -2
View File
@@ -172,7 +172,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `chunk-preflight` finds the mechanical facts a chunk starts from, and with `--worktree` makes its worktree; `loop-claude` runs it with `--worktree` before it routes each chunk, and `chunk-preflight --help` lists the facts.
- `command-patterns [<version>] [--calls <file>]` prints a table of the shell command patterns a release's archived chunks ran, such as `tea pr create`, for an audit of the release, and with `--calls <file>` saves each call as a JSON line; `command-patterns --help` states the columns and the line's keys.
- `deploy-release [--cut] [<checkout>]` moves `~/.claude`, or the checkout named, to the newest release, as Update says; with `--cut`, run in the development clone, it first cuts the next release, as Release says.
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints what each rung a review climbs to misses, and the lowest rung below them that, like each candidate above it, returns them; `judge-replay --help` states the replay. `judge-replay` and `review-replay` keep each run's prompt and stream, and the report they print, in a `replays/` folder of loop-claude's archive.
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints what each rung a review climbs to misses, and the lowest rung below them that, like each candidate above it, returns them; `judge-replay --help` states the replay.
- `larv-review <command>` holds larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's round count, the scoring run's trigger and setup; the comprehension history's format, its latest places and its count of how often seats agree; and each reviewer's prompt header. `larv-review --help` states them. `settings.json` → `hooks` runs `larv-review reviewer-context` as each subagent starts. A reviewer under `agents/` whose frontmatter has `omitClaudeMd: true` starts without `CLAUDE.md`, and the command hands it that file's "Shared goal", "Prose" and "Vocabulary" sections.
- `loop-claude <task>` runs a project's next chunks, each in its own tmux window, cycle after cycle; `loop-claude --help` states its contract.
- `metric-bench [history]` benches cheap static metrics of a project's Python against the places its comprehension history names as hardest; `metric-bench --help` states the metrics and the scores.
@@ -219,7 +219,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `rest_client/` sends a JSON request to a REST API and reads its credentials from an env file, for `notifier`, `atlassian` and `record_store`.
- `review_rules/` holds the larv-review rules that several tools apply: its passes, its review folder, its round commit titles and the comprehension history's format.
- `routing_figures/` measures loop-claude's archive, and each of its projects' default branches, for loop-claude's routing calls.
- `rung_replay/` holds what `judge_replay` and `review_replay` share: the parallel replay runner, the argument reader, the candidate rungs and the exits, and `judge_replay`'s rung search. It reads `loop_claude`'s retry count and models file, and finds that file through `loop_state`'s paths.
- `rung_replay/` holds what `judge_replay` and `review_replay` share: the parallel replay runner, the argument reader, the candidate rungs and the exits, and `judge_replay`'s rung search. It reads `loop_claude`'s retry count and models file, and finds that file through `loop_state`'s paths. Its `archive.py` keeps each replay's runs and its report in a `replays/` folder of loop-claude's archive.
- `test_env.py` serves the tests.
- `text_files/` reads a Markdown file's `##` sections, quotes a passage and joins names into a list, stamps times in UTC, and replaces a file in one step, private where it is new.
- `transcript_archive/` stores loop-claude's transcripts. Its `project_key.py` names each project's archive folder, loop-claude session and models-file entry.
+1 -1
View File
@@ -17,7 +17,7 @@ class Limited:
said: str
# --no-session-persistence keeps the runs out of the session picker, and Claude Code then writes no transcript; stream-json prints one, so a caller may keep it, and claude_reply reads the reply from its last line. -p needs --verbose with stream-json. No permission prompt, as no one answers it.
# No persistence, so these runs stay out of the session picker; Claude Code then writes no transcript, so stream-json, which prints one, lets a caller keep it. claude_reply reads the reply from the stream's last line. -p needs --verbose with stream-json. No permission prompt, as no one answers it.
HEADLESS = ["--no-session-persistence", "--output-format", "stream-json", "--verbose", "--permission-mode", "bypassPermissions"]
+4 -1
View File
@@ -27,7 +27,10 @@ JUDGE_REPLAYED in bin/loop_claude/routing.py, which routes that model's judge. j
keeps the judge to its own model's rungs; a rung above them that misses no interrupt is the
evidence to let the judge climb onto it.
Each run is kept with the report it ended in: `replays/<tool>-<arguments>-<UTC stamp>/` in loop-claude's archive holds a `.jsonl` file per run, its prompt then claude's stream, and `result.md`, the printed report.
It keeps each run and the report it prints in loop-claude's archive, in
replays/judge-replay-<model>-<UTC stamp>/ under ~/.claude/loop-claude-transcripts or $LOOP_CLAUDE_TRANSCRIPTS.
A run's .jsonl file starts with a line holding claude's arguments, working directory, prompt, exit code and
stderr, then holds claude's stream-json output; result.md holds the report.
Like the loop, it tries a judge three times running before it counts the reply as failed. A
verdict whose project, transcript or archived judge run cannot be read replays as failed too. Exit 1, saying why, on
+4 -1
View File
@@ -45,7 +45,10 @@ replay, and runs no SubagentStart hook there. So the replay without CLAUDE.md tu
instruction files off in its settings and opens the prompt with what the hook hands a reviewer, and
closes it with the header's line on the project principles.
Each run, the reviewer's tool calls included, is kept with the report it ended in: `replays/<tool>-<arguments>-<UTC stamp>/` in loop-claude's archive holds a `.jsonl` file per run, its prompt then claude's stream, and `result.md`, the printed report.
It keeps each run, a reviewer's tool calls included, and the report it prints in loop-claude's archive, in
replays/review-replay-<pass>-<model>[-omit-claude-md]-<UTC stamp>/ under ~/.claude/loop-claude-transcripts
or $LOOP_CLAUDE_TRANSCRIPTS. A run's .jsonl file starts with a line holding claude's arguments, working
directory, prompt, exit code and stderr, then holds claude's stream-json output; result.md holds the report.
Like the loop, it tries claude three times running before it counts a reply as failed. A failed
review or match leaves its round out, except on a rung below the project's, where it finds
+3 -3
View File
@@ -1,4 +1,4 @@
"""Keeps what judge-replay and review-replay ran beside what they printed, so a replay that decided a rung can be read again, re-priced or re-scored once the decision is questioned: docs/decisions.md → `loop-claude: the archive keeps the raw transcript and each judge run, …`."""
"""Keeps each run of judge-replay and review-replay, and the report each prints, in loop-claude's archive: docs/decisions.md → `loop-claude: the archive keeps the raw transcript and each judge run, …`."""
import dataclasses
import hashlib
import json
@@ -10,7 +10,7 @@ import time
from loop_effects.effects import Effects
# The run's file starts with this line: what claude read and where, none of which its stream carries. Its type is none claude writes.
# The type of the first line of a run's file, which holds what the stream lacks: claude's arguments, working directory, prompt, exit code and stderr. No event claude writes has this type.
HEADER = "replay_run"
RESULT = "result.md"
@@ -22,7 +22,7 @@ def replay_folder(root: pathlib.Path, tool: str, argv: list[str], now: float) ->
def archiving(effects: Effects, folder: pathlib.Path) -> Effects:
"""effects whose claude also writes each run into folder: a line with the arguments, the working directory and the prompt, then the stream claude printed. A run is kept whether it failed or not, and once where it repeats a run; a folder that cannot be written is said on stderr and costs the replay nothing."""
"""effects whose claude also writes each run into folder: a header line, then the stream claude printed. A run is kept whether it failed or not, and a run that repeats another word for word is kept once. A folder that can't be written is reported on stderr, and the replay goes on."""
claude = effects.claude
def run(args: list[str], cwd: str, prompt: str) -> subprocess.CompletedProcess[str]:
+10 -2
View File
@@ -182,9 +182,17 @@ Valid while `~/.claude` is a checkout of this repo.
Valid while claude takes `--session-id` and resumes by it, and the human runs loop-claude where tmux is installed.
## loop-claude: the archive keeps the raw transcript and each judge run, a failed archive stops the loop, and judge-replay and review-replay keep each run and their report under its replays/ folder
## loop-claude: the archive keeps the raw transcript and each judge run, a failed archive stops the loop, and a replay keeps its runs and report in a replays/ folder of the archive
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. The judge's runs are archived too, as `judge/<sha256>.json` in the chunk's folder: the prompt it read and its json reply. Together they are its whole transcript, since it runs with no tools and one turn. A run is kept whole and deduplicated by hash, and the outcome of the verdict, interrupt or rejected reply it produced names it on a `Judge run:` line. Keeping it whole costs little: a prompt is at most `JUDGE_PROMPT_CHARS`, 400,000 characters. Measured 2026-11-17, the archive's 911 judge lines come to at most 0.36 GB of prompts, against 3.5 GB in its 1,211 transcript copies, the largest 23.4 MB. The judge is still priced from its `Judge cost:` line, which is claude's own total for the run, and the Spent block counts the live reply's tokens; the archived reply keeps its `usage` for a later reader, and nothing reads it back. `judge-replay` replays a verdict on its archived prompt where there is one. The replays keep their own runs, since each decides which rung a review pass or the judge may run on and so must stay checkable and re-priceable once the decision is questioned (2026-11-17, item 304): `replays/<tool>-<arguments>-<UTC stamp>/` in the archive's root holds one `<sha256>.jsonl` per run, whether it failed or not, and `result.md`, the report the tool printed. A run's file is a header line with claude's arguments, working directory and prompt, then claude's stream-json output as printed, whose last line is the reply the replay read; a reviewer's tool calls and a judge's reply are in it. `claude_reply` takes stream-json in place of json for every caller, so the judge's runs are streams too, and its archive keeps the `JudgeRun` it did. A run that repeats another word for word is kept once. A folder that cannot be written is said on stderr and doesn't stop the replay. Valid while transcripts, and the judge's prompts, stay small enough to copy whole.
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. The judge's runs are archived too, as `judge/<sha256>.json` in the chunk's folder: the prompt it read and its json reply. Together they are its whole transcript, since it runs with no tools and one turn. A run is kept whole and deduplicated by hash, and the outcome of the verdict, interrupt or rejected reply it produced names it on a `Judge run:` line. Keeping it whole costs little: a prompt is at most `JUDGE_PROMPT_CHARS`, 400,000 characters. Measured 2026-11-17, the archive's 911 judge lines come to at most 0.36 GB of prompts, against 3.5 GB in its 1,211 transcript copies, the largest 23.4 MB. The judge is still priced from its `Judge cost:` line, which is claude's own total for the run, and the Spent block counts the live reply's tokens; the archived reply keeps its `usage` for a later reader, and nothing reads it back. `judge-replay` replays a verdict on its archived prompt where there is one. Valid while transcripts, and the judge's prompts, stay small enough to copy whole.
2026-11-17, lilleman, todo item 304. Serves goals 1.4 and 3. `judge-replay` and `review-replay` each decide which rung a review pass or the judge may run on, so their runs stay checkable and re-priceable once a decision is questioned. They keep their runs in `replays/<tool>-<arguments>-<UTC stamp>/` in the archive's root:
- One `<sha256>.jsonl` file per run, whether it failed or not. It starts with a header line holding claude's arguments, working directory, prompt, exit code and stderr, and goes on with claude's stream-json output as printed, whose last line is the reply the replay read. A reviewer's tool calls are in it.
- `result.md`, the report the tool printed.
- A run that repeats another word for word is kept once. A folder that can't be written is reported on stderr and doesn't stop the replay.
`claude_reply` reads stream-json for every caller, so the judge's runs are streams too; the archive's `JudgeRun` stays as it was, since a judge has no tools and one turn. Valid while transcripts stay small enough to copy whole.
## loop-claude: each verdict records the claude-config release and commit that ran it, the raw tokens each thread spent by model and kind since the previous outcome that recorded a spend, and the human's typed answer the judge quotes as reversing a choice the worker took on its own recommendation