Archive each review-replay and judge-replay run transcript beside its result #571
@@ -2,6 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- review-replay and judge-replay keep each run, with its prompt, claude's exit code and stream, and the report they print, in a `replays/<tool>-<arguments>-<UTC stamp>/` folder of loop-claude's archive, so a replay that decided a rung can be checked and re-priced later; claude's json replies are now stream-json, whose last line is the reply.
|
||||
- loop-claude's archive keeps each judge run's prompt and json reply in a judge/ folder beside its chunk's transcripts, and names the run on a `Judge run:` line of the outcome it produced; judge-replay replays a verdict on its archived prompt where the archive holds one.
|
||||
- A plan kept as a Jira epic takes no changelog bullet: `todo-item finish` there needs neither `--changelog` nor `--no-changelog`, and refuses `--changelog`, since the issues' history is the changelog.
|
||||
|
||||
|
||||
@@ -219,7 +219,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `rest_client/` sends a JSON request to a REST API and reads its credentials from an env file, for `notifier`, `atlassian` and `record_store`.
|
||||
- `review_rules/` holds the larv-review rules that several tools apply: its passes, its review folder, its round commit titles and the comprehension history's format.
|
||||
- `routing_figures/` measures loop-claude's archive, and each of its projects' default branches, for loop-claude's routing calls.
|
||||
- `rung_replay/` holds what `judge_replay` and `review_replay` share: the parallel replay runner, the argument reader, the candidate rungs and the exits, and `judge_replay`'s rung search. It reads `loop_claude`'s retry count and models file, and finds that file through `loop_state`'s paths.
|
||||
- `rung_replay/` holds what `judge_replay` and `review_replay` share: the parallel replay runner, the argument reader, the candidate rungs and the exits, and `judge_replay`'s rung search. It reads `loop_claude`'s retry count and models file, and finds that file through `loop_state`'s paths. Its `archive.py` keeps each replay's runs and its report in a `replays/` folder of loop-claude's archive.
|
||||
- `test_env.py` serves the tests.
|
||||
- `text_files/` reads a Markdown file's `##` sections, quotes a passage and joins names into a list, stamps times in UTC, and replaces a file in one step, private where it is new.
|
||||
- `transcript_archive/` stores loop-claude's transcripts. Its `project_key.py` names each project's archive folder, loop-claude session and models-file entry.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""One `claude -p --output-format json` run, as loop-claude's judge and the replays start it."""
|
||||
"""One `claude -p --output-format stream-json` run, as loop-claude's judge and the replays start it."""
|
||||
import dataclasses
|
||||
import subprocess
|
||||
import typing
|
||||
@@ -17,22 +17,23 @@ class Limited:
|
||||
said: str
|
||||
|
||||
|
||||
# The json reply claude_reply parses; no persistence, so its runs stay out of the session picker; no permission prompt, as no one answers it.
|
||||
HEADLESS = ["--no-session-persistence", "--output-format", "json", "--permission-mode", "bypassPermissions"]
|
||||
# No persistence, so these runs stay out of the session picker; Claude Code then writes no transcript, so stream-json, which prints one, lets a caller keep it. claude_reply reads the reply from the stream's last line. -p needs --verbose with stream-json. No permission prompt, as no one answers it.
|
||||
HEADLESS = ["--no-session-persistence", "--output-format", "stream-json", "--verbose", "--permission-mode", "bypassPermissions"]
|
||||
|
||||
|
||||
def claude_reply(claude: Claude, args: list[str], cwd: str, prompt: str, who: str) -> dict | Failure | Limited:
|
||||
"""claude's json reply to `claude -p <args>` and HEADLESS, or why there is none; who, such as "the judge", names the run in the message for a non-zero exit or a reply that is no json."""
|
||||
"""claude's reply to `claude -p <args>` and HEADLESS, the result event that ends its stream, or why there is none; who, such as "the judge", names the run in the message for a non-zero exit or a last line that isn't a json object."""
|
||||
try:
|
||||
proc = claude(["-p", *args, *HEADLESS], cwd, prompt)
|
||||
except OSError as failure:
|
||||
return Failure(f"could not start claude: {failure.strerror}")
|
||||
reply = json_object(proc.stdout)
|
||||
# The json output names the reset only in prose, such as "resets 7:40pm (Europe/Stockholm)", so its limit carries no reset time.
|
||||
last = proc.stdout.strip().rpartition("\n")[2]
|
||||
reply = json_object(last)
|
||||
# The result event names the reset only in prose, such as "resets 7:40pm (Europe/Stockholm)", so its limit carries no reset time; the stream's rate_limit_event holds it as a time, which nothing reads here yet.
|
||||
if reply is not None and hit_limit(reply):
|
||||
return Limited(None, limit_said(reply))
|
||||
if proc.returncode != 0:
|
||||
return Failure(proc.stderr.strip() or f"{who} exited {proc.returncode}")
|
||||
if reply is None:
|
||||
return Failure(f"{who} printed no json reply; it printed: {proc.stdout.strip() or 'nothing'}")
|
||||
return Failure(f"{who} printed no json reply; its last line was: {last or 'nothing'}")
|
||||
return reply
|
||||
|
||||
@@ -199,12 +199,19 @@ class ClaudeReply(unittest.TestCase):
|
||||
return subprocess.CompletedProcess(args, 0, "{}", "")
|
||||
|
||||
reply.claude_reply(run, ["--tools", ""], "/r", "p", "the judge")
|
||||
self.assertEqual(asked, [(["-p", "--tools", "", "--no-session-persistence", "--output-format", "json", "--permission-mode", "bypassPermissions"], "/r", "p")])
|
||||
self.assertEqual(asked, [(["-p", "--tools", "", "--no-session-persistence", "--output-format", "stream-json", "--verbose", "--permission-mode", "bypassPermissions"], "/r", "p")])
|
||||
|
||||
def test_reads_the_reply_from_the_streams_last_line(self):
|
||||
stream = "\n".join(json.dumps(event) for event in ({"type": "system", "subtype": "init"}, {"type": "assistant"}, {"type": "result", "result": "ok"}))
|
||||
self.assertEqual(reply.claude_reply(self.claude((0, stream + "\n", "")), [], "/r", "p", "the judge"), {"type": "result", "result": "ok"})
|
||||
|
||||
def test_a_stream_that_ends_in_no_json_names_only_its_last_line(self):
|
||||
self.assertEqual(reply.claude_reply(self.claude((0, '{"type": "system"}\nWarning: bye\n', "")), [], "/r", "p", "the judge"), Failure("the judge printed no json reply; its last line was: Warning: bye"))
|
||||
|
||||
def test_returns_the_json_reply_a_usage_limit_whatever_the_exit_code_or_why_there_is_none(self):
|
||||
self.assertEqual(reply.claude_reply(self.claude((0, '{"result": "ok"}', "")), [], "/r", "p", "the judge"), {"result": "ok"})
|
||||
self.assertEqual(reply.claude_reply(self.claude((0, "Warning: hook said hi\n", "")), [], "/r", "p", "the judge"), Failure("the judge printed no json reply; it printed: Warning: hook said hi"))
|
||||
self.assertEqual(reply.claude_reply(self.claude((0, "", "")), [], "/r", "p", "the judge"), Failure("the judge printed no json reply; it printed: nothing"))
|
||||
self.assertEqual(reply.claude_reply(self.claude((0, "Warning: hook said hi\n", "")), [], "/r", "p", "the judge"), Failure("the judge printed no json reply; its last line was: Warning: hook said hi"))
|
||||
self.assertEqual(reply.claude_reply(self.claude((0, "", "")), [], "/r", "p", "the judge"), Failure("the judge printed no json reply; its last line was: nothing"))
|
||||
limit = json.dumps({"type": "result", "api_error_status": 429, "result": "You've hit your limit"})
|
||||
self.assertEqual(reply.claude_reply(self.claude((1, limit, "")), [], "/r", "p", "the judge"), reply.Limited(None, "You've hit your limit"))
|
||||
self.assertEqual(reply.claude_reply(self.claude((2, "", " ")), [], "/r", "p", "the judge"), Failure("the judge exited 2"))
|
||||
|
||||
@@ -5,7 +5,7 @@ usage: judge-replay <model>
|
||||
<model> is a project model on the routing ladder this setup ships, such as claude-opus-5-5. The tool replays the
|
||||
verdicts archived for the projects that ~/.claude/loop-claude-models.json runs on it: the latest
|
||||
15 of each kind, finished, unfinished, interrupt and quoted answer. Where the archive kept the judge's own
|
||||
prompt (docs/decisions.md → `loop-claude: the archive keeps the raw transcript and each judge run`), the
|
||||
prompt (docs/decisions.md → `loop-claude: the archive keeps the raw transcript and each judge run, …`), the
|
||||
replay hands the judge that prompt, which holds what the judge read then. For a verdict from before the
|
||||
archive kept it, the replay builds the prompt from the transcript copy beside the verdict, with the task,
|
||||
premises and records each project holds now.
|
||||
@@ -27,6 +27,11 @@ JUDGE_REPLAYED in bin/loop_claude/routing.py, which routes that model's judge. j
|
||||
keeps the judge to its own model's rungs; a rung above them that misses no interrupt is the
|
||||
evidence to let the judge climb onto it.
|
||||
|
||||
It keeps each run and the report it prints in loop-claude's archive, in
|
||||
replays/judge-replay-<model>-<UTC stamp>/ under ~/.claude/loop-claude-transcripts or $LOOP_CLAUDE_TRANSCRIPTS.
|
||||
A run's .jsonl file starts with a line holding claude's arguments, working directory, prompt, exit code and
|
||||
stderr, then holds claude's stream-json output; result.md holds the report.
|
||||
|
||||
Like the loop, it tries a judge three times running before it counts the reply as failed. A
|
||||
verdict whose project, transcript or archived judge run cannot be read replays as failed too. Exit 1, saying why, on
|
||||
bad arguments, an unreadable archive or models file, no archived verdict of the model's projects,
|
||||
@@ -37,6 +42,7 @@ import typing
|
||||
|
||||
from transcript_archive.location import archive_root, transcript_of
|
||||
from transcript_archive.read import archived_outcomes
|
||||
from rung_replay.archive import archiving, print_report, replay_folder
|
||||
from rung_replay.command import arguments_or_exit, down_from, failed, limited_exit, projects_on
|
||||
from loop_claude.cycle.judge import Brief, judge_archived, judge_once
|
||||
from judge_replay.replay import Case, case_from, Kind, report, reproduced, sample, verdict_kind
|
||||
@@ -75,12 +81,14 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
if not cases:
|
||||
return failed(TOOL, f"no archived verdicts of projects on {project.model} in {archive_root(home, env)}; run loop-claude on a project on that model first")
|
||||
cases = sample(cases, PER_KIND)
|
||||
folder = replay_folder(archive_root(home, env), TOOL, argv, effects.now())
|
||||
effects = archiving(effects, folder)
|
||||
reference = Rung(project.model, REFERENCE_EFFORT)
|
||||
# The candidates stop at the lowest rung a worker routes to: no route uses a rung below it for the judge.
|
||||
replay = replayed(cases, reference, down_from(ladder, project, ladder.lowest_worker_rung(project)), lambda found, rung: rule(found, rung, home, effects), reproduced, "verdicts", climbs(ladder, project, reference))
|
||||
if isinstance(replay, Limited):
|
||||
return limited_exit(TOOL, replay)
|
||||
print(report(project.model, reference, cases, replay))
|
||||
print_report(folder, report(project.model, reference, cases, replay))
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
@@ -139,7 +139,7 @@ class Main(unittest.TestCase):
|
||||
result = reply(Rung(args[args.index("--model") + 1], args[args.index("--effort") + 1]), "said Verdict: interrupt" in prompt.rsplit("latest last:", 1)[-1])
|
||||
return subprocess.CompletedProcess(args, 0, json.dumps(result), "")
|
||||
|
||||
effects = Effects(claude=claude, claude_on_terminal=unused, discard_typed=unused, dm=unused, execv=unused, git=unused, input=unused, local_time=unused, memory=unused, now=unused, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused, tmux=unused, tmux_on_terminal=unused)
|
||||
effects = Effects(claude=claude, claude_on_terminal=unused, discard_typed=unused, dm=unused, execv=unused, git=unused, input=unused, local_time=unused, memory=unused, now=lambda: 1790000000.0, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused, tmux=unused, tmux_on_terminal=unused)
|
||||
out, err = io.StringIO(), io.StringIO()
|
||||
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
code = main.main(argv, {}, self.home, effects)
|
||||
@@ -174,6 +174,14 @@ class Main(unittest.TestCase):
|
||||
# The replay shows the judge the whole transcript, with no step-up marked in it.
|
||||
self.assertNotIn("[step]", self.prompts[0])
|
||||
|
||||
def test_keeps_each_run_and_the_printed_report_in_one_folder_of_the_archive_beside_the_projects(self):
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
self.assertEqual(code, 0, err)
|
||||
folder = self.home / ".claude" / "loop-claude-transcripts" / "replays" / "judge-replay-claude-opus-5-5-20260921T141320Z"
|
||||
self.assertEqual((folder / "result.md").read_text(encoding="utf-8"), out)
|
||||
runs = [json.loads(path.read_text(encoding="utf-8").splitlines()[0]) for path in folder.glob("*.jsonl")]
|
||||
self.assertEqual({run["prompt"] for run in runs}, set(self.prompts))
|
||||
|
||||
def test_hands_the_judge_the_prompt_the_archive_kept_and_rebuilds_the_prompt_of_a_verdict_with_none(self):
|
||||
kept = self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1"
|
||||
prompt = "From the goals file, OLD premises\n\nThe records: none\n\nThe task the worker was given:\nDo it.\n\nWhat the worker did and said, what the human typed on [human] lines and what loop-claude told it on [loop] lines, latest last:\nsaid Verdict: finished\n\nOLD rules"
|
||||
@@ -286,7 +294,7 @@ class JudgeArchivedRuns(unittest.TestCase):
|
||||
cwds.append(cwd)
|
||||
return subprocess.CompletedProcess(args, 0, json.dumps({"type": "result", "structured_output": output, "total_cost_usd": 0.25}), "")
|
||||
|
||||
effects = Effects(claude=claude, claude_on_terminal=unused, discard_typed=unused, dm=unused, execv=unused, git=unused, input=unused, local_time=unused, memory=unused, now=unused, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused, tmux=unused, tmux_on_terminal=unused)
|
||||
effects = Effects(claude=claude, claude_on_terminal=unused, discard_typed=unused, dm=unused, execv=unused, git=unused, input=unused, local_time=unused, memory=unused, now=lambda: 1790000000.0, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused, tmux=unused, tmux_on_terminal=unused)
|
||||
grounds = grounds_of(prompt, self.ENTRIES)
|
||||
return (grounds if isinstance(grounds, Failure) else judge_archived("/code/p", prompt, grounds, len(self.ENTRIES), REFERENCE, effects)), cwds
|
||||
|
||||
|
||||
@@ -699,7 +699,7 @@ class Cycles(unittest.TestCase):
|
||||
self.assertEqual(run.roles(), ["worker", "judge", "judge", "judge"])
|
||||
self.assertEqual(run.sleeps, [60, 60])
|
||||
self.assertIn("retrying the judge in a minute: judge down", run.stderr)
|
||||
self.assertIn("retrying the judge in a minute: the judge printed no json reply; it printed: nothing\n", run.stderr)
|
||||
self.assertIn("retrying the judge in a minute: the judge printed no json reply; its last line was: nothing\n", run.stderr)
|
||||
self.assertEqual(run.stderr.strip().splitlines()[-1], 'the judge failed three times running: the judge\'s reply has no answer of type string')
|
||||
|
||||
def test_waits_out_a_usage_limit_which_ends_a_run_of_failures(self):
|
||||
|
||||
@@ -45,6 +45,11 @@ replay, and runs no SubagentStart hook there. So the replay without CLAUDE.md tu
|
||||
instruction files off in its settings and opens the prompt with what the hook hands a reviewer, and
|
||||
closes it with the header's line on the project principles.
|
||||
|
||||
It keeps each run, a reviewer's tool calls included, and the report it prints in loop-claude's archive, in
|
||||
replays/review-replay-<pass>-<model>[-omit-claude-md]-<UTC stamp>/ under ~/.claude/loop-claude-transcripts
|
||||
or $LOOP_CLAUDE_TRANSCRIPTS. A run's .jsonl file starts with a line holding claude's arguments, working
|
||||
directory, prompt, exit code and stderr, then holds claude's stream-json output; result.md holds the report.
|
||||
|
||||
Like the loop, it tries claude three times running before it counts a reply as failed. A failed
|
||||
review or match leaves its round out, except on a rung below the project's, where it finds
|
||||
none. Exit 1, saying why, on bad arguments, an unreadable archive or models file, no archived
|
||||
@@ -56,6 +61,7 @@ import typing
|
||||
|
||||
from transcript_archive.location import archive_root
|
||||
from review_replay.rounds import archived_rounds, Round, sample
|
||||
from rung_replay.archive import archiving, print_report, replay_folder
|
||||
from rung_replay.command import arguments_or_exit, failed, limited_exit
|
||||
from review_replay.replay import codes_found, findings_replayed
|
||||
from review_replay.omit_comparison import compared
|
||||
@@ -89,11 +95,13 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
if not codes:
|
||||
return failed(TOOL, f"no archived {reviewed} round of projects on {project.model} in {archive_root(home, env)} holds a 🔴 or 🟡 on code it can find")
|
||||
chosen = sample(list(codes), LATEST)
|
||||
folder = replay_folder(archive_root(home, env), TOOL, argv, effects.now())
|
||||
effects = archiving(effects, folder)
|
||||
if without:
|
||||
report = compared(reviewed, project, chosen, lambda found, bare: findings_replayed(found, codes[found], project, project, home, effects, bare))
|
||||
if isinstance(report, Limited):
|
||||
return limited_exit(TOOL, report)
|
||||
print(report)
|
||||
print_report(folder, report)
|
||||
return 0
|
||||
|
||||
def replay_one(found: Round, rung: Rung) -> set[int] | Failure | Limited:
|
||||
@@ -105,7 +113,7 @@ def main(argv: list[str], env: typing.Mapping[str, str], home: pathlib.Path, eff
|
||||
recall = recalled(chosen, ladder, project, replay_one, f"{reviewed} rounds")
|
||||
if isinstance(recall, Limited):
|
||||
return limited_exit(TOOL, recall)
|
||||
print(recall_report(reviewed, project, recall))
|
||||
print_report(folder, recall_report(reviewed, project, recall))
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
@@ -267,12 +267,22 @@ class Main(unittest.TestCase):
|
||||
result = result if "api_error_status" in result else {"type": "result", "structured_output": {"found": [n + 1 for n in result["found"]]}}
|
||||
return subprocess.CompletedProcess(args, 0, json.dumps(result), "")
|
||||
|
||||
effects = Effects(claude=claude, claude_on_terminal=unused, discard_typed=unused, dm=unused, execv=unused, git=system_effects().git, input=unused, local_time=unused, memory=unused, now=unused, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused, tmux=unused, tmux_on_terminal=unused)
|
||||
effects = Effects(claude=claude, claude_on_terminal=unused, discard_typed=unused, dm=unused, execv=unused, git=system_effects().git, input=unused, local_time=unused, memory=unused, now=lambda: 1790000000.0, on_terminal=unused, python=unused, sleep=unused, stream_claude=unused, tmux=unused, tmux_on_terminal=unused)
|
||||
out, err = io.StringIO(), io.StringIO()
|
||||
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
code = main.main(argv, {}, self.home, effects)
|
||||
return code, out.getvalue(), err.getvalue()
|
||||
|
||||
def test_keeps_each_reviewer_and_match_run_and_the_printed_report_in_one_folder_of_the_archive(self):
|
||||
code, out, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0, 1]})
|
||||
self.assertEqual(code, 0, err)
|
||||
folder = self.home / ".claude" / "loop-claude-transcripts" / "replays" / "review-replay-stability-claude-opus-5-5-20260921T141320Z"
|
||||
self.assertEqual((folder / "result.md").read_text(encoding="utf-8"), out)
|
||||
runs = [json.loads(path.read_text(encoding="utf-8").splitlines()[0]) for path in folder.glob("*.jsonl")]
|
||||
# A run that repeats another word for word, as the fakes' do, is kept once.
|
||||
self.assertEqual({run["prompt"] for run in runs if "--agent" in run["args"]}, {prompt for *_, prompt, _ in self.reviews})
|
||||
self.assertTrue(any("--json-schema" in run["args"] for run in runs))
|
||||
|
||||
def test_without_claude_md_replays_each_round_three_times_with_it_and_once_without_on_the_project_rung(self):
|
||||
finds = iter(([0, 1], [0], [0, 1], [1]))
|
||||
code, out, err = self.run_main(["stability", "claude-opus-5-5", "--omit-claude-md"], lambda rung: {"found": next(finds)})
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Keeps each run of judge-replay and review-replay, and the report each prints, in loop-claude's archive: docs/decisions.md → `loop-claude: the archive keeps the raw transcript and each judge run, …`."""
|
||||
import dataclasses
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
from loop_effects.effects import Effects
|
||||
|
||||
# The type of the first line of a run's file, which holds what the stream lacks: claude's arguments, working directory, prompt, exit code and stderr. No event claude writes has this type.
|
||||
HEADER = "replay_run"
|
||||
RESULT = "result.md"
|
||||
|
||||
|
||||
def replay_folder(root: pathlib.Path, tool: str, argv: list[str], now: float) -> pathlib.Path:
|
||||
"""The folder a replay of tool, run with argv at now, keeps its runs and its result in, under the archive at root. It holds no outcomes file, so no reader of loop-claude's archive takes it for a chunk's."""
|
||||
name = "-".join([tool, *(arg.lstrip("-") for arg in argv), time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(now))])
|
||||
return root / "replays" / re.sub(r"[^A-Za-z0-9._-]", "_", name)
|
||||
|
||||
|
||||
def archiving(effects: Effects, folder: pathlib.Path) -> Effects:
|
||||
"""effects whose claude also writes each run into folder: a header line, then the stream claude printed. A run is kept whether it failed or not, and a run that repeats another word for word is kept once. A folder that can't be written is reported on stderr, and the replay goes on."""
|
||||
claude = effects.claude
|
||||
|
||||
def run(args: list[str], cwd: str, prompt: str) -> subprocess.CompletedProcess[str]:
|
||||
proc = claude(args, cwd, prompt)
|
||||
content = "\n".join([json.dumps({"type": HEADER, "args": args, "cwd": cwd, "prompt": prompt, "returncode": proc.returncode, "stderr": proc.stderr}, ensure_ascii=False), proc.stdout])
|
||||
try:
|
||||
folder.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
(folder / f"{hashlib.sha256(content.encode()).hexdigest()}.jsonl").write_text(content, encoding="utf-8")
|
||||
except OSError as failure:
|
||||
print(f"could not keep the run in {folder}: {failure}", file=sys.stderr, flush=True)
|
||||
return proc
|
||||
|
||||
return dataclasses.replace(effects, claude=run)
|
||||
|
||||
|
||||
def print_report(folder: pathlib.Path, report: str) -> None:
|
||||
"""Prints the report a replay ends with, and keeps it in folder, beside the runs."""
|
||||
print(report)
|
||||
try:
|
||||
folder.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
(folder / RESULT).write_text(report + "\n", encoding="utf-8")
|
||||
except OSError as failure:
|
||||
print(f"could not keep the result in {folder}: {failure}", file=sys.stderr, flush=True)
|
||||
@@ -1,12 +1,17 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from rung_replay.archive import archiving, print_report, replay_folder
|
||||
from rung_replay.command import down_from
|
||||
from loop_effects.effects import Effects
|
||||
from failure import Failure
|
||||
from rung_replay.search import first_reply, Miss, Replay, replayed, Tried
|
||||
from claude_code.reply import Limited
|
||||
@@ -79,5 +84,89 @@ class DownFrom(unittest.TestCase):
|
||||
self.assertEqual(down_from(LADDER, REFERENCE, MIDDLE), [REFERENCE, MIDDLE])
|
||||
|
||||
|
||||
def effects(claude):
|
||||
return Effects(claude=claude, **{name: unused for name in Effects.__dataclass_fields__ if name != "claude"})
|
||||
|
||||
|
||||
def unused(*_):
|
||||
raise AssertionError("not called")
|
||||
|
||||
|
||||
class ArchivedRuns(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.folder = pathlib.Path(self.enterContext(tempfile.TemporaryDirectory())) / "replays" / "tool-1"
|
||||
|
||||
def run_archived(self, replies):
|
||||
return archiving(effects(lambda args, cwd, prompt: replies.pop(0)), self.folder).claude
|
||||
|
||||
def kept(self):
|
||||
return sorted(self.folder.glob("*.jsonl"))
|
||||
|
||||
def test_keeps_each_run_as_the_prompt_it_read_then_the_stream_it_printed_and_returns_claudes_process_as_it_was(self):
|
||||
stream = '{"type": "system"}\n{"type": "result", "result": "ok"}\n'
|
||||
proc = subprocess.CompletedProcess(["claude"], 0, stream, "")
|
||||
run = self.run_archived([proc, subprocess.CompletedProcess(["claude"], 0, stream, "")])
|
||||
self.assertIs(run(["-p", "--agent", "prose-reviewer"], "/clone", "Review it."), proc)
|
||||
run(["-p", "--agent", "prose-reviewer"], "/clone", "Review that.")
|
||||
first, second = (path.read_text(encoding="utf-8").splitlines() for path in self.kept())
|
||||
prompts = sorted((json.loads(lines[0]) for lines in (first, second)), key=lambda header: header["prompt"])
|
||||
self.assertEqual(prompts, [
|
||||
{"type": "replay_run", "args": ["-p", "--agent", "prose-reviewer"], "cwd": "/clone", "prompt": prompt, "returncode": 0, "stderr": ""} for prompt in ("Review it.", "Review that.")
|
||||
])
|
||||
self.assertEqual(first[1:], stream.splitlines())
|
||||
|
||||
def test_keeps_the_runs_private_and_the_prompt_readable(self):
|
||||
self.run_archived([subprocess.CompletedProcess(["claude"], 0, "{}", "")])([], "/r", "Läs den.")
|
||||
self.assertEqual(self.folder.stat().st_mode & 0o777, 0o700)
|
||||
self.assertIn("Läs den.", self.kept()[0].read_text(encoding="utf-8"))
|
||||
|
||||
def test_keeps_a_run_that_failed_and_a_run_repeated_once(self):
|
||||
failed = subprocess.CompletedProcess(["claude"], 1, "half a stream", "boom")
|
||||
run = self.run_archived([failed, failed])
|
||||
run([], "/r", "p")
|
||||
run([], "/r", "p")
|
||||
self.assertEqual(len(self.kept()), 1)
|
||||
kept = self.kept()[0].read_text(encoding="utf-8").splitlines()
|
||||
self.assertEqual((json.loads(kept[0])["returncode"], json.loads(kept[0])["stderr"], kept[1]), (1, "boom", "half a stream"))
|
||||
|
||||
def test_keeps_nothing_where_claude_did_not_start_and_says_where_it_could_not_keep_a_run(self):
|
||||
def missing(args, cwd, prompt):
|
||||
raise FileNotFoundError(2, "No such file or directory")
|
||||
|
||||
with self.assertRaises(FileNotFoundError):
|
||||
archiving(effects(missing), self.folder).claude([], "/r", "p")
|
||||
self.assertFalse(self.folder.exists())
|
||||
# A file where the folder's parent should be.
|
||||
self.folder.parent.parent.mkdir(exist_ok=True)
|
||||
self.folder.parent.write_text("")
|
||||
proc = subprocess.CompletedProcess(["claude"], 0, "{}", "")
|
||||
err = io.StringIO()
|
||||
with contextlib.redirect_stderr(err):
|
||||
self.assertIs(self.run_archived([proc])([], "/r", "p"), proc)
|
||||
self.assertIn(f"could not keep the run in {self.folder}: ", err.getvalue())
|
||||
|
||||
def test_prints_the_report_and_keeps_it_beside_the_runs(self):
|
||||
out = io.StringIO()
|
||||
with contextlib.redirect_stdout(out):
|
||||
print_report(self.folder, "The report.")
|
||||
self.assertEqual(out.getvalue(), "The report.\n")
|
||||
self.assertEqual((self.folder / "result.md").read_text(encoding="utf-8"), "The report.\n")
|
||||
self.assertEqual(self.folder.stat().st_mode & 0o777, 0o700)
|
||||
|
||||
def test_says_on_stderr_where_it_could_not_keep_the_report_and_still_prints_it(self):
|
||||
self.folder.parent.parent.mkdir(exist_ok=True)
|
||||
self.folder.parent.write_text("")
|
||||
out, err = io.StringIO(), io.StringIO()
|
||||
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
print_report(self.folder, "The report.")
|
||||
self.assertEqual(out.getvalue(), "The report.\n")
|
||||
self.assertIn(f"could not keep the result in {self.folder}: ", err.getvalue())
|
||||
|
||||
def test_names_the_folder_by_the_tool_its_arguments_and_the_second_in_utc_under_replays(self):
|
||||
root = pathlib.Path("/archive")
|
||||
self.assertEqual(replay_folder(root, "review-replay", ["stability", "claude-opus-5-5", "--omit-claude-md"], 1790000000.0), root / "replays" / "review-replay-stability-claude-opus-5-5-omit-claude-md-20260921T141320Z")
|
||||
self.assertEqual(replay_folder(root, "judge-replay", ["a/b c"], 0.0), root / "replays" / "judge-replay-a_b_c-19700101T000000Z")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
+9
-1
@@ -182,10 +182,18 @@ Valid while `~/.claude` is a checkout of this repo.
|
||||
|
||||
Valid while claude takes `--session-id` and resumes by it, and the human runs loop-claude where tmux is installed.
|
||||
|
||||
## loop-claude: the archive keeps the raw transcript and each judge run, and a failed archive stops the loop
|
||||
## loop-claude: the archive keeps the raw transcript and each judge run, a failed archive stops the loop, and a replay keeps its runs and report in a replays/ folder of the archive
|
||||
|
||||
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. The judge's runs are archived too, as `judge/<sha256>.json` in the chunk's folder: the prompt it read and its json reply. Together they are its whole transcript, since it runs with no tools and one turn. A run is kept whole and deduplicated by hash, and the outcome of the verdict, interrupt or rejected reply it produced names it on a `Judge run:` line. Keeping it whole costs little: a prompt is at most `JUDGE_PROMPT_CHARS`, 400,000 characters. Measured 2026-11-17, the archive's 911 judge lines come to at most 0.36 GB of prompts, against 3.5 GB in its 1,211 transcript copies, the largest 23.4 MB. The judge is still priced from its `Judge cost:` line, which is claude's own total for the run, and the Spent block counts the live reply's tokens; the archived reply keeps its `usage` for a later reader, and nothing reads it back. `judge-replay` replays a verdict on its archived prompt where there is one. Valid while transcripts, and the judge's prompts, stay small enough to copy whole.
|
||||
|
||||
2026-11-17, lilleman, todo item 304. Serves goals 1.4 and 3. `judge-replay` and `review-replay` each decide which rung a review pass or the judge may run on, so their runs stay checkable and re-priceable once a decision is questioned. They keep their runs in `replays/<tool>-<arguments>-<UTC stamp>/` in the archive's root:
|
||||
|
||||
- One `<sha256>.jsonl` file per run, whether it failed or not. It starts with a header line holding claude's arguments, working directory, prompt, exit code and stderr, and goes on with claude's stream-json output as printed, whose last line is the reply the replay read. A reviewer's tool calls are in it.
|
||||
- `result.md`, the report the tool printed.
|
||||
- A run that repeats another word for word is kept once. A folder that can't be written is reported on stderr and doesn't stop the replay.
|
||||
|
||||
`claude_reply` reads stream-json for every caller, so the judge's runs are streams too; the archive's `JudgeRun` stays as it was, since a judge has no tools and one turn. Valid while transcripts stay small enough to copy whole.
|
||||
|
||||
## loop-claude: each verdict records the claude-config release and commit that ran it, the raw tokens each thread spent by model and kind since the previous outcome that recorded a spend, and the human's typed answer the judge quotes as reversing a choice the worker took on its own recommendation
|
||||
|
||||
2026-11-12, lilleman, todo item 305. Serves goals 1, 1.2 and 1.4. A change to the setup can be judged only by comparing chunks run before it with chunks run after, so each verdict, interrupt and quoted answer in loop-claude's archive records three things beside its judge lines.
|
||||
|
||||
@@ -2345,3 +2345,54 @@ Read most (opens and searches):
|
||||
- `bin/larv_review/test_larv_review.py` 3
|
||||
- `bin/project_config/main.py` 3
|
||||
- `bin/release_plan/formula.py` 3
|
||||
|
||||
## Reading cost, chunk 093397894c39, finished 2026-11-16T23:09:28Z
|
||||
|
||||
71806 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `bin/todo_item/main.py` 10
|
||||
- `bin/todo_item/test_todo_item.py` 10
|
||||
- `README.md` 5
|
||||
- `bin/project_config/test_project_config.py` 4
|
||||
- `bin/chunk_pr/main.py` 3
|
||||
- `CLAUDE.md` 2
|
||||
- `bin/plan_store/fakes.py` 2
|
||||
- `bin/plan_store/stores.py` 2
|
||||
- `bin/project_config/settings.py` 2
|
||||
- `docs/decisions.md` 2
|
||||
|
||||
## Reading cost, chunk 7cd3e6d607bf, finished 2026-11-16T23:18:20Z
|
||||
|
||||
89486 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `bin/deploy_release/test_deploy_release.py` 11
|
||||
- `bin/chunk_preflight/test_chunk_preflight.py` 5
|
||||
- `.gitignore` 4
|
||||
- `bin/chunk_preflight/gathered.py` 4
|
||||
- `bin/deploy_release/cut.py` 3
|
||||
- `bin/forge/pulls.py` 3
|
||||
- `docs/decisions.md` 3
|
||||
- `README.md` 2
|
||||
- `bin/chunk_pr/main.py` 2
|
||||
- `bin/chunk_preflight/main.py` 2
|
||||
|
||||
## Reading cost, chunk 080dc175f969, finished 2026-11-17T00:45:04Z
|
||||
|
||||
139513 new input tokens before the first commit.
|
||||
|
||||
Read most (opens and searches):
|
||||
|
||||
- `bin/loop_claude/test_loop_claude.py` 13
|
||||
- `README.md` 8
|
||||
- `bin/loop_claude/cycle/run.py` 5
|
||||
- `bin/judge_replay/main.py` 4
|
||||
- `bin/loop_claude/cycle/judge.py` 4
|
||||
- `bin/judge_replay/test_judge_replay.py` 3
|
||||
- `bin/transcript_archive/outcomes.py` 3
|
||||
- `bin/transcript_archive/test_transcript_archive.py` 3
|
||||
- `skills/larv-review/SKILL.md` 3
|
||||
- `bin/loop_claude/loop/close.py` 2
|
||||
|
||||
@@ -13,7 +13,6 @@
|
||||
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 306 | 0.12.0 | | | **Set a strategy that keeps loop-claude's archive and larv-review's review folders backed up and small over the years, compressing older entries without losing any.** | 3 | 2 | 4 | 7 | 5 | 3 by 5 | 3.9 |
|
||||
| 304 | 0.12.0 | | defect | **Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 2, 3 by 2 | 3.8 |
|
||||
| 387 | 0.12.0 | | | **Bind a project's rung to its ladder in one value, so the routing functions and `Cycle` take one argument for both.** | 3 | 1 | 3 | 5 | 5 | code health by 3 | 2.9 |
|
||||
| 338 | 0.12.0 | | owner | **Script each step that 0.9.0's chunks repeated by hand and a program can compute, filing one item per step, and file this audit again for the next release.** | 1 | 2 | 1 | 2 | 2 | 1.4 by 3, 1.5 by 3 | 2.3 |
|
||||
| 288 | 0.12.0 | | | **Point the two reading-cost decisions at code health and goal 1.4's counter-signal, in place of the goal 1.1 they cite.** | 1 | 2 | 1 | 2 | 1 | code health by 1, 1.4 by 1 | 2.2 |
|
||||
@@ -98,22 +97,6 @@ Touches:
|
||||
- `skills/larv-review/SKILL.md`: `## 6. Wrap up` (small)
|
||||
- `README.md`: `## Install` (small)
|
||||
|
||||
### 304. Archive each review-replay and judge-replay run's transcript beside its result, so a replay that decided a rung can be checked and re-priced later.
|
||||
|
||||
Today `review-replay` and `judge-replay` leave no transcript: both run claude through `claude_reply` in `bin/claude_code/reply.py`, whose `--no-session-persistence` makes Claude Code write no session file, as item 303 describes for the judge. A replay decides which rung a review pass or the judge may run on (`routing: a review pass runs below the project's rung only on a rung that a replay of its rounds shows…`, and the judge's rung on `JUDGE_REPLAYED`), so the runs behind a routing decision can't be read again, re-priced or re-scored once the decision is questioned. Goal 3 asks for the transcripts of unattended runs, the bench's included, to be backed up; `workflow-bench/bench.py` already keeps its claude stream in `logs/`.
|
||||
|
||||
Unlike the judge, a replayed reviewer uses tools over several turns, so its transcript is more than its prompt and reply. Recommended: have `claude_reply` take `--output-format stream-json --verbose` in place of `json`, keep `--no-session-persistence`, write the stream where the replay keeps its result, and read the reply from the stream's last `result` event. That also gives item 303 the judge's stream for free, so build on 303's archive layout and do both callers through the one runner.
|
||||
|
||||
Touches:
|
||||
|
||||
- `bin/claude_code/reply.py`: `claude_reply`
|
||||
- `bin/claude_code/test_claude_code.py`
|
||||
- `bin/review_replay/replay.py`: `reviewed_in`, `match`, `copy_reports`
|
||||
- `bin/review_replay/test_review_replay.py`
|
||||
- `bin/judge_replay/main.py`: `rule`
|
||||
- `bin/judge_replay/test_judge_replay.py`
|
||||
- `bin/loop_claude/cycle/judge.py`: `judge_args`, `run_judge` (small)
|
||||
|
||||
### 387. Bind a project's rung to its ladder in one value, so the routing functions and `Cycle` take one argument for both.
|
||||
|
||||
Filed by PR #553, which made the routing ladder a file a project may replace. Its stability round 1 found two places the comprehension seats name as hard, touched and no easier to read: `bin/loop_claude/routing.py` `unfinished_route` took `ladder` as a fourth parameter, and `Cycle` in `bin/loop_claude/cycle/run.py` gained the field `ladder`. The patch threaded the ladder as an argument, since a wider change did not fit a patch cut during an outage.
|
||||
|
||||
Reference in New Issue
Block a user