Gzip loop-claude's archive and the review folders 30 days after each file was last written, and read gzipped files by their plain name
CI / tests (pull_request) Failing after 2m59s
CI / tests (pull_request) Failing after 2m59s
This commit is contained in:
@@ -47,6 +47,9 @@
|
||||
!bin/command_lines/*.py
|
||||
!bin/command_patterns/
|
||||
!bin/command_patterns/*.py
|
||||
!bin/compress-archives
|
||||
!bin/compress_archives/
|
||||
!bin/compress_archives/*.py
|
||||
!bin/deploy-release
|
||||
!bin/deploy_release/
|
||||
!bin/deploy_release/*.py
|
||||
|
||||
@@ -73,6 +73,12 @@ Then pick a model with `/model`.
|
||||
|
||||
Link `loop-claude` onto `PATH`: `ln -s ~/.claude/bin/loop-claude ~/.local/bin/loop-claude`. Every claude it starts, the interactive ones included, skips every permission prompt. loop-claude runs each chunk in its own window of a tmux session per project, so it needs tmux 3.2 or later.
|
||||
|
||||
loop-claude keeps every transcript in `~/.claude/loop-claude-transcripts`, and larv-review each chunk's reports in `~/.local/state/larv-review`, outside `~/.claude`: back up both. Add a crontab line that gzips each of their files 30 days after it was last written, keeping every one, and records any failure:
|
||||
|
||||
```sh
|
||||
17 4 * * * $HOME/.claude/bin/compress-archives > /dev/null 2>> $HOME/.local/state/compress-archives.errors
|
||||
```
|
||||
|
||||
To get loop-claude's questions and stops as DMs from a bot, write the bot's token, your username or member ID and, for Mattermost, the server into `~/.claude-projects/<key>.env`, in the keys `~/.claude/bin/notify --help` names, run `chmod 600` on it, and add `env-file = "~/.claude-projects/<key>.env"` and `notifier = "mattermost"` (or `"slack"`) to the project's config, as Install → "Set up a project" describes.
|
||||
|
||||
### Set up a project
|
||||
@@ -171,6 +177,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `chunk-pr open|sync|ready|merge|clean` opens a chunk's pull request as a draft, brings its branch level with origin's default branch, marks it ready and merges it once its checks pass, with `gh` on a GitHub origin and `tea` elsewhere. The chunks of one project merge one at a time: in its turn, `merge` brings the branch level again, finishes the plan items its `--finish` flags name, and waits for the checks; once the pull request is merged, `clean` removes the chunk's worktree and branch and fast-forwards the main checkout. `chunk-pr --help` states the steps.
|
||||
- `chunk-preflight` finds the mechanical facts a chunk starts from, and with `--worktree` makes its worktree; `loop-claude` runs it with `--worktree` before it routes each chunk, and `chunk-preflight --help` lists the facts.
|
||||
- `command-patterns [<version>] [--calls <file>]` prints a table of the shell command patterns a release's archived chunks ran, such as `tea pr create`, for an audit of the release, and with `--calls <file>` saves each call as a JSON line; `command-patterns --help` states the columns and the line's keys.
|
||||
- `compress-archives` gzips each file of loop-claude's archive and the review folders that no tool has written for 30 days, keeping every one; Install's crontab line runs it daily, and `compress-archives --help` says which files.
|
||||
- `deploy-release [--cut] [<checkout>]` moves `~/.claude`, or the checkout named, to the newest release, as Update says; with `--cut`, run in the development clone, it first cuts the next release, as Release says.
|
||||
- `judge-replay <model>` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints what each rung a review climbs to misses, and the lowest rung below them that, like each candidate above it, returns them; `judge-replay --help` states the replay.
|
||||
- `larv-review <command>` holds larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's round count, the scoring run's trigger and setup; the comprehension history's format, its latest places and its count of how often seats agree; and each reviewer's prompt header. `larv-review --help` states them. `settings.json` → `hooks` runs `larv-review reviewer-context` as each subagent starts. A reviewer under `agents/` whose frontmatter has `omitClaudeMd: true` starts without `CLAUDE.md`, and the command hands it that file's "Shared goal", "Prose" and "Vocabulary" sections.
|
||||
@@ -221,7 +228,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `routing_figures/` measures loop-claude's archive, and each of its projects' default branches, for loop-claude's routing calls.
|
||||
- `rung_replay/` holds what `judge_replay` and `review_replay` share: the parallel replay runner, the argument reader, the candidate rungs and the exits, and `judge_replay`'s rung search. It reads `loop_claude`'s retry count and models file, and finds that file through `loop_state`'s paths.
|
||||
- `test_env.py` serves the tests.
|
||||
- `text_files/` reads a Markdown file's `##` sections, quotes a passage and joins names into a list, stamps times in UTC, and replaces a file in one step, private where it is new.
|
||||
- `text_files/` reads a Markdown file's `##` sections, quotes a passage and joins names into a list, stamps times in UTC, replaces a file in one step, private where it is new, and reads a file by its plain name where `compress-archives` gzipped it.
|
||||
- `transcript_archive/` stores loop-claude's transcripts. Its `project_key.py` names each project's archive folder, loop-claude session and models-file entry.
|
||||
|
||||
## Test
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import typing
|
||||
|
||||
from claude_code.entries import AssistantTurn, Entry, HumanTurn, LoopPrompt, parse_entry, Text, ToolCall
|
||||
from failure import Failure
|
||||
from text_files.compressed import read_stored
|
||||
|
||||
JUDGED_TOOL_INPUT = 400
|
||||
|
||||
@@ -14,9 +17,9 @@ class Typed(typing.NamedTuple):
|
||||
|
||||
|
||||
def read_transcript(path: str) -> list[Entry] | Failure:
|
||||
"""The entries of the transcript at path, or of its gzip where the archive compressed it."""
|
||||
try:
|
||||
with open(path, "rb") as handle:
|
||||
return [entry for raw in handle if (entry := parse_entry(raw))]
|
||||
return [entry for raw in io.BytesIO(read_stored(pathlib.Path(path))) if (entry := parse_entry(raw))]
|
||||
except OSError as failure:
|
||||
return Failure(f"could not read the transcript {path}: {failure.strerror}")
|
||||
|
||||
|
||||
Executable
+9
@@ -0,0 +1,9 @@
|
||||
#!/usr/bin/env python3
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
|
||||
from compress_archives import main
|
||||
|
||||
sys.exit(main.main(sys.argv[1:]))
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Compress every file of loop-claude's archive and the review folders that no tool writes to any more, keeping each one.
|
||||
|
||||
usage: compress-archives
|
||||
|
||||
It gzips each file last written over 30 days ago, beside its plain name as <name>.gz with the same
|
||||
times: in loop-claude's archive, $LOOP_CLAUDE_TRANSCRIPTS (default ~/.claude/loop-claude-transcripts),
|
||||
each transcript (*.jsonl); in the review folders, ~/.local/state/larv-review, every file. It writes
|
||||
the gzip, reads it back, and removes the plain file only once the gzip reads back its bytes. Every
|
||||
tool that reads these folders reads a gzipped file by its plain name. Install's crontab line runs it
|
||||
daily.
|
||||
|
||||
It prints how many files it compressed. Exit 1, saying why, at the first file it cannot compress,
|
||||
which it keeps plain with every file after it, or on any argument but --help.
|
||||
"""
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import time
|
||||
import typing
|
||||
|
||||
from failure import Failure
|
||||
from review_rules.review_folder import REPORTS
|
||||
from text_files.compressed import compress, SUFFIX
|
||||
from transcript_archive.location import archive_root
|
||||
|
||||
# Why 30 days: docs/decisions.md → `archives: loop-claude's archive and the review folders keep every file, and gzip each one 30 days after it was last written`.
|
||||
AGE_DAYS = 30
|
||||
|
||||
|
||||
def main(argv: list[str], env: typing.Mapping[str, str] = os.environ, home: pathlib.Path | None = None, now: float | None = None) -> int:
|
||||
if argv in (["-h"], ["--help"]):
|
||||
print(__doc__.strip())
|
||||
return 0
|
||||
if argv:
|
||||
print(f"unknown arguments {' '.join(argv)}; usage: compress-archives", file=sys.stderr)
|
||||
return 1
|
||||
home = home or pathlib.Path.home()
|
||||
before = (time.time() if now is None else now) - AGE_DAYS * 24 * 60 * 60
|
||||
count = 0
|
||||
for path in [*old_files(archive_root(home, env), "*.jsonl", before), *old_files(home / REPORTS, "*", before)]:
|
||||
if failed := compress(path):
|
||||
print(failed.reason, file=sys.stderr)
|
||||
return 1
|
||||
count += 1
|
||||
print(f"Compressed {count} files.")
|
||||
return 0
|
||||
|
||||
|
||||
def old_files(root: pathlib.Path, pattern: str, before: float) -> list[pathlib.Path]:
|
||||
"""Each plain file under root that pattern matches, last written before the time before, sorted; a link is left alone, since its target lies elsewhere."""
|
||||
found = []
|
||||
for path in root.rglob(pattern):
|
||||
try:
|
||||
if not path.name.endswith(SUFFIX) and path.is_file() and not path.is_symlink() and path.stat().st_mtime < before:
|
||||
found.append(path)
|
||||
# Removed while the walk ran, such as a file of review-replay's copy of a review folder.
|
||||
except FileNotFoundError:
|
||||
continue
|
||||
return sorted(found)
|
||||
@@ -0,0 +1,94 @@
|
||||
import contextlib
|
||||
import gzip
|
||||
import io
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from compress_archives import main
|
||||
from failure import Failure
|
||||
|
||||
NOW = 1790000000
|
||||
DAY = 24 * 60 * 60
|
||||
|
||||
|
||||
class Main(unittest.TestCase):
|
||||
"""A home whose archive and review folders each hold a file last written 31 days ago and one written 29 days ago."""
|
||||
|
||||
def setUp(self):
|
||||
self.home = pathlib.Path(self.enterContext(tempfile.TemporaryDirectory()))
|
||||
self.archive = self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1"
|
||||
self.reviews = self.home / ".local" / "state" / "larv-review" / "code" / "proj-item-1"
|
||||
for folder in (self.archive / "session" / "s" / "subagents", self.reviews / "probe"):
|
||||
folder.mkdir(parents=True)
|
||||
self.write(self.archive / "a.jsonl", 31)
|
||||
self.write(self.archive / "a.outcomes.md", 31)
|
||||
self.write(self.archive / "session" / "s" / "subagents" / "agent-a.jsonl", 31)
|
||||
self.write(self.archive / "session" / "s" / "subagents" / "agent-a.meta.json", 31)
|
||||
self.write(self.archive / "b.jsonl", 29)
|
||||
self.write(self.reviews / "stability-round-1.md", 31)
|
||||
self.write(self.reviews / "probe" / "out.js", 31)
|
||||
self.write(self.reviews / "branch-1.diff", 29)
|
||||
|
||||
def write(self, path, days_ago):
|
||||
path.write_bytes(f"{path.name}\n".encode())
|
||||
os.utime(path, (NOW - days_ago * DAY, NOW - days_ago * DAY))
|
||||
|
||||
def run_main(self, argv=(), env=None):
|
||||
out, err = io.StringIO(), io.StringIO()
|
||||
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
code = main.main(list(argv), env or {}, self.home, NOW)
|
||||
return code, out.getvalue(), err.getvalue()
|
||||
|
||||
def kept(self):
|
||||
return sorted(str(path.relative_to(self.home)) for path in self.home.rglob("*") if path.is_file())
|
||||
|
||||
def test_gzips_the_archives_transcripts_and_every_review_file_last_written_over_30_days_ago(self):
|
||||
code, out, err = self.run_main()
|
||||
self.assertEqual((code, err), (0, ""))
|
||||
self.assertEqual(self.kept(), [
|
||||
".claude/loop-claude-transcripts/proj/c1/a.jsonl.gz",
|
||||
".claude/loop-claude-transcripts/proj/c1/a.outcomes.md",
|
||||
".claude/loop-claude-transcripts/proj/c1/b.jsonl",
|
||||
".claude/loop-claude-transcripts/proj/c1/session/s/subagents/agent-a.jsonl.gz",
|
||||
".claude/loop-claude-transcripts/proj/c1/session/s/subagents/agent-a.meta.json",
|
||||
".local/state/larv-review/code/proj-item-1/branch-1.diff",
|
||||
".local/state/larv-review/code/proj-item-1/probe/out.js.gz",
|
||||
".local/state/larv-review/code/proj-item-1/stability-round-1.md.gz",
|
||||
])
|
||||
self.assertEqual(gzip.decompress((self.archive / "a.jsonl.gz").read_bytes()), b"a.jsonl\n")
|
||||
self.assertEqual(out, "Compressed 4 files.\n")
|
||||
self.assertEqual(self.run_main(), (0, "Compressed 0 files.\n", ""))
|
||||
|
||||
def test_reads_the_archive_where_loop_claude_transcripts_puts_it(self):
|
||||
moved = self.home / "elsewhere"
|
||||
(self.home / ".claude" / "loop-claude-transcripts").rename(moved)
|
||||
code, _, err = self.run_main(env={"LOOP_CLAUDE_TRANSCRIPTS": str(moved)})
|
||||
self.assertEqual((code, err), (0, ""))
|
||||
self.assertTrue((moved / "proj" / "c1" / "a.jsonl.gz").is_file())
|
||||
|
||||
def test_runs_where_neither_folder_exists_yet(self):
|
||||
with tempfile.TemporaryDirectory() as empty:
|
||||
self.home = pathlib.Path(empty)
|
||||
self.assertEqual(self.run_main(), (0, "Compressed 0 files.\n", ""))
|
||||
|
||||
def test_stops_at_the_first_failure_keeping_that_file_and_every_later_one(self):
|
||||
with mock.patch.object(main, "compress", return_value=Failure("could not compress x: full")):
|
||||
code, out, err = self.run_main()
|
||||
self.assertEqual((code, out, err), (1, "", "could not compress x: full\n"))
|
||||
self.assertTrue((self.archive / "a.jsonl").is_file())
|
||||
|
||||
def test_prints_its_help_and_refuses_other_arguments(self):
|
||||
code, out, _ = self.run_main(["--help"])
|
||||
self.assertEqual(code, 0)
|
||||
self.assertTrue(out.startswith("Compress every file"), out)
|
||||
self.assertEqual(self.run_main(["x"])[0], 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -12,6 +12,7 @@ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
from transcript_archive.outcomes import ANSWERED, FINISHED, INTERRUPTED, UNFINISHED
|
||||
from claude_code.entries import AssistantTurn, HumanTurn, LoopPrompt, Text
|
||||
from judge_replay.replay import Case, case_from, report, reproduced, sample, verdict_kind
|
||||
from text_files import compressed
|
||||
from project_config.test_project_config import configured
|
||||
from loop_effects.effects import Effects
|
||||
from failure import Failure
|
||||
@@ -225,6 +226,13 @@ class Main(unittest.TestCase):
|
||||
self.assertEqual(code, 0)
|
||||
self.assertIn("replayed failed: could not read the transcript", out)
|
||||
|
||||
def test_replays_a_verdict_whose_transcript_is_compressed(self):
|
||||
self.assertIsNone(compressed.compress(self.home / ".claude" / "loop-claude-transcripts" / "proj" / "c1" / "t0.jsonl"))
|
||||
code, out, err = self.run_main(["claude-opus-5-5"], lambda rung, interrupted: self.verdict(interrupted, f"said {INTERRUPTED if interrupted else FINISHED}"))
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertNotIn("replayed failed", out)
|
||||
self.assertIn("The reference judge, claude-opus-5-5 at xhigh effort, reproduced 2.", out)
|
||||
|
||||
def test_tries_the_projects_rung_first_and_retries_a_failed_reply_as_the_loop_does(self):
|
||||
def reply(rung, interrupted):
|
||||
return self.verdict(interrupted, "not in the transcript" if rung.model != "claude-opus-5-5" else f"said {INTERRUPTED if interrupted else FINISHED}")
|
||||
|
||||
@@ -16,6 +16,7 @@ from transcript_archive.write import archive
|
||||
from claude_code.entries import AssistantTurn
|
||||
from project_config.templates import BRANCH
|
||||
from reading_cost.measure import Checkout, Reading
|
||||
from text_files import compressed
|
||||
from failure import Failure
|
||||
from reading_cost import figures, main, measure
|
||||
from transcript_archive.read import finished_chunks, FinishedChunk
|
||||
@@ -173,6 +174,14 @@ class ReadingCostTool(unittest.TestCase):
|
||||
self.assertEqual((ran.returncode, ran.stdout), (1, ""))
|
||||
self.assertIn("no config names the checkout", ran.stderr)
|
||||
|
||||
def test_measures_a_chunk_whose_archived_transcripts_are_compressed(self):
|
||||
self.archive_chunk(1790000000)
|
||||
for copy in self.archive.rglob("*.jsonl"):
|
||||
self.assertIsNone(compressed.compress(copy))
|
||||
ran = self.run_tool(self.checkout)
|
||||
self.assertEqual(ran.returncode, 0, ran.stderr)
|
||||
self.assertIn("33 new input tokens before the first commit.", (self.checkout / "docs" / "reading-cost.md").read_text())
|
||||
|
||||
def test_appends_each_finished_chunk_once_in_the_order_they_finished_from_a_worktree(self):
|
||||
self.archive_chunk(1790000200)
|
||||
self.archive_chunk(1790000100)
|
||||
|
||||
@@ -16,6 +16,7 @@ from rung_replay.search import first_reply
|
||||
from record_store.stores import local_file, store
|
||||
from project_config.main import located
|
||||
from review_rules.reviewer_context import principles_line, reviewer_context
|
||||
from text_files.compressed import read_stored, SUFFIX
|
||||
from review_rules.review_folder import REPORTS, review_folder
|
||||
from model_routing.ladder import Rung
|
||||
from claude_code.entries import structured_output
|
||||
@@ -146,8 +147,14 @@ def copy_reports(found: Round, folder: pathlib.Path, reports: str) -> None:
|
||||
"""Copies into reports each report written to folder after found's chunk began and before its round did."""
|
||||
# The round's own report, any later one, and one from an earlier attempt at the item in the same worktree would each hand the reviewer the findings it is to find.
|
||||
for written in folder.iterdir() if folder.is_dir() else ():
|
||||
if written.is_file() and found.began <= written.stat().st_mtime < found.seconds:
|
||||
if not written.is_file() or not found.began <= written.stat().st_mtime < found.seconds:
|
||||
continue
|
||||
name = written.name.removesuffix(SUFFIX)
|
||||
if name == written.name:
|
||||
shutil.copy2(written, reports)
|
||||
# compress-archives gzipped it. A plain file of the same name replaced it later, as it would have replaced the plain file, so only that one is copied, where its time falls before the round.
|
||||
elif not (folder / name).is_file():
|
||||
(pathlib.Path(reports) / name).write_bytes(read_stored(folder / name))
|
||||
|
||||
|
||||
def redirected(prompt: str, home: pathlib.Path, folder: pathlib.Path, worktree: str, copies: Copies) -> str | Failure:
|
||||
|
||||
@@ -11,6 +11,7 @@ from transcript_archive.project_key import checkout_of
|
||||
from failure import Failure
|
||||
from claude_code.transcript import first_timestamped_line, read_transcript
|
||||
from rung_replay.command import projects_on
|
||||
from text_files.compressed import read_stored, stored_glob
|
||||
|
||||
# A line that opens with a 🔴 or 🟡 mark, after any list, table, heading or bold markup, as review-findings.md has reviewers report them.
|
||||
FINDING = re.compile(r"^[\s#>*+|-]*(?:\d+[.)]\s*)?[*_]*\s*(?:🔴|🟡)")
|
||||
@@ -126,8 +127,8 @@ def archived_rounds(reviewer: str, model: str, root: pathlib.Path, home: pathlib
|
||||
|
||||
|
||||
def chunk_rounds(reviewer: str, checkout: str, chunk: pathlib.Path) -> list[Round]:
|
||||
copies = sorted(chunk.glob("*.jsonl"))
|
||||
began = parse_entry(first_timestamped_line(copies[0].read_bytes())) if copies else None
|
||||
copies = stored_glob(chunk, "*.jsonl")
|
||||
began = parse_entry(first_timestamped_line(read_stored(copies[0]))) if copies else None
|
||||
if not began or not began.timestamp:
|
||||
return []
|
||||
found = []
|
||||
|
||||
@@ -15,6 +15,7 @@ import unittest.mock
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from review_replay.rounds import authored_by, lost_inputs, recorded_findings, reviewed_path, Round, round_from, sample, stamp_seconds
|
||||
from text_files import compressed
|
||||
from project_config.test_project_config import configured, write_config
|
||||
from loop_effects.effects import Effects, system_effects
|
||||
from claude_code.entries import entry_from
|
||||
@@ -378,6 +379,28 @@ class Main(unittest.TestCase):
|
||||
os.utime(folder / "stability-round-3.md", (attempt, attempt))
|
||||
return folder
|
||||
|
||||
def test_replays_rounds_whose_archived_transcripts_are_compressed(self):
|
||||
for copy in self.archive.rglob("*.jsonl"):
|
||||
self.assertIsNone(compressed.compress(copy))
|
||||
code, out, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [1] if rung == LOW else [0, 1]})
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertTrue(out.startswith("Replayed 1 stability rounds of claude-opus-5-5 projects,"), out)
|
||||
|
||||
def test_copies_a_compressed_report_from_before_the_round_under_its_plain_name(self):
|
||||
folder = self.review_folder_with_round_1()
|
||||
self.assertIsNone(compressed.compress(folder / "stability-round-1.md"))
|
||||
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to ~/.local/state/larv-review/code/proj-x/stability-round-2.md; the last round's is ~/.local/state/larv-review/code/proj-x/stability-round-1.md.")
|
||||
seen = []
|
||||
|
||||
def review(rung):
|
||||
named = pathlib.Path(re.search(r"report to (\S+)/stability-round-2\.md", self.reviews[-1][4]).group(1))
|
||||
seen.append((sorted(path.name for path in named.iterdir()), (named / "stability-round-1.md").read_text()))
|
||||
return {"type": "result", "result": f"review on {rung.model} {rung.effort}"}
|
||||
|
||||
code, _, err = self.run_main(["stability", "claude-opus-5-5"], lambda rung: {"found": [0, 1]}, review)
|
||||
self.assertEqual(code, 0, err)
|
||||
self.assertEqual(seen[0], (["stability-round-1.md"], "round 1\n"))
|
||||
|
||||
def test_points_the_prompts_review_folder_at_a_copy_of_the_reports_from_before_the_round_that_it_removes(self):
|
||||
folder = self.review_folder_with_round_1()
|
||||
self.set_prompt(f"Review branch x in {self.home}/code/proj-x. Write your report to ~/.local/state/larv-review/code/proj-x/stability-round-2.md; the last round's is ~/.local/state/larv-review/code/proj-x/stability-round-1.md.")
|
||||
|
||||
@@ -3,6 +3,7 @@ import contextlib
|
||||
import dataclasses
|
||||
import fcntl
|
||||
import functools
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import typing
|
||||
@@ -15,6 +16,7 @@ from routing_figures.chunk import Chunk, chunk_figures, ChunkFigures, Subagent
|
||||
from git_cli.git_repo import default_ref, Git
|
||||
from claude_code.entries import Entry, parse_entry
|
||||
from failure import Failure
|
||||
from text_files.compressed import read_stored
|
||||
from text_files.file_writes import replace_file
|
||||
from project_config.main import title_templates
|
||||
from project_config.templates import Titles
|
||||
@@ -115,8 +117,7 @@ def branch_history(checkout: str, home: pathlib.Path, titles: Titles, git: Git)
|
||||
|
||||
|
||||
def entries(path: pathlib.Path) -> list[Entry]:
|
||||
with open(path, "rb") as handle:
|
||||
return [entry for line in handle if (entry := parse_entry(line))]
|
||||
return [entry for line in io.BytesIO(read_stored(path)) if (entry := parse_entry(line))]
|
||||
|
||||
|
||||
def subagent(transcript: pathlib.Path) -> Subagent:
|
||||
|
||||
@@ -19,6 +19,7 @@ from project_config.templates import BARE, Titles
|
||||
from branch_chunks.cut import BranchChunk
|
||||
from routing_figures.calls import Call, calls, totals
|
||||
from routing_figures.chunk import Chunk, chunk_figures, ChunkFigures, Reports, Subagent
|
||||
from text_files import compressed
|
||||
from project_config.test_project_config import configured, write_config
|
||||
from routing_figures.state import due
|
||||
from claude_code.entries import entry_from
|
||||
@@ -352,6 +353,13 @@ class Archive(unittest.TestCase):
|
||||
self.assertEqual(found.chunks[0].dollars["subagent type"], {"worker": 2.0, "Explore": 2.0})
|
||||
self.assertEqual(archive.measure_archive(self.root, "/h", "2026-10-14T10:00:00Z", no_checkout), archive.ArchiveChunks([], ()))
|
||||
|
||||
def test_measures_a_chunk_whose_transcripts_are_compressed(self):
|
||||
self.chunk("home/x", "c1", stamped("2026-10-14T10:00:00Z", outcomes.finished(LOW_ROUTE, "m", "j")), [message("m1", "2026-10-14T09:00:00Z", call("Agent", "a1", subagent_type="Explore"))], ({"agentType": "Explore", "toolUseId": "a1"}, [message("s1", "2026-10-14T09:00:01Z")]))
|
||||
for copy in self.root.rglob("*.jsonl"):
|
||||
self.assertIsNone(compressed.compress(copy))
|
||||
found = archive.measure_archive(self.root, "/h", "", no_checkout)
|
||||
self.assertEqual(found.chunks[0].dollars["subagent type"], {"worker": 2.0, "Explore": 2.0})
|
||||
|
||||
def test_counts_each_chunks_later_fixes_from_its_projects_checkout(self):
|
||||
home = self.root / "home"
|
||||
env = {**isolated_env(), "GIT_CONFIG_GLOBAL": os.devnull, "HOME": str(home)}
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
"""A file kept plain, or gzipped beside its plain name as <name>.gz: reading it by its plain name, and compressing it.
|
||||
|
||||
Why gzip and when a file is compressed: docs/decisions.md → `archives: loop-claude's archive and the review folders keep every file, and gzip each one 30 days after it was last written`.
|
||||
"""
|
||||
import errno
|
||||
import gzip
|
||||
import os
|
||||
import pathlib
|
||||
import zlib
|
||||
|
||||
from failure import Failure
|
||||
from text_files.file_writes import replace_file
|
||||
|
||||
SUFFIX = ".gz"
|
||||
|
||||
|
||||
def stored(path: pathlib.Path) -> pathlib.Path | None:
|
||||
"""Where the file named path is kept: path itself, else its gzip; None where neither is."""
|
||||
# A plain file beside its gzip was written after it was compressed, so it holds the newer bytes. Whatever else stands at path, such as a folder, is read as path, so the read names it.
|
||||
return next((kept for kept in (path, gzipped(path)) if kept.exists()), None)
|
||||
|
||||
|
||||
def read_stored(path: pathlib.Path) -> bytes:
|
||||
"""The bytes of the file named path, read from its gzip where it is kept that way. Raises OSError, FileNotFoundError where neither is kept."""
|
||||
kept = stored(path)
|
||||
if kept is None:
|
||||
raise FileNotFoundError(2, "No such file or directory", str(path))
|
||||
if kept == path:
|
||||
return path.read_bytes()
|
||||
try:
|
||||
return gzip.decompress(kept.read_bytes())
|
||||
except (EOFError, zlib.error) as failure:
|
||||
raise OSError(errno.EIO, f"not a whole gzip: {failure}", str(kept)) from failure
|
||||
|
||||
|
||||
def stored_glob(folder: pathlib.Path, pattern: str) -> list[pathlib.Path]:
|
||||
"""The plain name of each file in folder that pattern matches, whether kept plain or gzipped, sorted."""
|
||||
return sorted({*folder.glob(pattern), *(path.with_name(path.name.removesuffix(SUFFIX)) for path in folder.glob(pattern + SUFFIX))})
|
||||
|
||||
|
||||
def compress(path: pathlib.Path) -> Failure | None:
|
||||
"""Replaces path with its gzip, which keeps path's times: writes the gzip, reads it back, and removes path only once it reads back path's bytes. A failure keeps path."""
|
||||
packed = gzipped(path)
|
||||
try:
|
||||
before = path.stat()
|
||||
data = path.read_bytes()
|
||||
if unwritten := replace_file(packed, gzip.compress(data, mtime=0)):
|
||||
return Failure(f"could not compress {path}: {unwritten.reason}")
|
||||
if gzip.decompress(packed.read_bytes()) != data:
|
||||
return Failure(f"could not compress {path}: {packed} reads back other bytes")
|
||||
os.utime(packed, ns=(before.st_atime_ns, before.st_mtime_ns))
|
||||
after = path.stat()
|
||||
# Written while it compressed: the plain file keeps the newer bytes, which readers prefer, and a later run compresses them.
|
||||
if (after.st_size, after.st_mtime_ns) == (before.st_size, before.st_mtime_ns):
|
||||
path.unlink()
|
||||
except (OSError, EOFError, zlib.error) as failure:
|
||||
return Failure(f"could not compress {path}: {failure}")
|
||||
return None
|
||||
|
||||
|
||||
def gzipped(path: pathlib.Path) -> pathlib.Path:
|
||||
return path.with_name(path.name + SUFFIX)
|
||||
@@ -0,0 +1,87 @@
|
||||
import gzip
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from failure import Failure
|
||||
from text_files import compressed
|
||||
|
||||
|
||||
class Compress(unittest.TestCase):
|
||||
def test_replaces_the_file_with_a_gzip_of_the_same_bytes_and_times(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
plain = pathlib.Path(root, "t.jsonl")
|
||||
plain.write_bytes(b'{"a": 1}\n')
|
||||
os.utime(plain, (1000, 2000))
|
||||
self.assertIsNone(compressed.compress(plain))
|
||||
self.assertFalse(plain.exists())
|
||||
packed = pathlib.Path(root, "t.jsonl.gz")
|
||||
self.assertEqual(gzip.decompress(packed.read_bytes()), b'{"a": 1}\n')
|
||||
self.assertEqual(packed.stat().st_mtime, 2000)
|
||||
|
||||
def test_a_copy_that_reads_back_other_bytes_keeps_the_original_and_fails(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
plain = pathlib.Path(root, "t.jsonl")
|
||||
plain.write_bytes(b"kept\n")
|
||||
with mock.patch.object(compressed.gzip, "compress", return_value=gzip.compress(b"other\n")):
|
||||
failed = compressed.compress(plain)
|
||||
self.assertIsInstance(failed, Failure)
|
||||
self.assertIn(str(plain), failed.reason)
|
||||
self.assertEqual(plain.read_bytes(), b"kept\n")
|
||||
|
||||
def test_a_file_written_while_it_compressed_stays_plain(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
plain = pathlib.Path(root, "t.jsonl")
|
||||
plain.write_bytes(b"one\n")
|
||||
packing = compressed.gzip.compress
|
||||
|
||||
def grow_meanwhile(data, **kwargs):
|
||||
with open(plain, "ab") as handle:
|
||||
handle.write(b"two\n")
|
||||
return packing(data, **kwargs)
|
||||
|
||||
with mock.patch.object(compressed.gzip, "compress", grow_meanwhile):
|
||||
self.assertIsNone(compressed.compress(plain))
|
||||
self.assertEqual(plain.read_bytes(), b"one\ntwo\n")
|
||||
self.assertEqual(compressed.read_stored(plain), b"one\ntwo\n")
|
||||
|
||||
|
||||
class Read(unittest.TestCase):
|
||||
def test_reads_a_file_kept_plain_or_gzipped_by_its_plain_name(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
plain = pathlib.Path(root, "a.jsonl")
|
||||
plain.write_bytes(b"a\n")
|
||||
pathlib.Path(root, "b.jsonl.gz").write_bytes(gzip.compress(b"b\n"))
|
||||
self.assertEqual(compressed.read_stored(plain), b"a\n")
|
||||
self.assertEqual(compressed.read_stored(pathlib.Path(root, "b.jsonl")), b"b\n")
|
||||
self.assertEqual(compressed.stored(pathlib.Path(root, "b.jsonl")), pathlib.Path(root, "b.jsonl.gz"))
|
||||
self.assertIsNone(compressed.stored(pathlib.Path(root, "c.jsonl")))
|
||||
with self.assertRaises(FileNotFoundError):
|
||||
compressed.read_stored(pathlib.Path(root, "c.jsonl"))
|
||||
|
||||
def test_the_plain_file_wins_over_an_older_gzip_beside_it(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
pathlib.Path(root, "a.jsonl").write_bytes(b"a\nmore\n")
|
||||
pathlib.Path(root, "a.jsonl.gz").write_bytes(gzip.compress(b"a\n"))
|
||||
self.assertEqual(compressed.read_stored(pathlib.Path(root, "a.jsonl")), b"a\nmore\n")
|
||||
|
||||
def test_a_truncated_gzip_fails_as_an_os_error(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
pathlib.Path(root, "a.jsonl.gz").write_bytes(gzip.compress(b"a\n" * 100)[:20])
|
||||
with self.assertRaises(OSError):
|
||||
compressed.read_stored(pathlib.Path(root, "a.jsonl"))
|
||||
|
||||
def test_globs_each_file_once_by_its_plain_name(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
for name in ("a.jsonl", "b.jsonl.gz", "c.jsonl", "c.jsonl.gz", "d.md"):
|
||||
pathlib.Path(root, name).write_bytes(b"")
|
||||
self.assertEqual(compressed.stored_glob(pathlib.Path(root), "*.jsonl"), [pathlib.Path(root, name) for name in ("a.jsonl", "b.jsonl", "c.jsonl")])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -22,6 +22,9 @@
|
||||
- judge/ holds each judge run an outcome names on its `Judge run:` line: the prompt the judge read and claude's
|
||||
json reply, named by the hash of that content (judge_run.JudgeRun). judge-replay reads the prompt back.
|
||||
|
||||
compress-archives gzips each .jsonl 30 days after it was last written, as <name>.jsonl.gz beside its plain name,
|
||||
and every reader reads it by that name (text_files.compressed); a plain file beside its gzip is the newer.
|
||||
|
||||
write.archive writes the layout, at every interrupt and as every cycle ends. read.py and the tools that
|
||||
measure finished chunks read it.
|
||||
"""
|
||||
|
||||
@@ -29,5 +29,5 @@ def outcomes_of(copy: pathlib.Path) -> pathlib.Path:
|
||||
|
||||
|
||||
def transcript_of(outcomes: pathlib.Path) -> pathlib.Path:
|
||||
"""The transcript copy an outcomes file sits beside; outcomes_of is its inverse."""
|
||||
"""The plain name of the transcript copy an outcomes file sits beside, which text_files.compressed reads whether gzipped or not; outcomes_of is its inverse."""
|
||||
return outcomes.with_name(outcomes.name.removesuffix(FILE_SUFFIX) + ".jsonl")
|
||||
|
||||
@@ -4,6 +4,7 @@ import pathlib
|
||||
from transcript_archive.location import chunk_folder, FILE_SUFFIX, transcript_of
|
||||
from failure import Failure
|
||||
from transcript_archive.outcomes import FINISHED_AT, spent_stamps, workers
|
||||
from text_files.compressed import read_stored, stored_glob
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
@@ -32,7 +33,7 @@ def finished_chunks(project_archive: pathlib.Path) -> list[FinishedChunk] | Fail
|
||||
copies.append((stamps[-1], outcomes))
|
||||
if copies:
|
||||
finished, outcomes = max(copies)
|
||||
found.append(FinishedChunk(folder.name, finished, transcript_of(outcomes), sorted(folder.glob("session/*/subagents/*.jsonl"))))
|
||||
found.append(FinishedChunk(folder.name, finished, transcript_of(outcomes), subagent_transcripts(folder)))
|
||||
return sorted(found, key=lambda chunk: (chunk.finished, chunk.folder))
|
||||
|
||||
|
||||
@@ -44,6 +45,11 @@ def archived_outcomes(root: pathlib.Path) -> list[tuple[str, pathlib.Path]] | Fa
|
||||
return Failure(f"could not read the archive {root}: {failure}")
|
||||
|
||||
|
||||
def subagent_transcripts(folder: pathlib.Path) -> list[pathlib.Path]:
|
||||
"""The plain name of each subagent's transcript archived in a chunk's folder, kept plain or gzipped, sorted."""
|
||||
return [path for session in sorted(folder.glob("session/*/subagents")) for path in stored_glob(session, "*.jsonl")]
|
||||
|
||||
|
||||
def chunk_outcomes(folder: pathlib.Path) -> list[pathlib.Path]:
|
||||
"""The outcomes file of each transcript copy in a chunk's folder, sorted."""
|
||||
return sorted(folder.glob(f"*{FILE_SUFFIX}"))
|
||||
@@ -52,7 +58,7 @@ def chunk_outcomes(folder: pathlib.Path) -> list[pathlib.Path]:
|
||||
def read_threads(chunk: FinishedChunk) -> list[bytes] | Failure:
|
||||
"""The worker's transcript, then each subagent's."""
|
||||
try:
|
||||
return [path.read_bytes() for path in (chunk.transcript, *chunk.subagents)]
|
||||
return [read_stored(path) for path in (chunk.transcript, *chunk.subagents)]
|
||||
except OSError as failure:
|
||||
return Failure(f"could not read {failure.filename}: {failure.strerror}")
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@ import unittest
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
|
||||
|
||||
from transcript_archive.project_key import checkout_of, project_key
|
||||
from text_files import compressed
|
||||
from failure import Failure
|
||||
from transcript_archive.judge_run import JudgeRun
|
||||
from transcript_archive import outcomes, read, write
|
||||
@@ -119,6 +120,47 @@ class CopyFor(unittest.TestCase):
|
||||
self.assertIsNone(write.copy_for(copy, b"second"))
|
||||
self.assertEqual(write.copy_for(copy, b"first, grown"), copy)
|
||||
|
||||
def test_reads_a_compressed_copy_as_the_kept_version(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
copy = pathlib.Path(temp, "agent-a.jsonl")
|
||||
copy.write_bytes(b"first\n")
|
||||
self.assertIsNone(compressed.compress(copy))
|
||||
self.assertIsNone(write.copy_for(copy, b"first\n"))
|
||||
self.assertEqual(write.copy_for(copy, b"first\nsecond\n"), copy)
|
||||
self.assertEqual(write.copy_for(copy, b"other\n"), copy.with_name(f"agent-a.jsonl.{hashlib.sha256(b'other' + bytes([10])).hexdigest()}"))
|
||||
|
||||
|
||||
class Compressed(unittest.TestCase):
|
||||
"""The archive, its transcript copies and subagents' transcripts gzipped as compress-archives leaves them."""
|
||||
|
||||
def archived(self, root: str) -> pathlib.Path:
|
||||
kept = pathlib.Path(root, "archive")
|
||||
transcript = pathlib.Path(root, "t.jsonl")
|
||||
transcript.write_text('{"timestamp": "2026-10-03T00:00:00Z"}\n')
|
||||
subagent = pathlib.Path(root, "t", "subagents", "agent-a.jsonl")
|
||||
subagent.parent.mkdir(parents=True)
|
||||
subagent.write_text('{"sub": 1}\n')
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.finished("w", "merged", "j"), NOW))
|
||||
for copy in kept.rglob("*.jsonl"):
|
||||
self.assertIsNone(compressed.compress(copy))
|
||||
return transcript
|
||||
|
||||
def test_a_compressed_copy_takes_the_next_outcome_and_is_not_copied_again(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
transcript = self.archived(root)
|
||||
kept = pathlib.Path(root, "archive")
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertIsNone(write.archive(kept, str(transcript), "Do it.", outcomes.stopped("w", "Ctrl-C"), NOW + 1))
|
||||
self.assertEqual([path.name.endswith(".gz") for path in kept.rglob("*.jsonl*")], [True, True])
|
||||
self.assertIn("Stopped: Ctrl-C", next(kept.glob("*/*.outcomes.md")).read_text())
|
||||
|
||||
def test_reads_a_finished_chunk_whose_copies_are_compressed(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
self.archived(root)
|
||||
(chunk,) = read.finished_chunks(pathlib.Path(root, "archive"))
|
||||
self.assertEqual(read.read_threads(chunk), [b'{"timestamp": "2026-10-03T00:00:00Z"}\n', b'{"sub": 1}\n'])
|
||||
|
||||
|
||||
class Outcomes(unittest.TestCase):
|
||||
def test_the_archive_stamps_what_it_reads_back(self):
|
||||
|
||||
@@ -7,6 +7,7 @@ from transcript_archive.location import chunk_folder, outcomes_of
|
||||
from failure import Failure
|
||||
from transcript_archive.judge_run import JudgeRun
|
||||
from text_files.file_writes import make_private, replace_file
|
||||
from text_files.compressed import read_stored, stored
|
||||
from claude_code.session import session_files
|
||||
from transcript_archive.outcomes import TASK
|
||||
from text_files.utc_stamp import utc
|
||||
@@ -29,7 +30,7 @@ def archive(project_archive: pathlib.Path, path: str, task: str, outcome: str, a
|
||||
copy = folder / f"{hashlib.sha256(raw).hexdigest()}.jsonl"
|
||||
outcomes = outcomes_of(copy)
|
||||
stamped = f"{utc(at)} {outcome}"
|
||||
if copy.exists() and outcomes.exists():
|
||||
if stored(copy) and outcomes.exists():
|
||||
return add_outcome(outcomes, stamped)
|
||||
# The copy lands last, so a copy on disk always has its outcomes and its session's files beside it.
|
||||
for dest, data in ((outcomes, f"{TASK}{task}\n\n{stamped}\n".encode()), (copy, raw)):
|
||||
@@ -41,7 +42,7 @@ def archive(project_archive: pathlib.Path, path: str, task: str, outcome: str, a
|
||||
|
||||
def keep_judge_run(folder: pathlib.Path, run: JudgeRun) -> Failure | None:
|
||||
kept = folder / run.file
|
||||
if kept.exists():
|
||||
if stored(kept):
|
||||
return None
|
||||
try:
|
||||
make_private(kept.parent)
|
||||
@@ -93,11 +94,12 @@ def files_under(folder: pathlib.Path) -> list[pathlib.Path]:
|
||||
|
||||
|
||||
# Claude Code rewrites some files whole, such as custom-title.json, so a copy is replaced only by a file that grew from it.
|
||||
# A grown copy lands plain beside its gzip, which compress-archives then replaces losing nothing, since the gzip holds a prefix of it.
|
||||
def copy_for(copy: pathlib.Path, data: bytes) -> pathlib.Path | None:
|
||||
if not copy.is_file():
|
||||
if not stored(copy):
|
||||
return copy
|
||||
kept = copy.read_bytes()
|
||||
kept = read_stored(copy)
|
||||
if data.startswith(kept):
|
||||
return copy if data != kept else None
|
||||
beside = copy.with_name(f"{copy.name}.{hashlib.sha256(data).hexdigest()}")
|
||||
return None if beside.exists() else beside
|
||||
return None if stored(beside) else beside
|
||||
|
||||
+16
-1
@@ -12,6 +12,7 @@ Each project principle states one rule that the decisions below apply; their ent
|
||||
|
||||
# Decisions no project principle covers
|
||||
|
||||
- archives: loop-claude's archive and the review folders keep every file, and gzip each one 30 days after it was last written
|
||||
- bin/: a test parametrizes its cases with `subTest`, since pytest's `parametrize` doesn't reach a unittest `TestCase`
|
||||
- bin/: the tools run on the host's Python, with no container, and so do their tests outside CI
|
||||
- bin/: the tools send no OTLP traces, logs or metrics, and loop-claude's archive holds the trace of each cycle
|
||||
@@ -58,6 +59,20 @@ Settling a call is the human's. The chunk that settles the calls cites, in `docs
|
||||
|
||||
# Decisions
|
||||
|
||||
## archives: loop-claude's archive and the review folders keep every file, and gzip each one 30 days after it was last written
|
||||
|
||||
2026-11-17, the worker of todo item 306. Serves goal 3, backing up the transcripts, and goal 1, whose measures read them: the routing figures, `reading-cost`, `score-outcomes` and both replays rest on the whole archive, so nothing is pruned. Measured on 2026-11-11, the archive held 6.3 GB and `~/.local/state/larv-review` 997 MB, both growing with every chunk.
|
||||
|
||||
- **What.** `compress-archives`, run daily from the crontab line in `README.md` → "Install", gzips each file last written over 30 days ago, as `<name>.gz` beside its plain name with the same times. In the archive it takes the transcripts, `*.jsonl`, which held 7.8 GB of its 7.9 GB on 2026-11-17; the outcomes files stay plain, since loop-claude appends to them every cycle. In the review folders it takes every file, since the probes' build output fills most of them.
|
||||
- **Safely.** It writes the gzip, reads it back, and removes the plain file only once the gzip reads back the same bytes and the plain file is unchanged since it was read. At the first failure it stops and exits 1, keeping that file plain, as a failed archive stops loop-claude.
|
||||
- **Readers.** Every tool reads a file by its plain name through `bin/text_files/compressed.py`, and a plain file beside its gzip wins, since it was written later. A session file that grows after its copy was compressed lands plain beside the gzip, which holds a prefix of it, so the next run's gzip loses nothing. A file rewritten whole in a review folder replaces what it held, as it did before.
|
||||
- **Why gzip.** Python 3.14's `compression.zstd` needs a Python built with libzstd, and this machine's lacks `_zstd`; gzip is in every build (goal 2). On 400 archived transcripts older than 30 days it kept 23% of their 609 MB, at about 29 MB a second.
|
||||
- **Why 30 days.** A chunk's files stop changing once it merges, and chunks run for hours or days. Since every reader reads both forms, the threshold sets only how long a file stays plain.
|
||||
- **Backup.** Both folders sit under `$HOME`, on the filesystem that holds `~/code` and `~/.claude` (ai-sandbox's `/dev/vda3` on 2026-11-17), so backing that volume up, as item 1 waits for, covers both. `README.md` → "Install" names them, since the review folders lie outside `~/.claude`.
|
||||
- **One writer.** Each file keeps one writer: the plain file the tool that writes it, its gzip `compress-archives`.
|
||||
|
||||
Valid while every reader of the two folders reads through `text_files/compressed.py`.
|
||||
|
||||
## bin/: a test parametrizes its cases with `subTest`, since pytest's `parametrize` doesn't reach a unittest `TestCase`
|
||||
|
||||
2026-11-12, lilleman, from the release review of 0.9.0. Applies `principles/technical-principles.md` → "Most code should have automated tests", whose "parametrize the cases" asks for one reported result per case. The tests are plain unittest, run by pytest, and pytest's `parametrize` doesn't work on a `TestCase` method; `subTest` is the unittest mechanism that does: each case reports and fails on its own, with its parameters named. A test that loops over its cases itself, outside `subTest`, or holds a branch, is still a breach. Valid while the tests stay in plain unittest.
|
||||
@@ -184,7 +199,7 @@ Valid while claude takes `--session-id` and resumes by it, and the human runs lo
|
||||
|
||||
## loop-claude: the archive keeps the raw transcript and each judge run, and a failed archive stops the loop
|
||||
|
||||
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. The judge's runs are archived too, as `judge/<sha256>.json` in the chunk's folder: the prompt it read and its json reply. Together they are its whole transcript, since it runs with no tools and one turn. A run is kept whole and deduplicated by hash, and the outcome of the verdict, interrupt or rejected reply it produced names it on a `Judge run:` line. Keeping it whole costs little: a prompt is at most `JUDGE_PROMPT_CHARS`, 400,000 characters. Measured 2026-11-17, the archive's 911 judge lines come to at most 0.36 GB of prompts, against 3.5 GB in its 1,211 transcript copies, the largest 23.4 MB. The judge is still priced from its `Judge cost:` line, which is claude's own total for the run, and the Spent block counts the live reply's tokens; the archived reply keeps its `usage` for a later reader, and nothing reads it back. `judge-replay` replays a verdict on its archived prompt where there is one. Valid while transcripts, and the judge's prompts, stay small enough to copy whole.
|
||||
2026-09-28, lilleman. Serves goals 1 and 3: saving each transcript locally is the first step of backing it up, and it shows afterwards what an unattended run did. A copy of the `.jsonl`, deduplicated by content hash, with its outcomes beside it; a rendering drops tool results, and a second-resolution stamp names two states within one second as one file. A failed archive exits 1, since a loop that loses its record silently fails the backup goal. The judge's runs are archived too, as `judge/<sha256>.json` in the chunk's folder: the prompt it read and its json reply. Together they are its whole transcript, since it runs with no tools and one turn. A run is kept whole and deduplicated by hash, and the outcome of the verdict, interrupt or rejected reply it produced names it on a `Judge run:` line. Keeping it whole costs little: a prompt is at most `JUDGE_PROMPT_CHARS`, 400,000 characters. Measured 2026-11-17, the archive's 911 judge lines come to at most 0.36 GB of prompts, against 3.5 GB in its 1,211 transcript copies, the largest 23.4 MB. The judge is still priced from its `Judge cost:` line, which is claude's own total for the run, and the Spent block counts the live reply's tokens; the archived reply keeps its `usage` for a later reader, and nothing reads it back. `judge-replay` replays a verdict on its archived prompt where there is one. Valid while transcripts, and the judge's prompts, stay small enough to copy whole; `archives: loop-claude's archive and the review folders keep every file, and gzip each one 30 days after it was last written` keeps the copies small on disk.
|
||||
|
||||
## loop-claude: each verdict records the claude-config release and commit that ran it, the raw tokens each thread spent by model and kind since the previous outcome that recorded a spend, and the human's typed answer the judge quotes as reversing a choice the worker took on its own recommendation
|
||||
|
||||
|
||||
@@ -65,7 +65,7 @@ Give a reviewer without Bash `git ls-files`'s output in its prompt, since some h
|
||||
|
||||
`~/.claude/bin/larv-review` holds this section's mechanical rules: the chunk's review folder, the round count, the paired ruling's base and vote count, the scoring run's trigger; the history's format, its latest places and its count of how often seats agree; and the header of each reviewer's prompt. Run it in the chunk's worktree; `~/.claude/bin/larv-review --help` states each rule. On exit 1, repair what its message names and run it again; when you cannot, ask the user.
|
||||
|
||||
**Reports.** Run `~/.claude/bin/larv-review reviewer-header <pass>` in the chunk's worktree, and open each prompt to an agent under `~/.claude/agents/` with what it prints; then say in a few lines what the change does and which files it touches. The command makes the chunk's review folder and writes the branch diff and the plan's listing into it. The header names that diff, the reviewer's full report, `<pass>-round-<n>.md`, in the folder, the folders of technical principles and review standards the project's config sets, the listing, and for architecture and stability the places too. For a reviewer whose definition starts it without `CLAUDE.md`, the header's last line names the file that holds the project principles and decision titles, which the reviewer reads first. For a relaunch one rung up, add `--rerun`: the report becomes `<pass>-round-<n>-rerun.md`, and the header names the report that asked. Before you answer a 🔴 or 🟡, read its entry: Grep the report for the finding's place, with the lines after it, rather than Read the whole report. rule-hook refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`. Keep the folder after the merge: `review-replay` replays a round from the reports its prompt names.
|
||||
**Reports.** Run `~/.claude/bin/larv-review reviewer-header <pass>` in the chunk's worktree, and open each prompt to an agent under `~/.claude/agents/` with what it prints; then say in a few lines what the change does and which files it touches. The command makes the chunk's review folder and writes the branch diff and the plan's listing into it. The header names that diff, the reviewer's full report, `<pass>-round-<n>.md`, in the folder, the folders of technical principles and review standards the project's config sets, the listing, and for architecture and stability the places too. For a reviewer whose definition starts it without `CLAUDE.md`, the header's last line names the file that holds the project principles and decision titles, which the reviewer reads first. For a relaunch one rung up, add `--rerun`: the report becomes `<pass>-round-<n>-rerun.md`, and the header names the report that asked. Before you answer a 🔴 or 🟡, read its entry: Grep the report for the finding's place, with the lines after it, rather than Read the whole report. rule-hook refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`. Keep the folder after the merge: `review-replay` replays a round from the reports its prompt names, and `compress-archives` gzips each file 30 days after it was last written.
|
||||
|
||||
**Precedent, then ask** — for the full scan:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user