Merge pull request 'Measure each chunk's reading cost from loop-claude's archive and keep it in its own record' (#225) from reading-cost into main

This commit was merged in pull request #225.
This commit is contained in:
2026-10-08 19:29:58 +02:00
22 changed files with 796 additions and 17 deletions
+3
View File
@@ -25,6 +25,9 @@
!bin/loop_claude/**/*.py
!bin/markdown_sections.py
!bin/rank_todo.py
!bin/reading-cost
!bin/reading_cost/
!bin/reading_cost/*.py
!bin/statusline.py
!bin/test_*.py
!docs/
+2
View File
@@ -24,6 +24,8 @@ Decisions live in `docs/decisions.md`:
- settings.json: Claude Code runs in auto mode with the dangerous-mode prompt skipped
- the repo may name bender, bender-infra and ai-sandbox
- todo.md: one table of items scored by the formula and goal weights it states, held to its bar, ranked by `bin/rank_todo.py`
- reading cost: larv-review measures each finished chunk from loop-claude's archive and appends the figure to `docs/reading-cost.md` in the next chunk's PR
- reading cost: a rise is the last 5 chunks' median above the 5 before them, and files or refreshes one plan item
## Working rules
+2 -1
View File
@@ -2,9 +2,10 @@
## Unreleased
- larv-review runs the new `bin/reading-cost`, which measures each chunk `loop-claude` finished from its archive: the new input tokens the worker and its subagents took in before the worker's first commit, and the files git tracks that they read most. It appends each figure the project's `docs/reading-cost.md` lacks. Once 10 chunks are measured, and the last 5 chunks' median is above the 5 before them, larv-review files one plan item naming the files read most, or refreshes it. The chunk that ships it runs `reading-cost --reset`, so the trend counts only chunks that finished after it.
- larv-review appends every scoring run to the project's `docs/comprehension-history.md`: the commit the seats read, each seat's five scores and the seats' mean, and the places each seat named. The ratchet compares a run with the history's last run on the default branch. The `comprehension floor:` entry in `docs/decisions.md` keeps only the decisions to ship under the floor; a project's next scoring run moves the last scores the entry still holds into the history. The seats do not read the history.
- The README's goals put quality with the fewest interrupts first, as goal 1; sharing the setup, backups and leaking nothing follow as goals 2–4.
- larv-review reruns the comprehension scoring run before merge when any commit since the last run changes more than `docs/decisions.md`, `docs/comprehension-history.md`, `todo.md` or `CHANGELOG.md`, so the recorded scores are the merged commit's. A score under the floor, or below the default branch's recorded scores, blocks the merge and is never filed in `todo.md`. A fix lifts either; your decision to ship under the floor lifts only a score under the floor. A rerun counts toward the three-round cap only when it reports a blocking score that no decision of yours covers.
- larv-review reruns the comprehension scoring run before merge when any commit since the last run changes more than `docs/decisions.md`, `docs/comprehension-history.md`, `docs/reading-cost.md`, `todo.md` or `CHANGELOG.md`, so the recorded scores are the merged commit's. A score under the floor, or below the default branch's recorded scores, blocks the merge and is never filed in `todo.md`. A fix lifts either; your decision to ship under the floor lifts only a score under the floor. A rerun counts toward the three-round cap only when it reports a blocking score that no decision of yours covers.
- larv-review skips the comprehension scoring run for a chunk whose plan items a project decision names. The decision also names the later item whose scoring run ends the suspension. Neither the comprehension floor nor the ratchet can block a skipped chunk, its summary cites the decision, and a chunk shipping its release's last item is always measured.
- README → Install and Update give the commands that check the latest tag out into `~/.claude`: on a new machine, whether or not Claude Code has already created the directory, and on one already on an older tag.
- A session commits changes Claude Code made to `settings.json` only once their diff holds no credential, and asks you to move any credential out of the file first.
+1
View File
@@ -67,6 +67,7 @@ Rename `## Unreleased` in `CHANGELOG.md` to the version, commit, tag that commit
- `loop-claude <task>` runs a project's next chunk cycle after cycle; `loop-claude --help` states its contract.
- `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format.
- `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure.
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
Test on the host, with `uv` on `PATH` to fetch mypy and the Python version Install names (`docs/decisions.md` → `bin/`): `python3 -m unittest discover -s bin -p 'test_*.py'`
+2 -2
View File
@@ -2,7 +2,7 @@ import dataclasses
import typing
from loop_claude.cycle.states import Active, Archived, Closing, Ended, Event, HumanAnswered, Interrupting, Judging, QUOTED_ANSWERS_PER_CYCLE, Ruled, State, transition, WorkerRan, Working
from loop_claude.cycle.archive import archive
from loop_claude.transcript_archive import archive, FINISHED, UNFINISHED
from loop_claude.claude_code.session import delete_session, Session, session_open, transcript_file
from loop_claude.effects import Effects
from failure import Failure
@@ -109,7 +109,7 @@ def question_for(ruling: Interrupt | QuotedAnswer) -> str:
def close(cycle: Cycle, transcript_path: str, verdict: Finished | Unfinished) -> Archived | Stop:
finished = isinstance(verdict, Finished)
if failure := archive(cycle.project_key, transcript_path, cycle.task, f"Verdict: {'finished' if finished else 'unfinished'}\n\nEvidence:\n{quoted(verdict.evidence)}", cycle.effects.now(), cycle.paths):
if failure := archive(cycle.project_key, transcript_path, cycle.task, f"{FINISHED if finished else UNFINISHED}\n\nEvidence:\n{quoted(verdict.evidence)}", cycle.effects.now(), cycle.paths):
return Stop(1, failure.reason)
if finished and (failure := delete_session(cycle.session)):
return Stop(1, failure.reason)
@@ -1,6 +1,8 @@
import dataclasses
import hashlib
import os
import pathlib
import re
import sys
import time
@@ -10,6 +12,20 @@ from file_writes import make_private, replace_file
from loop_claude.paths import Paths
from loop_claude.claude_code.session import session_files
FINISHED = "Verdict: finished"
UNFINISHED = "Verdict: unfinished"
STAMP = "%Y-%m-%dT%H:%M:%SZ"
STAMPED = r"\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ"
FINISHED_AT = re.compile(rf"^({STAMPED}) {re.escape(FINISHED)}$", re.MULTILINE)
@dataclasses.dataclass(frozen=True)
class FinishedChunk:
folder: str
finished: str
transcript: pathlib.Path
subagents: list[pathlib.Path]
def archive(project_key: str, path: str, task: str, outcome: str, at: float, paths: Paths) -> Failure | None:
try:
@@ -25,7 +41,7 @@ def archive(project_key: str, path: str, task: str, outcome: str, at: float, pat
return unkept
copy = folder / f"{hashlib.sha256(raw).hexdigest()}.jsonl"
outcomes = copy.with_suffix(".outcomes.md")
stamped = f"{time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime(at))} {outcome}"
stamped = f"{utc(at)} {outcome}"
if copy.exists() and outcomes.exists():
return add_outcome(outcomes, stamped)
# The copy lands last, so a copy on disk always has its outcomes and its session's files beside it.
@@ -85,3 +101,21 @@ def copy_for(copy: pathlib.Path, data: bytes) -> pathlib.Path | None:
return copy if data != kept else None
beside = copy.with_name(f"{copy.name}.{hashlib.sha256(data).hexdigest()}")
return None if beside.exists() else beside
def utc(at: float) -> str:
return time.strftime(STAMP, time.gmtime(at))
# One per chunk folder, its latest finished copy, in the order they finished.
def finished_chunks(project_archive: pathlib.Path) -> list[FinishedChunk] | Failure:
found = []
try:
for folder in sorted(path for path in project_archive.glob("*") if path.is_dir()):
copies = sorted((stamps[-1], outcomes) for outcomes in folder.glob("*.outcomes.md") if (stamps := FINISHED_AT.findall(outcomes.read_text(encoding="utf-8"))))
if copies:
finished, outcomes = copies[-1]
found.append(FinishedChunk(folder.name, finished, outcomes.with_name(outcomes.name.removesuffix(".outcomes.md") + ".jsonl"), sorted(folder.glob("session/*/subagents/*.jsonl"))))
except (OSError, UnicodeDecodeError) as failure:
return Failure(f"could not read the archive {project_archive}: {failure}")
return sorted(found, key=lambda chunk: (chunk.finished, chunk.folder))
+9
View File
@@ -0,0 +1,9 @@
#!/usr/bin/env python3
import pathlib
import sys
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from reading_cost import main
sys.exit(main.main(sys.argv[1:]))
View File
+20
View File
@@ -0,0 +1,20 @@
import pathlib
import typing
from failure import Failure
from loop_claude.transcript_archive import FinishedChunk
from loop_claude.paths import paths_from
from loop_claude.project.checkout import project_key
def project_archive(checkout_root: str, home: pathlib.Path, env: typing.Mapping[str, str]) -> pathlib.Path:
paths = paths_from(home, env)
return paths.transcript_archive / project_key(checkout_root, paths.code)
def read_threads(chunk: FinishedChunk) -> list[bytes] | Failure:
"""The worker's transcript, then each subagent's."""
try:
return [path.read_bytes() for path in (chunk.transcript, *chunk.subagents)]
except OSError as failure:
return Failure(f"could not read {failure.filename}: {failure.strerror}")
+71
View File
@@ -0,0 +1,71 @@
import collections
import re
import statistics
from failure import Failure
from loop_claude.transcript_archive import FinishedChunk, STAMPED
from reading_cost.measure import Reading
from markdown_sections import sections
FIGURE = re.compile(rf"^Reading cost, chunk (\w+), finished ({STAMPED})$")
PLACE = re.compile(r"^- `(.+)` (\d+)$")
PLACES_KEPT = 10
RESET = "Reading cost reset, "
RESET_AT = re.compile(rf"^{RESET}({STAMPED})$")
TOKENS = re.compile(r"^(\d+) new input tokens before the first commit\.$")
WINDOW = 5
def unheld(chunks: list[FinishedChunk], record: str) -> list[FinishedChunk]:
held = {match.group(1) for heading in sections(record) if (match := FIGURE.match(heading))}
return [chunk for chunk in chunks if chunk_id(chunk) not in held]
def chunk_id(chunk: FinishedChunk) -> str:
return chunk.folder[:12]
def figure(chunk: FinishedChunk, reading: Reading) -> str:
places = "".join(f"- `{path}` {times}\n" for path, times in ranked(reading.places))
return f"## Reading cost, chunk {chunk_id(chunk)}, finished {chunk.finished}\n\n{reading.tokens} new input tokens before the first commit.\n" + (f"\nRead most (opens and searches):\n\n{places}" if places else "")
# The figures after the last reset, in the order their chunks finished: each machine appends its own archive's chunks.
def readings_of(record: str) -> list[Reading] | Failure:
dated = []
resets = [""]
for heading, bodies in sections(record).items():
if reset := RESET_AT.match(heading):
resets.append(reset.group(1))
elif named := FIGURE.match(heading):
lines = bodies[0].splitlines() if bodies else []
tokens = [int(match.group(1)) for line in lines if (match := TOKENS.match(line))]
if len(tokens) != 1:
return Failure(f"the figure under \"## {heading}\" holds no single tokens line; restore it from git history")
dated.append((named.group(2), heading, Reading(tokens[0], collections.Counter({match.group(1): int(match.group(2)) for line in lines if (match := PLACE.match(line))}))))
elif heading.startswith("Reading cost"):
return Failure(f"\"## {heading}\" names no chunk and UTC time, or no reset time; restore it from git history")
return [reading for finished, _, reading in sorted(dated, key=lambda item: item[:2]) if finished > max(resets)]
def ranked(places: collections.Counter[str]) -> list[tuple[str, int]]:
return sorted(places.items(), key=lambda item: (-item[1], item[0]))[:PLACES_KEPT]
def merged(record: str, figures: list[str]) -> str:
return record.rstrip("\n") + "\n\n" + "\n".join(figures)
def trend(readings: list[Reading]) -> str:
if len(readings) < 2 * WINDOW:
return f"{len(readings)} chunks measured since the last reset; the trend needs {2 * WINDOW}."
last, earlier = median_tokens(readings[-WINDOW:]), median_tokens(readings[-2 * WINDOW:-WINDOW])
medians = f"the last {WINDOW} chunks' median is {last} new input tokens before their first commit, the {WINDOW} before them {earlier}."
if last <= earlier:
return f"Not rising: {medians}"
read_most = "".join(f"\n- `{path}` {times}" for path, times in ranked(sum((reading.places for reading in readings[-WINDOW:]), collections.Counter())))
return f"Rising: {medians}\n\nRead most over the last {WINDOW} (opens and searches):\n{read_most}"
def median_tokens(readings: list[Reading]) -> int:
return round(statistics.median(reading.tokens for reading in readings))
+132
View File
@@ -0,0 +1,132 @@
"""Measure each chunk loop-claude finished, append the figures to a project's record, and say whether they rise.
usage: reading-cost [--reset] [record]
Run it inside a project checkout or one of its worktrees. record defaults to docs/reading-cost.md
there, created when missing. It reads loop-claude's archive of the project,
$LOOP_CLAUDE_TRANSCRIPTS/<key> (default ~/.claude/loop-claude-transcripts), <key> being the
checkout's path relative to ~/code, or to / outside it. For each finished chunk the record does not
yet hold, it measures what the worker and its subagents read before the worker's first commit: their
new input tokens (uncached and cache-written; cache reads excluded), and how often they opened or
searched each file git tracks in the checkout. A chunk whose worker never committed is not measured.
Each figure is appended under a "## Reading cost, chunk <id>, finished <UTC time>" heading, in the
order the chunks finished.
--reset also appends a "## Reading cost reset, <UTC time>" heading: the trend then counts only
figures of chunks that finished after it.
It then compares the median tokens of the record's last 5 figures with the 5 before them, in the
order their chunks finished, and prints "Rising:" or "Not rising:" with both medians. Rising, it
lists the files the last 5 chunks read most. With fewer than 10 figures, it says so.
Exit 1, saying why, on unknown arguments, or when git cannot start or finds no checkout, the
archive or the record cannot be read or written, or a figure in the record cannot be read back.
"""
import dataclasses
import os
import pathlib
import subprocess
import sys
import time
from reading_cost.measure import Checkout, measure, Reading
from failure import Failure
from reading_cost.figures import figure, merged, readings_of, RESET, trend, unheld
from loop_claude.transcript_archive import finished_chunks, utc
from reading_cost.archive import project_archive, read_threads
from file_writes import replace_file
@dataclasses.dataclass(frozen=True)
class Updated:
appended: int
readings: list[Reading]
def main(argv: list[str]) -> int:
reset = argv[:1] == ["--reset"]
given = argv[1:] if reset else argv
if given in (["-h"], ["--help"]):
print(__doc__.strip())
return 0
if len(given) > 1 or given[:1] and given[0].startswith("-"):
print(f"unknown arguments {' '.join(given)}; usage: reading-cost [--reset] [record]", file=sys.stderr)
return 1
checkout = located()
if isinstance(checkout, Failure):
return failed(checkout)
home = pathlib.Path.home()
archive = project_archive(checkout.root, home, os.environ)
record = pathlib.Path(given[0]) if given else pathlib.Path(checkout.top, "docs", "reading-cost.md")
updated = update(record, archive, checkout, str(home), utc(time.time()) if reset else None)
if isinstance(updated, Failure):
return failed(updated)
print(f"Appended {updated.appended} reading-cost figure{'' if updated.appended == 1 else 's'} from {archive}{' and a reset' if reset else ''} to {record}.")
print(trend(updated.readings))
return 0
def update(record: pathlib.Path, archive: pathlib.Path, checkout: Checkout, home: str, reset_at: str | None) -> Updated | Failure:
text = read_record(record)
if isinstance(text, Failure):
return text
figures = new_figures(text, archive, checkout, home)
if isinstance(figures, Failure):
return figures
marks = [*figures, f"## {RESET}{reset_at}\n"] if reset_at else figures
text = merged(text, marks) if marks else text
readings = readings_of(text)
if isinstance(readings, Failure):
return readings
if marks and (unwritten := write_record(record, text)):
return unwritten
return Updated(len(figures), readings)
def new_figures(record: str, archive: pathlib.Path, checkout: Checkout, home: str) -> list[str] | Failure:
chunks = finished_chunks(archive)
if isinstance(chunks, Failure):
return chunks
figures = []
for chunk in unheld(chunks, record):
threads = read_threads(chunk)
if isinstance(threads, Failure):
return threads
if reading := measure(threads, checkout, home):
figures.append(figure(chunk, reading))
return figures
def write_record(record: pathlib.Path, text: str) -> Failure | None:
try:
record.parent.mkdir(parents=True, exist_ok=True)
except OSError as failure:
return Failure(f"could not write {record}: {failure.strerror}")
unwritten = replace_file(record, text.encode())
return Failure(f"could not write {record}: {unwritten.reason}") if unwritten else None
def failed(failure: Failure) -> int:
print(failure.reason, file=sys.stderr)
return 1
def located() -> Checkout | Failure:
try:
found = subprocess.run(["git", "rev-parse", "--path-format=absolute", "--show-toplevel", "--git-common-dir"], capture_output=True, text=True)
if found.returncode != 0:
return Failure(found.stderr.strip())
top, common = found.stdout.splitlines()
listed = subprocess.run(["git", "ls-files", "-z"], capture_output=True, cwd=top, text=True)
except (OSError, ValueError) as failure:
return Failure(f"could not find the checkout: {failure}")
if listed.returncode != 0:
return Failure(listed.stderr.strip())
# The common dir is the checkout's .git, whichever worktree runs this.
return Checkout(str(pathlib.Path(common).parent), top, frozenset(filter(None, listed.stdout.split("\0"))))
def read_record(record: pathlib.Path) -> str | Failure:
try:
return record.read_text(encoding="utf-8") if record.exists() else "# Reading cost\n"
except (OSError, UnicodeDecodeError) as failure:
return Failure(f"could not read {record}: {failure}")
+80
View File
@@ -0,0 +1,80 @@
import collections
import dataclasses
import re
from loop_claude.claude_code.entries import assistant_parts, parse_entry, tool_call
from reading_cost.shell_reads import Reads, resolved, shell_reads
COMMIT = re.compile(r"(^|[\s;&|(])git\s+(-C\s+\S+\s+)?commit\b")
@dataclasses.dataclass(frozen=True)
class Reading:
"""What the worker and its subagents took in before the worker's first commit."""
tokens: int
places: collections.Counter[str]
@dataclasses.dataclass(frozen=True)
class Checkout:
root: str
top: str
tracked: frozenset[str]
def measure(threads: list[bytes], checkout: Checkout, home: str) -> Reading | None:
worker, *subagents = [entries_of(thread) for thread in threads]
commit = commit_time(worker)
if commit is None:
return None
before = [[entry for entry in thread if entry.get("type") == "assistant" and str(entry.get("timestamp", "")) < commit] for thread in (worker, *subagents)]
reads = [reads_of(part, str(entry.get("cwd") or "/"), home) for thread in before for entry in thread for part in assistant_parts(entry)]
roots = [checkout.root, *(added for found in reads for added in found.added)]
places = collections.Counter(place for found in reads for path in found.read if (place := relative(path, roots)) in checkout.tracked)
return Reading(sum(map(tokens_in, before)), places)
def entries_of(raw: bytes) -> list[dict]:
return [entry for line in raw.splitlines() if (entry := parse_entry(line))]
def commit_time(worker: list[dict]) -> str | None:
for entry in worker:
for part in assistant_parts(entry):
name, arguments = call_of(part)
if name == "Bash" and COMMIT.search(str(arguments.get("command", ""))):
return str(entry.get("timestamp", ""))
return None
def tokens_in(entries: list[dict]) -> int:
# Claude Code writes one entry per content block of a message, each with the message's usage.
usages = {str(message.get("id") or entry.get("uuid")): message.get("usage") for entry in entries if isinstance(message := entry.get("message"), dict)}
return sum(count(usage, "input_tokens") + count(usage, "cache_creation_input_tokens") for usage in usages.values() if isinstance(usage, dict))
# A count Claude Code left out or wrote as another type adds nothing, so one odd entry leaves the rest of the figure standing.
def count(usage: dict, field: str) -> int:
value = usage.get(field)
return value if isinstance(value, int) else 0
def call_of(part: object) -> tuple[object, dict]:
call = tool_call(part)
return (call[0], call[1]) if call and isinstance(call[1], dict) else (None, {})
def reads_of(part: object, cwd: str, home: str) -> Reads:
name, arguments = call_of(part)
path = arguments.get("file_path") if name == "Read" else arguments.get("path") if name == "Grep" else None
if path:
return Reads([resolved(cwd, str(path), home)], [])
return shell_reads(str(arguments.get("command", "")), cwd, home) if name == "Bash" else Reads([], [])
# Relative to the deepest root holding it, so the checkout and each worktree name a file alike.
def relative(path: str, roots: list[str]) -> str | None:
holding = [root for root in roots if path.startswith(root.rstrip("/") + "/")]
return path[len(max(holding, key=len).rstrip("/")) + 1:] if holding else None
+91
View File
@@ -0,0 +1,91 @@
"""The files a shell command reads, and the worktrees it adds, as absolute paths."""
import dataclasses
import posixpath
import re
import shlex
import typing
NOT_A_PATH = re.compile(r"[*?\[\]{}$\\'\",=()<>|;&]|://|^\d+$")
# Their first operand is a pattern or a program, unless -e or -f gives it.
PATTERN_FIRST = {"awk", "egrep", "grep", "rg", "sed"}
READERS = {"awk", "bat", "cat", "egrep", "grep", "head", "less", "more", "nl", "rg", "sed", "tail", "wc"}
WORKTREE_OPTIONS_WITH_VALUE = {"-b", "-B", "--reason"}
@dataclasses.dataclass(frozen=True)
class Reads:
read: list[str]
added: list[str]
def shell_reads(command: str, cwd: str, home: str) -> Reads:
read: list[str] = []
added: list[str] = []
base = cwd
for tokens in shell_lines(command):
for words in segments(tokens):
if words[0] == "cd" and len(words) == 2:
base = resolved(base, words[1], home)
read += [resolved(base, word, home) for word in operands(words)]
added += [resolved(base, path, home) for path in worktree_added(words)]
return Reads(read, added)
def shell_lines(command: str) -> typing.Iterator[list[str]]:
"""Each line's tokens, skipping a heredoc's body, which feeds a program."""
ends = None
for line in command.splitlines():
if ends is not None:
ends = None if line.strip() == ends else ends
continue
try:
lexer = shlex.shlex(line, posix=True, punctuation_chars=True)
lexer.whitespace_split = True
tokens = list(lexer)
except ValueError:
continue
# The lexer splits <<-EOF into << and -EOF.
ends = next((tokens[index + 1].removeprefix("-") for index, token in enumerate(tokens[:-1]) if token == "<<"), None)
yield tokens
def segments(tokens: list[str]) -> list[list[str]]:
found: list[list[str]] = [[]]
target = False
for token in tokens:
if target:
target = False
elif all(char in "();<>|&" for char in token):
# A redirect's target is not an operand, so it does not count as a read.
target = "<" in token or ">" in token
found.append([])
else:
found[-1].append(token)
return [words for words in found if words]
def operands(words: list[str]) -> list[str]:
program = posixpath.basename(words[0])
if program not in READERS or (program == "sed" and any(word.startswith("-i") for word in words)):
return []
given = [word for word in words[1:] if not word.startswith("-")]
if program in PATTERN_FIRST and not {"-e", "-f"} & set(words):
given = given[1:]
return [word for word in given if ("/" in word or re.search(r"\.[A-Za-z0-9]+$", word)) and not NOT_A_PATH.search(word)]
# git worktree add [-b <branch>] <path> [<commit>]
def worktree_added(words: list[str]) -> list[str]:
if words[:1] != ["git"] or "worktree" not in words or words[words.index("worktree") + 1:][:1] != ["add"]:
return []
after = iter(words[words.index("worktree") + 2:])
for word in after:
if word in WORKTREE_OPTIONS_WITH_VALUE:
next(after, None)
elif not word.startswith("-"):
return [word]
return []
def resolved(base: str, path: str, home: str) -> str:
return posixpath.normpath(posixpath.join(base, home + path[1:] if path == "~" or path.startswith("~/") else path))
+2 -1
View File
@@ -16,12 +16,13 @@ import uuid
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from loop_claude.cycle import archive, interrupt, judge, retry, run, states, worker
from loop_claude import transcript_archive as archive
from loop_claude.project import checkout, model_choice, records
from loop_claude.effects import Effects
from loop_claude.claude_code import entries, session, transcript, trust
from failure import Failure
from loop_claude.cycle.judge_reply import Finished, Interrupt, QuotedAnswer, Unfinished, verdict_from
from loop_claude.cycle import interrupt, judge, retry, run, states, worker
from loop_claude.cycle.retry import Limited
from loop_claude.loop import main, reload
from loop_claude import paths
+6 -1
View File
@@ -18,6 +18,11 @@ import unittest
BIN = pathlib.Path(__file__).resolve().parent
TASK = "Do the next chunk."
# A git hook's GIT_DIR, the host's LOOP_CLAUDE_* or PYTHONPATH would point the program under test at the host's own state.
def host_env() -> dict[str, str]:
return {name: value for name, value in os.environ.items() if not name.startswith(("GIT_", "LOOP_CLAUDE_", "PYTHON"))}
# Plays claude from FAKE_CLAUDE_SCENARIO, one step per call, logging each call; a step for another role,
# or none left, is Ctrl-C to loop-claude, which otherwise loops forever.
FAKE_CLAUDE = """#!{python}
@@ -155,7 +160,7 @@ class Checkout:
self.root.mkdir(parents=True)
(self.root / "README.md").write_text(PREMISES)
self.script = BIN / "loop-claude"
self.env = {name: value for name, value in os.environ.items() if not name.startswith(("GIT_", "LOOP_CLAUDE_", "PYTHON"))}
self.env = host_env()
fake = self.temp / "fake-bin"
fake.mkdir()
(fake / "claude").write_text(FAKE_CLAUDE.format(python=sys.executable))
+161
View File
@@ -0,0 +1,161 @@
import collections
import contextlib
import hashlib
import io
import json
import os
import pathlib
import subprocess
import sys
import tempfile
import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from loop_claude.transcript_archive import archive, FINISHED, FINISHED_AT, finished_chunks, FinishedChunk, UNFINISHED, utc
from reading_cost.measure import Checkout, Reading
from failure import Failure
from reading_cost import figures, measure
from test_loop_claude_black_box import host_env
from loop_claude.paths import paths_from
from reading_cost.shell_reads import Reads, shell_reads
TOOL = pathlib.Path(__file__).resolve().parent / "reading-cost"
def said(timestamp, message_id, command=None, tokens=0, cwd="/c/p", **call):
part = {"type": "tool_use", "name": "Bash", "input": {"command": command}} if command else {"type": "tool_use", **call}
return {"type": "assistant", "timestamp": timestamp, "cwd": cwd, "message": {"id": message_id, "content": [part], "usage": {"input_tokens": tokens, "cache_creation_input_tokens": 2 * tokens, "cache_read_input_tokens": 1000}}}
def jsonl(entries):
return b"".join(json.dumps(entry).encode() + b"\n" for entry in entries)
def folder_name():
return hashlib.sha256(os.urandom(8)).hexdigest()
class ShellReads(unittest.TestCase):
def test_reads_operands_past_a_cd_and_skips_patterns_redirect_targets_and_heredoc_bodies(self):
command = "cd ../w && sed -n 1,5p src/a.py | grep -n x\ngrep -rn 'def x' bin/t.py > out.txt\npython3 - <<'EOF'\ncat secret.py\nEOF\ncat ~/README.md\nsed -i s/a/b/ c.py"
self.assertEqual(shell_reads(command, "/c/p", "/h"), Reads(["/c/w/src/a.py", "/c/w/bin/t.py", "/h/README.md"], []))
def test_ends_a_dashed_heredoc_at_its_marker(self):
self.assertEqual(shell_reads("cat <<-EOF > x.md\ncat secret.py\nEOF\ncat a.py", "/c/p", "/h"), Reads(["/c/p/a.py"], []))
def test_names_the_worktree_a_git_worktree_add_makes(self):
self.assertEqual(shell_reads("git worktree add -q -b topic ../p-topic origin/main", "/c/p", "/h"), Reads([], ["/c/p-topic"]))
class Measure(unittest.TestCase):
def test_counts_each_message_once_and_the_tracked_files_read_before_the_first_commit(self):
worker = [
said("1", "m1", tokens=10, name="Read", input={"file_path": "/c/p/README.md"}),
said("2", "m1", tokens=10, name="Read", input={"file_path": "/c/p/.env"}),
said("3", "m2", command="git worktree add ../w && cd ../w && cat src/a.py && cd src && cat b.py"),
said("4", "m3", command="cd /tmp && cat x.py"),
said("5", "m4", command="git -C ../w commit -m 'x'", tokens=100),
said("6", "m5", tokens=100, name="Read", input={"file_path": "/c/p/late.py"}),
]
subagent = [said("2", "s1", tokens=1, name="Grep", input={"path": "/c/w/src/a.py"}), said("7", "s2", tokens=100, name="Read", input={"file_path": "/c/p/late.py"})]
checkout = Checkout("/c/p", "/c/w", frozenset({"README.md", "late.py", "src/a.py", "src/b.py", "x.py"}))
self.assertEqual(measure.measure([jsonl(worker), jsonl(subagent)], checkout, "/h"), Reading(33, collections.Counter({"src/a.py": 2, "README.md": 1, "src/b.py": 1})))
def test_measures_nothing_for_a_worker_that_never_committed(self):
self.assertIsNone(measure.measure([jsonl([said("1", "m1", command="git log --grep commit")])], Checkout("/c/p", "/c/p", frozenset()), "/h"))
def test_skips_a_usage_that_is_not_a_count(self):
self.assertEqual(measure.tokens_in([{"message": {"id": "m", "usage": {"input_tokens": "7", "cache_creation_input_tokens": 2}}}]), 2)
class Figures(unittest.TestCase):
def test_rises_only_above_the_five_before_and_needs_ten(self):
flat = [Reading(100, collections.Counter({"a.py": 1}))] * 10
self.assertEqual(figures.trend(flat[:9]), "9 chunks measured since the last reset; the trend needs 10.")
self.assertTrue(figures.trend(flat).startswith("Not rising: the last 5 chunks' median is 100 new input tokens"))
self.assertTrue(figures.trend(flat[:9] + [Reading(3000000, collections.Counter())]).startswith("Not rising:"))
self.assertEqual(figures.trend(flat[:5] + [Reading(101, collections.Counter({"b.py": 2}))] * 5), "Rising: the last 5 chunks' median is 101 new input tokens before their first commit, the 5 before them 100.\n\nRead most over the last 5 (opens and searches):\n\n- `b.py` 10")
def test_a_figure_reads_back_as_the_reading_it_was_written_from_and_a_damaged_one_fails(self):
reading = Reading(7, collections.Counter({"b.py": 1, "a.py": 3}))
written = figures.figure(FinishedChunk("abcdef0123456789", "2026-10-08T00:00:00Z", pathlib.Path(), []), reading)
self.assertEqual(written, "## Reading cost, chunk abcdef012345, finished 2026-10-08T00:00:00Z\n\n7 new input tokens before the first commit.\n\nRead most (opens and searches):\n\n- `a.py` 3\n- `b.py` 1\n")
self.assertEqual(figures.readings_of(written), [reading])
self.assertEqual(figures.readings_of(written.replace("7 new", "seven new")), Failure('the figure under "## Reading cost, chunk abcdef012345, finished 2026-10-08T00:00:00Z" holds no single tokens line; restore it from git history'))
for damaged in (written.replace(", finished", ""), "## Reading cost reset, x\n"):
with self.subTest(damaged=damaged):
self.assertIsInstance(figures.readings_of(damaged), Failure)
def test_counts_only_figures_finished_after_the_last_reset_in_the_order_they_finished(self):
def written(chunk, finished, tokens):
return figures.figure(FinishedChunk(chunk, finished, pathlib.Path(), []), Reading(tokens, collections.Counter()))
history = "\n".join([written("a", "2026-10-01T00:00:00Z", 1), written("c", "2026-10-03T00:00:00Z", 3), "## Reading cost reset, 2026-10-02T00:00:00Z\n", written("b", "2026-10-04T00:00:00Z", 2)])
self.assertEqual(figures.readings_of(history), [Reading(3, collections.Counter()), Reading(2, collections.Counter())])
def test_the_archive_stamps_what_it_reads_back(self):
self.assertRegex(f"{utc(1790000000)} {FINISHED}", FINISHED_AT)
class ReadingCostTool(unittest.TestCase):
def setUp(self):
temp = tempfile.TemporaryDirectory()
self.addCleanup(temp.cleanup)
self.home = pathlib.Path(os.path.realpath(temp.name))
self.env = {**host_env(), "GIT_CONFIG_GLOBAL": os.devnull, "HOME": str(self.home)}
self.checkout = self.home / "code" / "proj"
self.checkout.mkdir(parents=True)
(self.checkout / "a.py").write_text("")
self.git("init", "-q")
self.git("add", "a.py")
self.git("-c", "user.name=t", "-c", "user.email=t@t", "commit", "-q", "-m", "x")
self.paths = paths_from(self.home, {})
def git(self, *args):
subprocess.run(["git", *args], cwd=self.checkout, check=True, env=self.env)
def archive_chunk(self, at, outcome=FINISHED):
transcript = self.home / ".claude" / "projects" / "-proj" / f"{folder_name()}.jsonl"
(transcript.with_suffix("") / "subagents").mkdir(parents=True)
worker = [said("1", "m1", tokens=10, cwd=str(self.checkout), name="Read", input={"file_path": str(self.checkout / "a.py")}), said("2", "m2", command="git commit -m x")]
transcript.write_bytes(jsonl([{"timestamp": str(at)}, *worker]))
(transcript.with_suffix("") / "subagents" / "agent-a.jsonl").write_bytes(jsonl([said("1", "s1", tokens=1)]))
with contextlib.redirect_stderr(io.StringIO()):
self.assertIsNone(archive("proj", str(transcript), "Do it.", f"{outcome}\n\nEvidence:\n> merged", at, self.paths))
def run_tool(self, cwd, *args):
return subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, cwd=cwd, env=self.env, text=True)
def test_appends_each_finished_chunk_once_in_the_order_they_finished_from_a_worktree(self):
self.archive_chunk(1790000200)
self.archive_chunk(1790000100)
self.archive_chunk(1790000000, UNFINISHED)
chunks = finished_chunks(self.paths.transcript_archive / "proj")
self.assertEqual([chunk.finished for chunk in chunks], ["2026-09-21T14:15:00Z", "2026-09-21T14:16:40Z"])
worktree = self.home / "code" / "proj-wt"
self.git("worktree", "add", "-q", str(worktree))
ran = self.run_tool(worktree)
self.assertEqual(ran.returncode, 0, ran.stderr)
history = worktree / "docs" / "reading-cost.md"
self.assertEqual(ran.stdout, f"Appended 2 reading-cost figures from {self.home / '.claude' / 'loop-claude-transcripts' / 'proj'} to {history}.\n2 chunks measured since the last reset; the trend needs 10.\n")
figure = "## Reading cost, chunk {}, finished {}\n\n33 new input tokens before the first commit.\n\nRead most (opens and searches):\n\n- `a.py` 1\n"
self.assertEqual(history.read_text(), "# Reading cost\n\n" + "\n".join(figure.format(chunk.folder[:12], chunk.finished) for chunk in chunks))
self.assertTrue(self.run_tool(worktree).stdout.startswith(f"Appended 0 reading-cost figures from"))
self.assertEqual(history.read_text().count("## Reading cost"), 2)
reset = self.run_tool(worktree, "--reset")
self.assertEqual(reset.stdout, f"Appended 0 reading-cost figures from {self.home / '.claude' / 'loop-claude-transcripts' / 'proj'} and a reset to {history}.\n0 chunks measured since the last reset; the trend needs 10.\n")
self.assertRegex(history.read_text(), r"\n## Reading cost reset, \d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ\n$")
def test_refuses_an_option_it_does_not_know(self):
ran = self.run_tool(self.checkout, "--rest")
self.assertEqual((ran.returncode, ran.stdout), (1, ""))
def test_exits_1_outside_a_checkout(self):
ran = self.run_tool(self.home)
self.assertEqual(ran.returncode, 1)
self.assertIn("not a git repository", ran.stderr)
if __name__ == "__main__":
unittest.main()
+1 -1
View File
@@ -11,5 +11,5 @@ class Types(unittest.TestCase):
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
if shutil.which("uv") is None:
self.fail("uv on PATH fetches mypy; see README.md's Test line")
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "-m", "rank_todo", "-m", "statusline", "-p", "loop_claude"], capture_output=True, cwd=ROOT / "bin", text=True)
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "-m", "rank_todo", "-p", "reading_cost", "-m", "statusline", "-p", "loop_claude"], capture_output=True, cwd=ROOT / "bin", text=True)
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
+153
View File
@@ -109,3 +109,156 @@ Inherited architect, would restructure first:
- `bin/loop_claude/project/checkout.py:32` `read_utf8`
- `agents/stability-reviewer.md:3`, `agents/systems-architect.md:3`: house style
- `workflow-bench/bench.py:26`: not runnable from the repo
## 2026-10-08T14:28:39Z, PR #225 at 13d8b16
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|---|---|---|---|---|---|
| Junior A | 7 | 6 | 7 | 6 | 6.5 |
| Mid A | 7 | 6.5 | 7 | 6 | 6.5 |
| Maintainability senior | 7.5 | 7 | 7.5 | 7 | 7 |
| Inherited architect | 7 | 6.5 | 6.5 | 7 | 7 |
| Mean | 7.13 | 6.50 | 7.00 | 6.50 | 6.75 |
Junior A, hardest first:
- `skills/larv-review/SKILL.md:79` "Architecture": Floor, Ratchet, Fresh, Suspended, and the history paragraph at `:95`
- `skills/larv-review/SKILL.md:53` "Round cap", with `:64` "Re-entry"
- `bin/loop_claude/cycle/judge.py:31` `run_judge`, with `grounded`, `verbatim` and `plain`
- `bin/loop_claude/cycle/states.py:69` `transition`, with `cycle/run.py:55` `step` and `:104` `question_for`
- `bin/loop_claude/claude_code/entries.py:28` `written_by_human`
- `bin/loop_claude/cycle/archive.py:28` `archive`, with `copy_for` and `finished_chunks`
- `bin/reading_cost.py:145` `measure`, with `bin/shell_reads.py:52` `segments` and `:67` `operands`
- `bin/rank_todo.py:408` `rank_details`
Mid A, hardest first:
- `skills/larv-review/SKILL.md:53` "Round cap", with "History", "Ratchet", "Fresh" and "Suspended"
- `bin/loop_claude/cycle/judge_reply.py:12` `JUDGE_RULES`, with `QUOTE_SOURCES`, `ANSWER_DESTINATIONS` and `ANSWERS_RECORDED`
- `bin/loop_claude/cycle/states.py:69` `transition`
- `bin/loop_claude/cycle/archive.py:28` `archive`, with `finished_chunks` and `FINISHED_AT`
- `bin/reading_cost.py:145` `measure`, with `located`, `relative` and `commit_time`
- `bin/shell_reads.py:67` `operands`
- `bin/loop_claude/claude_code/entries.py:28` `written_by_human`
- `bin/rank_todo.py:408` `rank_details`
Maintainability senior, hardest first:
- `skills/larv-review/SKILL.md:79` "Architecture"
- `bin/loop_claude/cycle/judge_reply.py:12` `JUDGE_RULES`, with `cycle/worker.py:41` `worker_args`
- `bin/loop_claude/cycle/judge.py:31` `run_judge`
- `bin/loop_claude/claude_code/entries.py:28` `written_by_human`
- `bin/loop_claude/cycle/archive.py:17` `FINISHED_AT`, with `finished_chunks` and `cycle/run.py:110` `close`
- `bin/reading_cost.py:145` `measure`, with `located`
- `bin/shell_reads.py:67` `operands`
- `bin/rank_todo.py:408` `rank_details`, with `read_scoring`
Inherited architect, would restructure first:
- `skills/larv-review/SKILL.md:53` "Round cap", and `:88` "Architecture" from History to Suspended
- `bin/loop_claude/cycle/archive.py:109` `finished_chunks`, with `bin/reading_cost.py:106` `located` and `bin/loop_claude/project/checkout.py:7` `checkout_root`
- `bin/` flat folder: tools, shared modules and tests
- `bin/loop_claude/project/checkout.py:32` `read_utf8`, `claude_code/transcript.py:27` `render`, `judge_reply.py:87` `Verdict` beside `states.py:24` `ruling`
- `bin/loop_claude/loop/main.py:9` the recorded-answers rule, beside `judge_reply.py:11` `ANSWERS_RECORDED`
## 2026-10-08T15:54:46Z, PR #225 at 1fc91a5
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|---|---|---|---|---|---|
| Junior A | 7 | 6 | 6.5 | 6 | 6.5 |
| Mid A | 7 | 6.5 | 7 | 6.5 | 7 |
| Maintainability senior | 7.5 | 6.5 | 7 | 6.5 | 7 |
| Inherited architect | 7 | 6 | 6.5 | 7 | 6.5 |
| Mean | 7.13 | 6.25 | 6.75 | 6.50 | 6.75 |
Junior A, hardest first:
- `skills/larv-review/SKILL.md:53` "Round cap", through "Re-entry", "Fresh" and "Suspended"
- `bin/loop_claude/cycle/states.py:69` `transition`, with `cycle/run.py:55` `step`
- `bin/reading_cost.py:180` `measure`
- `bin/reading_cost.py:242` `readings_of`, with `:171` `unheld` and `:236` `figure`
- `bin/loop_claude/cycle/judge.py:31` `run_judge`
- `bin/loop_claude/transcript_archive.py:29` `archive`, with `copy_for` and `finished_chunks`
- `bin/loop_claude/cycle/judge_reply.py:12` `JUDGE_RULES`
- `bin/rank_todo.py:408` `rank_details`
Mid A, hardest first:
- `skills/larv-review/SKILL.md:53` "Round cap", with "Architecture" from Floor to Suspended
- `bin/loop_claude/cycle/judge.py:31` `run_judge`
- `bin/loop_claude/cycle/states.py:69` `transition`
- `bin/reading_cost.py:180` `measure`, and `:242` `readings_of`
- `bin/loop_claude/transcript_archive.py:29` `archive`, `:95` `copy_for` and `:110` `finished_chunks`
- `bin/loop_claude/claude_code/entries.py:28` `written_by_human`
- `bin/loop_claude/loop/reload.py:50` `exec_if_edited`
- `bin/rank_todo.py:408` `rank_details`
Maintainability senior, hardest first:
- `skills/larv-review/SKILL.md:79` "Architecture"
- `skills/larv-review/SKILL.md:95` the move of an old entry's scores into the history
- `bin/loop_claude/cycle/judge_reply.py:12` `JUDGE_RULES`
- `bin/loop_claude/cycle/judge.py:31` `run_judge`
- `bin/reading_cost.py:180` `measure`, and `:242` `readings_of`
- `bin/loop_claude/transcript_archive.py:18` `FINISHED_AT`, with `finished_chunks` and `cycle/run.py:110` `close`
- `bin/reading_cost.py:141` `located`, beside `bin/loop_claude/project/checkout.py:7` `checkout_root`
- `bin/shell_reads.py:52` `segments`, and `:67` `operands`
Inherited architect, would restructure first:
- `bin/` by kind: tools, shared modules and tests
- `bin/loop_claude/transcript_archive.py:110` `finished_chunks`, read by `bin/reading_cost.py` through four `loop_claude` modules
- `docs/comprehension-history.md`, written by `bin/reading_cost.py:242` and by `skills/larv-review/SKILL.md:92`'s prose
- `skills/larv-review/SKILL.md:53` "Round cap"
- `bin/loop_claude/cycle/judge_reply.py:87` `Verdict` beside `states.py` `ruling`, and `project/checkout.py:32` `read_utf8`
## 2026-10-08T17:29:52Z, PR #225 at 8c9bf09
| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
|---|---|---|---|---|---|
| Junior A | 7 | 6.5 | 7 | 6.5 | 6.5 |
| Mid A | 7 | 7 | 7 | 6.5 | 7 |
| Maintainability senior | 7.5 | 6.5 | 7 | 7 | 7 |
| Inherited architect | 7.5 | 7 | 7 | 7.5 | 7 |
| Mean | 7.25 | 6.75 | 7.00 | 6.88 | 6.88 |
Junior A, hardest first:
- `bin/loop_claude/cycle/judge_reply.py:12` `JUDGE_RULES`, with `Reply` and `verdict_from`
- `bin/loop_claude/cycle/judge.py:31` `run_judge`, with `grounded`, `verbatim` and `plain`
- `bin/loop_claude/cycle/states.py:69` `transition`
- `bin/loop_claude/transcript_archive.py:30` `archive`, `copy_for` and `finished_chunks`
- `bin/reading_cost/measure.py:27` `measure`, with `bin/reading_cost/main.py:113` `located`
- `skills/larv-review/SKILL.md:95` the old entry's scores, and `:108` "Fresh"
- `bin/reading_cost/shell_reads.py:21` `shell_reads`, with `operands`, `segments` and `shell_lines`
- `bin/loop_claude/claude_code/entries.py:28` `written_by_human`
Mid A, hardest first:
- `skills/larv-review/SKILL.md:53` "Round cap", through "Architecture", "Fresh" and "Suspended"
- `bin/loop_claude/cycle/judge_reply.py:12` `JUDGE_RULES`, and `:90` `verdict_from`
- `bin/loop_claude/cycle/states.py:69` `transition`, with `cycle/run.py:35` `run_cycle` and `:104` `question_for`
- `bin/loop_claude/transcript_archive.py:30` `archive`, `copy_for` and `finished_chunks`
- `bin/loop_claude/claude_code/entries.py:28` `written_by_human`
- `bin/reading_cost/figures.py:34` `readings_of`
- `bin/reading_cost/shell_reads.py:52` `segments`, and `:67` `operands`
- `bin/reading_cost/main.py:113` `located`, beside `bin/loop_claude/project/checkout.py:7` `checkout_root`
Maintainability senior, hardest first:
- `bin/loop_claude/transcript_archive.py:30` `archive`, and `:111` `finished_chunks`
- `bin/reading_cost/figures.py:34` `readings_of`
- `bin/reading_cost/shell_reads.py:52` `segments`, and `:67` `operands`
- `bin/loop_claude/cycle/judge.py:31` `run_judge`
- `bin/reading_cost/main.py:113` `located`, and `measure.py:21` `Checkout`
- `skills/larv-review/SKILL.md:53` "Round cap", and `:108` "Fresh" and "Suspended"
- `bin/loop_claude/cycle/judge_reply.py:12` `JUDGE_RULES`
- `bin/rank_todo.py:408` `rank_details`
Inherited architect, would restructure first:
- `bin/loop_claude/claude_code/` and the read side of `bin/loop_claude/transcript_archive.py`, into a package both tools import
- `skills/larv-review/SKILL.md:58` "Count", with a checker
- `bin/loop_claude/cycle/judge_reply.py:12` `JUDGE_RULES`, beside `judge.py:53` `judge_prompt`
- `bin/loop_claude/project/checkout.py:7` `checkout_root` and `bin/reading_cost/main.py:113` `located`, and `checkout.py:32` `read_utf8`
- `bin/failure.py`, `bin/file_writes.py` and `bin/markdown_sections.py`, into one package
+9
View File
@@ -49,6 +49,7 @@ lilleman decided to ship each of these under the floor on the day listed. Each l
- 2026-10-03, PR #208: 7.5, 7.0, 7.13, 6.63, 6.88.
- 2026-10-04, PR #215: 7.38, 6.63, 7.0, 6.88, 6.88.
- 2026-10-08, PR #224: 7.00, 6.50, 7.00, 6.50, 6.75. lilleman shipped it below the ratchet as well: its 20 lines of skill prose left the code as it was, so the fall is the panel's noise between runs, which `todo.md` item 75 addresses.
- 2026-10-08, PR #225: 7.25, 6.75, 7.00, 6.88, 6.88. Above PR #224's run on every score but Shape, which held.
## loop-claude: each project's model lives in `~/.claude/loop-claude-models.json`, which only loop-claude writes
@@ -109,3 +110,11 @@ lilleman decided to ship each of these under the floor on the day listed. Each l
## the repo may name bender, bender-infra and ai-sandbox
2026-10-04, lilleman, answering the principles audit of PR #215. Serves goal 4, which keeps credentials, transcripts and memory files off the remote; the names of the host, its infrastructure project and the sandbox VM are none of those. Valid while the repo names them without their addresses or credentials.
## reading cost: larv-review measures each finished chunk from loop-claude's archive and appends the figure to `docs/reading-cost.md` in the next chunk's PR
2026-10-08, lilleman. Serves goal 1.1: comprehension measured from real work, at no model cost (goal 1.4). loop-claude archives from the main checkout, outside any PR, so a tracked file it wrote there would dirty that checkout. So `bin/reading-cost` measures each finished chunk `docs/reading-cost.md` lacks from the archive, in the next chunk's worktree. A PR that never merges then loses no figure, and loop-claude needs no change. The figures keep a file of their own, so `docs/comprehension-history.md` keeps one writer, the scoring run. Serves goal 4 by naming only files git tracks, and token counts. Rejected: loop-claude opening a PR of its own, one more PR per chunk and a tie to the forge; the worker measuring its own transcript before merging, which misses the chunk's tail. The figure counts new input tokens, uncached and cache-written, because workers read mostly through Bash and subagents, where parsed paths alone miss what they read. Valid while loop-claude archives every finished chunk with its subagents' transcripts.
## reading cost: a rise is the last 5 chunks' median above the 5 before them, and files or refreshes one plan item
2026-10-08, lilleman. Serves goal 1.1: comprehension never falls, so any rise counts. Per-chunk figures are noisy: a flat trend rises about every other chunk. So a rise refreshes the item and never refiles it, and a chunk that sees no rise leaves the item as it is. The chunk that ships the item resets the trend, so later figures do not refile it for work that chunk already did (goal 1.4). A median, because one chunk that ran a panel of readers before its first commit measured 2.99M tokens against a typical 80k–370k, and a mean let it decide the trend alone. No published standard sets this threshold. Valid while figures arrive once per chunk.
+1 -1
View File
@@ -27,7 +27,7 @@ You are one seat of a comprehension panel. Tools: Read, Grep, Glob, and Bash onl
```
Target: the project checked out at {checkout} (branch `{branch}`). Read ONLY there — never {main}, which is a different version.
Depth 1, the whole project: read README.md, AGENTS.md and CLAUDE.md (the charter), then every non-test tracked file: {files}. Do NOT open todo.md, docs/decisions.md or docs/comprehension-history.md (the plan, the decision log and past panels' scores), nor any review output. You may open a test file when the code does not answer a question; say when you did.
Depth 1, the whole project: read README.md, AGENTS.md and CLAUDE.md (the charter), then every non-test tracked file: {files}. Do NOT open todo.md, docs/decisions.md, docs/comprehension-history.md or docs/reading-cost.md (the plan, the decision log, past panels' scores and past chunks' reading cost), nor any review output. You may open a test file when the code does not answer a question; say when you did.
Your harness pre-loads these files into your context before your first tool call, from a DIFFERENT version (main): {preloaded}. Treat the worktree's own copies as the ones under review. A pre-loaded AGENTS.md may list decision titles; do not use them as evidence.
+3 -1
View File
@@ -24,6 +24,8 @@ Words: `~/.claude/CLAUDE.md` → "Vocabulary". What a reviewer reports: ~/.claud
Follow `~/.claude/CLAUDE.md` → "Git, commits, and PRs". Never commit to the default branch. Commit, push with upstream set, and open a **draft** PR so CI cycles while the review runs. No remote → review locally and say so in the summary.
**Reading cost.** Run `~/.claude/bin/reading-cost` in the chunk's worktree and commit what it appends to `docs/reading-cost.md`; the commit re-enters no pass. Resolve a merge conflict in that file by putting the default branch's figures first and dropping a figure for a chunk the default branch already holds. On `Rising:`, file a plan item titled `**Lower the reading cost of the files chunks read most before their first commit.**`, scored per `~/.claude/CLAUDE.md` → "Todo items" with the tool's medians and files as its detail; where the plan already holds it, refresh its detail instead. The item names the next release, or the one after when this chunk ships the next release's last item. On exit 1, repair what its message names and run it again; when the message names no repair, file it as a `question` item, or refresh the question item an earlier run filed, and go on. Leave the plan as it is on any other output. The chunk that ships that item runs the tool with `--reset` instead, so the next rise is measured on chunks that finished after it.
## 3. Passes
A pass is one kind of review, a row of the table, named by its first column. Its trigger says when it runs on a change it has not seen; its reviewer reports the findings.
@@ -103,7 +105,7 @@ Fix the places the seats name under the lowest dimension and under each dimensio
Put each run's five scores in the summary, and append the run to the history on this branch. List in the entry each decision to ship under the floor, and file its places in `todo.md` per `~/.claude/CLAUDE.md` → "Todo items". A project without the entry gets it at its first scoring run, titled `comprehension floor: every dimension and the overall at 7.0 or above` and applying KISS (`~/.claude/principles/technical-principles.md`).
**Fresh.** Where the chunk ran a scoring run, the architecture pass is done only once its last scoring run measured the commit to merge. That run may predate commits that change only `docs/decisions.md`, `docs/comprehension-history.md`, `todo.md` or `CHANGELOG.md`. Any other commit after it, a prose fix included, because the seats read the docs, calls for a rescore: the scoring run again, once every other pass is done, the auto-reviewer included. A rescore reporting a 🔴 that no decision to ship answers is an architecture round. Any other rescore is not a round: append only its run to the history, in a commit titled `Rescore: …`. A `Rescore:` commit re-enters no pass: push it and wait for the forge's checks (§4 step 3), not for an auto-review.
**Fresh.** Where the chunk ran a scoring run, the architecture pass is done only once its last scoring run measured the commit to merge. That run may predate commits that change only `docs/decisions.md`, `docs/comprehension-history.md`, `docs/reading-cost.md`, `todo.md` or `CHANGELOG.md`. Any other commit after it, a prose fix included, because the seats read the docs, calls for a rescore: the scoring run again, once every other pass is done, the auto-reviewer included. A rescore reporting a 🔴 that no decision to ship answers is an architecture round. Any other rescore is not a round: append only its run to the history, in a commit titled `Rescore: …`. A `Rescore:` commit re-enters no pass: push it and wait for the forge's checks (§4 step 3), not for an auto-review.
**Suspended.** A decision in the project's `docs/decisions.md`, held before the chunk began, may name plan items to ship without a scoring run. A named item counts as `todo.md` worded it before the chunk began. The decision must name the later plan item whose chunk ends the suspension with a scoring run. A chunk may ship named items only with each other. Such a chunk skips the scoring run, so neither the floor nor the ratchet can block it, and its summary cites the decision. A chunk shipping its release's last item runs the scoring run, even when every item it ships is named.
+12 -8
View File
@@ -6,7 +6,7 @@
`Bar = 9`
`Next ID = 83`
`Next ID = 84`
| Goal | W |
|---|---|
@@ -24,7 +24,6 @@
| ID | Release | Exempt | Item | R | S | A | G | Goals | Score |
|---|---|---|---|---|---|---|---|---|---|
| 79 | 0.1.0 | | **Measure each chunk's reading cost from its transcript as loop-claude archives it, append it to `docs/comprehension-history.md`, and file a plan item naming the most-read places when the trend rises.** | 2 | 4 | 10 | 9 | 1.1, 1.4 | 34.8 |
| 72 | 0.1.0 | | **Rerun after a fix only the pass whose finding it answers, over the fix's diff alone; close a pass on a round of only 🟢, and give prose one round.** | 4 | 4 | 10 | 9 | 1.4, 1.5 | 32.3 |
| 75 | 0.1.0 | decision | **Decide the ratchet by a paired run: Mid A and the inherited architect read the whole project at the default branch and at the PR head, and rule whether it reads better, the same or worse once the PR lands; add seats only until the votes settle.** | 5 | 5 | 10 | 9 | 1.1, 1.4 | 31.6 |
| 73 | 0.1.0 | | **Have each reviewer write its full report to a file and return the worker only its findings' one-line summaries.** | 3 | 4 | 10 | 8 | 1.4 | 31.4 |
@@ -38,6 +37,7 @@
| 59 | 0.1.0 | | **Bound a run of unfinished verdicts on one chunk, so a worker and judge that disagree cannot cycle unattended without end.** | 3 | 3 | 6 | 6 | 1.4 | 19.8 |
| 12 | 0.1.0 | | **Run the test suite in CI on every PR, or record in `docs/decisions.md` why the local run is enough.** | 2 | 4 | 4 | 6 | 1.1 | 16.9 |
| 80 | 0.1.0 | | **State in README → Install that every machine taking this checkout must be a sandbox, as `settings.json`'s auto mode assumes.** | 1 | 1 | 5 | 5 | 4 | 16.8 |
| 83 | 0.1.0 | | **Group `bin/` by tool, so each tool's modules and tests sit together.** | 3 | 4 | 4 | 6 | 1.1 | 15.9 |
| 5 | 0.1.0 | decision | **Key a project on its checkout's full path, and refuse to start while another session uses that key.** | 5 | 5 | 5 | 6 | 1.1 | 15.6 |
| 67 | 0.1.0 | principle | **Point stability-reviewer at the technical principles in place of its own restatement of them.** | 2 | 3 | 4 | 5 | 1.3 | 14.9 |
| 66 | 0.1.0 | principle | **Hold systems-architect's method thresholds only while the comprehension panel finds the flagged code hard.** | 3 | 3 | 4 | 5 | 1.1 | 14.2 |
@@ -55,10 +55,6 @@
## Details
### 79. Measure each chunk's reading cost from its transcript as loop-claude archives it, append it to `docs/comprehension-history.md`, and file a plan item naming the most-read places when the trend rises.
The human wants comprehension measured from real work. The archived transcript already records what the worker opened and searched before its first edit, and how much it read per line it changed; a script derives both at archive time with no model call. One chunk's figure is noise, since tasks differ; the trend over many chunks is the signal. The files read most often before an edit name where to act. Decide who commits the figure, and in which PR: loop-claude archives from the main checkout, outside any PR, and a tracked file it writes there dirties that checkout. Give these entries a heading the ratchet does not read as a scoring run.
### 72. Rerun after a fix only the pass whose finding it answers, over the fix's diff alone; close a pass on a round of only 🟢, and give prose one round.
The human ranks review cost first. An audit of the four costliest archived chunks (adf-codec, loop-claude's quoted answers, visthus networking, fejkdata on-demand loading; 2026-10-08) found nearly every finding that mattered in round 1 of its pass. Later rounds mostly re-checked the previous fix, flipped wording that earlier rounds wrote, or reported a bug the previous fix created; reviewers contradicted each other on the same CHANGELOG line across rounds. Estimated 55–75% of review effort in those chunks changed nothing a user would notice. Keep a round 2 where a fix touched behaviour: it caught real bugs the fix introduced.
@@ -93,7 +89,9 @@ The README line covers the bench's transcripts in `workflow-bench/logs/` too (go
### 62. Lift Locality, Self-sufficiency and the overall to the comprehension floor, at the places the seats name.
Raised by the principles audit of PR #215 against KISS, in `principles/technical-principles.md`, and goal 1.1: the `comprehension floor:` entry in `docs/decisions.md` lists seven decisions to ship under the floor, and 0.1.0 waits on the floor. PR #215's last run put Locality at 6.63 and Self-sufficiency and the overall at 6.88. Its seats named, under Locality: the judge's contract spread over `cycle/judge_reply.py`, `cycle/worker.py` and `cycle/run.py`; `remaining`, which is `Unfinished.evidence` under another name, threaded through `loop/main.py`, `loop/reload.py`'s `LoopRun`, `cycle/run.py` and `worker_args`; the `quoted_answers` cap held by case order in `cycle/states.py`'s `transition` and restated in `run.py`'s `question_for`; the four outcome texts `run.py` builds inline for `archive`. Under Self-sufficiency: `skills/larv-review/SKILL.md`, which needs `CLAUDE.md` → "Vocabulary" open beside it; the archive layout, defined only in `bin/loop_claude/loop/main.py`'s docstring and not at `cycle/archive.py`; `JUDGE_RULES`' precedence in `cycle/judge_reply.py`, restated by `verdict_from`'s branch order and the `Reply` field descriptions; the `sys.path` insert in `bin/loop-claude` that the `from failure import` lines rely on, unstated at the import sites.
Raised by the principles audit of PR #215 against KISS, in `principles/technical-principles.md`, and goal 1.1: the `comprehension floor:` entry in `docs/decisions.md` lists seven decisions to ship under the floor, and 0.1.0 waits on the floor. PR #215's last run put Locality at 6.63 and Self-sufficiency and the overall at 6.88. Its seats named, under Locality: the judge's contract spread over `cycle/judge_reply.py`, `cycle/worker.py` and `cycle/run.py`; `remaining`, which is `Unfinished.evidence` under another name, threaded through `loop/main.py`, `loop/reload.py`'s `LoopRun`, `cycle/run.py` and `worker_args`; the `quoted_answers` cap held by case order in `cycle/states.py`'s `transition` and restated in `run.py`'s `question_for`; the four outcome texts `run.py` builds inline for `archive`. Under Self-sufficiency: `skills/larv-review/SKILL.md`, which needs `CLAUDE.md` → "Vocabulary" open beside it; the archive layout, defined only in `bin/loop_claude/loop/main.py`'s docstring and not at `transcript_archive.py`; `JUDGE_RULES`' precedence in `cycle/judge_reply.py`, restated by `verdict_from`'s branch order and the `Reply` field descriptions; the `sys.path` insert in `bin/loop-claude` that the `from failure import` lines rely on, unstated at the import sites.
PR #225's last run, at 8c9bf09, put Locality at 6.75 and Self-sufficiency and the overall at 6.88. Its seats named: the judge's policy spread over `loop/main.py`'s docstring, `cycle/judge_reply.py`'s `JUDGE_RULES`, `cycle/judge.py`'s `judge_prompt` and `cycle/worker.py`'s `worker_args`; `loop_claude/claude_code/` and the read side of `transcript_archive.py`, which `bin/reading_cost/` also imports; two meanings of a checkout's root in `project/checkout.py`'s `checkout_root` and `reading_cost/main.py`'s `located`; and `skills/larv-review/SKILL.md` §3's round count, which only prose enforces.
### 21. Give `settings.json`'s model one writer, and sort its keys.
@@ -117,10 +115,16 @@ The repo has no CI, so larv-review's gate is the local run, and larv-review's Ga
Raised by the principles audit of PR #224. The `settings.json` decision is valid while every machine taking this checkout is a sandbox, but Install asks for a sandbox only for `loop-claude`, so a new machine is never told.
### 83. Group `bin/` by tool, so each tool's modules and tests sit together.
Raised by the architect on PR #225 and by the inherited architect of PR #224's scoring run: `bin/` is one flat folder of tools, the modules they share and the repo's checks, with `loop_claude`'s tests beside them instead of beside the package. `loop_claude/claude_code/` parses Claude Code's own formats and `bin/reading_cost/measure.py` imports it too, so it is a package of its own.
### 5. Key a project on its checkout's full path, and refuse to start while another session uses that key.
The key names the project's session, archive folder and models entry. `project_key` drops the leading `/`, so `~/code/a` and `/a` share all three, and one loop can resume or delete the other's session.
`bin/reading-cost` keys the archive by the main checkout, the parent of git's common dir, since it runs in a chunk's worktree; `loop-claude` keys it by `--show-toplevel`, so a loop started inside a worktree archives where `reading-cost` never looks. Derive both keys one way.
### 67. Point stability-reviewer at the technical principles in place of its own restatement of them.
DRY, in `principles/technical-principles.md`: `agents/stability-reviewer.md` restates principles in its own words, and the copy has drifted: it asks for "properly randomised UUIDs" where the principle requires UUID v7.
@@ -167,7 +171,7 @@ Claude Code owns that file and writes it. `trust` reads it, changes it and write
### 4. Name each chunk's archive folder by the chunk's start time, so `<key>/` lists chunks in the order they ran.
Today the folder is the SHA-256 of the transcript's first timestamped line, which tells the human nothing; that line carries the start time. Drop the fallback to the first line, which every chunk of a project shares, and give the tests' fake transcripts ISO timestamps.
Today the folder is the SHA-256 of the transcript's first timestamped line, which tells the human nothing; that line carries the start time. Drop the fallback to the first line, which every chunk of a project shares, and give the tests' fake transcripts ISO timestamps. `bin/reading-cost` takes a chunk's id from the folder's first 12 characters and skips a chunk whose id `docs/reading-cost.md` holds, so give it a way to match a figure written under the old name.
### 52. Record why this checkout lives in Claude Code's own directory, or move it out and link it in.