Answer architecture round 1: measure from loop-claude's archive in the CLI, file the reading-cost item only on a rise, keep only tracked files

This commit is contained in:
2026-10-08 16:10:23 +02:00
parent 1b8142e7e5
commit 13d8b167a7
17 changed files with 468 additions and 327 deletions
+1
View File
@@ -26,6 +26,7 @@
!bin/markdown_sections.py
!bin/rank_todo.py
!bin/reading_cost.py
!bin/shell_reads.py
!bin/statusline.py
!bin/test_*.py
!docs/
+2 -2
View File
@@ -24,8 +24,8 @@ Decisions live in `docs/decisions.md`:
- settings.json: Claude Code runs in auto mode with the dangerous-mode prompt skipped
- the repo may name bender, bender-infra and ai-sandbox
- todo.md: one table of items scored by the formula and goal weights it states, held to its bar, ranked by `bin/rank_todo.py`
- reading cost: loop-claude keeps each finished chunk's figure in its archive, and the next chunk's larv-review moves it into `docs/comprehension-history.md`
- reading cost: a rise is the last 5 chunks' mean above the 5 before them, and keeps one standing plan item
- reading cost: larv-review measures each finished chunk from loop-claude's archive and appends the figure to `docs/comprehension-history.md` in the next chunk's PR
- reading cost: a rise is the last 5 chunks' mean above the 5 before them, and files or refreshes one plan item that only its own chunk or the human removes
## Working rules
+1 -1
View File
@@ -2,7 +2,7 @@
## Unreleased
- `loop-claude` measures each finished chunk's reading cost: the tokens the worker and its subagents took in before the worker's first commit, and the project's files they read most. It keeps the figure in `$LOOP_CLAUDE_TRANSCRIPTS/<key>/reading-cost.md`. larv-review runs the new `bin/reading_cost.py`, which appends each figure the project's `docs/comprehension-history.md` lacks. When the last 5 chunks' mean is above the 5 before them, larv-review keeps one plan item naming the files read most; when not, it deletes that item.
- larv-review runs the new `bin/reading_cost.py`, which measures each chunk `loop-claude` finished from its archive: the tokens the worker and its subagents took in before the worker's first commit, and the files git tracks that they read most. It appends each figure the project's `docs/comprehension-history.md` lacks. When the last 5 chunks' mean is above the 5 before them, larv-review files one plan item naming the files read most, or refreshes it; only its own chunk or you remove it.
- larv-review appends every scoring run to the project's `docs/comprehension-history.md`: the commit the seats read, each seat's five scores and the seats' mean, and the places each seat named. The ratchet compares a run with the history's last run on the default branch. The `comprehension floor:` entry in `docs/decisions.md` keeps only the decisions to ship under the floor; a project's next scoring run moves the last scores the entry still holds into the history. The seats do not read the history.
- The README's goals put quality with the fewest interrupts first, as goal 1; sharing the setup, backups and leaking nothing follow as goals 2–4.
- larv-review reruns the comprehension scoring run before merge when any commit since the last run changes more than `docs/decisions.md`, `docs/comprehension-history.md`, `todo.md` or `CHANGELOG.md`, so the recorded scores are the merged commit's. A score under the floor, or below the default branch's recorded scores, blocks the merge and is never filed in `todo.md`. A fix lifts either; your decision to ship under the floor lifts only a score under the floor. A rerun counts toward the three-round cap only when it reports a blocking score that no decision of yours covers.
+1 -1
View File
@@ -67,7 +67,7 @@ Rename `## Unreleased` in `CHANGELOG.md` to the version, commit, tag that commit
- `loop-claude <task>` runs a project's next chunk cycle after cycle; `loop-claude --help` states its contract.
- `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`; `rank_todo.py --help` states the format.
- `reading_cost.py [history]` moves `loop-claude`'s reading-cost figures into a project's comprehension history and says whether they rise; `reading_cost.py --help` states the measure.
- `reading_cost.py [history]` measures the chunks `loop-claude` finished into a project's comprehension history and says whether the figures rise; `reading_cost.py --help` states the measure.
- `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it.
Test on the host, with `uv` on `PATH` to fetch mypy and the Python version Install names (`docs/decisions.md` → `bin/`): `python3 -m unittest discover -s bin -p 'test_*.py'`
+35 -7
View File
@@ -1,6 +1,8 @@
import dataclasses
import hashlib
import os
import pathlib
import re
import sys
import time
@@ -10,12 +12,25 @@ from file_writes import make_private, replace_file
from loop_claude.paths import Paths
from loop_claude.claude_code.session import session_files
FINISHED = "Verdict: finished"
UNFINISHED = "Verdict: unfinished"
FINISHED_AT = re.compile(rf"^(\S+) {re.escape(FINISHED)}$", re.MULTILINE)
@dataclasses.dataclass(frozen=True)
class FinishedChunk:
folder: str
finished: str
transcript: pathlib.Path
subagents: list[pathlib.Path]
def archive(project_key: str, path: str, task: str, outcome: str, at: float, paths: Paths) -> Failure | None:
try:
with open(path, "rb") as handle:
raw = handle.read()
folder = paths.transcript_archive / project_key / chunk_of(raw)
# Each chunk's first timestamped line is its own: a finished chunk's session is deleted, so the next chunk's transcript starts afresh.
folder = paths.transcript_archive / project_key / hashlib.sha256(first_timestamped_line(raw)).hexdigest()
make_private(folder)
except OSError as failure:
return unarchived(failure.filename, failure.strerror)
@@ -24,7 +39,7 @@ def archive(project_key: str, path: str, task: str, outcome: str, at: float, pat
return unkept
copy = folder / f"{hashlib.sha256(raw).hexdigest()}.jsonl"
outcomes = copy.with_suffix(".outcomes.md")
stamped = f"{time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime(at))} {outcome}"
stamped = f"{utc(at)} {outcome}"
if copy.exists() and outcomes.exists():
return add_outcome(outcomes, stamped)
# The copy lands last, so a copy on disk always has its outcomes and its session's files beside it.
@@ -35,11 +50,6 @@ def archive(project_key: str, path: str, task: str, outcome: str, at: float, pat
return None
# Each chunk's first timestamped line is its own: a finished chunk's session is deleted, so the next chunk's transcript starts afresh.
def chunk_of(raw: bytes) -> str:
return hashlib.sha256(first_timestamped_line(raw)).hexdigest()
def unarchived(where: object, why: str | None) -> Failure:
return Failure(f"could not archive the transcript: {where}: {why}")
@@ -89,3 +99,21 @@ def copy_for(copy: pathlib.Path, data: bytes) -> pathlib.Path | None:
return copy if data != kept else None
beside = copy.with_name(f"{copy.name}.{hashlib.sha256(data).hexdigest()}")
return None if beside.exists() else beside
def utc(at: float) -> str:
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(at))
# One per chunk folder, its latest finished copy, in the order they finished.
def finished_chunks(project_archive: pathlib.Path) -> list[FinishedChunk] | Failure:
found = []
try:
for folder in sorted(path for path in project_archive.glob("*") if path.is_dir()):
copies = sorted((stamps[-1], outcomes) for outcomes in folder.glob("*.outcomes.md") if (stamps := FINISHED_AT.findall(outcomes.read_text(encoding="utf-8"))))
if copies:
finished, outcomes = copies[-1]
found.append(FinishedChunk(folder.name, finished, outcomes.with_name(outcomes.name.removesuffix(".outcomes.md") + ".jsonl"), sorted(folder.glob("session/*/subagents/*.jsonl"))))
except (OSError, UnicodeDecodeError) as failure:
return Failure(f"could not read the archive {project_archive}: {failure}")
return sorted(found, key=lambda chunk: (chunk.finished, chunk.folder))
-181
View File
@@ -1,181 +0,0 @@
import collections
import dataclasses
import os
import pathlib
import posixpath
import re
import shlex
import time
from failure import Failure
from file_writes import make_private, replace_file
from loop_claude.claude_code.entries import assistant_parts, parse_entry, tool_call
from loop_claude.cycle.archive import chunk_of
from loop_claude.paths import Paths
COMMIT = re.compile(r"(^|[\s;&|(])git\s+(-C\s+\S+\s+)?commit\b")
# Each reads the files it names; for these, the first operand is a pattern or a program unless -e or -f gives it.
READERS = {"awk", "bat", "cat", "egrep", "grep", "head", "less", "more", "nl", "rg", "sed", "tail", "wc"}
PATTERN_FIRST = {"awk", "egrep", "grep", "rg", "sed"}
NOT_A_PATH = re.compile(r"[*?\[\]{}$\\'\",=()<>|;&]|://|^\d+$")
PLACES_KEPT = 10
HEADING = "## Reading cost, chunk "
TOKENS = re.compile(r"^(\d+) tokens read before the first commit\.$", re.MULTILINE)
@dataclasses.dataclass(frozen=True)
class Reading:
"""What the worker and its subagents took in before the worker's first commit."""
tokens: int
places: collections.Counter[str]
def record_reading(project_key: str, transcript: str, at: float, paths: Paths) -> Failure | None:
try:
raw = pathlib.Path(transcript).read_bytes()
subagents = [path.read_bytes() for path in sorted(pathlib.Path(transcript.removesuffix(".jsonl"), "subagents").glob("*.jsonl"))]
except OSError as failure:
return Failure(f"could not measure the reading cost: {failure.filename}: {failure.strerror}")
reading = measure(raw, subagents)
if reading is None:
return None
return keep_reading(pending_file(project_key, paths), entry(chunk_of(raw)[:12], time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(at)), reading))
def pending_file(project_key: str, paths: Paths) -> pathlib.Path:
return paths.transcript_archive / project_key / "reading-cost.md"
def measure(transcript: bytes, subagents: list[bytes]) -> Reading | None:
worker = entries_of(transcript)
commit = next((entry.get("timestamp", "") for entry in worker if any(commits(part) for part in assistant_parts(entry))), None)
if commit is None:
return None
tokens = 0
places: collections.Counter[str] = collections.Counter()
for thread in (worker, *map(entries_of, subagents)):
# Claude Code writes one entry per content block of a message, each with the message's usage.
counted: set[str] = set()
for entry in thread:
if entry.get("type") != "assistant" or entry.get("timestamp", "") >= commit:
continue
message = entry.get("message") or {}
if (key := str(message.get("id") or entry.get("uuid"))) not in counted:
counted.add(key)
usage = message.get("usage") or {}
tokens += int(usage.get("input_tokens") or 0) + int(usage.get("cache_creation_input_tokens") or 0)
for part in assistant_parts(entry):
places.update(read_paths(part, entry.get("cwd") or ""))
return Reading(tokens, places)
def entries_of(raw: bytes) -> list[dict]:
return [entry for line in raw.splitlines() if (entry := parse_entry(line))]
def arguments_of(part: object) -> tuple[object, dict]:
call = tool_call(part)
return (call[0], call[1]) if call and isinstance(call[1], dict) else (None, {})
def commits(part: object) -> bool:
name, arguments = arguments_of(part)
return name == "Bash" and bool(COMMIT.search(str(arguments.get("command", ""))))
def read_paths(part: object, cwd: str) -> list[str]:
name, arguments = arguments_of(part)
if name == "Read":
return place(str(arguments.get("file_path", "")), cwd, cwd)
if name == "Grep" and arguments.get("path"):
return place(str(arguments["path"]), cwd, cwd)
if name == "Bash":
return bash_reads(str(arguments.get("command", "")), cwd)
return []
def bash_reads(command: str, cwd: str) -> list[str]:
found = []
base = cwd
lines = iter(command.splitlines())
for line in lines:
try:
lexer = shlex.shlex(line, posix=True, punctuation_chars=True)
lexer.whitespace_split = True
tokens = list(lexer)
except ValueError:
continue
for words in segments(tokens):
if words[0] == "cd" and len(words) == 2:
base = posixpath.normpath(posixpath.join(base, os.path.expanduser(words[1])))
found += [kept for path in operands(words) for kept in place(path, base, cwd)]
if "<<" in tokens or "<<-" in tokens:
marker = tokens[tokens.index("<<" if "<<" in tokens else "<<-") + 1:][:1]
for body in lines:
if marker and body.strip() == marker[0]:
break
return found
def segments(tokens: list[str]) -> list[list[str]]:
found: list[list[str]] = [[]]
skip = False
for token in tokens:
if skip:
skip = False
elif token and all(char in "();<>|&" for char in token):
# A redirect's target is written or fed, not read by the command.
skip = "<" in token or ">" in token
found.append([])
else:
found[-1].append(token)
return [words for words in found if words]
def operands(words: list[str]) -> list[str]:
program = posixpath.basename(words[0])
if program not in READERS or (program == "sed" and any(word.startswith("-i") for word in words)):
return []
given = [word for word in words[1:] if not word.startswith("-")]
if program in PATTERN_FIRST and not any(word in ("-e", "-f") for word in words):
given = given[1:]
return [word for word in given if ("/" in word or re.search(r"\.[A-Za-z0-9]+$", word)) and not NOT_A_PATH.search(word)]
# Relative to the folder it was read in, a worktree or the checkout, so both name a file alike; a file outside both is no place of the project.
def place(path: str, base: str, cwd: str) -> list[str]:
full = posixpath.normpath(posixpath.join(base, os.path.expanduser(path)))
roots = [root.rstrip("/") + "/" for root in (base, cwd) if root]
return next(([full.removeprefix(root)] for root in roots if full.startswith(root)), [])
def entry(chunk: str, finished: str, reading: Reading) -> str:
places = "".join(f"- `{path}` {count}\n" for path, count in sorted(reading.places.items(), key=lambda item: (-item[1], item[0]))[:PLACES_KEPT])
return f"{HEADING}{chunk}, finished {finished}\n\n{reading.tokens} tokens read before the first commit.\n" + (f"\nRead most:\n\n{places}" if places else "")
def keep_reading(pending: pathlib.Path, block: str) -> Failure | None:
try:
make_private(pending.parent)
kept = pending.read_text(encoding="utf-8") if pending.exists() else ""
except (OSError, UnicodeDecodeError) as failure:
return Failure(f"could not keep the reading cost in {pending}: {failure}")
if block.splitlines()[0] in kept.splitlines():
return None
if unwritten := replace_file(pending, (kept + ("\n" if kept else "") + block).encode()):
return Failure(f"could not keep the reading cost in {pending}: {unwritten.reason}")
return None
def blocks(text: str) -> list[str]:
"""The reading-cost entries of a markdown file, in order, each from its heading to the next heading."""
found = []
for section in re.split(r"(?m)^(?=## )", text):
if section.startswith(HEADING):
found.append(section.rstrip("\n") + "\n")
return found
def tokens_of(block: str) -> int:
match = TOKENS.search(block)
return int(match.group(1)) if match else 0
+3 -4
View File
@@ -2,7 +2,7 @@ import dataclasses
import typing
from loop_claude.cycle.states import Active, Archived, Closing, Ended, Event, HumanAnswered, Interrupting, Judging, QUOTED_ANSWERS_PER_CYCLE, Ruled, State, transition, WorkerRan, Working
from loop_claude.cycle.archive import archive
from loop_claude.cycle.archive import archive, FINISHED, UNFINISHED
from loop_claude.claude_code.session import delete_session, Session, session_open, transcript_file
from loop_claude.effects import Effects
from failure import Failure
@@ -12,7 +12,6 @@ from loop_claude.cycle.judge import judge
from loop_claude.project.model_choice import Models
from loop_claude.paths import Paths
from loop_claude.claude_code.transcript import read_transcript, render
from loop_claude.cycle.reading_cost import record_reading
from loop_claude.cycle.retry import retried
from loop_claude.cycle.worker import run_worker, worker_args
from loop_claude.stop import Stop
@@ -110,9 +109,9 @@ def question_for(ruling: Interrupt | QuotedAnswer) -> str:
def close(cycle: Cycle, transcript_path: str, verdict: Finished | Unfinished) -> Archived | Stop:
finished = isinstance(verdict, Finished)
if failure := archive(cycle.project_key, transcript_path, cycle.task, f"Verdict: {'finished' if finished else 'unfinished'}\n\nEvidence:\n{quoted(verdict.evidence)}", cycle.effects.now(), cycle.paths):
if failure := archive(cycle.project_key, transcript_path, cycle.task, f"{FINISHED if finished else UNFINISHED}\n\nEvidence:\n{quoted(verdict.evidence)}", cycle.effects.now(), cycle.paths):
return Stop(1, failure.reason)
if finished and (failure := record_reading(cycle.project_key, transcript_path, cycle.effects.now(), cycle.paths) or delete_session(cycle.session)):
if finished and (failure := delete_session(cycle.session)):
return Stop(1, failure.reason)
return Archived()
+1 -4
View File
@@ -32,9 +32,6 @@ time. Claude Code keeps a folder per session beside its transcript, holding the
transcripts among other files; that folder, and every other file named after the session except
the transcript, are copied into <chunk>/session/. A file there is replaced only by a version that
grew from it. A file Claude Code rewrote lands beside its copy as <name>.<hash of its content>.
When a chunk finishes, the reading cost of its work before the worker's first commit is kept in
$LOOP_CLAUDE_TRANSCRIPTS/<key>/reading-cost.md; reading_cost.py --help says what it holds and how
it reaches the project.
Every claude loop-claude starts runs the project's model, saved per <key> in
~/.claude/loop-claude-models.json and read each cycle; a project without an entry, as on its first
@@ -53,7 +50,7 @@ Exit 1, saying why, when:
- the models file cannot be read or written
- claude leaves no transcript
- a transcript cannot be read or archived
- a finished chunk's reading cost cannot be measured or kept, or its session deleted
- a finished chunk's session cannot be deleted
- claude cannot start for an interrupt
- python cannot start for a reload
- a reload cannot read the state the running script handed it, likely because the edit changed that
Executable → Regular
+207 -49
View File
@@ -1,24 +1,27 @@
#!/usr/bin/env python3
"""Move loop-claude's reading-cost figures into a project's history, and say whether they rise.
"""Measure each chunk loop-claude finished, append the figures to a project's history, and say whether they rise.
usage: reading_cost.py [history]
Run it inside a project checkout or one of its worktrees. history defaults to
docs/comprehension-history.md there, created when missing. When loop-claude finishes a chunk, it
measures what the worker and its subagents read before the worker's first commit: the tokens they
took in, and the project's files they opened or searched, with how often. It keeps each figure in
$LOOP_CLAUDE_TRANSCRIPTS/<key>/reading-cost.md (default ~/.claude/loop-claude-transcripts), <key>
being the checkout's path relative to ~/code, or to / outside it. This appends to history each
figure it does not yet hold, under a "## Reading cost, chunk <id>, finished <UTC time>" heading.
docs/comprehension-history.md there, created when missing. It reads loop-claude's archive of the
project, $LOOP_CLAUDE_TRANSCRIPTS/<key> (default ~/.claude/loop-claude-transcripts), <key> being
the checkout's path relative to ~/code, or to / outside it. For each chunk the judge ruled
finished that history does not hold, it measures what the worker and its subagents read before
the worker's first commit: the tokens they took in, and how often they opened or searched each
file git tracks in the checkout. A chunk whose worker never committed is not measured. Each
figure is appended under a "## Reading cost, chunk <id>, finished <UTC time>" heading, in the
order the chunks finished.
It then compares the mean tokens of the history's last 5 figures with the 5 before them, and prints
"Rising:" or "Not rising:" with both means. Rising, it lists the files the last 5 chunks read most.
With fewer than 10 figures, it says so.
Exit 1, saying why, when git finds no checkout, or the figures or the history cannot be read or
written.
Exit 1, saying why, when git cannot start or finds no checkout, the archive or the history cannot
be read or written, or a figure in the history cannot be read back.
"""
import collections
import dataclasses
import os
import pathlib
import re
@@ -27,63 +30,218 @@ import sys
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from loop_claude.cycle.reading_cost import blocks, pending_file, PLACES_KEPT, tokens_of
from loop_claude.claude_code.entries import assistant_parts, parse_entry, tool_call
from failure import Failure
from loop_claude.cycle.archive import finished_chunks, FinishedChunk
from loop_claude.paths import paths_from
from loop_claude.project.checkout import project_key
from shell_reads import Reads, resolved, shell_reads
from file_writes import replace_file
from markdown_sections import sections
COMMIT = re.compile(r"(^|[\s;&|(])git\s+(-C\s+\S+\s+)?commit\b")
HEADING = "Reading cost, chunk "
PLACE = re.compile(r"^- `(.+)` (\d+)$")
PLACES_KEPT = 10
TOKENS = re.compile(r"^(\d+) tokens read before the first commit\.$")
WINDOW = 5
PLACE = re.compile(r"^- `(.+)` (\d+)$", re.MULTILINE)
@dataclasses.dataclass(frozen=True)
class Reading:
"""What the worker and its subagents took in before the worker's first commit."""
tokens: int
places: collections.Counter[str]
@dataclasses.dataclass(frozen=True)
class Checkout:
root: str
top: str
tracked: frozenset[str]
def main(argv: list[str]) -> int:
if argv[:1] in (["-h"], ["--help"]) or len(argv) > 1:
print(__doc__.strip(), file=sys.stdout if argv[:1] in (["-h"], ["--help"]) else sys.stderr)
return 0 if argv[:1] in (["-h"], ["--help"]) else 1
found = subprocess.run(["git", "rev-parse", "--path-format=absolute", "--show-toplevel", "--git-common-dir"], capture_output=True, text=True)
if found.returncode != 0:
print(found.stderr.strip(), file=sys.stderr)
return 1
top, common = found.stdout.splitlines()
paths = paths_from(pathlib.Path.home(), os.environ)
# The common dir is the checkout's .git, whichever worktree runs this.
pending = pending_file(project_key(str(pathlib.Path(common).parent), paths.code), paths)
history = pathlib.Path(argv[0]) if argv else pathlib.Path(top, "docs", "comprehension-history.md")
try:
figures = blocks(pending.read_text(encoding="utf-8")) if pending.exists() else []
text = history.read_text(encoding="utf-8") if history.exists() else "# Comprehension history\n"
except (OSError, UnicodeDecodeError) as failure:
print(f"could not read the reading cost: {failure}", file=sys.stderr)
return 1
held = set(text.splitlines())
new = [block for block in figures if block.splitlines()[0] not in held]
if new:
text = text.rstrip("\n") + "\n\n" + "\n".join(new)
helped = argv[:1] in (["-h"], ["--help"])
if helped or len(argv) > 1:
print(__doc__.strip(), file=sys.stdout if helped else sys.stderr)
return 0 if helped else 1
home = pathlib.Path.home()
paths = paths_from(home, os.environ)
checkout = located()
if isinstance(checkout, Failure):
return failed(checkout)
history = pathlib.Path(argv[0]) if argv else pathlib.Path(checkout.top, "docs", "comprehension-history.md")
text = read_history(history)
chunks = finished_chunks(paths.transcript_archive / project_key(checkout.root, paths.code))
if isinstance(text, Failure):
return failed(text)
if isinstance(chunks, Failure):
return failed(chunks)
figures = []
for chunk in unheld(chunks, text):
threads = read_threads(chunk)
if isinstance(threads, Failure):
return failed(threads)
if reading := measure(threads, checkout, str(home)):
figures.append(figure(chunk, reading))
if figures:
text = merged(text, figures)
history.parent.mkdir(parents=True, exist_ok=True)
if unwritten := replace_file(history, text.encode()):
print(f"could not write {history}: {unwritten.reason}", file=sys.stderr)
return 1
print(f"Appended {len(new)} reading-cost figures to {history}.")
print(trend(blocks(text)))
return failed(Failure(f"could not write {history}: {unwritten.reason}"))
readings = readings_of(text)
if isinstance(readings, Failure):
return failed(readings)
print(f"Appended {len(figures)} reading-cost figures to {history}.")
print(trend(readings))
return 0
def trend(figures: list[str]) -> str:
if len(figures) < 2 * WINDOW:
return f"{len(figures)} chunks measured; the trend needs {2 * WINDOW}."
last, before = figures[-WINDOW:], figures[-2 * WINDOW:-WINDOW]
means = f"the last {WINDOW} chunks read {mean(last)} tokens on average before their first commit, the {WINDOW} before them {mean(before)}."
if mean(last) <= mean(before):
def failed(failure: Failure) -> int:
print(failure.reason, file=sys.stderr)
return 1
def located() -> Checkout | Failure:
try:
found = subprocess.run(["git", "rev-parse", "--path-format=absolute", "--show-toplevel", "--git-common-dir"], capture_output=True, text=True)
if found.returncode != 0:
return Failure(found.stderr.strip())
top, common = found.stdout.splitlines()
listed = subprocess.run(["git", "ls-files", "-z"], capture_output=True, cwd=top, text=True)
except (OSError, ValueError) as failure:
return Failure(f"could not find the checkout: {failure}")
if listed.returncode != 0:
return Failure(listed.stderr.strip())
# The common dir is the checkout's .git, whichever worktree runs this.
return Checkout(str(pathlib.Path(common).parent), top, frozenset(filter(None, listed.stdout.split("\0"))))
def read_history(history: pathlib.Path) -> str | Failure:
try:
return history.read_text(encoding="utf-8") if history.exists() else "# Comprehension history\n"
except (OSError, UnicodeDecodeError) as failure:
return Failure(f"could not read {history}: {failure}")
def read_threads(chunk: FinishedChunk) -> list[bytes] | Failure:
"""The worker's transcript, then each subagent's."""
try:
return [path.read_bytes() for path in (chunk.transcript, *chunk.subagents)]
except OSError as failure:
return Failure(f"could not read {failure.filename}: {failure.strerror}")
def unheld(chunks: list[FinishedChunk], history: str) -> list[FinishedChunk]:
held = {heading.removeprefix(HEADING).split(",")[0] for heading in sections(history) if heading.startswith(HEADING)}
return [chunk for chunk in chunks if chunk_id(chunk) not in held]
def chunk_id(chunk: FinishedChunk) -> str:
return chunk.folder[:12]
def measure(threads: list[bytes], checkout: Checkout, home: str) -> Reading | None:
worker, *subagents = [entries_of(thread) for thread in threads]
commit = commit_time(worker)
if commit is None:
return None
tokens = 0
reads = Reads([], [])
for thread in (worker, *subagents):
read_before = [entry for entry in thread if entry.get("type") == "assistant" and str(entry.get("timestamp", "")) < commit]
tokens += tokens_in(read_before)
for entry in read_before:
for part in assistant_parts(entry):
found = reads_of(part, str(entry.get("cwd") or "/"), home)
reads.read.extend(found.read)
reads.added.extend(found.added)
roots = [checkout.root, *reads.added]
return Reading(tokens, collections.Counter(place for path in reads.read if (place := relative(path, roots)) in checkout.tracked))
def entries_of(raw: bytes) -> list[dict]:
return [entry for line in raw.splitlines() if (entry := parse_entry(line))]
def commit_time(worker: list[dict]) -> str | None:
for entry in worker:
for part in assistant_parts(entry):
name, arguments = call_of(part)
if name == "Bash" and COMMIT.search(str(arguments.get("command", ""))):
return str(entry.get("timestamp", ""))
return None
def tokens_in(entries: list[dict]) -> int:
# Claude Code writes one entry per content block of a message, each with the message's usage.
usages = {str(message.get("id") or entry.get("uuid")): message.get("usage") for entry in entries if isinstance(message := entry.get("message"), dict)}
return sum(count(usage, "input_tokens") + count(usage, "cache_creation_input_tokens") for usage in usages.values() if isinstance(usage, dict))
def count(usage: dict, field: str) -> int:
value = usage.get(field)
return value if isinstance(value, int) else 0
def call_of(part: object) -> tuple[object, dict]:
call = tool_call(part)
return (call[0], call[1]) if call and isinstance(call[1], dict) else (None, {})
def reads_of(part: object, cwd: str, home: str) -> Reads:
name, arguments = call_of(part)
path = arguments.get("file_path") if name == "Read" else arguments.get("path") if name == "Grep" else None
if path:
return Reads([resolved(cwd, str(path), home)], [])
return shell_reads(str(arguments.get("command", "")), cwd, home) if name == "Bash" else Reads([], [])
# Relative to the deepest root holding it, so the checkout and each worktree name a file alike.
def relative(path: str, roots: list[str]) -> str | None:
holding = [root for root in roots if path.startswith(root.rstrip("/") + "/")]
return path[len(max(holding, key=len).rstrip("/")) + 1:] if holding else None
def figure(chunk: FinishedChunk, reading: Reading) -> str:
places = "".join(f"- `{path}` {times}\n" for path, times in ranked(reading.places))
return f"## {HEADING}{chunk_id(chunk)}, finished {chunk.finished}\n\n{reading.tokens} tokens read before the first commit.\n" + (f"\nRead most:\n\n{places}" if places else "")
def readings_of(history: str) -> list[Reading] | Failure:
readings = []
for heading, bodies in sections(history).items():
if heading.startswith(HEADING):
lines = bodies[0].splitlines()
tokens = [int(match.group(1)) for line in lines if (match := TOKENS.match(line))]
if len(tokens) != 1:
return Failure(f"the figure under \"## {heading}\" holds no single tokens line; restore it from git history")
readings.append(Reading(tokens[0], collections.Counter({match.group(1): int(match.group(2)) for line in lines if (match := PLACE.match(line))})))
return readings
def ranked(places: collections.Counter[str]) -> list[tuple[str, int]]:
return sorted(places.items(), key=lambda item: (-item[1], item[0]))[:PLACES_KEPT]
def merged(history: str, figures: list[str]) -> str:
return history.rstrip("\n") + "\n\n" + "\n".join(figures)
def trend(readings: list[Reading]) -> str:
if len(readings) < 2 * WINDOW:
return f"{len(readings)} chunks measured; the trend needs {2 * WINDOW}."
last, earlier = readings[-WINDOW:], readings[-2 * WINDOW:-WINDOW]
means = f"the last {WINDOW} chunks read {mean(last)} tokens on average before their first commit, the {WINDOW} before them {mean(earlier)}."
if mean(last) <= mean(earlier):
return f"Not rising: {means}"
places: collections.Counter[str] = collections.Counter()
for block in last:
places.update({path: int(count) for path, count in PLACE.findall(block)})
read_most = "".join(f"\n- `{path}` {count}" for path, count in sorted(places.items(), key=lambda item: (-item[1], item[0]))[:PLACES_KEPT])
read_most = "".join(f"\n- `{path}` {times}" for path, times in ranked(sum((reading.places for reading in last), collections.Counter())))
return f"Rising: {means}\n\nRead most over the last {WINDOW}:\n{read_most}"
def mean(figures: list[str]) -> int:
return round(sum(map(tokens_of, figures)) / len(figures))
def mean(readings: list[Reading]) -> int:
return round(sum(reading.tokens for reading in readings) / len(readings))
if __name__ == "__main__":
+91
View File
@@ -0,0 +1,91 @@
"""The files a shell command reads, and the worktrees it adds, as absolute paths."""
import dataclasses
import posixpath
import re
import shlex
import typing
HEREDOC = {"<<", "<<-"}
NOT_A_PATH = re.compile(r"[*?\[\]{}$\\'\",=()<>|;&]|://|^\d+$")
# Their first operand is a pattern or a program, unless -e or -f gives it.
PATTERN_FIRST = {"awk", "egrep", "grep", "rg", "sed"}
READERS = {"awk", "bat", "cat", "egrep", "grep", "head", "less", "more", "nl", "rg", "sed", "tail", "wc"}
WORKTREE_OPTIONS_WITH_VALUE = {"-b", "-B", "--reason"}
@dataclasses.dataclass(frozen=True)
class Reads:
read: list[str]
added: list[str]
def shell_reads(command: str, cwd: str, home: str) -> Reads:
reads = Reads([], [])
base = cwd
for tokens in shell_lines(command):
for words in segments(tokens):
if words[0] == "cd" and len(words) == 2:
base = resolved(base, words[1], home)
reads.read.extend(resolved(base, word, home) for word in operands(words))
reads.added.extend(resolved(base, path, home) for path in worktree_added(words))
return reads
def shell_lines(command: str) -> typing.Iterator[list[str]]:
"""Each line's tokens, skipping a heredoc's body, which feeds a program rather than naming files."""
ends = None
for line in command.splitlines():
if ends is not None:
ends = None if line.strip() == ends else ends
continue
try:
lexer = shlex.shlex(line, posix=True, punctuation_chars=True)
lexer.whitespace_split = True
tokens = list(lexer)
except ValueError:
continue
marker = next((tokens[index + 1] for index, token in enumerate(tokens[:-1]) if token in HEREDOC), None)
ends = marker
yield tokens
def segments(tokens: list[str]) -> list[list[str]]:
found: list[list[str]] = [[]]
target = False
for token in tokens:
if target:
target = False
elif all(char in "();<>|&" for char in token):
# A redirect's target is written or fed, not read by the command.
target = "<" in token or ">" in token
found.append([])
else:
found[-1].append(token)
return [words for words in found if words]
def operands(words: list[str]) -> list[str]:
program = posixpath.basename(words[0])
if program not in READERS or (program == "sed" and any(word.startswith("-i") for word in words)):
return []
given = [word for word in words[1:] if not word.startswith("-")]
if program in PATTERN_FIRST and not {"-e", "-f"} & set(words):
given = given[1:]
return [word for word in given if ("/" in word or re.search(r"\.[A-Za-z0-9]+$", word)) and not NOT_A_PATH.search(word)]
# git worktree add [-b <branch>] <path> [<commit>]
def worktree_added(words: list[str]) -> list[str]:
if words[:1] != ["git"] or "worktree" not in words or words[words.index("worktree") + 1:][:1] != ["add"]:
return []
after = iter(words[words.index("worktree") + 2:])
for word in after:
if word in WORKTREE_OPTIONS_WITH_VALUE:
next(after, None)
elif not word.startswith("-"):
return [word]
return []
def resolved(base: str, path: str, home: str) -> str:
return posixpath.normpath(posixpath.join(base, home + path[1:] if path == "~" or path.startswith("~/") else path))
+2 -43
View File
@@ -1,5 +1,4 @@
import ast
import collections
import contextlib
import dataclasses
import hashlib
@@ -17,7 +16,7 @@ import uuid
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from loop_claude.cycle import archive, interrupt, judge, reading_cost, retry, run, states, worker
from loop_claude.cycle import archive, interrupt, judge, retry, run, states, worker
from loop_claude.project import checkout, model_choice, records
from loop_claude.effects import Effects
from loop_claude.claude_code import entries, session, transcript, trust
@@ -189,24 +188,6 @@ class Cycles(unittest.TestCase):
self.assertEqual(self.archived_outcomes(), ["Task: Do it.\n\n2026-09-21T14:13:20Z Verdict: finished\n\nEvidence:\n> working\n"])
self.assertIsNone(session.transcript_file(self.session))
def test_a_finished_chunk_that_committed_keeps_its_reading_cost_once(self):
def committing(args):
Worker(self.paths.claude_code_transcripts.parent)(args)
with open(session.transcript_file(self.session), "a") as handle:
handle.write(json.dumps({**assistant({"type": "tool_use", "name": "Bash", "input": {"command": "git commit -m x"}}), "timestamp": "9"}) + "\n")
return Streamed(0)
fake = FakeEffects(streams=[committing], claude=[judged("finished")])
self.assertEqual(run.run_cycle(self.cycle(fake)), Finished("working"))
[kept] = (self.paths.transcript_archive / "proj").glob("reading-cost.md")
self.assertRegex(kept.read_text(), r"^## Reading cost, chunk [0-9a-f]{12}, finished 2026-09-21T14:13:20Z\n\n0 tokens read before the first commit\.\n$")
self.assertIsNone(reading_cost.keep_reading(kept, kept.read_text()))
self.assertEqual(len(reading_cost.blocks(kept.read_text())), 1)
def test_a_finished_chunk_that_never_committed_keeps_no_reading_cost(self):
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)], claude=[judged("finished")])
self.assertEqual(run.run_cycle(self.cycle(fake)), Finished("working"))
self.assertFalse((self.paths.transcript_archive / "proj" / "reading-cost.md").exists())
def test_an_unfinished_chunks_transcript_is_archived_and_the_next_cycle_resumes_its_session_with_the_judges_evidence(self):
fake = FakeEffects(streams=[Worker(self.paths.claude_code_transcripts.parent)] * 2, claude=[judged("unfinished")] * 2)
self.assertEqual(run.run_cycle(self.cycle(fake)), Unfinished("working"))
@@ -443,28 +424,6 @@ class Records(unittest.TestCase):
self.assertEqual(records.read_records(root, claude_md), Stop(1, f"AGENTS.md in {root} is not UTF-8 on line 2 - the judge cannot read the records."))
class ReadingCost(unittest.TestCase):
def test_counts_each_message_once_and_the_projects_files_read_before_the_first_commit(self):
def said(timestamp, message_id, command=None, tokens=0, **call):
part = {"type": "tool_use", "name": "Bash", "input": {"command": command}} if command else {"type": "tool_use", **call}
return {"type": "assistant", "timestamp": timestamp, "cwd": "/c/p", "message": {"id": message_id, "content": [part], "usage": {"input_tokens": tokens, "cache_creation_input_tokens": 2 * tokens, "cache_read_input_tokens": 1000}}}
worker = [
said("1", "m1", tokens=10, name="Read", input={"file_path": "/c/p/README.md"}),
said("2", "m1", tokens=10, name="Read", input={"file_path": "/tmp/log.txt"}),
said("3", "m2", command="cd ../w && sed -n 1,5p src/a.py | grep -n x\ngrep -rn 'def x' bin/t.py > out.txt\npython3 - <<'EOF'\ncat secret.py\nEOF\ncat README.md"),
said("4", "m3", command="git -C ../w commit -m 'x'", tokens=100),
said("5", "m4", tokens=100, name="Read", input={"file_path": "/c/p/late.py"}),
]
subagent = [said("2", "s1", tokens=1, name="Grep", input={"path": "src/a.py", "pattern": "x"}), said("6", "s2", tokens=100, name="Read", input={"file_path": "/c/p/late.py"})]
lines = lambda entries: b"".join(json.dumps(entry).encode() + b"\n" for entry in entries)
reading = reading_cost.measure(lines(worker), [lines(subagent)])
self.assertEqual(reading, reading_cost.Reading(33, collections.Counter({"README.md": 2, "src/a.py": 2, "bin/t.py": 1})))
self.assertEqual(reading_cost.entry("abc", "t", reading), "## Reading cost, chunk abc, finished t\n\n33 tokens read before the first commit.\n\nRead most:\n\n- `README.md` 2\n- `src/a.py` 2\n- `bin/t.py` 1\n")
def test_measures_nothing_for_a_worker_that_never_committed(self):
self.assertIsNone(reading_cost.measure(json.dumps(assistant({"type": "tool_use", "name": "Bash", "input": {"command": "git log --grep commit"}})).encode(), []))
class Render(unittest.TestCase):
def test_joins_assistant_text_tool_calls_human_turns_and_loop_prompts_and_skips_the_rest(self):
said = transcript.render([
@@ -656,7 +615,7 @@ class ExitContract(unittest.TestCase):
("cycle/run.py", "rule", "1", "entries.reason", read),
("cycle/run.py", "ask", "1", "failure.reason", read),
("cycle/run.py", "close", "1", "failure.reason", read),
("cycle/run.py", "close", "1", "failure.reason", "a finished chunk's reading cost cannot be measured or kept, or its session deleted"),
("cycle/run.py", "close", "1", "failure.reason", "a finished chunk's session cannot be deleted"),
("loop/main.py", "loop_run_from", "0 if task else 1", "__doc__.strip()", "no task is given"),
("loop/main.py", "loop_run_from", "1", 'f"the edited loop-claude cannot read the state', "a reload cannot read the state the running script handed it"),
("loop/reload.py", "exec_if_edited", "1", "f'could not reload loop-claude:", "python cannot start for a reload"),
+6 -1
View File
@@ -18,6 +18,11 @@ import unittest
BIN = pathlib.Path(__file__).resolve().parent
TASK = "Do the next chunk."
# A git hook's GIT_DIR, the host's LOOP_CLAUDE_* or PYTHONPATH would point the program under test at the host's own state.
def host_env() -> dict[str, str]:
return {name: value for name, value in os.environ.items() if not name.startswith(("GIT_", "LOOP_CLAUDE_", "PYTHON"))}
# Plays claude from FAKE_CLAUDE_SCENARIO, one step per call, logging each call; a step for another role,
# or none left, is Ctrl-C to loop-claude, which otherwise loops forever.
FAKE_CLAUDE = """#!{python}
@@ -155,7 +160,7 @@ class Checkout:
self.root.mkdir(parents=True)
(self.root / "README.md").write_text(PREMISES)
self.script = BIN / "loop-claude"
self.env = {name: value for name, value in os.environ.items() if not name.startswith(("GIT_", "LOOP_CLAUDE_", "PYTHON"))}
self.env = host_env()
fake = self.temp / "fake-bin"
fake.mkdir()
(fake / "claude").write_text(FAKE_CLAUDE.format(python=sys.executable))
+104 -25
View File
@@ -1,3 +1,8 @@
import collections
import contextlib
import hashlib
import io
import json
import os
import pathlib
import subprocess
@@ -5,48 +10,122 @@ import sys
import tempfile
import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from loop_claude.cycle.archive import archive, FINISHED, finished_chunks, FinishedChunk, UNFINISHED
from reading_cost import Checkout, Reading
from failure import Failure
from test_loop_claude_black_box import host_env
from loop_claude.paths import paths_from
import reading_cost
from shell_reads import Reads, shell_reads
TOOL = pathlib.Path(__file__).resolve().parent / "reading_cost.py"
def figure(chunk, tokens, place="a.py"):
return f"## Reading cost, chunk {chunk}, finished t\n\n{tokens} tokens read before the first commit.\n\nRead most:\n\n- `{place}` 2\n"
def said(timestamp, message_id, command=None, tokens=0, cwd="/c/p", **call):
part = {"type": "tool_use", "name": "Bash", "input": {"command": command}} if command else {"type": "tool_use", **call}
return {"type": "assistant", "timestamp": timestamp, "cwd": cwd, "message": {"id": message_id, "content": [part], "usage": {"input_tokens": tokens, "cache_creation_input_tokens": 2 * tokens, "cache_read_input_tokens": 1000}}}
def jsonl(entries):
return b"".join(json.dumps(entry).encode() + b"\n" for entry in entries)
def folder_name():
return hashlib.sha256(os.urandom(8)).hexdigest()
class ShellReads(unittest.TestCase):
def test_reads_operands_past_a_cd_and_skips_patterns_redirect_targets_and_heredoc_bodies(self):
command = "cd ../w && sed -n 1,5p src/a.py | grep -n x\ngrep -rn 'def x' bin/t.py > out.txt\npython3 - <<'EOF'\ncat secret.py\nEOF\ncat ~/README.md\nsed -i s/a/b/ c.py"
self.assertEqual(shell_reads(command, "/c/p", "/h"), Reads(["/c/w/src/a.py", "/c/w/bin/t.py", "/h/README.md"], []))
def test_names_the_worktree_a_git_worktree_add_makes(self):
self.assertEqual(shell_reads("git worktree add -q -b topic ../p-topic origin/main", "/c/p", "/h"), Reads([], ["/c/p-topic"]))
class Measure(unittest.TestCase):
def test_counts_each_message_once_and_the_tracked_files_read_before_the_first_commit(self):
worker = [
said("1", "m1", tokens=10, name="Read", input={"file_path": "/c/p/README.md"}),
said("2", "m1", tokens=10, name="Read", input={"file_path": "/c/p/.env"}),
said("3", "m2", command="git worktree add ../w && cd ../w && cat src/a.py && cd src && cat b.py"),
said("4", "m3", command="cd /tmp && cat x.py"),
said("5", "m4", command="git -C ../w commit -m 'x'", tokens=100),
said("6", "m5", tokens=100, name="Read", input={"file_path": "/c/p/late.py"}),
]
subagent = [said("2", "s1", tokens=1, name="Grep", input={"path": "/c/w/src/a.py"}), said("7", "s2", tokens=100, name="Read", input={"file_path": "/c/p/late.py"})]
checkout = Checkout("/c/p", "/c/w", frozenset({"README.md", "late.py", "src/a.py", "src/b.py", "x.py"}))
self.assertEqual(reading_cost.measure([jsonl(worker), jsonl(subagent)], checkout, "/h"), Reading(33, collections.Counter({"src/a.py": 2, "README.md": 1, "src/b.py": 1})))
def test_measures_nothing_for_a_worker_that_never_committed(self):
self.assertIsNone(reading_cost.measure([jsonl([said("1", "m1", command="git log --grep commit")])], Checkout("/c/p", "/c/p", frozenset()), "/h"))
def test_skips_a_usage_that_is_not_a_count(self):
self.assertEqual(reading_cost.tokens_in([{"message": {"id": "m", "usage": {"input_tokens": "7", "cache_creation_input_tokens": 2}}}]), 2)
class Figures(unittest.TestCase):
def test_rises_only_above_the_five_before_and_needs_ten(self):
flat = [Reading(100, collections.Counter({"a.py": 1}))] * 10
self.assertEqual(reading_cost.trend(flat[:9]), "9 chunks measured; the trend needs 10.")
self.assertTrue(reading_cost.trend(flat).startswith("Not rising: the last 5 chunks read 100 tokens"))
self.assertEqual(reading_cost.trend(flat[:5] + [Reading(101, collections.Counter({"b.py": 2}))] * 5), "Rising: the last 5 chunks read 101 tokens on average before their first commit, the 5 before them 100.\n\nRead most over the last 5:\n\n- `b.py` 10")
def test_a_figure_reads_back_as_the_reading_it_was_written_from_and_a_damaged_one_fails(self):
reading = Reading(7, collections.Counter({"b.py": 1, "a.py": 3}))
written = reading_cost.figure(FinishedChunk("abcdef0123456789", "t", pathlib.Path(), []), reading)
self.assertEqual(written, "## Reading cost, chunk abcdef012345, finished t\n\n7 tokens read before the first commit.\n\nRead most:\n\n- `a.py` 3\n- `b.py` 1\n")
self.assertEqual(reading_cost.readings_of(written), [reading])
self.assertEqual(reading_cost.readings_of(written.replace("7 tokens", "seven tokens")), Failure('the figure under "## Reading cost, chunk abcdef012345, finished t" holds no single tokens line; restore it from git history'))
class ReadingCostTool(unittest.TestCase):
def setUp(self):
temp = tempfile.TemporaryDirectory()
self.addCleanup(temp.cleanup)
self.home = pathlib.Path(temp.name)
self.home = pathlib.Path(os.path.realpath(temp.name))
self.env = {**host_env(), "GIT_CONFIG_GLOBAL": os.devnull, "HOME": str(self.home)}
self.checkout = self.home / "code" / "proj"
self.checkout.mkdir(parents=True)
subprocess.run(["git", "init", "-q"], cwd=self.checkout, check=True)
subprocess.run(["git", "-c", "user.name=t", "-c", "user.email=t@t", "commit", "-q", "--allow-empty", "-m", "x"], cwd=self.checkout, check=True)
self.pending = self.home / "archive" / "proj" / "reading-cost.md"
self.pending.parent.mkdir(parents=True)
(self.checkout / "a.py").write_text("")
self.git("init", "-q")
self.git("add", "a.py")
self.git("-c", "user.name=t", "-c", "user.email=t@t", "commit", "-q", "-m", "x")
self.paths = paths_from(self.home, {})
def run_tool(self, cwd, *args):
env = {**os.environ, "HOME": str(self.home), "LOOP_CLAUDE_TRANSCRIPTS": str(self.home / "archive")}
return subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, cwd=cwd, env=env, text=True)
def git(self, *args):
subprocess.run(["git", *args], cwd=self.checkout, check=True, env=self.env)
def test_appends_each_figure_once_from_a_worktree_and_reports_a_rise_with_the_places_read_most(self):
self.pending.write_text("\n".join(figure(f"c{n}", 100) for n in range(5)) + "\n" + "\n".join(figure(f"d{n}", 101, "b.py") for n in range(5)))
def archive_chunk(self, at, outcome=FINISHED):
transcript = self.home / ".claude" / "projects" / "-proj" / f"{folder_name()}.jsonl"
(transcript.with_suffix("") / "subagents").mkdir(parents=True)
worker = [said("1", "m1", tokens=10, cwd=str(self.checkout), name="Read", input={"file_path": str(self.checkout / "a.py")}), said("2", "m2", command="git commit -m x")]
transcript.write_bytes(jsonl([{"timestamp": str(at)}, *worker]))
(transcript.with_suffix("") / "subagents" / "agent-a.jsonl").write_bytes(jsonl([said("1", "s1", tokens=1)]))
with contextlib.redirect_stderr(io.StringIO()):
self.assertIsNone(archive("proj", str(transcript), "Do it.", f"{outcome}\n\nEvidence:\n> merged", at, self.paths))
def run_tool(self, cwd):
return subprocess.run([sys.executable, str(TOOL)], capture_output=True, cwd=cwd, env=self.env, text=True)
def test_appends_each_finished_chunk_once_in_the_order_they_finished_from_a_worktree(self):
self.archive_chunk(1790000200)
self.archive_chunk(1790000100)
self.archive_chunk(1790000000, UNFINISHED)
chunks = finished_chunks(self.paths.transcript_archive / "proj")
self.assertEqual([chunk.finished for chunk in chunks], ["2026-09-21T14:15:00Z", "2026-09-21T14:16:40Z"])
worktree = self.home / "code" / "proj-wt"
subprocess.run(["git", "worktree", "add", "-q", str(worktree)], cwd=self.checkout, check=True)
self.git("worktree", "add", "-q", str(worktree))
ran = self.run_tool(worktree)
self.assertEqual(ran.returncode, 0, ran.stderr)
history = worktree / "docs" / "comprehension-history.md"
self.assertEqual(ran.stdout, f"Appended 10 reading-cost figures to {history}.\nRising: the last 5 chunks read 101 tokens on average before their first commit, the 5 before them 100.\n\nRead most over the last 5:\n\n- `b.py` 10\n")
self.assertTrue(history.read_text().startswith("# Comprehension history\n\n## Reading cost, chunk c0, finished t\n"))
again = self.run_tool(worktree)
self.assertTrue(again.stdout.startswith(f"Appended 0 reading-cost figures to {history}.\nRising:"))
self.assertEqual(history.read_text().count("## Reading cost"), 10)
def test_reports_an_equal_mean_as_not_rising_and_too_few_figures_as_such(self):
self.pending.write_text("\n".join(figure(f"c{n}", 100) for n in range(9)))
self.assertEqual(self.run_tool(self.checkout).stdout.splitlines()[1], "9 chunks measured; the trend needs 10.")
with open(self.pending, "a") as handle:
handle.write("\n" + figure("c9", 100))
self.assertTrue(self.run_tool(self.checkout).stdout.splitlines()[1].startswith("Not rising: the last 5 chunks read 100 tokens"))
self.assertEqual(ran.stdout, f"Appended 2 reading-cost figures to {history}.\n2 chunks measured; the trend needs 10.\n")
figure = "## Reading cost, chunk {}, finished {}\n\n33 tokens read before the first commit.\n\nRead most:\n\n- `a.py` 1\n"
self.assertEqual(history.read_text(), "# Comprehension history\n\n" + "\n".join(figure.format(chunk.folder[:12], chunk.finished) for chunk in chunks))
self.assertTrue(self.run_tool(worktree).stdout.startswith(f"Appended 0 reading-cost figures to {history}."))
self.assertEqual(history.read_text().count("## Reading cost"), 2)
def test_exits_1_outside_a_checkout(self):
ran = self.run_tool(self.home)
+1 -1
View File
@@ -11,5 +11,5 @@ class Types(unittest.TestCase):
self.assertIsNotNone(floor, "README.md's Install names `Python 3.N or newer`")
if shutil.which("uv") is None:
self.fail("uv on PATH fetches mypy; see README.md's Test line")
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "-m", "rank_todo", "-m", "statusline", "-p", "loop_claude"], capture_output=True, cwd=ROOT / "bin", text=True)
ran = subprocess.run(["uv", "tool", "run", "--python", "%d.%d" % floor, "mypy==2.3.1", "--cache-dir=/dev/null", "--check-untyped-defs", "--disallow-untyped-defs", "-m", "rank_todo", "-m", "reading_cost", "-m", "shell_reads", "-m", "statusline", "-p", "loop_claude"], capture_output=True, cwd=ROOT / "bin", text=True)
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
+4 -4
View File
@@ -110,10 +110,10 @@ lilleman decided to ship each of these under the floor on the day listed. Each l
2026-10-04, lilleman, answering the principles audit of PR #215. Serves goal 4, which keeps credentials, transcripts and memory files off the remote; the names of the host, its infrastructure project and the sandbox VM are none of those. Valid while the repo names them without their addresses or credentials.
## reading cost: loop-claude keeps each finished chunk's figure in its archive, and the next chunk's larv-review moves it into `docs/comprehension-history.md`
## reading cost: larv-review measures each finished chunk from loop-claude's archive and appends the figure to `docs/comprehension-history.md` in the next chunk's PR
2026-10-08, lilleman. Serves goal 1.1: comprehension measured from real work, at no model cost (goal 1.4). loop-claude archives from the main checkout, outside any PR, so a tracked file it wrote there would dirty that checkout; the figure waits in the archive, and `bin/reading_cost.py` appends each one the history lacks, so a PR that never merges loses none. Rejected: loop-claude opening a PR of its own, one more PR per chunk and a tie to the forge; the worker measuring its own transcript before merging, which misses the chunk's tail. The figure counts tokens taken in, because workers read mostly through Bash and subagents, where parsed paths alone miss what they read. Valid while loop-claude archives every finished chunk.
2026-10-08, lilleman. Serves goal 1.1: comprehension measured from real work, at no model cost (goal 1.4). loop-claude archives from the main checkout, outside any PR, so a tracked file it wrote there would dirty that checkout; `bin/reading_cost.py` instead measures each finished chunk the history lacks from the archive, in the next chunk's worktree, so a PR that never merges loses no figure and loop-claude needs no change. Serves goal 4 by naming only files git tracks, and token counts. Rejected: loop-claude opening a PR of its own, one more PR per chunk and a tie to the forge; the worker measuring its own transcript before merging, which misses the chunk's tail. The figure counts tokens taken in, because workers read mostly through Bash and subagents, where parsed paths alone miss what they read. Valid while loop-claude archives every finished chunk with its subagents' transcripts.
## reading cost: a rise is the last 5 chunks' mean above the 5 before them, and keeps one standing plan item
## reading cost: a rise is the last 5 chunks' mean above the 5 before them, and files or refreshes one plan item that only its own chunk or the human removes
2026-10-08, lilleman. Serves goal 1.1: comprehension never falls, so any rise counts. Per-chunk figures are noisy, so a flat trend rises about every other chunk; one standing item, refreshed rather than refiled, keeps that from flooding the plan. No published standard sets this threshold. Valid while figures arrive once per chunk.
2026-10-08, lilleman. Serves goal 1.1: comprehension never falls, so any rise counts. Per-chunk figures are noisy, so a flat trend rises about every other chunk; the item is refreshed, never refiled, and a chunk that sees no rise leaves it, so the plan does not churn with the noise (goal 1.4). No published standard sets this threshold. Valid while figures arrive once per chunk.
+2 -2
View File
@@ -24,7 +24,7 @@ Words: `~/.claude/CLAUDE.md` → "Vocabulary". What a reviewer reports: ~/.claud
Follow `~/.claude/CLAUDE.md` → "Git, commits, and PRs". Never commit to the default branch. Commit, push with upstream set, and open a **draft** PR so CI cycles while the review runs. No remote → review locally and say so in the summary.
**Reading cost.** Run `~/.claude/bin/reading_cost.py` in the chunk's worktree and commit what it appends to `docs/comprehension-history.md`; the commit re-enters no pass. Then keep at most one plan item titled `**Lower the reading cost of the files chunks read most before their first commit.**`. On `Rising:`, file it, or refresh its detail, with the tool's means and files. On `Not rising:`, delete it.
**Reading cost.** Run `~/.claude/bin/reading_cost.py` in the chunk's worktree and commit what it appends to `docs/comprehension-history.md`; the commit re-enters no pass. On `Rising:`, file a plan item titled `**Lower the reading cost of the files chunks read most before their first commit.**` under the next release, with no Exempt reason and the tool's means and files as its detail; where the plan already holds it, refresh its detail instead. Leave the plan as it is on any other output.
## 3. Passes
@@ -89,7 +89,7 @@ Then the comprehension floor, which the scoring run measures. Four simulated rea
Two files of the project keep the floor's records:
- **History.** `docs/comprehension-history.md` keeps every scoring run, appended and never edited. A run is a `## <UTC time, ISO 8601>, <PR or branch> at <commit>` heading naming when the seats finished and the commit they read; a table of each seat's five scores, with a last row holding the four seats' mean of each; and under each seat, labelled `<seat>, hardest first:` or, for the architect, `<seat>, would restructure first:`, its places, each as `path:line` and the unit's name. A `## Reading cost` heading opens a reading-cost figure (§2), not a run. Appending a run re-enters no pass. Resolve a merge conflict in the file by putting the default branch's runs first.
- **History.** `docs/comprehension-history.md` keeps every scoring run, appended and never edited. A run is a `## <UTC time, ISO 8601>, <PR or branch> at <commit>` heading naming when the seats finished and the commit they read; a table of each seat's five scores, with a last row holding the four seats' mean of each; and under each seat, labelled `<seat>, hardest first:` or, for the architect, `<seat>, would restructure first:`, its places, each as `path:line` and the unit's name. A `## Reading cost` heading opens a reading-cost figure (§2), not a run. Appending a run re-enters no pass. Resolve a merge conflict in the file by putting the default branch's runs and figures first, and dropping a figure for a chunk the default branch already holds.
- **Entry.** `docs/decisions.md` keeps one entry whose title opens `comprehension floor:`. It holds each decision to ship under the floor, with its scores.
A `comprehension floor:` entry written before the history may still hold the last scoring run's five scores. While the default branch's history holds no run, the ratchet compares with those scores. The next scoring run moves them, and only them, into the history as its first run: a heading of the date or time the entry gives and whatever it names the run by, its commit included where named, and a table with the mean alone. The rest of the entry stays.
+7 -2
View File
@@ -6,7 +6,7 @@
`Bar = 9`
`Next ID = 83`
`Next ID = 84`
| Goal | W |
|---|---|
@@ -37,6 +37,7 @@
| 59 | 0.1.0 | | **Bound a run of unfinished verdicts on one chunk, so a worker and judge that disagree cannot cycle unattended without end.** | 3 | 3 | 6 | 6 | 1.4 | 19.8 |
| 12 | 0.1.0 | | **Run the test suite in CI on every PR, or record in `docs/decisions.md` why the local run is enough.** | 2 | 4 | 4 | 6 | 1.1 | 16.9 |
| 80 | 0.1.0 | | **State in README → Install that every machine taking this checkout must be a sandbox, as `settings.json`'s auto mode assumes.** | 1 | 1 | 5 | 5 | 4 | 16.8 |
| 83 | 0.1.0 | | **Group `bin/` by tool, so each tool's modules and tests sit together.** | 3 | 4 | 4 | 6 | 1.1 | 15.9 |
| 5 | 0.1.0 | decision | **Key a project on its checkout's full path, and refuse to start while another session uses that key.** | 5 | 5 | 5 | 6 | 1.1 | 15.6 |
| 67 | 0.1.0 | principle | **Point stability-reviewer at the technical principles in place of its own restatement of them.** | 2 | 3 | 4 | 5 | 1.3 | 14.9 |
| 66 | 0.1.0 | principle | **Hold systems-architect's method thresholds only while the comprehension panel finds the flagged code hard.** | 3 | 3 | 4 | 5 | 1.1 | 14.2 |
@@ -112,6 +113,10 @@ The repo has no CI, so larv-review's gate is the local run, and larv-review's Ga
Raised by the principles audit of PR #224. The `settings.json` decision is valid while every machine taking this checkout is a sandbox, but Install asks for a sandbox only for `loop-claude`, so a new machine is never told.
### 83. Group `bin/` by tool, so each tool's modules and tests sit together.
Raised by the architect on PR #225 and by the inherited architect of PR #224's scoring run: `bin/` is one flat folder of tools, the modules they share and the repo's checks, with `loop_claude`'s tests beside them instead of beside the package.
### 5. Key a project on its checkout's full path, and refuse to start while another session uses that key.
The key names the project's session, archive folder and models entry. `project_key` drops the leading `/`, so `~/code/a` and `/a` share all three, and one loop can resume or delete the other's session.
@@ -162,7 +167,7 @@ Claude Code owns that file and writes it. `trust` reads it, changes it and write
### 4. Name each chunk's archive folder by the chunk's start time, so `<key>/` lists chunks in the order they ran.
Today the folder is the SHA-256 of the transcript's first timestamped line, which tells the human nothing; that line carries the start time. Drop the fallback to the first line, which every chunk of a project shares, and give the tests' fake transcripts ISO timestamps.
Today the folder is the SHA-256 of the transcript's first timestamped line, which tells the human nothing; that line carries the start time. Drop the fallback to the first line, which every chunk of a project shares, and give the tests' fake transcripts ISO timestamps. `bin/reading_cost.py` takes a chunk's id from the folder's first 12 characters and skips a chunk whose id the history holds, so give it a way to match a figure written under the old name.
### 52. Record why this checkout lives in Claude Code's own directory, or move it out and link it in.