File items 398, 399 and 400, and move 339, 341 and 298 to 0.13.0 #588

Merged
lilleman merged 2 commits from plan-0.12.0 into main 2026-11-18 08:33:15 +01:00
+91 -46
View File
@@ -6,23 +6,26 @@
`Bar = 0.1`
`Next ID = 398`
`Next ID = 401`
## Items
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 398 | 0.12.0 | | | **Make each reasonable fix an outside reviewer asks for on a PR trigger a reflection that changes the setup so review would have caught it, in this repo first.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 6, 1.2 by 2 | 2.9 |
| 399 | 0.12.0 | | | **Rerun the pass that raised a 🟡 when its fix changes what a test asserts or whether it runs.** | 2 | 2 | 3 | 5 | 3 | 1.1 by 6 | 2.2 |
| 400 | 0.12.0 | | | **Forbid a test that decides at run time whether to check anything in `principles/technical-principles.md`, and point stability-reviewer's Tests checks at it.** | 1 | 2 | 3 | 4 | 2 | 1.1 by 5 | 2.0 |
| 299 | 0.12.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
| 311 | 0.12.0 | | | **Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.0 |
| 316 | 0.12.0 | | | **Launch comprehension seats as an agent type of their own under `agents/`, without `CLAUDE.md` and the skill and agent listings a general-purpose subagent carries.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
| 339 | 0.12.0 | | | **Lower the reading cost of the files a chunk reads most before its first commit.** | 2 | 2 | 1 | 3 | 3 | 1.4 by 3 | 1.0 |
| 341 | 0.12.0 | 311 | | **Hand each worker's claude a lower Read limit through `CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS`, kept only where replays of the archived review rounds find what two unchanged replays find.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.0 |
| 310 | 0.12.0 | | | **Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.** | 3 | 2 | 3 | 6 | 4 | 1.4 by 3 | 0.9 |
| 298 | 0.12.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 0.9 |
| 365 | 0.13.0 | | | **Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
| 368 | 0.13.0 | | | **Call a release review due, call a release ready to cut, and tell whether a chunk owes a scoring run, for a plan outside the checkout, from the project repo's history or from Jira.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.1 by 2 | 1.2 |
| 309 | 0.13.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.2 |
| 366 | 0.13.0 | | | **Count the chunks of a project whose plan sits outside the checkout in the routing figures, from the project repo's history or from Jira.** | 3 | 2 | 2 | 5 | 4 | 1.6 by 2, 1.4 by 2 | 1.1 |
| 339 | 0.13.0 | | | **Lower the reading cost of the files a chunk reads most before its first commit.** | 2 | 2 | 1 | 3 | 3 | 1.4 by 3 | 1.0 |
| 341 | 0.13.0 | 311 | | **Hand each worker's claude a lower Read limit through `CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS`, kept only where replays of the archived review rounds find what two unchanged replays find.** | 2 | 2 | 3 | 5 | 3 | 1.4 by 3 | 1.0 |
| 298 | 0.13.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 0.9 |
| 395 | 0.13.0 | | decision | **Keep each block a save didn't change, with its localId, and the spacing paragraphs beside it as Jira or Confluence holds them.** | 3 | 2 | 2 | 5 | 5 | 1.6 by 4 | 0.8 |
| 367 | 0.13.0 | | | **Check that every write to a Jira plan leaves it ranked, as rule-hook's `todo-rank-check` checks `todo.md` today.** | 2 | 2 | 3 | 5 | 3 | 1.6 by 2 | 0.7 |
| 77 | 0.14.0 | | | **Teach `bin/metric-bench` Go and TypeScript, rerun it once two projects' comprehension histories name places, and bring into review only a metric that tracks the panel.** | 4 | 2 | 1 | 5 | 6 | 1.1 by 7, 1.4 by 7 | 2.1 |
@@ -65,6 +68,48 @@
## Details
### 398. Make each reasonable fix an outside reviewer asks for on a PR trigger a reflection that changes the setup so review would have caught it, in this repo first.
Applies to any reviewer outside the review passes: a forge's auto-reviewer such as CodeRabbit, a human, or any other. When such a reviewer's feedback on a PR is reasonable and we fix it, we also ask why our setup didn't catch it first, and change the setup so it would. A fix that only answers the comment leaves the same gap open for the next chunk.
Where the change goes, in this order of preference:
1. This repo's core: the reviewers' agents, `larv-review`, the hooks and tools, so every project gains.
2. The technical principles or review standards, where a rule was missing or too vague.
3. The project's own settings or instructions (`config.toml`, `AGENTS.md`), only where the gap is that project's alone.
Where the cause is a gap in none of these, say so in the chunk's summary rather than inventing a change. File a change too big for the chunk as its own item (`~/.claude/CLAUDE.md` → "Working loop"). The reflection is part of answering the feedback, so `larv-review` §4 step 6 and §3 → "Other scales" are the places to start.
Touches:
- `skills/larv-review/SKILL.md`: `## 4. Gate` (small), `## 3. Passes` (small)
- `CLAUDE.md`: `## Working loop` (small)
### 399. Rerun the pass that raised a 🟡 when its fix changes what a test asserts or whether it runs.
In a client project's pull request, stability round 1 raised a 🟡: an end-to-end payroll test passes on a used database. It proposed asserting the created row counts. The answer commit added that assertion behind a `test.skip` that runs when every count is zero. A broken payroll also creates zero rows, so a regression became a skip. No pass reviewed the fix, because a 🟡 fix reruns none (`skills/larv-review/SKILL.md` §3 → "Re-entry"). The forge's auto-reviewer found it.
Neither the gate nor the mutation run catches a weakened test, since a skipped test counts as passing. The rerun reviews only the answers (`review-findings.md`), and the cap of two rounds still ends the pass.
Revise the decision `review: a fix for a 🔴 reruns the pass that found it…` in `docs/decisions.md`. Its 2026-10-29 entry holds only "while 🔴 findings in later chunks, fixing code a 🟡 fix wrote, stay rare".
Touches:
- `skills/larv-review/SKILL.md`: `**Re-entry.**` (small)
- `docs/decisions.md`: `## review: a fix for a 🔴 reruns the pass that found it, unless that pass is at its cap, and each pass whose row in larv-review §3's re-entry table the fix meets`
### 400. Forbid a test that decides at run time whether to check anything in `principles/technical-principles.md`, and point stability-reviewer's Tests checks at it.
Today the rule is only implied, by "fails when that behaviour breaks" and "states what it needs and gets it". Proposed bullet under the tests principle: "A test doesn't decide at run time whether to check anything: no conditional skip, early return, or catch around its assertions. A test that needs a state creates it, or fails and names what is missing. BANNED: `test.skip(created.every((count) => count === 0))`, since a broken payroll also creates zero rows. GOOD: check the precondition before the action, failing with its cause, then `expect(created).toEqual([1, 2, 1])`."
Add one line to section 11 of `agents/stability-reviewer.md` that points at the bullet.
Source: a client project's pull request, where a fix hid a regression behind a conditional skip.
Touches:
- `principles/technical-principles.md`: the tests bullet group, `A valuable test checks behaviour` (small)
- `agents/stability-reviewer.md`: `### 11. Tests` (small)
### 299. Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.
The judge's LLM still decides three things: the verdict kind, the escalation it quotes, and whether a release is ready to cut (`docs/decisions.md` → `routing: loop-claude reads the items a chunk takes …`). This item replays the verdict kind. No single rule found so far separates interrupts: a "?" in the worker's final message appears in 48 interrupts and misses 66. A smaller LLM already failed the replay (`docs/decisions.md` → `routing: on a model judge-replay covered, …`: Sonnet returned another kind on 3 interrupts, Haiku missed 7). Untested are combined code rules, a decision table, and a small classifier trained and run on this machine on the archived verdicts (372 finished, 234 interrupt, 42 unfinished, 20 quoted). Run each offline, nothing sent to an outside service (goal 4), and count disagreements by kind; the bar is zero missed interrupts. A hosted decision model is out of scope, since it would send transcripts to an outside service.
@@ -105,36 +150,6 @@ Touches:
- `bin/rule_hook/test_rule_hook.py`: `ReviewerRung`
- `bin/larv_review/test_scoring_setup.py`
### 339. Lower the reading cost of the files a chunk reads most before its first commit.
`reading-cost` printed `Rising:` on 2026-11-12: the last 5 chunks' median is 128,199 new input tokens before their first commit, the 5 before them 101,795. Read most over the last 5, as opens and searches: `todo.md` 34, `docs/decisions.md` 26, `README.md` 19, `skills/larv-review/SKILL.md` 13, `bin/loop_claude/routing.py` 8, `CHANGELOG.md` 6. The chunk that ships this runs `reading-cost --reset`.
Touches:
- `README.md`
- `skills/larv-review/SKILL.md`
- `bin/loop_claude/routing.py`
### 341. Hand each worker's claude a lower Read limit through `CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS`, kept only where replays of the archived review rounds find what two unchanged replays find.
Raised by item 312's research round, 2026-11-12. Over the 53 chunks archived since 2026-09-30, Read results held about 3.17M tokens (characters / 4) in 1,617 reads. Results over 20,000 characters, about 5,000 tokens, held 479k tokens beyond that size in 115 reads: prose reviewers 144k, stability reviewers 119k, product-owner reviewers 102k, general-purpose subagents 65k, principles auditors 28k, architects 21k. Workers had none, under the reading rule in their prompt. Measured with a script over loop-claude's archive that sums each Read tool_result's characters by the subagent's `agentType`. The decision of 2026-10-27 found Read results out of a PostToolUse hook's reach; this variable reaches them.
`CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS` overrides the Read tool's default limit of 25,000 tokens (https://code.claude.com/docs/en/env-vars, read 2026-11-12: "Override the default token limit for file reads, which is 25,000 tokens."). Claude Code 2.1.322's binary holds the name. A read over the limit fails and asks for an offset and a limit, so an agent may page through the same file and save nothing; the replay measures that.
loop-claude can hand it as it hands `CLAUDE_CODE_AUTO_COMPACT_WINDOW`, and subagents inherit it, the principles auditors included. The choice of 2026-10-27 that the principles audit reads the project whole holds as long as it pages through what it reads.
The replay: give `review-replay` a way to set the variable, then replay the archived prose, stability, product-owner and principles rounds at 5,000 tokens, beside two unchanged replays of the same rounds. Keeps quality where each pass finds every finding the two unchanged replays both find, or misses no more than the most an unchanged replay misses, as `review-replay --omit-claude-md` counts it. Keep it only where the replays' Read characters also fall; item 311's stability replay can share the baseline runs.
Blocked by item 311, 2026-11-17: item 311's chunk already generalizes `review-replay`'s `--omit-claude-md` comparison into a `Change` of the replayed reviewer's environment (`bin/review_replay/changes.py`, `change_comparison.py`) and edits the same lines of `worker_environment`, `HELP` and `launch`; this item adds its own `Change` for `CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS` on top of it once it merges.
Touches:
- `bin/loop_claude/cycle/worker.py`: `COMPACT_WINDOW` (small)
- `bin/loop_claude/cycle/run.py`: `worker_environment` (small)
- `bin/loop_claude/loop/usage.py`: `HELP` (small)
- `bin/review_replay/main.py`: `(module)` (small), `main`
- `bin/review_replay/replay.py`: `launch`, `NO_CLAUDE_MD` (small)
- `bin/review_replay/test_review_replay.py`
- `bin/loop_claude/test_loop_claude.py`: `Cycles` (small), `HelpLimits` (small)
### 310. Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.
281 worker resumes after the 1-hour cache expired rewrote a whole context, $379 of the $9,154 measured from 2026-09-30 to 2026-11-11 (4.1%). A compaction while the cache is warm reads the context at a twentieth of the input price and leaves about 20k tokens to rewrite later.
@@ -152,18 +167,6 @@ Touches:
- `bin/loop_claude/test_loop_claude.py`: `FakeEffects`, `Interrupts`
- `bin/loop_effects/test_effects.py`
### 298. Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.
153 `Judge cost:` entries total $109.76, a median of about $0.76 a verdict at every rung, and the cost follows the size of the records, not the transcript: about 76k characters in log and 188k in claude-config. In 224 of 251 recent verdicts (89%) the evidence sits in the worker's final message. A missed interrupt is the costly error, so the bar is zero changed interrupts and zero new rejections across the replay. The judge is about 4.5% of dollars, so the saving is small; each cycle also gets faster.
Touches:
- `bin/loop_claude/cycle/judge.py`: `Brief`, `judge_prompt`, `JUDGE_PROMPT_CHARS`
- `bin/loop_claude/project/records.py`: `read_records`
- `bin/judge_replay/main.py`: `rule` (small)
- `bin/judge_replay/replay.py`: `case_from` (small)
- `bin/loop_claude/test_loop_claude.py`: `JudgePrompt`, `Records`
- `bin/judge_replay/test_judge_replay.py`
### 365. Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.
lilleman's call, 2026-11-12: 0.10.0 and 0.11.0 switch the history tools off for a plan outside the checkout, and 0.13.0 holds only their ports. In Jira a dropped item, which `Drop item` commit titles mark today, is an issue resolved by a resolution the config names.
@@ -205,6 +208,48 @@ Touches:
One of 0.13.0's history ports, per lilleman's call of 2026-11-12.
### 339. Lower the reading cost of the files a chunk reads most before its first commit.
`reading-cost` printed `Rising:` on 2026-11-12: the last 5 chunks' median is 128,199 new input tokens before their first commit, the 5 before them 101,795. Read most over the last 5, as opens and searches: `todo.md` 34, `docs/decisions.md` 26, `README.md` 19, `skills/larv-review/SKILL.md` 13, `bin/loop_claude/routing.py` 8, `CHANGELOG.md` 6. The chunk that ships this runs `reading-cost --reset`.
Touches:
- `README.md`
- `skills/larv-review/SKILL.md`
- `bin/loop_claude/routing.py`
### 341. Hand each worker's claude a lower Read limit through `CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS`, kept only where replays of the archived review rounds find what two unchanged replays find.
Raised by item 312's research round, 2026-11-12. Over the 53 chunks archived since 2026-09-30, Read results held about 3.17M tokens (characters / 4) in 1,617 reads. Results over 20,000 characters, about 5,000 tokens, held 479k tokens beyond that size in 115 reads: prose reviewers 144k, stability reviewers 119k, product-owner reviewers 102k, general-purpose subagents 65k, principles auditors 28k, architects 21k. Workers had none, under the reading rule in their prompt. Measured with a script over loop-claude's archive that sums each Read tool_result's characters by the subagent's `agentType`. The decision of 2026-10-27 found Read results out of a PostToolUse hook's reach; this variable reaches them.
`CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS` overrides the Read tool's default limit of 25,000 tokens (https://code.claude.com/docs/en/env-vars, read 2026-11-12: "Override the default token limit for file reads, which is 25,000 tokens."). Claude Code 2.1.322's binary holds the name. A read over the limit fails and asks for an offset and a limit, so an agent may page through the same file and save nothing; the replay measures that.
loop-claude can hand it as it hands `CLAUDE_CODE_AUTO_COMPACT_WINDOW`, and subagents inherit it, the principles auditors included. The choice of 2026-10-27 that the principles audit reads the project whole holds as long as it pages through what it reads.
The replay: give `review-replay` a way to set the variable, then replay the archived prose, stability, product-owner and principles rounds at 5,000 tokens, beside two unchanged replays of the same rounds. Keeps quality where each pass finds every finding the two unchanged replays both find, or misses no more than the most an unchanged replay misses, as `review-replay --omit-claude-md` counts it. Keep it only where the replays' Read characters also fall; item 311's stability replay can share the baseline runs.
Blocked by item 311, 2026-11-17: item 311's chunk already generalizes `review-replay`'s `--omit-claude-md` comparison into a `Change` of the replayed reviewer's environment (`bin/review_replay/changes.py`, `change_comparison.py`) and edits the same lines of `worker_environment`, `HELP` and `launch`; this item adds its own `Change` for `CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS` on top of it once it merges.
Touches:
- `bin/loop_claude/cycle/worker.py`: `COMPACT_WINDOW` (small)
- `bin/loop_claude/cycle/run.py`: `worker_environment` (small)
- `bin/loop_claude/loop/usage.py`: `HELP` (small)
- `bin/review_replay/main.py`: `(module)` (small), `main`
- `bin/review_replay/replay.py`: `launch`, `NO_CLAUDE_MD` (small)
- `bin/review_replay/test_review_replay.py`
- `bin/loop_claude/test_loop_claude.py`: `Cycles` (small), `HelpLimits` (small)
### 298. Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.
153 `Judge cost:` entries total $109.76, a median of about $0.76 a verdict at every rung, and the cost follows the size of the records, not the transcript: about 76k characters in log and 188k in claude-config. In 224 of 251 recent verdicts (89%) the evidence sits in the worker's final message. A missed interrupt is the costly error, so the bar is zero changed interrupts and zero new rejections across the replay. The judge is about 4.5% of dollars, so the saving is small; each cycle also gets faster.
Touches:
- `bin/loop_claude/cycle/judge.py`: `Brief`, `judge_prompt`, `JUDGE_PROMPT_CHARS`
- `bin/loop_claude/project/records.py`: `read_records`
- `bin/judge_replay/main.py`: `rule` (small)
- `bin/judge_replay/replay.py`: `case_from` (small)
- `bin/loop_claude/test_loop_claude.py`: `JudgePrompt`, `Records`
- `bin/judge_replay/test_judge_replay.py`
### 395. Keep each block a save didn't change, with its localId, and the spacing paragraphs beside it as Jira or Confluence holds them.
Today a Jira detail edited in one line, or a Confluence record with one entry appended, is written back whole by adf-codec's portable pair, which drops each block's localId and the empty paragraphs Jira uses for spacing. A save whose text is the text a read showed already sends nothing (`docs/decisions.md` → `project config: the loop shows a Jira description and a Confluence page as adf-codec's portable markdown, …`). Match the saved markdown's top-level blocks against the held document's, each as the portable pair reads it, and send the held block where they match. The decision's limit on partial saves ends with this item.