Predict touches for release 0.12.0 items #570

Merged
lilleman merged 2 commits from predict-touches into main 2026-11-17 01:46:55 +01:00
+104 -1
View File
@@ -153,12 +153,26 @@ Both `docs/decisions.md` entries under `reading cost: a rise is …` and `readin
Open before building: the lossless markdown is less readable than the portable one (upstream larvit/adf-codec#185), and agents and the judge read the plan's details from it. Decide how a detail stays readable, for example by reading lossless and showing a person a portable view. Test that an unedited issue and page save back identical. The item ships alone because its R is above 5.
Touches:
- `bin/plan_store/jira_epic.py`: `JiraPlan` (small), `description_of` (small)
- `bin/record_store/confluence_page.py`: `ConfluencePage` (small)
- `bin/plan_store/test_jira_epic.py`: `Editing`, `Filing`
- `bin/record_store/test_confluence_page.py`: `ConfluencePageTest`
### 319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
On SWE-bench Verified with Haiku 4.5 and Opus 4.7, handing the strong model the weak model's transcript solved 69.2% at $1.61 a task; starting the strong model fresh solved 79.2% at $0.90 (arXiv 2608.24358, The Handoff Tax). A model switch also rereads the whole context uncached (Claude Code prompt caching docs). A worker that steps from Sonnet 5.5 to Opus 5.5 by `--resume` is the raw handoff; an effort-only step within one model keeps the cache and isn't. The escalated subagent, which starts fresh with the trigger's evidence, already follows the pattern that won.
Bench it first on `workflow-bench`: resume against a fresh start with a brief. The brief is the chunk's item, its branch and diff, open findings and the trigger. lilleman approved on 2026-11-11. Revise the decision `routing: a worker whose run ends asking for a higher rung…`.
Touches:
- `bin/loop_claude/cycle/run.py`: `Cycle`, `run_cycle`, `worker_start`
- `bin/loop_claude/cycle/prompts.py`: `resumed`, `stepped_to`
- `bin/loop_claude/cycle/worker.py`: `worker_args`
- `bin/loop_claude/cycle/states.py`
- `bin/loop_claude/test_loop_claude.py`: `Cycles`, `WorkerArgs`
- `workflow-bench/bench.py`: `run`, `main`
### 309. Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.
Seats cost $1,143 of the $9,154 measured from 2026-09-30 to 2026-11-11 (12.5%), scoring runs $470 of that. A scoring run today is four seats, run twice on one head: eight seat runs. Two seats on the second pass make it six, and two on both passes four.
@@ -167,24 +181,65 @@ Why it should hold quality: the seats are one model with different personas, so
Decide once `agreement` has its 10 commits scored twice. Keep four seats on any pass that files a floor item, since its places come from seat reports. Revise the decision `review: the comprehension check records how hard a project is to read…`, which names the four-seat run.
Touches:
- `bin/larv_review/seats.py`: `SCORING_SEATS`, `scores`
- `bin/larv_review/history_commands.py`: `record_scoring`
- `bin/larv_review/scoring_setup.py`: `SEATS`, `prepare_scoring`
- `bin/larv_review/main.py`: `(module)` (small)
- `bin/larv_review/test_larv_review.py`
- `bin/larv_review/test_scoring_setup.py`
- `skills/larv-review/SKILL.md`: `### Comprehension check`
- `skills/comprehension-panel/SKILL.md`: `## The scoring run` (small)
### 299. Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.
The judge's LLM still decides three things: the verdict kind, the escalation it quotes, and whether a release is ready to cut (`docs/decisions.md` → `routing: loop-claude reads the items a chunk takes …`). This item replays the verdict kind. No single rule found so far separates interrupts: a "?" in the worker's final message appears in 48 interrupts and misses 66. A smaller LLM already failed the replay (`docs/decisions.md` → `routing: on a model judge-replay covered, …`: Sonnet returned another kind on 3 interrupts, Haiku missed 7). Untested are combined code rules, a decision table, and a small classifier trained and run on this machine on the archived verdicts (372 finished, 234 interrupt, 42 unfinished, 20 quoted). Run each offline, nothing sent to an outside service (goal 4), and count disagreements by kind; the bar is zero missed interrupts. A hosted decision model is out of scope, since it would send transcripts to an outside service.
Touches:
- `bin/judge_replay/main.py`: `main`, `archived_cases` (small)
- `bin/judge_replay/replay.py`: `Case`, `verdict_kind`, `report`
- `bin/judge_replay/test_judge_replay.py`
### 311. Hand each worker's claude a Bash output ceiling that saves longer output to a file, and refuse a Read of a session transcript under `~/.claude/projects`.
Workers took 7,870 Bash results over 3,000 characters, $305 of carried context, against the reading rule in their prompt; stability reviewers another $145. `BASH_MAX_OUTPUT_LENGTH` (or `bashOutputMaxChars`) makes Claude Code save a longer result to a file and hand the model its path and the first 2,000 characters. loop-claude can hand it as it hands `CLAUDE_CODE_AUTO_COMPACT_WINDOW`. A ceiling of 6,000 characters would have removed $228 of carried output, 10,000 characters $122. Reads of old session transcripts cost $46; a rule-hook rule can refuse them, but not a Read of the files Claude Code saves a long Bash result to: they sit under `~/.claude/projects/<project>/<session>/tool-results/`, the archive holds 85 of them (item 312's round, 2026-11-12), and the saved-to-file ceiling hands the model their path to read.
Subagents inherit the variable, and the decision of 2026-10-27 found a 12k cap cost stability reviewers a finding (3 of 6 against 4 of 6). That cap left no file to read, and the difference is inside the noise that decision reports. So keep the ceiling only where a replay of the archived stability rounds finds what two baseline replays find, and join item 260's comparison.
Touches:
- `bin/loop_claude/cycle/worker.py`: `COMPACT_WINDOW` (small)
- `bin/loop_claude/cycle/run.py`: `worker_environment` (small)
- `bin/loop_claude/loop/usage.py`: `HELP` (small)
- `bin/rule_hook/rules.py`: `RuleName` (small), `RULES` (small)
- `bin/rule_hook/main.py`: `breaches_of` (small)
- `bin/rule_hook/test_rule_hook.py`: `RuleNames` (small)
- `bin/review_replay/replay.py`: `launch` (small)
- `bin/loop_claude/test_loop_claude.py`: `Cycles` (small), `HelpLimits` (small)
### 316. Launch comprehension seats as an agent type of their own under `agents/`, without `CLAUDE.md` and the skill and agent listings a general-purpose subagent carries.
Seats run as general-purpose subagents, so each starts with the skill and agent listings, about 7,500 tokens, at $9.00 per 1,000 prefix tokens over the archive measured from 2026-09-30 to 2026-11-11: about 1.3% of spend. An agent under `agents/` with Read, Grep, Glob and Bash, `omitClaudeMd: true`, and the seat's persona in its prompt drops both; a seat reads the project as a newcomer does, which is what it simulates. Keep each seat's prompt from `skills/comprehension-panel/seat-prompts.md`. Check that a scoring run's scores on one head stay within the spread two runs of today's seats show. lilleman approved on 2026-11-11.
Touches:
- `agents/comprehension-seat.md`
- `bin/rule_hook/reviewer_rung.py`: `SEAT`, `review_kind`
- `bin/larv_review/scoring_setup.py`: `SEATS`, `prepare_scoring`
- `bin/loop_claude/cycle/prompts.py`: `review_launches` (small)
- `skills/comprehension-panel/seat-prompts.md`: `## Opening`, `(top)` (small)
- `skills/comprehension-panel/SKILL.md`: `## The scoring run` (small)
- `skills/larv-review/SKILL.md`: `### Comprehension check` (small)
- `bin/rule_hook/test_rule_hook.py`: `ReviewerRung`
- `bin/larv_review/test_scoring_setup.py`
### 339. Lower the reading cost of the files a chunk reads most before its first commit.
`reading-cost` printed `Rising:` on 2026-11-12: the last 5 chunks' median is 128,199 new input tokens before their first commit, the 5 before them 101,795. Read most over the last 5, as opens and searches: `todo.md` 34, `docs/decisions.md` 26, `README.md` 19, `skills/larv-review/SKILL.md` 13, `bin/loop_claude/routing.py` 8, `CHANGELOG.md` 6. The chunk that ships this runs `reading-cost --reset`.
Touches:
- `README.md`
- `skills/larv-review/SKILL.md`
- `bin/loop_claude/routing.py`
### 341. Hand each worker's claude a lower Read limit through `CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS`, kept only where replays of the archived review rounds find what two unchanged replays find.
Raised by item 312's research round, 2026-11-12. Over the 53 chunks archived since 2026-09-30, Read results held about 3.17M tokens (characters / 4) in 1,617 reads. Results over 20,000 characters, about 5,000 tokens, held 479k tokens beyond that size in 115 reads: prose reviewers 144k, stability reviewers 119k, product-owner reviewers 102k, general-purpose subagents 65k, principles auditors 28k, architects 21k. Workers had none, under the reading rule in their prompt. Measured with a script over loop-claude's archive that sums each Read tool_result's characters by the subagent's `agentType`. The decision of 2026-10-27 found Read results out of a PostToolUse hook's reach; this variable reaches them.
@@ -195,6 +250,15 @@ loop-claude can hand it as it hands `CLAUDE_CODE_AUTO_COMPACT_WINDOW`, and subag
The replay: give `review-replay` a way to set the variable, then replay the archived prose, stability, product-owner and principles rounds at 5,000 tokens, beside two unchanged replays of the same rounds. Keeps quality where each pass finds every finding the two unchanged replays both find, or misses no more than the most an unchanged replay misses, as `review-replay --omit-claude-md` counts it. Keep it only where the replays' Read characters also fall; item 311's stability replay can share the baseline runs.
Touches:
- `bin/loop_claude/cycle/worker.py`: `COMPACT_WINDOW` (small)
- `bin/loop_claude/cycle/run.py`: `worker_environment` (small)
- `bin/loop_claude/loop/usage.py`: `HELP` (small)
- `bin/review_replay/main.py`: `(module)` (small), `main`
- `bin/review_replay/replay.py`: `launch`, `NO_CLAUDE_MD` (small)
- `bin/review_replay/test_review_replay.py`
- `bin/loop_claude/test_loop_claude.py`: `Cycles` (small), `HelpLimits` (small)
### 310. Compact a waiting worker's session shortly before its prompt cache expires, only when the human has not answered an interrupt by then.
281 worker resumes after the 1-hour cache expired rewrote a whole context, $379 of the $9,154 measured from 2026-09-30 to 2026-11-11 (4.1%). A compaction while the cache is warm reads the context at a twentieth of the input price and leaves about 20k tokens to rewrite later.
@@ -203,16 +267,40 @@ lilleman, 2026-11-11: do it on a timer, so an answer that comes in time doesn't
Probe first that `claude -p "/compact" --resume <id>` compacts a session: the Agent SDK documents `/compact` as a prompt, the CLI pages don't show it. Claude Code compacts idle interactive sessions itself ("Compacted while idle"), but not `-p` runs. Check: the cost entries of 20 chunks with an interrupt, before and after.
Touches:
- `bin/loop_claude/cycle/interrupt.py`: `interrupt`, `answered`, `typed_answer`, `WorkerSession`
- `bin/loop_effects/effects.py`: `Effects`, `system_effects`
- `bin/claude_code/tokens.py`: `tokens_of` (small)
- `bin/claude_code/reply.py`: `claude_reply` (small)
- `bin/loop_claude/loop/usage.py`: `HELP` (small)
- `bin/loop_claude/test_loop_claude.py`: `FakeEffects`, `Interrupts`
- `bin/loop_effects/test_effects.py`
### 393. Step an L 3 worker one rung up and an L 4 worker two up from the project's rung, as lilleman ruled on 2026-11-14.
lilleman ruled on 2026-11-14 (`docs/decisions.md` → the routing ladder entry, the 2026-11-14 note) that an L 3 task steps one rung up from the project's rung and an L 4 task two up, in every project and for delegated subagents. `bin/model_routing/ladder.py`'s `WORKER_STEPS` still reads `{1: -1, 2: -1, 3: 0, 4: 1}`. The 0.11.0 release review's principles pass found the gap, and lilleman chose on 2026-11-16 to file it for 0.12.0.
Set `WORKER_STEPS` to `{1: -1, 2: -1, 3: 1, 4: 2}`. Then re-derive, from the ruling, the expected rungs of the 26 routing tests that pin the old steps: `bin/model_routing/test_ladder.py`, `bin/subagent_rung/test_subagent_rung.py`, and the routing, step and worker-argument tests in `bin/loop_claude/test_loop_claude.py` and `bin/loop_claude/test_black_box.py`. On an Opus 5.5 project, an L 3 worker or subagent then runs on Fable 5.2 at low, and an escalation stops without lilleman's yes at Opus 5.5 at xhigh; an L 4 one runs on Fable 5.2 at medium, and stops at Fable 5.2 at high. Reword the 2026-11-14 note's "hold until" sentence once the steps change, and cite the routing figures' stamp 2026-11-14T00:59:50Z in `routing-calls-raised.md` as settled.
Touches:
- `bin/model_routing/ladder.py`: `WORKER_STEPS` (small)
- `bin/model_routing/test_ladder.py`: `Ladder`
- `bin/subagent_rung/test_subagent_rung.py`: `SubagentRung`
- `bin/loop_claude/test_loop_claude.py`: `Routing`, `Steps`, `WorkerArgs`
- `bin/loop_claude/test_black_box.py`: `Models`
### 298. Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.
153 `Judge cost:` entries total $109.76, a median of about $0.76 a verdict at every rung, and the cost follows the size of the records, not the transcript: about 76k characters in log and 188k in claude-config. In 224 of 251 recent verdicts (89%) the evidence sits in the worker's final message. A missed interrupt is the costly error, so the bar is zero changed interrupts and zero new rejections across the replay. The judge is about 4.5% of dollars, so the saving is small; each cycle also gets faster.
Touches:
- `bin/loop_claude/cycle/judge.py`: `Brief`, `judge_prompt`, `JUDGE_PROMPT_CHARS`
- `bin/loop_claude/project/records.py`: `read_records`
- `bin/judge_replay/main.py`: `rule` (small)
- `bin/judge_replay/replay.py`: `case_from` (small)
- `bin/loop_claude/test_loop_claude.py`: `JudgePrompt`, `Records`
- `bin/judge_replay/test_judge_replay.py`
### 331. Reword four decisions to expire when the rungs their replays tested change, not whenever a model joins the ladder.
The architect's 🔀 answer on PR #493 (item 292): four decisions expire "when a new model joins the ladder". Fable 5.2 joined above every project model's rung, and none of these replays tested a rung above the project's, so the event fired with nothing new to test. Every later ladder rebuild would fire all four again.
@@ -228,6 +316,10 @@ lilleman approved rewording all four as above on 2026-11-12.
The product-owner pass of item 373's chunk asked it, 2026-11-15: `bin/touch_predictions/prediction.py` → `predicted` needs a line that is exactly `Touches:` and bullets that open `- `. A Jira issue's description reaches it through `bin/adf_codec/`, whose library chooses the bullet marker and may escape backticks or a trailing colon. No test shows that a list written in Jira comes out in that shape. Where it doesn't, read the shape the codec writes too.
Touches:
- `bin/touch_predictions/prediction.py`: `predicted` (small), `bullet` (small)
- `bin/touch_predictions/test_touch_predictions.py`: `PredictedTest`
### 327. Have `larv-review` print a round commit's title prefix, counted from the round's report, and have rule-hook refuse a round title whose counts differ from it.
The 0.8.0 audit (item 284) measured, over the 35 chunks that shipped 0.7.0: 72 commit calls in 33 chunks titled `Answer <pass> round <n>: R 🔴, Y 🟡 (K misread); <what changed>`, about 11k command tokens, most of them the summary. Each worker counts R, Y and K by hand. All three are in the round's report, `<pass>-round-<n>.md` in the review folder: R and Y are its finding headings, and K counts the findings the reviewer marked `[misread]`, as `bin/review_rules/round_titles.py` says. That module parses the counts for `score-outcomes`, which reports them as goal 1.1's signal, and rule-hook's `round_cap.py` checks only the round number, so nothing checks today that the counts match the report. Of the 8 titles whose report the audit could match, none miscounted; one report wrote its markers at line start rather than in `###` headings, so the reports' format varies by reviewer.
@@ -236,6 +328,17 @@ Add a `larv-review` subcommand that prints a round title's prefix, up to and inc
Left with the model: the summary after the `;`, and answering the findings.
Touches:
- `bin/larv_review/main.py`: `(module)` (small), `COMMANDS` (small)
- `bin/review_rules/round_titles.py`
- `bin/rule_hook/rules.py`: `title_breaches`, `RULES` (small)
- `bin/rule_hook/test_rule_hook.py`: `Commits`
- `bin/larv_review/test_larv_review.py`
- `bin/review_rules/test_round_titles.py`
- `skills/larv-review/review-findings.md`: `## Report` (small)
- `skills/larv-review/SKILL.md`: `## 3. Passes` (small)
- `README.md`: `## Tools` (small)
### 389. Set a Jira issue's "is blocked by" links from `todo-item edit --blocked-by`, on an issue the loop filed.
Filed by item 382's chunk on 2026-11-16. `JiraPlan.edit` refuses an edit of Blocked by, saying to link the issue in Jira, so a worker cannot mark an item it filed as blocked once it exists. Read each link's id from the search's `issuelinks`, make a link per new blocker as `add` does, and delete one per blocker removed, after `missing_blocker`'s check. Say in `docs/decisions.md` whether the loop may link an issue a person filed; the decision on the Jira epic as a plan says the loop writes only the scores of such an issue.
@@ -243,7 +346,7 @@ Filed by item 382's chunk on 2026-11-16. `JiraPlan.edit` refuses an edit of Bloc
Touches:
- `bin/plan_store/jira_epic.py`: `Issue`, `issue_of`, `JiraPlan.edit`, `revision_of`
- `bin/plan_store/test_jira_epic.py`: `Editing`
- `bin/todo_item/main.py`: the help
- `bin/todo_item/main.py`: lines 1-77 (small)
### 365. Have `score-outcomes` read the finished and dropped items of a plan outside the checkout, from the project repo's history or from Jira's issue changelogs.