Answer prose round 1: 0 🔴, 7 🟡 (0 misread); the codec pads on a space at both ends too, and the Valid lines name the event that expires them

This commit is contained in:
2026-11-17 03:51:51 +01:00
parent f56962cd8d
commit 2e44ec95ca
3 changed files with 10 additions and 11 deletions
+2 -2
View File
@@ -10,7 +10,7 @@ The spelling, which the decision `todo.md: an item of the next release predicts
A path alone is the whole file, and (small) marks a part the item changes only a little. file_parts says how a part is named.
A path or a name that holds a backtick is spelt as a Markdown code span that holds one: in a longer run of backticks than any it holds, with a space inside each end, as in `` ## The `foo` tool ``. So every path and name on one line has a spelling. A description kept in Jira reaches here through adf_codec, which spells such a name without the spaces where it neither opens nor ends in a backtick, as ``a`b.md``; that is read as the same name."""
A path or a name that holds a backtick is spelt as a Markdown code span that holds one: in a longer run of backticks than any it holds, with a space inside each end, as in `` ## The `foo` tool ``. So every path and name on one line has a spelling. A description kept in Jira reaches here through adf_codec, which pads a span only where its text opens or ends in a backtick, or opens and ends in a space. So it spells a`b.md as ``a`b.md``; code_span reads that as the name too."""
import dataclasses
import re
@@ -84,7 +84,7 @@ def code_span(line: str, at: int) -> tuple[str, int] | None:
if len(run) > 1:
if not text.strip():
return None
# As a Markdown code span reads: a space inside each end keeps a backtick at either end of the text apart from the run, and goes. adf_codec pads only that case, so no space is also the text.
# A space at both ends is padding, which keeps a backtick at either end of the text apart from the run. Without one at both ends, the span is its text.
if len(text) >= 3 and text.startswith(" ") and text.endswith(" "):
text = text[1:-1]
return (text, opened.end() + closed.end()) if text else None
@@ -135,8 +135,7 @@ class PredictedTest(unittest.TestCase):
self.assertEqual(predicted("Touches:\n- `a.md`: `` ## The `foo` tool `` (small), ``` `` ```, `` `x` ``\n"), {"a.md": (Part("## The `foo` tool", small=True), Part("``"), Part("`x`"))})
self.assertEqual(rendered({"a.md": (Part("## The `foo` tool", small=True), Part("``"))}), "- `a.md`: `` ## The `foo` tool `` (small), ``` `` ```")
def test_a_longer_run_without_the_spaces_the_text_does_not_need_reads_as_its_text(self) -> None:
# The spelling adf_codec gives a code mark: it pads a longer run only where the text opens or ends in a backtick.
def test_a_longer_run_without_padding_spaces_is_the_text_between_its_runs(self) -> None:
self.assertEqual(predicted("Touches:\n- ``a`b.md``: ``## The `foo` tool`` (small), ``` `` ```, `` `x` ``\n"), {"a`b.md": (Part("## The `foo` tool", small=True), Part("``"), Part("`x`"))})
def test_a_longer_run_with_a_space_at_one_end_only_keeps_it(self) -> None:
@@ -149,7 +148,7 @@ class PredictedTest(unittest.TestCase):
self.assertEqual(predicted(convert.adf_to_markdown(description)), predicted(DETAIL))
@needs_docker
def test_a_list_typed_in_jiras_editor_reads_as_the_item_it_names(self) -> None:
def test_a_list_typed_in_jiras_editor_reads_as_its_files_and_parts(self) -> None:
read = convert.adf_to_markdown(JIRA_TOUCHES)
self.assertEqual(predicted(read), {
"bin/a.py": (Part("main"), Part("Plan.branch", small=True)),
@@ -168,7 +167,7 @@ class PredictedTest(unittest.TestCase):
touches = {"my file.md": (Part("## todo.md: under `Touches:` in its detail"), Part("## ``a`` and `b`", small=True), Part("`"), Part("## ends in `x`"), Part(" x "), Part("lines 2-3")), "a`b.md": (), "c, d: e.md": (Part("# T"),)}
self.assertEqual(predicted(f"Touches:\n\n{rendered(touches)}\n"), touches)
def test_a_longer_run_without_its_end_or_a_name_is_a_failure(self) -> None:
def test_a_bullet_with_an_unclosed_run_a_blank_name_or_a_stray_separator_is_a_failure(self) -> None:
for bullet in ("- `a.md`: `` ## A", "- `a.md`: `` ## A `", "- `a.md`: `` ``", "- `a.md`: ", "- `a.md`:", "- `a.md`: `f`, ", "- `a.md`: `f`,`g`", "- ``", "- `a.md` (small)", "- `a.md`: `f` (small) (small)"):
with self.subTest(bullet):
self.assertIsInstance(predicted(f"Touches:\n\n{bullet}\n"), Failure)
+5 -5
View File
@@ -281,11 +281,11 @@ Valid while no published standard sets either number, the reading-cost window st
## reading cost: a rise is the last 5 chunks' median above the 5 before them, and files or refreshes one plan item
2026-10-08, lilleman. Serves code health and goal 1.4's counter-signal, a rising reading cost: so any rise counts. Per-chunk figures are noisy: a flat trend rises about every other chunk. So a rise refreshes the item and never refiles it, and a chunk that sees no rise leaves the item as it is. The chunk that ships the item resets the trend, so later figures do not refile it for work that chunk already did (goal 1.4). A median, because one chunk that ran a panel of readers before its first commit measured 2.99M tokens against a typical 80k–370k, and a mean let it decide the trend alone. No published standard sets this threshold. Valid while figures arrive once per chunk.
2026-10-08, lilleman. Serves code health and goal 1.4, whose counter-signal is a rising reading cost: any rise counts. Per-chunk figures are noisy: a flat trend rises about every other chunk. So a rise refreshes the item and never refiles it, and a chunk that sees no rise leaves the item as it is. The chunk that ships the item resets the trend, so later figures do not refile it for work that chunk already did (goal 1.4). A median, because one chunk that ran a panel of readers before its first commit measured 2.99M tokens against a typical 80k–370k, and a mean let it decide the trend alone. No published standard sets this threshold. Valid while figures arrive once per chunk.
## reading cost: larv-review measures each finished chunk from loop-claude's archive and appends the figure to `docs/reading-cost.md` in the next chunk's PR
2026-10-08, lilleman. Serves code health and goal 1.4's counter-signal: reading cost measured from real work, at no model cost (goal 1.4). loop-claude archives from the main checkout, outside any PR, so a tracked file it wrote there would dirty that checkout. So `bin/reading-cost` measures each finished chunk `docs/reading-cost.md` lacks from the archive, in the next chunk's worktree. A PR that never merges then loses no figure, and loop-claude needs no change. The figures keep a file of their own, so `docs/comprehension-history.md` keeps one writer, the comprehension check. Serves goal 4 by naming only files git tracks, and token counts. Rejected: loop-claude opening a PR of its own, one more PR per chunk and a tie to the forge; the worker measuring its own transcript before merging, which misses the chunk's tail. The figure counts new input tokens, uncached and cache-written, because workers read mostly through Bash and subagents, where parsed paths alone miss what they read. Valid while loop-claude archives every finished chunk with its subagents' transcripts.
2026-10-08, lilleman. Serves code health and goal 1.4, whose counter-signal is reading cost: measured from real work, at no model cost. loop-claude archives from the main checkout, outside any PR, so a tracked file it wrote there would dirty that checkout. So `bin/reading-cost` measures each finished chunk `docs/reading-cost.md` lacks from the archive, in the next chunk's worktree. A PR that never merges then loses no figure, and loop-claude needs no change. The figures keep a file of their own, so `docs/comprehension-history.md` keeps one writer, the comprehension check. Serves goal 4 by naming only files git tracks, and token counts. Rejected: loop-claude opening a PR of its own, one more PR per chunk and a tie to the forge; the worker measuring its own transcript before merging, which misses the chunk's tail. The figure counts new input tokens, uncached and cache-written, because workers read mostly through Bash and subagents, where parsed paths alone miss what they read. Valid while loop-claude archives every finished chunk with its subagents' transcripts.
## review: `/security-review` is the security pass of record, except where the full scan runs on the same change and replaces it
@@ -408,7 +408,7 @@ Valid while the ladder orders its rungs by competence.
## review: each pass keeps a reviewer of its own
2026-11-09, lilleman asked for the trials. Serves goals 1.1 and 1.4. Merged reviewers were replayed on archived first rounds, beside the separate reviewers (`reviewer-merge-trials/README.md`). Each merge cost a third to a half less, and in each, one absorbed pass lost findings. Stability, product-owner and prose as one reviewer recovered 15 of 34 prose findings, against 22 and 23 for two separate runs. The architect and the principles auditor as one recovered 5 of 17 principles findings, against 10 for one separate run, and lost both principles 🔴. Stability and product-owner as one, run twice, recovered 4 and 5 of 16 product-owner findings, against 7 and 10 for two separate runs. Folding override-reviewer into stability had no data: no archived override round holds a 🔴 or 🟡. Valid until the rung the trials ran on, Opus 5.5 at medium, changes, or a reviewer's definition changes: replay the merge then, with the scripts in `reviewer-merge-trials/`.
2026-11-09, lilleman asked for the trials. Serves goals 1.1 and 1.4. Merged reviewers were replayed on archived first rounds, beside the separate reviewers (`reviewer-merge-trials/README.md`). Each merge cost a third to a half less, and in each, one absorbed pass lost findings. Stability, product-owner and prose as one reviewer recovered 15 of 34 prose findings, against 22 and 23 for two separate runs. The architect and the principles auditor as one recovered 5 of 17 principles findings, against 10 for one separate run, and lost both principles 🔴. Stability and product-owner as one, run twice, recovered 4 and 5 of 16 product-owner findings, against 7 and 10 for two separate runs. Folding override-reviewer into stability had no data: no archived override round holds a 🔴 or 🟡. Valid until a reviewer's definition changes, and for a project only while its rung is Opus 5.5 at medium, the rung the trials ran on: replay the merge then, with the scripts in `reviewer-merge-trials/`.
## review: each reviewer under `agents/` writes its full report into the chunk's review folder under `~/.local/state/larv-review/` and returns the worker its verdict and one line per finding; rule-hook refuses its Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`
@@ -462,7 +462,7 @@ The base is the last measured commit: the default branch's newest merged change
2026-10-28, claude, through todo item 233. Serves goal 1.4. One auditor per round checks the README's goals with code health, and each principles file that applies. It reads the project once, where one auditor per goal set or file ran 3.8 times per chunk before.
A trial checked that it keeps goal 1.1. It replayed the 6 latest first rounds that launched two or more auditors, at most two per project, from this repo, `log`, `visthus` and `adf-codec`: 15 auditors, whose reports held 40 🔴 and 🟡 findings. Each round ran on the project's rung as `review-replay` replays it, once with its separate auditors and once with one auditor handed everything they checked, its new definition passed through `--agents`. A replayed reviewer finds a different part of its findings each time: the separate auditors found only 10 of their own 40 again. The one auditor found 12, among them 8 of those 10. So it finds as much as the separate auditors do. It missed two 🟡s on a missing or weak test, and found 4 findings they missed. No principles file keeps an auditor of its own. Asked whether those two misses should bring back separate auditors for the technical principles and for the goals with code health, lilleman chose to try the one auditor. Valid until the auditor's definition changes or the rung a project runs its auditors on, which this trial replayed, changes: replay the pass then, as this trial did.
A trial checked that it keeps goal 1.1. It replayed the 6 latest first rounds that launched two or more auditors, at most two per project, from this repo, `log`, `visthus` and `adf-codec`: 15 auditors, whose reports held 40 🔴 and 🟡 findings. Each round ran on the project's rung as `review-replay` replays it, once with its separate auditors and once with one auditor handed everything they checked, its new definition passed through `--agents`. A replayed reviewer finds a different part of its findings each time: the separate auditors found only 10 of their own 40 again. The one auditor found 12, among them 8 of those 10. So it finds as much as the separate auditors do. It missed two 🟡s on a missing or weak test, and found 4 findings they missed. No principles file keeps an auditor of its own. Asked whether those two misses should bring back separate auditors for the technical principles and for the goals with code health, lilleman chose to try the one auditor. Valid until the auditor's definition changes or a project's rung changes from the one this trial replayed it on: replay the pass then, as this trial did.
2026-10-29, lilleman: it runs only in the release review. `review: the architecture and principles passes run only in the release review…` says why.
@@ -525,7 +525,7 @@ The runs, each on the latest 10 rounds of Opus 5.5 projects that hold a 🔴 or
The prose pass went first under the new bar: a missed prose finding breaks no behaviour, and a later change fixes it cheaply. Sonnet 5.5 missed 44 to 55% of what the project's rung found again, in the prose pass as in both stability runs, so it is unlikely to reach the new bar either; and Opus 5.5 at low, the likeliest rung, fell 17 points short on the pass whose misses cost least. The architecture, principles, overrides and product-owner passes are not replayed yet: `todo.md` item 196 keeps their replays in the plan.
Valid until the rungs below the project's rung change, which calls for a replay of every pass, or a pass's reviewer prompt changes, which calls for a replay of that pass. `review-replay` tests only those rungs, so a rung added above the project's rung changes nothing here.
Valid until the rungs below the project's rung change, which calls for a replay of every pass, or a pass's reviewer prompt changes, which calls for a replay of that pass.
## routing: a worker delegates a task only where it needs just the task's conclusion, and each subagent runs on the rung its task's L picks from the project's rung, never below the project's rung when it files or scores todo.md items