Answer stability round 1: 0 🔴, 2 🟡; name the rename fix in the finished message and the changelog, read a bad newest mark as unknown, refuse a self-rename, share the release parser, test the strip

This commit is contained in:
2026-10-18 09:47:44 +02:00
parent 28a3f9c5e9
commit 1f0a046dd4
7 changed files with 41 additions and 19 deletions
+1 -1
View File
@@ -4,7 +4,7 @@
- Each reviewer under `agents/` now writes its full report to a file that larv-review names in the reviewer's prompt, in a folder per chunk under `~/.local/state/larv-review/`. It returns only its verdict and one line per finding, so a review takes less of the worker's context. The worker reads a finding's full entry before it fixes or answers a 🔴 or 🟡. Each reviewer now holds Write. `rule-hook` refuses a reviewer's Edit and MultiEdit, and any Write outside `~/.local/state/larv-review/`; no override waives this rule. Each chunk's folder stays after the merge, so `stability-replay` can replay a round that reads an earlier round's report. It replays on a copy of the folder.
- `loop-claude` asks an interrupt's question on its own terminal and resumes the worker headless with your one-line answer, so the loop no longer waits for you to leave a claude session. Type `/talk` instead to open the worker's session in claude, as every interrupt did before, to read the diff or discuss first. A release ready to cut opens that session at once. Pressing Enter on an empty answer stops the loop, exit 2. Keys typed before the question shows are dropped, and a pasted answer keeps only its first line; type `/talk` for a longer one. The judge counts that answer as yours, like one typed in claude: the chunk stays open until the answer is recorded, where a later chunk could need it.
- `todo.md` may mark a renamed release in a `## Renamed` section, one line per rename, such as `- 0.2.1 renamed to 0.3.0`. `chunk-preflight` and `loop-claude` then skip that release when they pick the next one, and larv-review's scoring-run trigger calls no run for it. Without the mark, a release named since the last tag whose items all moved to other releases reads as finished: `chunk-preflight` says it is ready to cut and the loop stops for you, and the trigger calls a scoring run. `rank_todo.py` fails on a `## Renamed` line of another shape.
- `todo.md` may mark a renamed release in a `## Renamed` section, one line per rename, such as `- 0.2.1 renamed to 0.3.0`. `chunk-preflight` and `loop-claude` then skip that release when they pick the next one, and larv-review's scoring-run trigger calls no run for it. Without the mark, a release named since the last tag whose items all moved to other releases reads as finished: `chunk-preflight` says it is ready to cut and the loop stops for you, and the trigger calls a scoring run. `rank_todo.py` fails on a `## Renamed` line of another shape, and on an item filed under a release it marks. Before you move a project to this release, add the line to every project whose `todo.md` renamed a release since its last tag, or its loop stops for that release's cut.
- When a worker's run ends with a question and a request for a higher rung, `loop-claude` now steps the worker one rung up as the judge interrupts, or as it answers the question with a quote from `README.md` or another file it may quote. The judge must quote a passage showing that the escalation trigger the run cites fired. Before, the step was lost for an answered question, and the worker carried on from the answer on its old rung. The step has the same limits as the step on an unfinished verdict, and no later ruling of that cycle carries a second escalation. The archive's interrupt and quoted-answer outcomes now name the worker's rung.
- A scoring run due because a release's items ran out costs four more seats: it is done twice on one head. `larv-review agreement` compares how often Mid A and the inherited architect vote alike in paired rulings with how often their overall scores move the same way between the two runs. After 10 such pairs it gives a verdict, and on `Assumption fails:` larv-review asks you whether to keep the paired ruling.
- `workflow-bench/bench.py` gives each run a fixed list of allowed tools. Before, it skipped every permission check. A run may launch agents, run `/security-review`, edit its clone (not the clone's `.git/` or `.claude/`) and its report, and read its clone and the copy of `~/.claude` it benches. For git it gets one command, `workflow-bench/review_git.py`, and no bare `git`. The wrapper runs `log`, `diff`, `show`, `status`, `ls-files`, `add` and `commit -m` on the run's clone, whatever the working directory. It allows only the options it lists for each, and refuses an argument that names a path outside the clone. git runs with no global or system config, hook, pager or editor. A run loads no settings or MCP servers from `~/.claude` or from the case's clone, so their hooks and extra directories do not apply to it. Every other call is denied. Expect a benched review that needs another git command, or to run a project's code or tests, to say in its report that it could not.
+1 -1
View File
@@ -97,7 +97,7 @@ def planned(todo: str | Failure, cut: LastCut, default: str) -> Plan | NoItem:
return NoItem(f"{where} files items under release {version(earliest)}, which is not after the last cut, {cut.tag}; ask the human which release they belong to.")
gap = f"{where} holds no item for the next release after the last cut, {cut.tag}, and"
if stands == "finished":
return NoItem(f"{gap} work has landed since, so that release is finished and ready to cut, and the working loop stops here.")
return NoItem(f"{gap} work has landed since, so that release is finished and ready to cut, and the working loop stops here. If todo.md renamed that release instead, add `- <old release> renamed to <new release>` under its `## Renamed`.")
if stands == "unplanned":
return NoItem(f"{gap} no work has landed since, so that release is to be planned first.")
if stands == "unknown":
+3 -1
View File
@@ -376,7 +376,9 @@ class LarvReviewTool(unittest.TestCase):
self.assertEqual(self.run_tool("needs-scoring-run").stdout, "Scoring run due: the chunk leaves release 0.1.0 with no plan item.\n")
self.commit("Mark 0.1.0 renamed", {"todo.md": todo(*rows) + "\n## Renamed\n\n- 0.1.0 renamed to 0.2.0\n"})
self.assertEqual(self.run_tool("needs-scoring-run").stdout, "No scoring run due.\n")
self.commit("Break the mark", {"todo.md": todo(*rows) + "\n## Renamed\n\nsoon\n"})
def test_refuses_a_checkout_plan_whose_renamed_section_does_not_parse(self):
self.commit("Break the mark", {"todo.md": todo("| 1 | 0.2.0 | | **Do it.** | 1 | 1 | 3 | 1 | 1 | 1 | 9.0 |") + "\n## Renamed\n\nsoon\n"})
broken = self.run_tool("needs-scoring-run")
self.assertEqual(broken.returncode, 1)
self.assertIn("## Renamed", broken.stderr)
+4 -4
View File
@@ -61,8 +61,9 @@ A valid todo.md; ## Details and each item's detail are optional:
- ## Items holds the table and nothing else: an order goes in the scores or the Blocked by column.
- ## Renamed is optional: one line `- 0.2.1 renamed to 0.3.0` per release the plan renamed. The tools
skip a marked release when they pick the next one, and call no scoring run for it; they read only
the old release, the new one is for the reader. No item may sit under a marked release. A release
whose items all moved to other releases without a mark reads as finished, so the human cuts it.
the old release, the new one is for the reader. No item may sit under a marked release. Delete a
line once a tag at or past its release is cut. A release whose items all moved to other releases
without a mark reads as finished, so the human cuts it.
- The table holds the columns above. One missing L, E or I fails, and so does one with a column this
version does not know, since writing it would drop that column.
- A table in the old shape, with C, A, G and Goals columns in place of L, E, I and Moves, still
@@ -208,8 +209,7 @@ def rank(text: str, readme: str, decisions: str) -> Ranked | Failure:
if isinstance(marked := renames(text), Failure):
return marked
if live := next((item for item in items if item.release in marked), None):
release = ".".join(map(str, live.release))
return Failure(f"todo.md marks release {release} renamed but files item {live.id} under it; drop the mark or move the item")
return Failure(f"todo.md marks release {release_name(live)} renamed but files item {live.id} under it; drop the mark or move the item")
scored = ordered(items, scoring)
if isinstance(scored, Failure):
return scored
+15 -4
View File
@@ -468,7 +468,7 @@ class Standing(unittest.TestCase):
("the plan shifted one minor", [plan((1, "0.2.1"), (2, "0.3.0")), plan((1, "0.3.0"), (2, "0.4.0"), renamed=[("0.2.1", "0.3.0"), ("0.3.0", "0.4.0")])], {(0, 3, 0), (0, 4, 0)}),
("the mark stays in later plans", [plan((1, "0.2.1")), plan((1, "0.3.0"), renamed=[("0.2.1", "0.3.0")]), plan((1, "0.3.0"), (2, "0.3.0"), renamed=[("0.2.1", "0.3.0")])], {(0, 3, 0)}),
("renamed, then named again", [plan((1, "0.2.1")), plan((1, "0.3.0"), renamed=[("0.2.1", "0.3.0")]), plan((1, "0.3.0"), (3, "0.2.1"), renamed=[("0.2.1", "0.3.0")])], {(0, 2, 1), (0, 3, 0)}),
("a mark that does not parse", [plan((1, "0.2.1")), plan((1, "0.3.0")) + "\n## Renamed\n\nsoon\n"], {(0, 2, 1)}),
("an older plan's mark that does not parse", [plan((1, "0.2.1")), plan((1, "0.3.0")) + "\n## Renamed\n\nsoon\n", plan((1, "0.3.0"))], {(0, 2, 1), (0, 3, 0)}),
)
for name, texts, named in cases:
with self.subTest(name):
@@ -477,7 +477,8 @@ class Standing(unittest.TestCase):
def test_reads_the_releases_the_renamed_section_names(self):
self.assertEqual(todo_table.renames("## Items\n\n## Details\n"), set())
self.assertEqual(todo_table.renames("## Renamed\n\n- 0.2.1 renamed to 0.3.0\n- 0.3.0 renamed to 0.4.0\n\n## Details\n"), {(0, 2, 1), (0, 3, 0)})
for bad in ("soon", "- 0.2.1 is now 0.3.0", "- 0.2 renamed to 0.3.0", "- 0.2.1 renamed to 0.3.0 later"):
self.assertEqual(todo_table.renames("## Renamed\n\n- 0.2.1 renamed to 0.3.0 \n"), {(0, 2, 1)})
for bad in ("soon", "- 0.2.1 is now 0.3.0", "- 0.2 renamed to 0.3.0", "- 0.2.1 renamed to 0.3.0 later", "- 0.2.1 renamed to 0.2.1"):
with self.subTest(bad):
self.assertIsInstance(todo_table.renames(f"## Renamed\n\n{bad}\n"), Failure)
@@ -485,9 +486,19 @@ class Standing(unittest.TestCase):
failed = rank_todo.rank(TODO + "\n## Renamed\n\nsoon\n", README, DECISIONS)
self.assertIsInstance(failed, Failure)
self.assertIn("## Renamed", failed.reason)
self.assertIsInstance(rank_todo.rank(TODO + "\n## Renamed\n\n- 0.2.1 renamed to 0.3.0\n", README, DECISIONS), rank_todo.Ranked)
def test_ranks_a_plan_with_a_valid_renamed_section(self):
self.assertIsInstance(rank_todo.rank(TODO + "\n## Renamed\n\n- 0.3.0 renamed to 0.4.0\n", README, DECISIONS), rank_todo.Ranked)
def test_fails_on_an_item_filed_under_a_release_the_plan_marks_renamed(self):
live = rank_todo.rank(TODO + "\n## Renamed\n\n- 0.2.0 renamed to 0.3.0\n", README, DECISIONS)
self.assertEqual(live, Failure("todo.md marks release 0.2.0 renamed but files item 1 under it; drop the mark or move the item"))
self.assertIsInstance(live, Failure)
self.assertIn("marks release 0.2.0 renamed but files item 1 under it", live.reason)
def test_fails_to_name_releases_when_the_newest_plan_marks_one_in_a_form_it_cannot_read(self):
plan = "## Items\n\n| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |\n|---|---|---|---|---|---|---|---|---|---|---|---|\n| 1 | 0.3.0 | | | **Do 1.** | 1 | 1 | 1 | 1 | 1 | 1 by 1 | 10.0 |\n"
self.assertIsInstance(todo_table.named_releases([plan, plan + "\n## Renamed\n\nsoon\n"]), Failure)
self.assertEqual(todo_table.named_releases([plan + "\n## Renamed\n\nsoon\n", plan]), {(0, 3, 0)})
class Main(unittest.TestCase):
+16 -7
View File
@@ -24,7 +24,8 @@ Standing = typing.Literal["next", "cut", "finished", "unplanned", "unknown"]
# todo.md's ## Details section, its body the first group.
DETAILS = re.compile(r"^## Details\n(.*?)(?=^## |\Z)", re.M | re.S)
# One line of todo.md's ## Renamed section, the release it renamed and the one it became.
RENAME = re.compile(r"- ([0-9]+\.[0-9]+\.[0-9]+) renamed to [0-9]+\.[0-9]+\.[0-9]+")
RELEASE = r"[0-9]+\.[0-9]+\.[0-9]+"
RENAME = re.compile(rf"- ({RELEASE}) renamed to ({RELEASE})")
# Each goal an item names, in its order, with how far the item moves it; None in OLD_SHAPE, whose Goals cell names no move.
Moves = tuple[tuple[str, int | None], ...]
@@ -205,9 +206,11 @@ def landed(cut: LastCut) -> int | Failure:
def named_releases(todos: list[str] | Failure) -> set[tuple[int, int, int]] | Failure:
"""Each release the todo.md texts, oldest first, file items under, leaving out one that a later text marks renamed, unless a text after that files items under it again; a text that does not parse names none."""
"""Each release the todo.md texts, oldest first, file items under, leaving out one that a later text marks renamed, unless a text after that files items under it again. A text whose items do not parse names none; the newest text's marks failing to parse fails the whole, since a release left unmarked would read as finished."""
if isinstance(todos, Failure):
return todos
if todos and isinstance(newest := renames(todos[-1]), Failure):
return newest
named: set[tuple[int, int, int]] = set()
for text in todos:
plan, marked = items(text), renames(text)
@@ -224,8 +227,9 @@ def renames(todo: str) -> set[tuple[int, int, int]] | Failure:
rename = RENAME.fullmatch(line.strip())
if rename is None:
return Failure(f"todo.md's ## Renamed section holds {line.strip()!r}, not a line such as `- 0.2.1 renamed to 0.3.0`")
major, minor, patch = (int(part) for part in rename.group(1).split("."))
marked.add((major, minor, patch))
if rename.group(1) == rename.group(2):
return Failure(f"todo.md's ## Renamed section renames {rename.group(1)} to itself; name the release it became")
marked.add(release_of(rename.group(1)))
return marked
@@ -318,8 +322,13 @@ def parse_row(row: str, table: Table) -> Item | Failure:
moves = shape.read_goals(named["ID"], named[shape.goals_column])
if isinstance(moves, Failure):
return moves
major, minor, patch = (int(part) for part in named["Release"].split("."))
return Item(int(named["ID"]), (major, minor, patch), named["Exempt"], named["Item"], scores, moves, blocked_by)
return Item(int(named["ID"]), release_of(named["Release"]), named["Exempt"], named["Item"], scores, moves, blocked_by)
def release_of(version: str) -> tuple[int, int, int]:
"""The release a RELEASE-shaped version names."""
major, minor, patch = (int(part) for part in version.split("."))
return major, minor, patch
def check_cells(row: str, named: dict[str, str]) -> Failure | None:
@@ -327,7 +336,7 @@ def check_cells(row: str, named: dict[str, str]) -> Failure | None:
id_, release, exempt, tagline = named["ID"], named["Release"], named["Exempt"], named["Item"]
if not re.fullmatch(r"[0-9]+", id_) or int(id_) < 1:
return Failure(f"the ID {id_!r} is not a positive whole number: {row}")
if not re.fullmatch(r"[0-9]+\.[0-9]+\.[0-9]+", release):
if not re.fullmatch(RELEASE, release):
return Failure(f"item {id_}'s release {release!r} is not a semver version")
if exempt not in EXEMPTIONS:
return Failure(f"item {id_}'s Exempt {exempt!r} is not empty or one of {', '.join(EXEMPTIONS[1:])}")
+1 -1
View File
@@ -5,7 +5,7 @@ paths:
# Todo items (`todo.md`)
- **`todo.md` is mandatory in every project, in the shape `rank_todo.py --help` shows**, which `~/.claude/bin/rank_todo.py` checks and ranks; its `--help` shows a valid file. `## Scoring` holds the formula and the bar; `## Items` is one table of every item; `## Details` holds each item's detail under its ID. `## Renamed`, when a release was renamed, holds one line `- 0.2.1 renamed to 0.3.0` per rename, and no item under a marked release; a release whose items all moved elsewhere without that line reads as finished, ready to cut. Cite an item by its ID. The formula and bar are the project's own to tune.
- **`todo.md` is mandatory in every project, in the shape `rank_todo.py --help` shows**, which `~/.claude/bin/rank_todo.py` checks and ranks; its `--help` shows a valid file. `## Scoring` holds the formula and the bar; `## Items` is one table of every item; `## Details` holds each item's detail under its ID. `## Renamed`, when a release was renamed, holds one line `- 0.2.1 renamed to 0.3.0` per rename, no item under a marked release, and a line gone once a tag at or past its release is cut; a release whose items all moved elsewhere without that line reads as finished, ready to cut. Cite an item by its ID. The formula and bar are the project's own to tune.
- **Order items only by score and the Blocked by column.** An item's Blocked by cell lists the IDs of the items that must be done before it starts, such as `12, 14`; `rank_todo.py --help` says how it ranks them. No text in `todo.md` outside the table orders items: where an item must go first, score it so, and say why in its detail.
- **Plan only on the project's rung or above.** loop-claude's prompt tells you when you run below it. Below it, never score or re-score an item, move it between releases, split a milestone into items, delete an item under the bar, rewrite a filed item's tagline or detail, or choose or replace an item in `docs/score-anchors.md`. Draft each new item, each change to a filed item, and each change to the anchors, instead. Before your next commit, launch one subagent to apply the drafts, score them and set their release. Launch it on the rung `CLAUDE.md`'s delegation rule gives a subagent that files or scores items. Keep the scores and releases it writes. For a security finding, launch it at once. Any rung may remove a finished item's row and detail, and score a delegated task's L. A subagent launched for another task leaves `todo.md` alone.
- **An item's release allows the bump that semver read strictly asks for.** An item that changes what exists needs a major, or before 1.0.0 a minor. One that only adds needs a minor, and a fix a patch.