Answer stability round 1: 1 🔴, 6 🟡 (0 misread); sort the imports, give up after 90 minutes inside the background command's limit, test the fail-closed reads, and say which settings are paths

This commit is contained in:
2026-11-14 00:55:03 +01:00
parent 6cdbcbc08b
commit 27b5409479
8 changed files with 49 additions and 30 deletions
+1 -1
View File
@@ -2,7 +2,7 @@
## Unreleased
- `chunk-pr merge` waits for a human's approval of the pull request where the project's config sets `wait-for-approval = true`. It reads the approval from GitHub, as the branch protection asks for it, and merges once the pull request is approved. Set it for a client whose pull requests need another person's approval.
- `chunk-pr merge` waits for a human's approval of the pull request where the project's config sets `wait-for-approval = true`: it merges once GitHub reports the pull request approved, and gives up after 90 minutes. It reads approvals on GitHub only. Set it for a client whose pull requests need another person's approval.
- The technical principles ask for code in files and folders before `// --- name ---` banner comments. They also ask for an entry point a newcomer finds at once: the root holds it, the README and the files a tool reads only from there.
- In a client project, the loop reads the team's own `CLAUDE.md` and `AGENTS.md` and proposes each change that aligns them with this setup in a pull request of its own. Where the team declines or doesn't answer, the loop follows the team's rule unless it breaks a project goal, and records the conflict as a decision in the `decisions` record.
- The new `project-record open <record>` prints the file in which an agent reads and edits one of a project's five records (`decisions`, `score-anchors`, `comprehension-history`, `reading-cost` and `changelog`), and `project-record save <record>` stores the edit. Every tool now reads and writes a record the same one way, so that a later release can keep a record outside files. No config needs a change.
+2 -2
View File
@@ -79,7 +79,7 @@ To get loop-claude's questions and stops as Mattermost DMs from a bot, write the
Each project the loop runs in has a folder, `~/.claude-projects/<key>/`, where `<key>` is the project's name, and a config in it, `config.toml`. The folder is a git repo of its own, so the config has a history and a client's files never enter this repo. The loop commits each change it makes to a file there, such as the plan, in that repo, titled as its commits in the checkout. A project is a whole git checkout, never a subfolder of one.
The config names the project's main checkout and the path of every file the loop keeps for it. No setting has a default: `project-config` fails and names each setting a config leaves out. A path is `checkout:<path>` inside the checkout, `folder:<path>` inside the config's folder, or one that starts with `/` or `~/`. Every tool reads and writes each file at the path its setting gives, and `chunk-preflight` and `todo-item` print those paths for the agents. Five of the settings name the project's records: `decisions`, `score-anchors`, `comprehension-history`, `reading-cost` and `changelog`. Keep each file's name, such as `todo.md`: the rules for a file load by its name. A project of your own keeps every file but the principles in its checkout:
The config names the project's main checkout and the path of every file the loop keeps for it. No path setting has a default: `project-config` fails and names each setting a config leaves out. A path is `checkout:<path>` inside the checkout, `folder:<path>` inside the config's folder, or one that starts with `/` or `~/`. Every tool reads and writes each file at the path its setting gives, and `chunk-preflight` and `todo-item` print those paths for the agents. Five of the settings name the project's records: `decisions`, `score-anchors`, `comprehension-history`, `reading-cost` and `changelog`. Keep each file's name, such as `todo.md`: the rules for a file load by its name. A project of your own keeps every file but the principles in its checkout:
```toml
checkout = "~/code/log"
@@ -96,7 +96,7 @@ principles = "~/.claude/principles"
A project may also set `env-file`, the file of `KEY=value` credentials its tools read, such as `env-file = "~/.claude-projects/log.env"`. It is a path that starts with `/` or `~/`, outside the checkout and the config's folder, so that no git remote receives it. A tool that needs a key fails and names each one the file lacks. No failure message, prompt, log line or archived loop output holds a token from the file. rule-hook refuses a Read or a Grep of the file and a Bash command that names it or searches a folder holding it with `ag`, `grep -r` or `rg`, unless the tool skips the file as hidden or ignored, so a worker doesn't read it in passing; `rule-hook --help` says which calls it checks. A project without the setting has no credentials, and never reads another project's. `project-config env-file` prints the path.
A project whose pull requests need a human's approval sets `wait-for-approval = true` in its config. `chunk-pr merge` then waits, after the checks pass, until GitHub reports the pull request approved as the branch protection asks, and merges it. It refuses where a reviewer requested changes, where the protection asks for no review, on Gitea, and where nobody approves within 2 hours; run it again once the pull request is approved. Left out, the setting is false.
A project whose pull requests need a human's approval sets `wait-for-approval = true` in its config. `chunk-pr merge` then waits, once the checks have passed, until GitHub reports the pull request approved as its branch protection requires, and merges it. It refuses on Gitea, where a reviewer requested changes and where the protection requires no review. It gives up after 90 minutes without an approval; run it again once the pull request is approved. Left out, the setting is false.
For a client's project, use one sandbox VM per client, so that one client's code, transcripts and credentials never share a machine with another's. What the client's repo should not hold lives in the config's folder, at paths written as `folder:<path>`, as `project-config --help` shows: the plan, the working rules in an `AGENTS.md` I write for the agents, the records, and the goals and audience, as `goals-and-audience = "folder:goals.md"`. Name that file `goals.md`, so the rules for goals and audience load when an agent writes it. A file outside the checkout has no commits in the checkout's history, so four things are not supported for it yet. `score-outcomes` and loop-claude's routing figures count a project's chunks only where `todo.md`, `docs/comprehension-history.md` and `docs/reading-cost.md` are at those paths in the checkout; otherwise each says so, and the loop goes on. For a plan or a comprehension history outside the checkout, `larv-review` says it cannot tell whether a scoring run is owed, and for such a history it cannot find a paired ruling's base. For a plan outside the checkout, chunk-preflight and loop-claude don't see a release whose last item has shipped, and say that its release review and its cut are yours to call. And rule-hook checks a plan outside the checkout on every call, not only when it changed.
+8 -8
View File
@@ -4,25 +4,25 @@ import pathlib
import time
import typing
from failure import Failure
from chunk_preflight.gathered import Forge
from deploy_release.forge import Run
from failure import Failure
BIN = pathlib.Path(__file__).resolve().parents[1]
# A human answers in hours, so merge polls once a minute. It gives up after 2 hours, the longest a background command may run: whoever asked for the merge runs it again once the pull request is approved.
# A human answers in hours, so merge polls once a minute and gives up after 90 minutes, which leaves room for the checks wait inside the 2 hours a background command may run, so that the give-up message prints.
POLL_SECONDS = 60
WAIT_SECONDS = 2 * 60 * 60
WAIT_SECONDS = 90 * 60
def awaited(forge: Forge, run: Run) -> bool | Failure:
"""Whether merge waits for an approval: the project config's `wait-for-approval`. A config that cannot be read fails, so that no unreadable config merges without the approval it may require. Only GitHub reports approvals, so a Gitea project that sets it fails too."""
"""Whether merge waits for an approval: the project config's `wait-for-approval`. A config that cannot be read fails, so that merge never skips an approval the config may require. chunk-pr reads approvals on GitHub only, so a Gitea project that sets it fails too."""
answer = run([str(BIN / "project-config"), "wait-for-approval"])
if isinstance(answer, Failure):
return answer
if answer.strip() not in ("true", "false"):
return Failure(f"project-config wait-for-approval printed {answer.strip()[:80]!r}, not true or false")
if answer.strip() == "true" and forge != "gh":
return Failure("the project's config sets wait-for-approval, which chunk-pr can read on GitHub only; set it to false and approve this pull request by hand")
return Failure("the project's config sets wait-for-approval, but chunk-pr reads approvals on GitHub only, so it cannot merge this pull request; ask the human to merge it")
return answer.strip() == "true"
@@ -41,9 +41,9 @@ def approved(number: str, run: Run, sleep: typing.Callable[[float], None] = time
if decision == "CHANGES_REQUESTED":
return Failure(f"a reviewer requested changes on pull request {number}")
if not decision:
return Failure("wait-for-approval is set, but the branch protection requires no review, so GitHub reports no decision; require a review in the protection, or set wait-for-approval to false")
return Failure("wait-for-approval is set, but the branch protection requires no review, so GitHub reports no decision; ask the human to require a review in the protection or to set wait-for-approval to false")
if decision != "REVIEW_REQUIRED":
return Failure(f"GitHub reports the review decision {decision!r} on pull request {number}, which chunk-pr doesn't know")
return Failure(f"GitHub reports the review decision {decision!r} on pull request {number}, and chunk-pr doesn't know that decision")
if waited < WAIT_SECONDS:
sleep(POLL_SECONDS)
return Failure(f"pull request {number} wasn't approved within {WAIT_SECONDS // 3600} hours; run chunk-pr merge again once it is")
return Failure(f"pull request {number} wasn't approved within {WAIT_SECONDS // 60} minutes; run chunk-pr merge again once it is")
+5 -4
View File
@@ -32,10 +32,11 @@ merge waits for the pull request's checks, and those origin's default branch
merge commit, the rebase and the squash. It also refuses where `larv-review
scoring-run-owed` exits 1, because the chunk owes a scoring run it has not recorded.
Where the project's config sets `wait-for-approval = true` (`project-config --help`), it
then waits, polling once a minute, until GitHub reports the pull request approved as its
branch protection asks, and refuses where a reviewer requested changes, where the
protection asks for no review, on Gitea, where it cannot read the config, and where no
approval came within 2 hours; run it again once the pull request is approved.
also waits, once the checks have passed, until GitHub reports the pull request approved
as its branch protection requires. It refuses at once where a reviewer requested changes,
where the protection requires no review, on Gitea, and where it cannot read the config.
After 90 minutes without an approval it refuses too; run it again once the pull request
is approved.
clean acts once the forge reports the branch's pull request merged at the commit checked out,
by any merge method. It fetches origin, fast-forwards the main checkout to origin's
default branch, removes the chunk's worktree, then deletes the local branch; origin's
+22 -4
View File
@@ -41,6 +41,8 @@ class Checkout:
self.statuses: list[str] = []
# What GitHub reports as the pull request's review decision on each read; the last one repeats.
self.decisions = ["APPROVED"]
# Set, what GitHub prints instead of a decision's JSON.
self.unreadable: str | Failure | None = None
self.ran: list[list[str]] = []
self.answers: dict[tuple[str, ...], str | Failure] = {
("git", "remote", "get-url", "origin"): f"{origin}\n",
@@ -64,6 +66,8 @@ class Checkout:
return json.dumps([{"number": 12, "head": {"ref": "item-7", "label": "item-7", "sha": HEAD, "repo": {"full_name": "me/project"}}, "base": {"repo": {"full_name": "me/project"}}}] if self.open else [])
if command == TEA_STATUS:
return self.statuses.pop(0) if self.statuses else json.dumps({"total_count": 1, "state": "success", "statuses": [{"context": "CI / tests (pull_request)", "status": "success"}]})
if command == GH_DECISION and self.unreadable is not None:
return self.unreadable
if command == GH_DECISION:
return json.dumps({"reviewDecision": self.decisions.pop(0) if len(self.decisions) > 1 else self.decisions[0]})
if command == TEA_BRANCH:
@@ -304,15 +308,29 @@ class Merge(unittest.TestCase):
self.assertEqual(merged(checkout, slept.append), "Merged pull request #12 (merge).")
self.assertEqual(slept, [60, 60])
def test_an_approval_that_never_comes_leaves_it_unmerged_after_two_hours(self):
def test_an_approval_that_never_comes_leaves_it_unmerged_after_90_minutes(self):
checkout = self.requiring_approval(GITHUB, "REVIEW_REQUIRED")
slept: list[float] = []
failed = merged(checkout, slept.append)
self.assertIsInstance(failed, Failure)
self.assertIn("2 hours", failed.reason)
self.assertEqual(sum(slept), 2 * 60 * 60)
self.assertIn("90 minutes", failed.reason)
self.assertEqual(sum(slept), 90 * 60)
self.assertEqual(checkout.acted(), [])
def test_an_approval_that_comes_in_the_last_minute_still_merges(self):
checkout = self.requiring_approval(GITHUB, *["REVIEW_REQUIRED"] * 90, "APPROVED")
slept: list[float] = []
self.assertEqual(merged(checkout, slept.append), "Merged pull request #12 (merge).")
self.assertEqual(sum(slept), 90 * 60)
def test_a_review_decision_it_cannot_read_or_does_not_know_leaves_it_unmerged(self):
for answer in (Failure("gh pr view failed"), "not json", "[]", "{}", json.dumps({"reviewDecision": "DISMISSED"})):
with self.subTest(answer):
checkout = self.requiring_approval()
checkout.unreadable = answer
self.assertIsInstance(step(["merge"], checkout), Failure)
self.assertEqual(checkout.acted(), [])
def test_requested_changes_leave_it_unmerged_without_waiting(self):
checkout = self.requiring_approval(GITHUB, "CHANGES_REQUESTED")
slept: list[float] = []
@@ -330,7 +348,7 @@ class Merge(unittest.TestCase):
self.assertIn("requires no review", failed.reason)
self.assertEqual(checkout.acted(), [])
def test_gitea_with_wait_for_approval_leaves_it_unmerged_since_only_github_reports_approvals(self):
def test_gitea_with_wait_for_approval_leaves_it_unmerged_since_chunk_pr_reads_approvals_on_github_only(self):
checkout = self.requiring_approval(GITEA)
failed = step(["merge"], checkout)
self.assertIsInstance(failed, Failure)
+7 -6
View File
@@ -6,7 +6,7 @@ Run inside a checkout, or a worktree of one. It finds the config that names the
~/.claude-projects/<key>/config.toml, and prints each setting and its absolute path, as in
`plan: /home/me/code/acme/todo.md`. Given a setting, it prints only that setting's path.
The config is TOML. `checkout` names the project's main checkout, and every other setting is a path:
The config is TOML. `checkout` names the project's main checkout, and every other setting but `wait-for-approval` is a path:
"checkout:<path>" inside the checkout, "folder:<path>" inside the config's own folder, or one that
starts with / or ~/. In a worktree, a "checkout:" path is the worktree's own file.
@@ -21,18 +21,19 @@ starts with / or ~/. In a worktree, a "checkout:" path is the worktree's own fil
goals-and-audience = "folder:goals.md"
principles = "folder:principles"
`env-file` is the one optional setting: the file of KEY=value credentials the project's tools read,
`env-file` is an optional setting: the file of KEY=value credentials the project's tools read,
a path that starts with / or ~/ and lies outside the checkout and the config's folder, such as
"~/.claude-projects/acme-billing.env". Where it is left out, the project has no credentials file, and
a tool that needs one fails naming the setting. It is printed with the others only where it is set.
`wait-for-approval` is the other optional setting, true or false: whether `chunk-pr merge` waits for a human's
approval of the pull request, as GitHub's branch protection asks for it, before it merges. Where it is left out it
is false. Asked for by name it prints true or false; it is listed with the others only where it is true.
approval of the pull request, as its branch protection requires on GitHub, before it merges. Left out, it is
false. `project-config wait-for-approval` prints true or false; the full listing shows the setting only where
it is true.
Exit 1, naming each fault on stderr, where no config names the checkout, or its config leaves out a
setting, names an unknown one or misspells a path, and where `env-file` is asked for and not set.
No setting has a default.
setting, names an unknown one, misspells a path or sets `wait-for-approval` to anything but true or
false, and where `env-file` is asked for and not set. No path setting has a default.
"""
import pathlib
import sys
+3 -4
View File
@@ -1,6 +1,6 @@
"""A project's settings, read from the one config that names its checkout: `~/.claude-projects/<key>/config.toml`.
No setting has a default, so that a client's project never runs on another project's files: a missing config or setting is a Failure that names it."""
No path setting has a default, so that a client's project never runs on another project's files: a missing config or setting is a Failure that names it."""
import dataclasses
import pathlib
import tomllib
@@ -13,9 +13,9 @@ CONFIG = "config.toml"
SETTINGS = ("plan", "working-rules", "decisions", "score-anchors", "comprehension-history", "reading-cost", "changelog", "goals-and-audience", "principles")
# They also picked a prefix over bare relative paths: each line says its own place, and a relative path with no prefix fails instead of landing somewhere by default.
SPELLING = '"checkout:<path>" for a path in the checkout, "folder:<path>" for one in the config\'s folder, or a path that starts with / or ~/'
# The one setting a project may leave out: the file of KEY=value credentials its tools read. It has no default either: a project without it asks for no credentials, and never reads another project's.
# A path setting a project may leave out: the file of KEY=value credentials its tools read. It has no default either: a project without it asks for no credentials, and never reads another project's.
ENV_FILE = "env-file"
# The other optional setting: whether `chunk-pr merge` waits for a human's approval of the pull request. It is a boolean, false where left out, so a project that doesn't set it merges as soon as its checks pass.
# The optional boolean setting: whether `chunk-pr merge` waits for a human's approval of the pull request. False where left out.
WAIT_FOR_APPROVAL = "wait-for-approval"
OUTSIDE = "a path that starts with / or ~/ and lies outside the checkout and the config's folder, so that no git remote receives the credentials"
@@ -32,7 +32,6 @@ class ProjectConfig:
paths: dict[str, pathlib.Path]
# The config's `env-file`, or None where it sets none.
env_file: pathlib.Path | None = None
# The config's `wait-for-approval`, false where it sets none.
wait_for_approval: bool = False
def tracked(self, setting: str) -> str | None:
+1 -1
View File
@@ -202,7 +202,7 @@ Launch prose-reviewer with what the diff touched and every section this review c
1. Read the finished PR against the goals file's goals and code health, unless that file voids it: does anything in it work against one? It breaks a maintain goal → do not merge; redesign the change or drop its item. It works against an achieve goal → do not merge; ask the human, stating the conflict.
2. Run the comprehension check (§3) where it runs; it runs once. When `origin/<default branch>` moves after that, return to §4, and keep the ruling and scoring run already recorded.
3. Remove the chunk's items from `todo.md` per `~/.claude/CLAUDE.md` → "Working loop" in the branch, commit and push that; the commit re-enters no pass.
4. Aligned and every pass done → **merge** with `~/.claude/bin/chunk-pr merge`, run in the background, since its wait can outlast the Bash tool's timeout. It waits for the pull request's checks to pass, and for a human's approval where the project's config sets `wait-for-approval` (`chunk-pr --help`), then takes the first merge method the repository allows on its forge. A repo whose `AGENTS.md` names another method merges that way with the forge's CLI instead, once the pull request's checks pass and `~/.claude/bin/larv-review scoring-run-owed` exits 0. A repo withholds this in its `AGENTS.md` → leave the PR ready and unmerged, and say which line withheld it.
4. Aligned and every pass done → **merge** with `~/.claude/bin/chunk-pr merge`, run in the background with its `timeout` set to 7200000 ms, since its wait can outlast the Bash tool's default. It waits for the pull request's checks to pass, and for a human's approval where the project's config sets `wait-for-approval` (`chunk-pr --help`), then takes the first merge method the repository allows on its forge. A repo whose `AGENTS.md` names another method merges that way with the forge's CLI instead, once the pull request's checks pass, `~/.claude/bin/larv-review scoring-run-owed` exits 0 and, where `~/.claude/bin/project-config wait-for-approval` prints `true`, GitHub reports the pull request approved. A repo withholds this in its `AGENTS.md` → leave the PR ready and unmerged, and say which line withheld it.
5. Merged → run `~/.claude/bin/chunk-pr clean` in the chunk's worktree, and put a failure it prints in the summary.
6. Stop. The next chunk gets a fresh session.