Answer stability round 1: 1 🔴, 6 🟡 (0 misread); sort the imports, give up after 90 minutes inside the background command's limit, test the fail-closed reads, and say which settings are paths
This commit is contained in:
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- `chunk-pr merge` waits for a human's approval of the pull request where the project's config sets `wait-for-approval = true`. It reads the approval from GitHub, as the branch protection asks for it, and merges once the pull request is approved. Set it for a client whose pull requests need another person's approval.
|
||||
- `chunk-pr merge` waits for a human's approval of the pull request where the project's config sets `wait-for-approval = true`: it merges once GitHub reports the pull request approved, and gives up after 90 minutes. It reads approvals on GitHub only. Set it for a client whose pull requests need another person's approval.
|
||||
- The technical principles ask for code in files and folders before `// --- name ---` banner comments. They also ask for an entry point a newcomer finds at once: the root holds it, the README and the files a tool reads only from there.
|
||||
- In a client project, the loop reads the team's own `CLAUDE.md` and `AGENTS.md` and proposes each change that aligns them with this setup in a pull request of its own. Where the team declines or doesn't answer, the loop follows the team's rule unless it breaks a project goal, and records the conflict as a decision in the `decisions` record.
|
||||
- The new `project-record open <record>` prints the file in which an agent reads and edits one of a project's five records (`decisions`, `score-anchors`, `comprehension-history`, `reading-cost` and `changelog`), and `project-record save <record>` stores the edit. Every tool now reads and writes a record the same one way, so that a later release can keep a record outside files. No config needs a change.
|
||||
|
||||
@@ -79,7 +79,7 @@ To get loop-claude's questions and stops as Mattermost DMs from a bot, write the
|
||||
|
||||
Each project the loop runs in has a folder, `~/.claude-projects/<key>/`, where `<key>` is the project's name, and a config in it, `config.toml`. The folder is a git repo of its own, so the config has a history and a client's files never enter this repo. The loop commits each change it makes to a file there, such as the plan, in that repo, titled as its commits in the checkout. A project is a whole git checkout, never a subfolder of one.
|
||||
|
||||
The config names the project's main checkout and the path of every file the loop keeps for it. No setting has a default: `project-config` fails and names each setting a config leaves out. A path is `checkout:<path>` inside the checkout, `folder:<path>` inside the config's folder, or one that starts with `/` or `~/`. Every tool reads and writes each file at the path its setting gives, and `chunk-preflight` and `todo-item` print those paths for the agents. Five of the settings name the project's records: `decisions`, `score-anchors`, `comprehension-history`, `reading-cost` and `changelog`. Keep each file's name, such as `todo.md`: the rules for a file load by its name. A project of your own keeps every file but the principles in its checkout:
|
||||
The config names the project's main checkout and the path of every file the loop keeps for it. No path setting has a default: `project-config` fails and names each setting a config leaves out. A path is `checkout:<path>` inside the checkout, `folder:<path>` inside the config's folder, or one that starts with `/` or `~/`. Every tool reads and writes each file at the path its setting gives, and `chunk-preflight` and `todo-item` print those paths for the agents. Five of the settings name the project's records: `decisions`, `score-anchors`, `comprehension-history`, `reading-cost` and `changelog`. Keep each file's name, such as `todo.md`: the rules for a file load by its name. A project of your own keeps every file but the principles in its checkout:
|
||||
|
||||
```toml
|
||||
checkout = "~/code/log"
|
||||
@@ -96,7 +96,7 @@ principles = "~/.claude/principles"
|
||||
|
||||
A project may also set `env-file`, the file of `KEY=value` credentials its tools read, such as `env-file = "~/.claude-projects/log.env"`. It is a path that starts with `/` or `~/`, outside the checkout and the config's folder, so that no git remote receives it. A tool that needs a key fails and names each one the file lacks. No failure message, prompt, log line or archived loop output holds a token from the file. rule-hook refuses a Read or a Grep of the file and a Bash command that names it or searches a folder holding it with `ag`, `grep -r` or `rg`, unless the tool skips the file as hidden or ignored, so a worker doesn't read it in passing; `rule-hook --help` says which calls it checks. A project without the setting has no credentials, and never reads another project's. `project-config env-file` prints the path.
|
||||
|
||||
A project whose pull requests need a human's approval sets `wait-for-approval = true` in its config. `chunk-pr merge` then waits, after the checks pass, until GitHub reports the pull request approved as the branch protection asks, and merges it. It refuses where a reviewer requested changes, where the protection asks for no review, on Gitea, and where nobody approves within 2 hours; run it again once the pull request is approved. Left out, the setting is false.
|
||||
A project whose pull requests need a human's approval sets `wait-for-approval = true` in its config. `chunk-pr merge` then waits, once the checks have passed, until GitHub reports the pull request approved as its branch protection requires, and merges it. It refuses on Gitea, where a reviewer requested changes and where the protection requires no review. It gives up after 90 minutes without an approval; run it again once the pull request is approved. Left out, the setting is false.
|
||||
|
||||
For a client's project, use one sandbox VM per client, so that one client's code, transcripts and credentials never share a machine with another's. What the client's repo should not hold lives in the config's folder, at paths written as `folder:<path>`, as `project-config --help` shows: the plan, the working rules in an `AGENTS.md` I write for the agents, the records, and the goals and audience, as `goals-and-audience = "folder:goals.md"`. Name that file `goals.md`, so the rules for goals and audience load when an agent writes it. A file outside the checkout has no commits in the checkout's history, so four things are not supported for it yet. `score-outcomes` and loop-claude's routing figures count a project's chunks only where `todo.md`, `docs/comprehension-history.md` and `docs/reading-cost.md` are at those paths in the checkout; otherwise each says so, and the loop goes on. For a plan or a comprehension history outside the checkout, `larv-review` says it cannot tell whether a scoring run is owed, and for such a history it cannot find a paired ruling's base. For a plan outside the checkout, chunk-preflight and loop-claude don't see a release whose last item has shipped, and say that its release review and its cut are yours to call. And rule-hook checks a plan outside the checkout on every call, not only when it changed.
|
||||
|
||||
|
||||
@@ -4,25 +4,25 @@ import pathlib
|
||||
import time
|
||||
import typing
|
||||
|
||||
from failure import Failure
|
||||
from chunk_preflight.gathered import Forge
|
||||
from deploy_release.forge import Run
|
||||
from failure import Failure
|
||||
|
||||
BIN = pathlib.Path(__file__).resolve().parents[1]
|
||||
# A human answers in hours, so merge polls once a minute. It gives up after 2 hours, the longest a background command may run: whoever asked for the merge runs it again once the pull request is approved.
|
||||
# A human answers in hours, so merge polls once a minute and gives up after 90 minutes, which leaves room for the checks wait inside the 2 hours a background command may run, so that the give-up message prints.
|
||||
POLL_SECONDS = 60
|
||||
WAIT_SECONDS = 2 * 60 * 60
|
||||
WAIT_SECONDS = 90 * 60
|
||||
|
||||
|
||||
def awaited(forge: Forge, run: Run) -> bool | Failure:
|
||||
"""Whether merge waits for an approval: the project config's `wait-for-approval`. A config that cannot be read fails, so that no unreadable config merges without the approval it may require. Only GitHub reports approvals, so a Gitea project that sets it fails too."""
|
||||
"""Whether merge waits for an approval: the project config's `wait-for-approval`. A config that cannot be read fails, so that merge never skips an approval the config may require. chunk-pr reads approvals on GitHub only, so a Gitea project that sets it fails too."""
|
||||
answer = run([str(BIN / "project-config"), "wait-for-approval"])
|
||||
if isinstance(answer, Failure):
|
||||
return answer
|
||||
if answer.strip() not in ("true", "false"):
|
||||
return Failure(f"project-config wait-for-approval printed {answer.strip()[:80]!r}, not true or false")
|
||||
if answer.strip() == "true" and forge != "gh":
|
||||
return Failure("the project's config sets wait-for-approval, which chunk-pr can read on GitHub only; set it to false and approve this pull request by hand")
|
||||
return Failure("the project's config sets wait-for-approval, but chunk-pr reads approvals on GitHub only, so it cannot merge this pull request; ask the human to merge it")
|
||||
return answer.strip() == "true"
|
||||
|
||||
|
||||
@@ -41,9 +41,9 @@ def approved(number: str, run: Run, sleep: typing.Callable[[float], None] = time
|
||||
if decision == "CHANGES_REQUESTED":
|
||||
return Failure(f"a reviewer requested changes on pull request {number}")
|
||||
if not decision:
|
||||
return Failure("wait-for-approval is set, but the branch protection requires no review, so GitHub reports no decision; require a review in the protection, or set wait-for-approval to false")
|
||||
return Failure("wait-for-approval is set, but the branch protection requires no review, so GitHub reports no decision; ask the human to require a review in the protection or to set wait-for-approval to false")
|
||||
if decision != "REVIEW_REQUIRED":
|
||||
return Failure(f"GitHub reports the review decision {decision!r} on pull request {number}, which chunk-pr doesn't know")
|
||||
return Failure(f"GitHub reports the review decision {decision!r} on pull request {number}, and chunk-pr doesn't know that decision")
|
||||
if waited < WAIT_SECONDS:
|
||||
sleep(POLL_SECONDS)
|
||||
return Failure(f"pull request {number} wasn't approved within {WAIT_SECONDS // 3600} hours; run chunk-pr merge again once it is")
|
||||
return Failure(f"pull request {number} wasn't approved within {WAIT_SECONDS // 60} minutes; run chunk-pr merge again once it is")
|
||||
|
||||
@@ -32,10 +32,11 @@ merge waits for the pull request's checks, and those origin's default branch
|
||||
merge commit, the rebase and the squash. It also refuses where `larv-review
|
||||
scoring-run-owed` exits 1, because the chunk owes a scoring run it has not recorded.
|
||||
Where the project's config sets `wait-for-approval = true` (`project-config --help`), it
|
||||
then waits, polling once a minute, until GitHub reports the pull request approved as its
|
||||
branch protection asks, and refuses where a reviewer requested changes, where the
|
||||
protection asks for no review, on Gitea, where it cannot read the config, and where no
|
||||
approval came within 2 hours; run it again once the pull request is approved.
|
||||
also waits, once the checks have passed, until GitHub reports the pull request approved
|
||||
as its branch protection requires. It refuses at once where a reviewer requested changes,
|
||||
where the protection requires no review, on Gitea, and where it cannot read the config.
|
||||
After 90 minutes without an approval it refuses too; run it again once the pull request
|
||||
is approved.
|
||||
clean acts once the forge reports the branch's pull request merged at the commit checked out,
|
||||
by any merge method. It fetches origin, fast-forwards the main checkout to origin's
|
||||
default branch, removes the chunk's worktree, then deletes the local branch; origin's
|
||||
|
||||
@@ -41,6 +41,8 @@ class Checkout:
|
||||
self.statuses: list[str] = []
|
||||
# What GitHub reports as the pull request's review decision on each read; the last one repeats.
|
||||
self.decisions = ["APPROVED"]
|
||||
# Set, what GitHub prints instead of a decision's JSON.
|
||||
self.unreadable: str | Failure | None = None
|
||||
self.ran: list[list[str]] = []
|
||||
self.answers: dict[tuple[str, ...], str | Failure] = {
|
||||
("git", "remote", "get-url", "origin"): f"{origin}\n",
|
||||
@@ -64,6 +66,8 @@ class Checkout:
|
||||
return json.dumps([{"number": 12, "head": {"ref": "item-7", "label": "item-7", "sha": HEAD, "repo": {"full_name": "me/project"}}, "base": {"repo": {"full_name": "me/project"}}}] if self.open else [])
|
||||
if command == TEA_STATUS:
|
||||
return self.statuses.pop(0) if self.statuses else json.dumps({"total_count": 1, "state": "success", "statuses": [{"context": "CI / tests (pull_request)", "status": "success"}]})
|
||||
if command == GH_DECISION and self.unreadable is not None:
|
||||
return self.unreadable
|
||||
if command == GH_DECISION:
|
||||
return json.dumps({"reviewDecision": self.decisions.pop(0) if len(self.decisions) > 1 else self.decisions[0]})
|
||||
if command == TEA_BRANCH:
|
||||
@@ -304,15 +308,29 @@ class Merge(unittest.TestCase):
|
||||
self.assertEqual(merged(checkout, slept.append), "Merged pull request #12 (merge).")
|
||||
self.assertEqual(slept, [60, 60])
|
||||
|
||||
def test_an_approval_that_never_comes_leaves_it_unmerged_after_two_hours(self):
|
||||
def test_an_approval_that_never_comes_leaves_it_unmerged_after_90_minutes(self):
|
||||
checkout = self.requiring_approval(GITHUB, "REVIEW_REQUIRED")
|
||||
slept: list[float] = []
|
||||
failed = merged(checkout, slept.append)
|
||||
self.assertIsInstance(failed, Failure)
|
||||
self.assertIn("2 hours", failed.reason)
|
||||
self.assertEqual(sum(slept), 2 * 60 * 60)
|
||||
self.assertIn("90 minutes", failed.reason)
|
||||
self.assertEqual(sum(slept), 90 * 60)
|
||||
self.assertEqual(checkout.acted(), [])
|
||||
|
||||
def test_an_approval_that_comes_in_the_last_minute_still_merges(self):
|
||||
checkout = self.requiring_approval(GITHUB, *["REVIEW_REQUIRED"] * 90, "APPROVED")
|
||||
slept: list[float] = []
|
||||
self.assertEqual(merged(checkout, slept.append), "Merged pull request #12 (merge).")
|
||||
self.assertEqual(sum(slept), 90 * 60)
|
||||
|
||||
def test_a_review_decision_it_cannot_read_or_does_not_know_leaves_it_unmerged(self):
|
||||
for answer in (Failure("gh pr view failed"), "not json", "[]", "{}", json.dumps({"reviewDecision": "DISMISSED"})):
|
||||
with self.subTest(answer):
|
||||
checkout = self.requiring_approval()
|
||||
checkout.unreadable = answer
|
||||
self.assertIsInstance(step(["merge"], checkout), Failure)
|
||||
self.assertEqual(checkout.acted(), [])
|
||||
|
||||
def test_requested_changes_leave_it_unmerged_without_waiting(self):
|
||||
checkout = self.requiring_approval(GITHUB, "CHANGES_REQUESTED")
|
||||
slept: list[float] = []
|
||||
@@ -330,7 +348,7 @@ class Merge(unittest.TestCase):
|
||||
self.assertIn("requires no review", failed.reason)
|
||||
self.assertEqual(checkout.acted(), [])
|
||||
|
||||
def test_gitea_with_wait_for_approval_leaves_it_unmerged_since_only_github_reports_approvals(self):
|
||||
def test_gitea_with_wait_for_approval_leaves_it_unmerged_since_chunk_pr_reads_approvals_on_github_only(self):
|
||||
checkout = self.requiring_approval(GITEA)
|
||||
failed = step(["merge"], checkout)
|
||||
self.assertIsInstance(failed, Failure)
|
||||
|
||||
@@ -6,7 +6,7 @@ Run inside a checkout, or a worktree of one. It finds the config that names the
|
||||
~/.claude-projects/<key>/config.toml, and prints each setting and its absolute path, as in
|
||||
`plan: /home/me/code/acme/todo.md`. Given a setting, it prints only that setting's path.
|
||||
|
||||
The config is TOML. `checkout` names the project's main checkout, and every other setting is a path:
|
||||
The config is TOML. `checkout` names the project's main checkout, and every other setting but `wait-for-approval` is a path:
|
||||
"checkout:<path>" inside the checkout, "folder:<path>" inside the config's own folder, or one that
|
||||
starts with / or ~/. In a worktree, a "checkout:" path is the worktree's own file.
|
||||
|
||||
@@ -21,18 +21,19 @@ starts with / or ~/. In a worktree, a "checkout:" path is the worktree's own fil
|
||||
goals-and-audience = "folder:goals.md"
|
||||
principles = "folder:principles"
|
||||
|
||||
`env-file` is the one optional setting: the file of KEY=value credentials the project's tools read,
|
||||
`env-file` is an optional setting: the file of KEY=value credentials the project's tools read,
|
||||
a path that starts with / or ~/ and lies outside the checkout and the config's folder, such as
|
||||
"~/.claude-projects/acme-billing.env". Where it is left out, the project has no credentials file, and
|
||||
a tool that needs one fails naming the setting. It is printed with the others only where it is set.
|
||||
|
||||
`wait-for-approval` is the other optional setting, true or false: whether `chunk-pr merge` waits for a human's
|
||||
approval of the pull request, as GitHub's branch protection asks for it, before it merges. Where it is left out it
|
||||
is false. Asked for by name it prints true or false; it is listed with the others only where it is true.
|
||||
approval of the pull request, as its branch protection requires on GitHub, before it merges. Left out, it is
|
||||
false. `project-config wait-for-approval` prints true or false; the full listing shows the setting only where
|
||||
it is true.
|
||||
|
||||
Exit 1, naming each fault on stderr, where no config names the checkout, or its config leaves out a
|
||||
setting, names an unknown one or misspells a path, and where `env-file` is asked for and not set.
|
||||
No setting has a default.
|
||||
setting, names an unknown one, misspells a path or sets `wait-for-approval` to anything but true or
|
||||
false, and where `env-file` is asked for and not set. No path setting has a default.
|
||||
"""
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""A project's settings, read from the one config that names its checkout: `~/.claude-projects/<key>/config.toml`.
|
||||
|
||||
No setting has a default, so that a client's project never runs on another project's files: a missing config or setting is a Failure that names it."""
|
||||
No path setting has a default, so that a client's project never runs on another project's files: a missing config or setting is a Failure that names it."""
|
||||
import dataclasses
|
||||
import pathlib
|
||||
import tomllib
|
||||
@@ -13,9 +13,9 @@ CONFIG = "config.toml"
|
||||
SETTINGS = ("plan", "working-rules", "decisions", "score-anchors", "comprehension-history", "reading-cost", "changelog", "goals-and-audience", "principles")
|
||||
# They also picked a prefix over bare relative paths: each line says its own place, and a relative path with no prefix fails instead of landing somewhere by default.
|
||||
SPELLING = '"checkout:<path>" for a path in the checkout, "folder:<path>" for one in the config\'s folder, or a path that starts with / or ~/'
|
||||
# The one setting a project may leave out: the file of KEY=value credentials its tools read. It has no default either: a project without it asks for no credentials, and never reads another project's.
|
||||
# A path setting a project may leave out: the file of KEY=value credentials its tools read. It has no default either: a project without it asks for no credentials, and never reads another project's.
|
||||
ENV_FILE = "env-file"
|
||||
# The other optional setting: whether `chunk-pr merge` waits for a human's approval of the pull request. It is a boolean, false where left out, so a project that doesn't set it merges as soon as its checks pass.
|
||||
# The optional boolean setting: whether `chunk-pr merge` waits for a human's approval of the pull request. False where left out.
|
||||
WAIT_FOR_APPROVAL = "wait-for-approval"
|
||||
OUTSIDE = "a path that starts with / or ~/ and lies outside the checkout and the config's folder, so that no git remote receives the credentials"
|
||||
|
||||
@@ -32,7 +32,6 @@ class ProjectConfig:
|
||||
paths: dict[str, pathlib.Path]
|
||||
# The config's `env-file`, or None where it sets none.
|
||||
env_file: pathlib.Path | None = None
|
||||
# The config's `wait-for-approval`, false where it sets none.
|
||||
wait_for_approval: bool = False
|
||||
|
||||
def tracked(self, setting: str) -> str | None:
|
||||
|
||||
@@ -202,7 +202,7 @@ Launch prose-reviewer with what the diff touched and every section this review c
|
||||
1. Read the finished PR against the goals file's goals and code health, unless that file voids it: does anything in it work against one? It breaks a maintain goal → do not merge; redesign the change or drop its item. It works against an achieve goal → do not merge; ask the human, stating the conflict.
|
||||
2. Run the comprehension check (§3) where it runs; it runs once. When `origin/<default branch>` moves after that, return to §4, and keep the ruling and scoring run already recorded.
|
||||
3. Remove the chunk's items from `todo.md` per `~/.claude/CLAUDE.md` → "Working loop" in the branch, commit and push that; the commit re-enters no pass.
|
||||
4. Aligned and every pass done → **merge** with `~/.claude/bin/chunk-pr merge`, run in the background, since its wait can outlast the Bash tool's timeout. It waits for the pull request's checks to pass, and for a human's approval where the project's config sets `wait-for-approval` (`chunk-pr --help`), then takes the first merge method the repository allows on its forge. A repo whose `AGENTS.md` names another method merges that way with the forge's CLI instead, once the pull request's checks pass and `~/.claude/bin/larv-review scoring-run-owed` exits 0. A repo withholds this in its `AGENTS.md` → leave the PR ready and unmerged, and say which line withheld it.
|
||||
4. Aligned and every pass done → **merge** with `~/.claude/bin/chunk-pr merge`, run in the background with its `timeout` set to 7200000 ms, since its wait can outlast the Bash tool's default. It waits for the pull request's checks to pass, and for a human's approval where the project's config sets `wait-for-approval` (`chunk-pr --help`), then takes the first merge method the repository allows on its forge. A repo whose `AGENTS.md` names another method merges that way with the forge's CLI instead, once the pull request's checks pass, `~/.claude/bin/larv-review scoring-run-owed` exits 0 and, where `~/.claude/bin/project-config wait-for-approval` prints `true`, GitHub reports the pull request approved. A repo withholds this in its `AGENTS.md` → leave the PR ready and unmerged, and say which line withheld it.
|
||||
5. Merged → run `~/.claude/bin/chunk-pr clean` in the chunk's worktree, and put a failure it prints in the summary.
|
||||
6. Stop. The next chunk gets a fresh session.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user