Answer product-owner round 1: 0 🔴, 2 🟡 (0 misread); print the checks even where the epic stays, stop a do-epic that hangs, and state the sandbox's needs in the README

This commit is contained in:
2026-11-19 03:09:36 +01:00
parent 098bfddef9
commit b00470996b
6 changed files with 81 additions and 20 deletions
+2 -2
View File
@@ -191,7 +191,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
- `compress-archives` gzips each transcript in loop-claude's archive and each file in the review folders 30 days after it last changed, keeping all of them; Install's crontab line runs it daily.
- `deploy-release [--cut] [<checkout>]` moves `~/.claude`, or the checkout named, to the newest release, as Update says; with `--cut`, run in the development clone, it first cuts the next release, as Release says.
- `do-epic <EPIC-KEY>` runs the chunks of one Jira epic whose items span repos, or have none, each in its own window of the tmux session `epic-<EPIC-KEY>`, on the same cycle, judge and interrupt as loop-claude. The epic's description is its record: goals, audience, scoring and decisions. An open issue runs once a label names its repo, in Jira's backlog order, as many at once as a line such as `**Max parallel chunks:** 2` in the epic's description allows, 1 without it, and as each repo's `parallel-chunks` allows. A chunk whose pull request waits on a gate parks. do-epic closes the epic only once you confirm every goal. Install → "Set up a project" says how to set it up; `do-epic --help` states the flags and the contract.
- `do-epic-e2e` drives `do-epic`'s `--file`, `--record-decision` and `--close-epic` against an epic it makes in a sandbox project on a real Jira site and deletes when it ends, as Test says. It isn't part of `run-tests`.
- `do-epic-e2e` runs `do-epic` against a sandbox epic on a real Jira site, as Test says.
- `judge-replay <model> [--offline]` replays loop-claude's judge on archived verdicts of the projects run on the model. It prints what each rung a review climbs to misses, and the lowest rung below those that returns every verdict, as each candidate above it does. With `--offline` it doesn't call a judge: it counts what a "?" rule, a decision table and a classifier trained on the archive decide of each verdict's kind from the worker's final message, and how many interrupts each misses. `judge-replay --help` states both replays.
- `larv-review <command>` holds larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's round count and the opening of its round commit's title, the scoring run's trigger and setup; the comprehension history's format, its latest places and its count of how often seats agree; and each reviewer's prompt header. `larv-review --help` states them. `settings.json` → `hooks` runs `larv-review reviewer-context` as each subagent starts. A reviewer under `agents/` whose frontmatter has `omitClaudeMd: true` starts without `CLAUDE.md`, and the command hands it that file's "Shared goal", "Prose" and "Vocabulary" sections.
- `loop-claude <task>` runs a project's next chunks, each in its own tmux window, cycle after cycle; `loop-claude --help` states its contract.
@@ -253,7 +253,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
Test on the host with `bin/run-tests`, which runs pytest through `uvx`, pinned in `bin/run_tests/main.py`. It prints each failing test with its error and pytest's summary line, and saves the whole output to a file whose path it prints. It passes on any test paths or pytest options it is given, such as `bin/run-tests bin/chunk_pr -x`. It needs `uv` on `PATH`, which fetches pytest, pytest-xdist and mypy, a Docker daemon, Linux, and a user other than root; a test that lacks one fails, naming it. pytest-xdist runs the tests in one worker per CPU of a heavy slot (`docs/decisions.md` → `bin/: the tools support only the latest stable Python, …`). `python3 -m unittest discover -s bin -p 'test_*.py'` runs the same tests one at a time, with the same needs. Test the bench's scripts with `python3 -m unittest discover -s workflow-bench -p 'test_*.py'`.
`bin/do-epic-e2e` checks `do-epic` against a real Jira site, which `run-tests` never reaches. Set `DO_EPIC_E2E_JIRA_URL` to the sandbox site's address, `DO_EPIC_E2E_PROJECT` to its project's key and `DO_EPIC_E2E_ENV_FILE` to the absolute path of the file holding the sandbox account's `ATLASSIAN_EMAIL` and `ATLASSIAN_API_TOKEN`. The account must be able to create, transition, comment on and delete issues there. It fails, naming each setting that is missing; it makes an epic with issues, runs `do-epic` against it in a home folder of its own, and deletes the epic and its issues when it ends. It needs a Docker daemon, as `run-tests` does.
`bin/do-epic-e2e` checks `do-epic` against a real Jira site, which `run-tests` never reaches. It makes an epic with issues and sub-tasks in a sandbox project, runs `do-epic --file`, `--record-decision` and `--close-epic` against it in a home folder of its own, and deletes the epic and its issues when it ends. The sandbox project must be team-managed, with a `Done` transition, and its account must be able to create, edit, transition, comment on and delete issues there. Set `DO_EPIC_E2E_JIRA_URL` to the sandbox site's address, `DO_EPIC_E2E_ENV_FILE` to the absolute path of the file holding the account's `ATLASSIAN_EMAIL` and `ATLASSIAN_API_TOKEN`, and `DO_EPIC_E2E_PROJECT` to the project's key. The command fails, naming each setting that is missing. It needs a Docker daemon, as `run-tests` does.
CI runs `ci.sh` on every pull request, and `main` doesn't merge anything until it passes. The script runs the same tests on the committed HEAD in a pinned Python container. Run it under `run-heavy` to reproduce a CI failure, with Docker on `PATH` and its socket at `/var/run/docker.sock`, which the container shares.
+10 -6
View File
@@ -1,4 +1,4 @@
"""The do-epic-e2e command: runs do-epic's commands that need no tmux session and no worker against a sandbox epic on a real Jira site (docs/decisions.md → `do-epic: drives one Jira epic …`). It is no part of `run-tests`."""
"""The do-epic-e2e command: runs do-epic's commands that need no tmux session and no worker against a sandbox epic on a real Jira site (docs/decisions.md → `do-epic: drives one Jira epic …`)."""
import pathlib
import sys
import typing
@@ -22,6 +22,8 @@ def main(argv: list[str], env: typing.Mapping[str, str], launcher: pathlib.Path)
if isinstance(found, Failure):
print(f"do-epic-e2e cannot start: {found.reason}", file=sys.stderr)
return 1
report = scenario.Report()
left: sandbox.Leftover | None = None
try:
with sandbox.epic_made(found, "do-epic e2e sandbox epic") as key:
if isinstance(key, Failure):
@@ -29,12 +31,14 @@ def main(argv: list[str], env: typing.Mapping[str, str], launcher: pathlib.Path)
return 1
print(f"Sandbox epic {key} in {found.jira_url}")
report = scenario.run(found, key, launcher)
except sandbox.Leftover as left:
print(f"do-epic-e2e: {left}", file=sys.stderr)
return 1
except sandbox.Leftover as failure:
left = failure
# The checks are printed even where the epic could not be deleted.
for line in report.held:
print(f"ok {line}")
for line in report.broke:
print(f"FAIL {line}")
print(f"{len(report.held)} held, {len(report.broke)} broke; the sandbox epic is deleted")
return 1 if report.broke else 0
print(f"{len(report.held)} held, {len(report.broke)} broke")
if left:
print(f"do-epic-e2e: {left}", file=sys.stderr)
return 1 if report.broke or left else 0
+2 -2
View File
@@ -8,7 +8,7 @@ from do_epic.e2e.settings import Sandbox
from failure import Failure
EPIC_TYPE = "Epic"
# How a team-managed project's workflow moves an issue to done, as do-epic's config names it.
# How a team-managed project's workflow moves an issue to done, as do-epic's `jira-finish-transition` and `jira-finish-status` name it.
FINISH_TRANSITION = "Done"
FINISH_STATUS = "Done"
DESCRIPTION = """**Language:** English
@@ -24,7 +24,7 @@ The person who runs do-epic's e2e command.
class Leftover(Exception):
"""The sandbox epic could not be deleted, so it and its issues are still on the site."""
"""The sandbox epic or one of its issues could not be deleted, so some of them may still be on the site."""
def api(sandbox: Sandbox, path: str) -> str:
+13 -6
View File
@@ -1,4 +1,4 @@
"""What the e2e command does to a sandbox epic: runs the do-epic commands that need no tmux session and no worker, each as a process of its own against the real Jira site, and checks what the site holds after each."""
"""What the e2e command does to a sandbox epic: runs `do-epic --file`, `--record-decision` and `--close-epic`, each as a process of its own against the real Jira site, and checks what the site holds after each."""
import dataclasses
import os
import pathlib
@@ -16,6 +16,8 @@ from model_routing.ladder_file import read_ladder, SHIPPED
from project_config.settings import Epic, Finish
from record_store.jira_description import EpicDescription
# Seconds one do-epic command may take.
TIMEOUT = 300
RULE = "A label that equals a repo's name names that repo.\n"
@@ -32,13 +34,16 @@ class Report:
@dataclasses.dataclass(frozen=True)
class Machine:
"""A home folder of its own, holding the do-epic settings that point at the sandbox, so that nothing of the real home is read or written."""
"""A home folder of its own, holding the do-epic settings that point at the sandbox, so that do-epic writes its state there. The sandbox's env-file stays where it is."""
home: pathlib.Path
launcher: pathlib.Path
def do_epic(self, *args: str, stdin: str = "") -> subprocess.CompletedProcess[str]:
env = {**os.environ, "HOME": str(self.home)}
return subprocess.run([str(self.launcher), *args], input=stdin, env=env, capture_output=True, text=True, timeout=300, check=False)
try:
return subprocess.run([str(self.launcher), *args], input=stdin, env=env, capture_output=True, text=True, timeout=TIMEOUT, check=False)
except subprocess.TimeoutExpired:
return subprocess.CompletedProcess(args, 124, "", f"do-epic {' '.join(args)} ran over {TIMEOUT} seconds and was stopped")
def machine_made(home: pathlib.Path, sandbox: Sandbox, launcher: pathlib.Path) -> Machine | Failure:
@@ -81,7 +86,7 @@ def read(report: Report, store: JiraEpic, what: str) -> list[EpicIssue]:
def items_filed(report: Report, store: JiraEpic, epic_key: str) -> None:
"""The epic's own issues, which a person files in Jira: an item, its sub-task and a second item."""
"""Files an item, its sub-task and a second item in the epic, and checks that a read answers them in the tracker's order."""
parent = store.file(epic_key, "Sandbox item with a sub-task", "The first item.", (HUMAN_LABEL,))
second = store.file(epic_key, "Sandbox second item", "", ())
child = store.file(parent, "Sandbox sub-task", "", ()) if isinstance(parent, str) else parent
@@ -109,10 +114,12 @@ def decision_recorded(report: Report, machine: Machine, record: epic_record.Epic
def closed(report: Report, machine: Machine, store: JiraEpic, epic_key: str) -> None:
early = machine.do_epic(epic_key, finish.CLOSE, stdin="Too early.\n")
report.check(early.returncode == 1 and " open" in early.stderr and store.closed() is False, f"`do-epic {finish.CLOSE}` refuses while issues are open and leaves the epic open (exit {early.returncode}: {early.stderr.strip()})")
still = store.closed()
report.check(early.returncode == 1 and " open" in early.stderr and still is False, f"`do-epic {finish.CLOSE}` refuses while issues are open and leaves the epic open (exit {early.returncode}: {early.stderr.strip()}; epic closed: {still!r})")
for issue in read(report, store, "before finishing them"):
done = store.finish_issue(issue.key)
report.check(done is None, f"finish {issue.key}" + (f": {done.reason}" if isinstance(done, Failure) else ""))
late = machine.do_epic(epic_key, finish.CLOSE, stdin="Everything the sandbox epic held is done.\n")
report.check(late.returncode == 0 and "closed it" in late.stdout, f"`do-epic {finish.CLOSE}` posts the summary and closes the epic (exit {late.returncode}: {late.stdout.strip() or late.stderr.strip()})")
report.check(store.closed() is True, "Jira counts the epic as done")
now = store.closed()
report.check(now is True, f"Jira counts the epic as done (found {now!r})")
+3 -3
View File
@@ -1,4 +1,4 @@
"""What the e2e command needs to know about its sandbox: three environment variables, so that neither the site nor the credentials' place is in the repo (docs/decisions.md → `do-epic: drives one Jira epic …`)."""
"""What the e2e command needs to know about its sandbox: three environment variables, so that neither the site nor the credentials' place is in the repo."""
import dataclasses
import pathlib
import re
@@ -15,7 +15,7 @@ PROJECT_KEY = r"[A-Z][A-Z0-9]+"
MEANING = {
JIRA_URL: "the sandbox Jira site's https address without a path, such as https://acme.atlassian.net",
ENV_FILE: "the absolute path of the file holding the sandbox account's ATLASSIAN_EMAIL and ATLASSIAN_API_TOKEN",
PROJECT: "the key of the sandbox project, a team-managed space where the account may create, edit, transition, comment on and delete issues, such as SBX",
PROJECT: "the key of the sandbox project, a team-managed space where the account may create, edit, transition (to Done), comment on and delete issues, such as SBX",
}
@@ -27,7 +27,7 @@ class Sandbox:
def read(env: typing.Mapping[str, str]) -> Sandbox | Failure:
"""The sandbox that env names; a Failure naming each setting that is missing or misspelt, and what the sandbox's env-file lacks."""
"""The sandbox that env names; a Failure naming each setting that is missing or malformed, and why the sandbox's env-file cannot be used."""
values = {name: env.get(name, "").strip() for name in MEANING}
url, project = values[JIRA_URL], values[PROJECT]
env_file = pathlib.Path(values[ENV_FILE]) if values[ENV_FILE] else None
+51 -1
View File
@@ -2,11 +2,14 @@ import contextlib
import io
import pathlib
import sys
import typing
import tempfile
import unittest
from unittest import mock
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2]))
from do_epic.e2e import main
from do_epic.e2e import main, sandbox, scenario
from do_epic.e2e.settings import ENV_FILE, JIRA_URL, PROJECT
@@ -30,3 +33,50 @@ class MainTest(unittest.TestCase):
code, out, err = self.run_main(["--status"])
self.assertEqual((code, out), (1, ""))
self.assertIn("usage: do-epic-e2e", err)
class ReportedTest(unittest.TestCase):
def setUp(self) -> None:
made = tempfile.TemporaryDirectory()
self.addCleanup(made.cleanup)
env_file = pathlib.Path(made.name) / "jira.env"
env_file.write_text("ATLASSIAN_EMAIL=a@example.com\nATLASSIAN_API_TOKEN=secret\n", encoding="utf-8")
env_file.chmod(0o600)
self.env = {JIRA_URL: "https://acme.atlassian.net", ENV_FILE: str(env_file), PROJECT: "SBX"}
def run_scenario(self, deleted: bool) -> tuple[int, str, str]:
@contextlib.contextmanager
def epic_made(*args: object) -> typing.Iterator[str]:
yield "SBX-1"
if not deleted:
raise sandbox.Leftover("SBX-1 is still there")
out, err = io.StringIO(), io.StringIO()
report = scenario.Report(["a check held"], ["a check broke"])
with mock.patch.object(sandbox, "epic_made", epic_made), mock.patch.object(scenario, "run", return_value=report):
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
code = main.main([], self.env, pathlib.Path("do-epic"))
return code, out.getvalue(), err.getvalue()
def test_prints_each_check_and_fails_where_one_broke(self) -> None:
code, out, err = self.run_scenario(True)
self.assertEqual((code, err), (1, ""))
self.assertIn("ok a check held\nFAIL a check broke\n1 held, 1 broke", out)
def test_prints_the_checks_and_names_the_epic_left_behind_when_it_cannot_be_deleted(self) -> None:
code, out, err = self.run_scenario(False)
self.assertEqual(code, 1)
self.assertIn("ok a check held", out)
self.assertIn("SBX-1 is still there", err)
class MachineTest(unittest.TestCase):
def test_stops_a_do_epic_that_runs_over_its_time_and_says_so(self) -> None:
with tempfile.TemporaryDirectory() as folder:
slow = pathlib.Path(folder) / "slow"
slow.write_text("#!/bin/sh\nsleep 30\n", encoding="utf-8")
slow.chmod(0o755)
with mock.patch.object(scenario, "TIMEOUT", 1):
done = scenario.Machine(pathlib.Path(folder), slow).do_epic("ABC-1")
self.assertEqual(done.returncode, 124)
self.assertIn("ran over 1 seconds", done.stderr)