Answer product-owner round 1: 0 🔴, 2 🟡 (0 misread); print the checks even where the epic stays, stop a do-epic that hangs, and state the sandbox's needs in the README
This commit is contained in:
@@ -191,7 +191,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
- `compress-archives` gzips each transcript in loop-claude's archive and each file in the review folders 30 days after it last changed, keeping all of them; Install's crontab line runs it daily.
|
||||
- `deploy-release [--cut] [<checkout>]` moves `~/.claude`, or the checkout named, to the newest release, as Update says; with `--cut`, run in the development clone, it first cuts the next release, as Release says.
|
||||
- `do-epic <EPIC-KEY>` runs the chunks of one Jira epic whose items span repos, or have none, each in its own window of the tmux session `epic-<EPIC-KEY>`, on the same cycle, judge and interrupt as loop-claude. The epic's description is its record: goals, audience, scoring and decisions. An open issue runs once a label names its repo, in Jira's backlog order, as many at once as a line such as `**Max parallel chunks:** 2` in the epic's description allows, 1 without it, and as each repo's `parallel-chunks` allows. A chunk whose pull request waits on a gate parks. do-epic closes the epic only once you confirm every goal. Install → "Set up a project" says how to set it up; `do-epic --help` states the flags and the contract.
|
||||
- `do-epic-e2e` drives `do-epic`'s `--file`, `--record-decision` and `--close-epic` against an epic it makes in a sandbox project on a real Jira site and deletes when it ends, as Test says. It isn't part of `run-tests`.
|
||||
- `do-epic-e2e` runs `do-epic` against a sandbox epic on a real Jira site, as Test says.
|
||||
- `judge-replay <model> [--offline]` replays loop-claude's judge on archived verdicts of the projects run on the model. It prints what each rung a review climbs to misses, and the lowest rung below those that returns every verdict, as each candidate above it does. With `--offline` it doesn't call a judge: it counts what a "?" rule, a decision table and a classifier trained on the archive decide of each verdict's kind from the worker's final message, and how many interrupts each misses. `judge-replay --help` states both replays.
|
||||
- `larv-review <command>` holds larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's round count and the opening of its round commit's title, the scoring run's trigger and setup; the comprehension history's format, its latest places and its count of how often seats agree; and each reviewer's prompt header. `larv-review --help` states them. `settings.json` → `hooks` runs `larv-review reviewer-context` as each subagent starts. A reviewer under `agents/` whose frontmatter has `omitClaudeMd: true` starts without `CLAUDE.md`, and the command hands it that file's "Shared goal", "Prose" and "Vocabulary" sections.
|
||||
- `loop-claude <task>` runs a project's next chunks, each in its own tmux window, cycle after cycle; `loop-claude --help` states its contract.
|
||||
@@ -253,7 +253,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a
|
||||
|
||||
Test on the host with `bin/run-tests`, which runs pytest through `uvx`, pinned in `bin/run_tests/main.py`. It prints each failing test with its error and pytest's summary line, and saves the whole output to a file whose path it prints. It passes on any test paths or pytest options it is given, such as `bin/run-tests bin/chunk_pr -x`. It needs `uv` on `PATH`, which fetches pytest, pytest-xdist and mypy, a Docker daemon, Linux, and a user other than root; a test that lacks one fails, naming it. pytest-xdist runs the tests in one worker per CPU of a heavy slot (`docs/decisions.md` → `bin/: the tools support only the latest stable Python, …`). `python3 -m unittest discover -s bin -p 'test_*.py'` runs the same tests one at a time, with the same needs. Test the bench's scripts with `python3 -m unittest discover -s workflow-bench -p 'test_*.py'`.
|
||||
|
||||
`bin/do-epic-e2e` checks `do-epic` against a real Jira site, which `run-tests` never reaches. Set `DO_EPIC_E2E_JIRA_URL` to the sandbox site's address, `DO_EPIC_E2E_PROJECT` to its project's key and `DO_EPIC_E2E_ENV_FILE` to the absolute path of the file holding the sandbox account's `ATLASSIAN_EMAIL` and `ATLASSIAN_API_TOKEN`. The account must be able to create, transition, comment on and delete issues there. It fails, naming each setting that is missing; it makes an epic with issues, runs `do-epic` against it in a home folder of its own, and deletes the epic and its issues when it ends. It needs a Docker daemon, as `run-tests` does.
|
||||
`bin/do-epic-e2e` checks `do-epic` against a real Jira site, which `run-tests` never reaches. It makes an epic with issues and sub-tasks in a sandbox project, runs `do-epic --file`, `--record-decision` and `--close-epic` against it in a home folder of its own, and deletes the epic and its issues when it ends. The sandbox project must be team-managed, with a `Done` transition, and its account must be able to create, edit, transition, comment on and delete issues there. Set `DO_EPIC_E2E_JIRA_URL` to the sandbox site's address, `DO_EPIC_E2E_ENV_FILE` to the absolute path of the file holding the account's `ATLASSIAN_EMAIL` and `ATLASSIAN_API_TOKEN`, and `DO_EPIC_E2E_PROJECT` to the project's key. The command fails, naming each setting that is missing. It needs a Docker daemon, as `run-tests` does.
|
||||
|
||||
CI runs `ci.sh` on every pull request, and `main` doesn't merge anything until it passes. The script runs the same tests on the committed HEAD in a pinned Python container. Run it under `run-heavy` to reproduce a CI failure, with Docker on `PATH` and its socket at `/var/run/docker.sock`, which the container shares.
|
||||
|
||||
|
||||
+10
-6
@@ -1,4 +1,4 @@
|
||||
"""The do-epic-e2e command: runs do-epic's commands that need no tmux session and no worker against a sandbox epic on a real Jira site (docs/decisions.md → `do-epic: drives one Jira epic …`). It is no part of `run-tests`."""
|
||||
"""The do-epic-e2e command: runs do-epic's commands that need no tmux session and no worker against a sandbox epic on a real Jira site (docs/decisions.md → `do-epic: drives one Jira epic …`)."""
|
||||
import pathlib
|
||||
import sys
|
||||
import typing
|
||||
@@ -22,6 +22,8 @@ def main(argv: list[str], env: typing.Mapping[str, str], launcher: pathlib.Path)
|
||||
if isinstance(found, Failure):
|
||||
print(f"do-epic-e2e cannot start: {found.reason}", file=sys.stderr)
|
||||
return 1
|
||||
report = scenario.Report()
|
||||
left: sandbox.Leftover | None = None
|
||||
try:
|
||||
with sandbox.epic_made(found, "do-epic e2e sandbox epic") as key:
|
||||
if isinstance(key, Failure):
|
||||
@@ -29,12 +31,14 @@ def main(argv: list[str], env: typing.Mapping[str, str], launcher: pathlib.Path)
|
||||
return 1
|
||||
print(f"Sandbox epic {key} in {found.jira_url}")
|
||||
report = scenario.run(found, key, launcher)
|
||||
except sandbox.Leftover as left:
|
||||
print(f"do-epic-e2e: {left}", file=sys.stderr)
|
||||
return 1
|
||||
except sandbox.Leftover as failure:
|
||||
left = failure
|
||||
# The checks are printed even where the epic could not be deleted.
|
||||
for line in report.held:
|
||||
print(f"ok {line}")
|
||||
for line in report.broke:
|
||||
print(f"FAIL {line}")
|
||||
print(f"{len(report.held)} held, {len(report.broke)} broke; the sandbox epic is deleted")
|
||||
return 1 if report.broke else 0
|
||||
print(f"{len(report.held)} held, {len(report.broke)} broke")
|
||||
if left:
|
||||
print(f"do-epic-e2e: {left}", file=sys.stderr)
|
||||
return 1 if report.broke or left else 0
|
||||
|
||||
@@ -8,7 +8,7 @@ from do_epic.e2e.settings import Sandbox
|
||||
from failure import Failure
|
||||
|
||||
EPIC_TYPE = "Epic"
|
||||
# How a team-managed project's workflow moves an issue to done, as do-epic's config names it.
|
||||
# How a team-managed project's workflow moves an issue to done, as do-epic's `jira-finish-transition` and `jira-finish-status` name it.
|
||||
FINISH_TRANSITION = "Done"
|
||||
FINISH_STATUS = "Done"
|
||||
DESCRIPTION = """**Language:** English
|
||||
@@ -24,7 +24,7 @@ The person who runs do-epic's e2e command.
|
||||
|
||||
|
||||
class Leftover(Exception):
|
||||
"""The sandbox epic could not be deleted, so it and its issues are still on the site."""
|
||||
"""The sandbox epic or one of its issues could not be deleted, so some of them may still be on the site."""
|
||||
|
||||
|
||||
def api(sandbox: Sandbox, path: str) -> str:
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""What the e2e command does to a sandbox epic: runs the do-epic commands that need no tmux session and no worker, each as a process of its own against the real Jira site, and checks what the site holds after each."""
|
||||
"""What the e2e command does to a sandbox epic: runs `do-epic --file`, `--record-decision` and `--close-epic`, each as a process of its own against the real Jira site, and checks what the site holds after each."""
|
||||
import dataclasses
|
||||
import os
|
||||
import pathlib
|
||||
@@ -16,6 +16,8 @@ from model_routing.ladder_file import read_ladder, SHIPPED
|
||||
from project_config.settings import Epic, Finish
|
||||
from record_store.jira_description import EpicDescription
|
||||
|
||||
# Seconds one do-epic command may take.
|
||||
TIMEOUT = 300
|
||||
RULE = "A label that equals a repo's name names that repo.\n"
|
||||
|
||||
|
||||
@@ -32,13 +34,16 @@ class Report:
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Machine:
|
||||
"""A home folder of its own, holding the do-epic settings that point at the sandbox, so that nothing of the real home is read or written."""
|
||||
"""A home folder of its own, holding the do-epic settings that point at the sandbox, so that do-epic writes its state there. The sandbox's env-file stays where it is."""
|
||||
home: pathlib.Path
|
||||
launcher: pathlib.Path
|
||||
|
||||
def do_epic(self, *args: str, stdin: str = "") -> subprocess.CompletedProcess[str]:
|
||||
env = {**os.environ, "HOME": str(self.home)}
|
||||
return subprocess.run([str(self.launcher), *args], input=stdin, env=env, capture_output=True, text=True, timeout=300, check=False)
|
||||
try:
|
||||
return subprocess.run([str(self.launcher), *args], input=stdin, env=env, capture_output=True, text=True, timeout=TIMEOUT, check=False)
|
||||
except subprocess.TimeoutExpired:
|
||||
return subprocess.CompletedProcess(args, 124, "", f"do-epic {' '.join(args)} ran over {TIMEOUT} seconds and was stopped")
|
||||
|
||||
|
||||
def machine_made(home: pathlib.Path, sandbox: Sandbox, launcher: pathlib.Path) -> Machine | Failure:
|
||||
@@ -81,7 +86,7 @@ def read(report: Report, store: JiraEpic, what: str) -> list[EpicIssue]:
|
||||
|
||||
|
||||
def items_filed(report: Report, store: JiraEpic, epic_key: str) -> None:
|
||||
"""The epic's own issues, which a person files in Jira: an item, its sub-task and a second item."""
|
||||
"""Files an item, its sub-task and a second item in the epic, and checks that a read answers them in the tracker's order."""
|
||||
parent = store.file(epic_key, "Sandbox item with a sub-task", "The first item.", (HUMAN_LABEL,))
|
||||
second = store.file(epic_key, "Sandbox second item", "", ())
|
||||
child = store.file(parent, "Sandbox sub-task", "", ()) if isinstance(parent, str) else parent
|
||||
@@ -109,10 +114,12 @@ def decision_recorded(report: Report, machine: Machine, record: epic_record.Epic
|
||||
|
||||
def closed(report: Report, machine: Machine, store: JiraEpic, epic_key: str) -> None:
|
||||
early = machine.do_epic(epic_key, finish.CLOSE, stdin="Too early.\n")
|
||||
report.check(early.returncode == 1 and " open" in early.stderr and store.closed() is False, f"`do-epic {finish.CLOSE}` refuses while issues are open and leaves the epic open (exit {early.returncode}: {early.stderr.strip()})")
|
||||
still = store.closed()
|
||||
report.check(early.returncode == 1 and " open" in early.stderr and still is False, f"`do-epic {finish.CLOSE}` refuses while issues are open and leaves the epic open (exit {early.returncode}: {early.stderr.strip()}; epic closed: {still!r})")
|
||||
for issue in read(report, store, "before finishing them"):
|
||||
done = store.finish_issue(issue.key)
|
||||
report.check(done is None, f"finish {issue.key}" + (f": {done.reason}" if isinstance(done, Failure) else ""))
|
||||
late = machine.do_epic(epic_key, finish.CLOSE, stdin="Everything the sandbox epic held is done.\n")
|
||||
report.check(late.returncode == 0 and "closed it" in late.stdout, f"`do-epic {finish.CLOSE}` posts the summary and closes the epic (exit {late.returncode}: {late.stdout.strip() or late.stderr.strip()})")
|
||||
report.check(store.closed() is True, "Jira counts the epic as done")
|
||||
now = store.closed()
|
||||
report.check(now is True, f"Jira counts the epic as done (found {now!r})")
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""What the e2e command needs to know about its sandbox: three environment variables, so that neither the site nor the credentials' place is in the repo (docs/decisions.md → `do-epic: drives one Jira epic …`)."""
|
||||
"""What the e2e command needs to know about its sandbox: three environment variables, so that neither the site nor the credentials' place is in the repo."""
|
||||
import dataclasses
|
||||
import pathlib
|
||||
import re
|
||||
@@ -15,7 +15,7 @@ PROJECT_KEY = r"[A-Z][A-Z0-9]+"
|
||||
MEANING = {
|
||||
JIRA_URL: "the sandbox Jira site's https address without a path, such as https://acme.atlassian.net",
|
||||
ENV_FILE: "the absolute path of the file holding the sandbox account's ATLASSIAN_EMAIL and ATLASSIAN_API_TOKEN",
|
||||
PROJECT: "the key of the sandbox project, a team-managed space where the account may create, edit, transition, comment on and delete issues, such as SBX",
|
||||
PROJECT: "the key of the sandbox project, a team-managed space where the account may create, edit, transition (to Done), comment on and delete issues, such as SBX",
|
||||
}
|
||||
|
||||
|
||||
@@ -27,7 +27,7 @@ class Sandbox:
|
||||
|
||||
|
||||
def read(env: typing.Mapping[str, str]) -> Sandbox | Failure:
|
||||
"""The sandbox that env names; a Failure naming each setting that is missing or misspelt, and what the sandbox's env-file lacks."""
|
||||
"""The sandbox that env names; a Failure naming each setting that is missing or malformed, and why the sandbox's env-file cannot be used."""
|
||||
values = {name: env.get(name, "").strip() for name in MEANING}
|
||||
url, project = values[JIRA_URL], values[PROJECT]
|
||||
env_file = pathlib.Path(values[ENV_FILE]) if values[ENV_FILE] else None
|
||||
|
||||
@@ -2,11 +2,14 @@ import contextlib
|
||||
import io
|
||||
import pathlib
|
||||
import sys
|
||||
import typing
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2]))
|
||||
|
||||
from do_epic.e2e import main
|
||||
from do_epic.e2e import main, sandbox, scenario
|
||||
from do_epic.e2e.settings import ENV_FILE, JIRA_URL, PROJECT
|
||||
|
||||
|
||||
@@ -30,3 +33,50 @@ class MainTest(unittest.TestCase):
|
||||
code, out, err = self.run_main(["--status"])
|
||||
self.assertEqual((code, out), (1, ""))
|
||||
self.assertIn("usage: do-epic-e2e", err)
|
||||
|
||||
|
||||
class ReportedTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
made = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(made.cleanup)
|
||||
env_file = pathlib.Path(made.name) / "jira.env"
|
||||
env_file.write_text("ATLASSIAN_EMAIL=a@example.com\nATLASSIAN_API_TOKEN=secret\n", encoding="utf-8")
|
||||
env_file.chmod(0o600)
|
||||
self.env = {JIRA_URL: "https://acme.atlassian.net", ENV_FILE: str(env_file), PROJECT: "SBX"}
|
||||
|
||||
def run_scenario(self, deleted: bool) -> tuple[int, str, str]:
|
||||
@contextlib.contextmanager
|
||||
def epic_made(*args: object) -> typing.Iterator[str]:
|
||||
yield "SBX-1"
|
||||
if not deleted:
|
||||
raise sandbox.Leftover("SBX-1 is still there")
|
||||
|
||||
out, err = io.StringIO(), io.StringIO()
|
||||
report = scenario.Report(["a check held"], ["a check broke"])
|
||||
with mock.patch.object(sandbox, "epic_made", epic_made), mock.patch.object(scenario, "run", return_value=report):
|
||||
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
code = main.main([], self.env, pathlib.Path("do-epic"))
|
||||
return code, out.getvalue(), err.getvalue()
|
||||
|
||||
def test_prints_each_check_and_fails_where_one_broke(self) -> None:
|
||||
code, out, err = self.run_scenario(True)
|
||||
self.assertEqual((code, err), (1, ""))
|
||||
self.assertIn("ok a check held\nFAIL a check broke\n1 held, 1 broke", out)
|
||||
|
||||
def test_prints_the_checks_and_names_the_epic_left_behind_when_it_cannot_be_deleted(self) -> None:
|
||||
code, out, err = self.run_scenario(False)
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("ok a check held", out)
|
||||
self.assertIn("SBX-1 is still there", err)
|
||||
|
||||
|
||||
class MachineTest(unittest.TestCase):
|
||||
def test_stops_a_do_epic_that_runs_over_its_time_and_says_so(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
slow = pathlib.Path(folder) / "slow"
|
||||
slow.write_text("#!/bin/sh\nsleep 30\n", encoding="utf-8")
|
||||
slow.chmod(0o755)
|
||||
with mock.patch.object(scenario, "TIMEOUT", 1):
|
||||
done = scenario.Machine(pathlib.Path(folder), slow).do_epic("ABC-1")
|
||||
self.assertEqual(done.returncode, 124)
|
||||
self.assertIn("ran over 1 seconds", done.stderr)
|
||||
|
||||
Reference in New Issue
Block a user