diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index 417e0a8b..01d231fa 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -1,6 +1,12 @@ name: CI on: pull_request +# A push to a pull request cancels that pull request's older runs; the merge gate reads only the head commit's status. +# The group is the ref: refs/pull//head for a pull request, the branch for a push, so no two refs cancel each other's runs. +concurrency: + group: ci-${{ github.ref }} + cancel-in-progress: true + jobs: tests: runs-on: docker-host diff --git a/.gitignore b/.gitignore index 9f3ba4ff..9f5b4fa9 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,7 @@ !.gitignore !CHANGELOG.md !CLAUDE.md +!LICENSE.md !README.md !ci.sh !settings.json @@ -30,6 +31,8 @@ !bin/adf_codec/node_shim/package-lock.json !bin/adf_codec/node_shim/package.json !bin/adf_codec/node_shim/shim.mjs +!bin/agent_cycle/ +!bin/agent_cycle/*.py !bin/atlassian/ !bin/atlassian/*.py !bin/branch_chunks/ @@ -53,6 +56,11 @@ !bin/deploy-release !bin/deploy_release/ !bin/deploy_release/*.py +!bin/do-epic +!bin/do_epic/ +!bin/do_epic/*.py +!bin/do_epic/backend/ +!bin/do_epic/backend/*.py !bin/failure.py !bin/forge/ !bin/forge/*.py diff --git a/CHANGELOG.md b/CHANGELOG.md index df8ee3d3..f9f82176 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,33 @@ ## Unreleased +- do-epic gives a `do-epic-human` issue that blocks another its own window: it DMs you once through the notifier of one of the epic's repos, takes no running slot, and polls until you close the issue in Jira, when the issues it held up may start. +- do-epic tells an epic chunk's worker how its repo ships the change: a bullet under the repo's changelog where the repo publishes something, none where its plan is in Jira, and no `--finish` for `chunk-pr merge`. +- `do-epic --status` says which chunk asks you a question, and since when each parked chunk has awaited its gate. +- do-epic moves a chunk's issue to done in Jira when the chunk finishes, and the split item it belongs to once its last sub-task is done, so no finished chunk starts again and the chunks it blocks start. It moves them by the transition `jira-finish-transition` and `jira-finish-status` name, so set both; without them a finishing chunk stops with that reason. +- Describe do-epic in the README: its entry under Tools, and its machine settings and `repo-rule.md` under Set up a project. +- do-epic finishes an epic: once none of its issues is open, a finishing chunk files a release review of each repo the epic changed, as an issue labelled `do-epic-release-review`, and asks you to confirm each goal. `do-epic --close-epic` then posts the summary piped in as a comment on the epic and closes it in Jira, by the transition `jira-finish-transition` names, to the status `jira-finish-status` names. +- do-epic parks a chunk whose pull request waits on its checks, an approval or a new review: the chunk frees its place for the next one, its tmux window says what it awaits, and its worker goes on when the gate opens or a reviewer comments, even after a reboot. `do-epic --status` prints what each open issue does. +- A chunk that `do-epic` runs files a finding about an epic goal as an issue of the epic, with `do-epic --file`, and records a decision in the epic's description, with `--record-decision`; its judge reads the epic's Scoring and Decisions before the repo's records. +- Added `do-epic `: it runs the chunks of one Jira epic, each in its own window of the tmux session `epic-`. As many run at once as the line `**Max parallel chunks:** 2` in the epic's description allows, one without it; a repo takes as many as its `parallel-chunks`, and none while loop-claude runs chunks in it. loop-claude in turn starts no chunk in a checkout where do-epic runs one. +- Refuse in rule-hook a Bash or Monitor command that waits forever: a sleeping loop with no time or count cap, and a `pgrep -f` or `pkill -f` whose pattern its own command line matches. +- A project's config may list its team's convention files in `coding-conventions`, which larv-review's reviewer agents read beside the principles and the standards, the team's rule winning where they conflict. larv-review opens the pull request after its own review where a push to the branch starts CI, and files an item to add a mutation tool where `mutate-changes` covers none of a language. The technical principles gain rules for tests on a shared environment, "Never mock a service your own team runs" and "Don't loosen a type when refactoring"; stability-reviewer checks a change that stores a value differently; and every finding of an outside reviewer now changes the setup, a deterministic tool first. +- do-epic's epic backends rank a child issue just before another, in Jira and in memory, and do-epic can re-rank Jira's backlog to an epic's ranking: it moves only the issues the loop filed, so each issue a person filed keeps its place, and it stops at the first rank Jira refuses and names it. Nothing calls it yet. +- Comprehension seats run as the `comprehension-seat` agent under `agents/`, which starts without `CLAUDE.md` and the skill and agent listings, and `larv-review prepare-scoring` no longer takes `--preloaded`; after updating, launch each seat with `subagent_type: comprehension-seat`. +- do-epic splits an epic item that changes several repos into one Jira sub-task per repo, each labelled `repo:`, with a headless claude call predicting the repos from the code; the item is marked `split-by-repo` and counts as done once every sub-task is, and an issue a person filed gets a comment for the team in the epic's language. The epic backends gain `comment`, in the Jira backend and the in-memory one. +- do-epic's epic backends read each issue's scores, its blockers and whether the loop filed it, in the Jira backend and the in-memory one; the Jira backend marks each issue it files as the loop's. Nothing calls it yet. +- The technical principles now forbid a test that decides at run time whether to check anything, by a conditional skip, early return or catch around its assertions, and stability-reviewer flags one. +- do-epic's scorer orders an epic's items by score, then by Blocked by: an item's Moves names a goal as `epic 1.1`, `code health` or `repo 2`, and the epic's achieve goals outweigh the achieve goals of every repo. Nothing calls it yet. +- `todo-item add` and `todo-item edit --detail` refuse a detail whose `Touches:` list touch-predictions cannot read, and `rank_todo.py --check` fails on each item holding one, so rule-hook refuses a commit of such a plan: after updating, run `~/.claude/bin/rank_todo.py --check` in each project and fix each list it names. +- loop-claude hands each worker's claude a Bash output ceiling of 6,000 characters that saves a longer result to a file, rule-hook refuses a Read of a session transcript under ~/.claude/projects, and review-replay takes --bash-ceiling and --read-limit to test such a limit against archived review rounds. +- do-epic picks each epic item's repo, or that it is Claude's work or a human task, by repo-rule.md and a headless claude call, records the pick as a label on the issue and reads it back without a new call; where the rule leaves it unclear it returns a question for the human. The epic backends gain a label write. +- judge-replay --offline replays every archived verdict through a "?" rule, a decision table and a classifier trained on the archive, with no judge call, and counts the interrupts each misses. +- do-epic finds an epic repo's checkout and project config by the repo's name, and clones a missing checkout from the host and owner of the epic's other repos, asking the human where they differ or the clone fails. +- do-epic reads its machine-wide settings from ~/.claude-projects/do-epic/config.toml and its repo rule from repo-rule.md beside it, and refuses to start while either is missing or misspelt. +- `project-record save` now writes a goals file kept as a Jira epic's description, except a change to `## Goals` or `## Audience`, which stays an edit in Jira. +- Add do-epic's epic backend: one interface that reads a Jira epic's child issues and their sub-tasks, with labels, assignee, status and parent, and files new ones, met by a Jira backend and an in-memory fake that pass the same contract tests. +- Fix for an outside reviewer's comment on a PR now ends with a change to the setup that would have caught it, in claude-config first, then the principles and standards, then the project. +- larv-review reruns the pass that raised a ๐ŸŸก when the fix removes or loosens an assertion, changes an expected value, or skips, deletes or guards a test. Other ๐ŸŸก fixes still rerun no pass. - Add `markdown-section `, which prints the section of a markdown file under the heading the fragment picks, to the next heading of its level or a higher one. It fails, listing the headings, where several hold the fragment. - loop-claude and subagent-rung run an L 3 task one rung above the project's rung and an L 4 task two above it: on an Opus 5.5 project, Fable 5.2 at low and at medium. - Add `larv-review round-title `, which prints a review round commit's title prefix with the ๐Ÿ”ด, ๐ŸŸก and misread counts taken from the round's report. rule-hook refuses a round commit whose counts differ from that report, where the report exists and can be counted; a report must head each finding `### ๐Ÿ”ด` or `### ๐ŸŸก`, optionally followed by ` [misread]`. diff --git a/CLAUDE.md b/CLAUDE.md index 5934ac61..7c4a0645 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,6 +18,12 @@ How work proceeds in any repo, unless the repo's `AGENTS.md` says otherwise. The - **Ask, don't guess.** Three choices are always the human's: a goal; a release, meaning the items planned into it and when it is cut; and a change to loop-claude's routing ladder, its rungs included. Any other rule that asks the human's approval reserves that choice too. Ask about a reserved choice. For any other choice, first check whether the goals already answer it. If they nearly do, propose a change to the goals and show how it answers this case: an answer settles one question, a goal settles every question like it. Otherwise take the option you would recommend, and record it with its reason: in the item's detail, or, where it governs later work, as the next sentence says. A recommendation you can give a reason for is not a guess. Ask only where you have no recommendation. Write every answer once, where it belongs, never only in chat: a goal in the goals file, a working rule in `AGENTS.md`, a standing choice in a comment at the one place it governs, or in `docs/decisions.md` once made in more than one place. When an answer holds a goal and a choice, write the goal in the goals file, and record as a choice only what the goal leaves open. - **Bench a syntax with the people who write it.** Before a new or changed spelling of something users read or write ships โ€” markdown, HTML, a config key, a CLI flag โ€” a panel of fresh-context readers, seated as those users from the goals file's `## Audience`, picks it. - **Implement the simplest change that solves the item, and ship it with `/larv-review`.** Whenever a test can check the change, write that test first and see it fail. Take its expected values from the item, never from the code. larv-review's reviewers hold the change to the technical principles, and a finding against one names it. Read a file in the principles folder the project's config sets (`~/.claude/bin/project-config principles` prints it) only when the item changes it, a finding or a record you write names it, or you decline a finding. +- **Change the setup for each finding an outside reviewer raises on a PR, deterministic tools first.** An outside reviewer is a forge's auto-reviewer or a human, not one of larv-review's reviewer agents. For each finding of theirs, whether you fix it or decline it, ask why our review didn't raise it first, and change the setup so it would; a fix that only answers the comment leaves the same gap open for the next chunk. Put the change in the first place that fits: + 1. A deterministic tool that would have caught it, such as a rule-hook rule, a gate check, a linter setting or `mutate-changes`: a tool catches the fault every time, and a reviewer only when it notices. + 2. A technical principle or a review standard, where a rule was missing or too vague: in claude-config's files, or in the project's own principles folder. + 3. Anything else: the reviewers' agents, `larv-review`'s text, or the project's own settings or `AGENTS.md`. A client repo's `AGENTS.md` changes by a proposal of its own, per "A client project's `CLAUDE.md` and `AGENTS.md`" above. + + Change claude-config's files where every project gains, and the project's own only where the gap is that project's alone. In claude-config, the change is a fix of the chunk. In another project, a change to the project's own files is a fix of the chunk too, and a change to claude-config's is an item in claude-config's plan: start a worktree of claude-config off `origin/main`, run `todo-item add` in it, commit the plan file, and ship it as a pull request of its own with `chunk-pr`, since claude-config's default branch may not take a direct push. Describe the gap without the client's name, code or comment text. File a change too big for the chunk as its own item. Where no setup gap caused the finding, as when the finding is wrong, say so in the chunk's summary rather than inventing a change. - **Remove a finished item from the plan, and revise any decision that cites it.** If the repo publishes something others consume โ€” a package, an image, an API โ€” reword the item for them and move it to `CHANGELOG.md` under `## Unreleased`. A changelog is for the public, never for the next agent. Otherwise delete it; git history is the record. `~/.claude/bin/todo-item finish ` removes the item with its detail, and adds the bullet you pass with `--changelog `; it refuses while a decision cites the item. A plan kept in Jira takes no bullet, since the issues' history is its changelog: `finish` there refuses `--changelog`. A chunk finishes its items as it merges, through `chunk-pr merge --finish `, which takes the same flags (`larv-review` ยง5). - **Releases are the human's.** The version bump and the publishing credentials are theirs, whatever another rule says. Prepare everything else, so cutting a release is one decision. - **Open each plan item or ticket with a bold one-sentence tagline**, so the taglines alone say what a release holds. The tagline states the target state, checkable when done: an imperative verb plus what holds afterwards. A present-tense fact reads as the current state and doesn't say anything about what to do. Add "today X" only where the direction is otherwise ambiguous. GOOD: `**Accept undefined in MetadataValue and drop such keys on output.**` BANNED: `MetadataValue accepts undefined; such keys are dropped on output`. @@ -128,7 +134,7 @@ Claude Code reads this section by its heading when it compacts a conversation. K - **chunk** โ€” the unit of work, shipped as one PR; "Working loop" says what it holds. - **cycle** โ€” in `loop-claude`, worker runs and the judge's verdict on each, until the chunk closes or is left for the next cycle. In between, the human answers any interrupt, or the worker resumes with an answer the judge quoted from the premises or the judge's records. - **gate** โ€” the merge gate: the checks a PR passes before it merges. -- **goals file** โ€” the file holding a project's `## Goals` and `## Audience`: the one its config names as `goals-and-audience`. Read it in the file `~/.claude/bin/project-record open goals-and-audience` prints. It is `README.md`, unless a client's project keeps it in the config's folder, or in a Jira epic's description, which the loop only reads. +- **goals file** โ€” the file holding a project's `## Goals` and `## Audience`: the one its config names as `goals-and-audience`. Read it in the file `~/.claude/bin/project-record open goals-and-audience` prints. It is `README.md`, unless a client's project keeps it in the config's folder, or in a Jira epic's description, whose `## Goals` and `## Audience` the loop only reads. - **interrupt** โ€” `loop-claude` asking the human, on its terminal, for a decision that is theirs. They type a one-line answer there, or `/talk` to open the worker's session there with `claude --resume`. A release ready to cut opens that session at once. A routing call opens claude on that terminal too, in `~/code/claude-config`, the checkout where this setup is developed. - **judge** โ€” the headless claude that reads a worker's transcript each cycle and decides whether to interrupt and whether the chunk is finished. - **judge's records** โ€” the files the judge reads beside the premises: the project's `AGENTS.md` and `docs/decisions.md`, then the human's `~/.claude/CLAUDE.md` and the `.md` files in `~/.claude/rules/`, then those in the project's review standards. diff --git a/LICENSE.md b/LICENSE.md new file mode 100644 index 00000000..562f0376 --- /dev/null +++ b/LICENSE.md @@ -0,0 +1,21 @@ +# MIT License + +Copyright (c) 2026 Larv IT AB + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index 430fa690..3446b6cf 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,7 @@ Then pick a model with `/model`. Link `loop-claude` onto `PATH`: `ln -s ~/.claude/bin/loop-claude ~/.local/bin/loop-claude`. Every claude it starts, the interactive ones included, skips every permission prompt. loop-claude runs each chunk in its own window of a tmux session per project, so it needs tmux 3.2 or later. -Back up two folders: `~/.claude/loop-claude-transcripts`, where loop-claude keeps every transcript, and `~/.local/state/larv-review`, where larv-review keeps each chunk's reports. To keep them small, add a crontab line that gzips each transcript and review file 30 days after it last changed, and logs each run with its date: +Back up three folders: `~/.claude/loop-claude-transcripts`, where loop-claude keeps every transcript, `~/.claude/do-epic-transcripts`, where do-epic keeps its chunks' transcripts, and `~/.local/state/larv-review`, where larv-review keeps each chunk's reports. To keep loop-claude's transcripts and the reports small, add a crontab line that gzips each transcript and review file 30 days after it last changed, and logs each run with its date: ```sh 17 4 * * * { date; $HOME/.claude/bin/compress-archives; } >> $HOME/.local/state/compress-archives.log 2>&1 @@ -109,6 +109,8 @@ A project may also set `env-file`, the file of `KEY=value` credentials its tools A project may also set `touch-predictions`, a path spelt as `changelog`'s is, such as `touch-predictions = "checkout:docs/touch-predictions.md"`. `touch-predictions record` appends each chunk's predicted files and parts there, beside those it touched. A project that leaves it out keeps no such record. +A project whose team wrote its own coding conventions lists the files in `coding-conventions`, each spelt as `principles` is, such as `coding-conventions = ["checkout:CONVENTIONS.md"]`. Each reviewer agent larv-review launches reads them beside the principles and the standards, and the team's rule wins where they conflict. + A project may also set `notifier` to `"mattermost"` or `"slack"`, the service that sends loop-claude's DMs. The `env-file` then holds that service's keys. A project that sets none gets no DMs, whatever its env file holds. `project-config notifier` prints the setting. A project's config may set four more things, each optional, as `project-config --help` describes. `language` names the language its owners write in, where that isn't English: the worker writes the project's prose in it, and the prose reviewer judges the prose in it. `commit-title`, `pull-request-title` and `branch` are templates for a project that titles and names these in another way than `CLAUDE.md` โ†’ "Git, commits, and PRs" says. `chunk-preflight` names the branch and the worktree by the `branch` template. On a branch that fits it, rule-hook refuses a commit, and `chunk-pr` a pull request, whose title doesn't fit its template; on any other branch, such as a release review's, a title is the summary alone. A tool that reads a commit title, such as a round's `Answer round :`, reads the template's `{summary}`: it tries `commit-title`, then `pull-request-title`, since a squash merge titles the commit as the pull request, then takes the title whole. While a project's config holds a fault, rule-hook refuses every commit there, and the tools that read its titles fail naming the fault. @@ -134,7 +136,7 @@ The loop reads such a plan and files items into it, each as a Task under the epi `todo-item edit` and `move` change only an issue the loop filed. Of an issue a person filed, `edit` sets only the scores, and posts its `--comment` on the issue to tell the team why; the summary, description and Fix Version stay the team's to change in Jira, as do its links. On an issue the loop filed, `edit --blocked-by` makes and deletes the links to the epic's open items. For `todo-item finish`, the config names the workflow transition that finishes an item and the status it leads to, both as Jira shows them: `jira-finish-transition = "Mark as done"` and `jira-finish-status = "Done"`. Finish then moves the item's issue by that transition. It refuses a transition that leads to another status, or to one Jira doesn't count as done, and fails where the config sets neither. The four things not supported for a plan outside the checkout, above, are not supported for a plan in Jira either. -A client that keeps its goals and audience in an epic's description names that epic as the goals file: `goals-and-audience = "jira:https://acme.atlassian.net/browse/ABC-12"`, which may be the plan's own epic. Such a config keeps its plan in Jira too, and reads both with the same `env-file`. The description needs `## Goals` and `## Audience` as headings, level-2 headings in Jira's editor. The loop only reads the description, so edit it in Jira; `project-record open goals-and-audience` prints a file holding its text. Whoever may edit that description sets the premises every worker and judge follows, so keep its edit rights to the people who own the project's goals. +A client that keeps its goals and audience in an epic's description names that epic as the goals file: `goals-and-audience = "jira:https://acme.atlassian.net/browse/ABC-12"`, which may be the plan's own epic. Such a config keeps its plan in Jira too, and reads both with the same `env-file`. The description needs `## Goals` and `## Audience` as headings, level-2 headings in Jira's editor. `project-record open goals-and-audience` prints a file holding the description's text, and `project-record save` writes it back, except a change to `## Goals` or `## Audience`: edit those two in Jira. A save rewrites the whole description from markdown, which keeps no mention, macro or block id that markdown can't hold. Whoever may edit that description sets the premises every worker and judge follows, so keep its edit rights to the people who own the project's goals. To name a chunk's branch, worktree, commits and pull request by the issue key, as in `ABC-123` and `ABC-123 - โ€ฆ`, set these three templates, with your Jira project's key in place of `ABC`: `branch = "ABC-{id}"`, `commit-title = "ABC-{id} - {summary}"` and `pull-request-title = "ABC-{id} - {summary}"`. `project-config --help` says how the templates work. @@ -148,6 +150,13 @@ cd ~/code/acme-billing && ~/.claude/bin/project-config The last line prints each setting's path once the config is complete. +`do-epic` keeps its settings per machine, not per project, in `~/.claude-projects/do-epic/`, two files for every epic: + +- `config.toml` sets `jira-url`, the site, such as `https://acme.atlassian.net`; `env-file`, the file holding `ATLASSIAN_EMAIL` and `ATLASSIAN_API_TOKEN`, a path that starts with `/` or `~/`, outside the clone folder and every folder of `~/.claude-projects`; and `repoless-model`, the model that works an item with no repo. It may also set `clone-folder`, where a missing repo is cloned, `~/code` if left out; `language`, the language of Jira text, English if left out; and `jira-finish-transition` with `jira-finish-status`, which name the workflow transition that moves a finished chunk's issue, and closes a finished epic, and the status it leads to, such as `"Mark as done"` and `"Done"`. Set both or neither: without them do-epic moves no issue to done and closes no epic, and a chunk that finishes stops with that reason. +- `repo-rule.md` is prose that says how to tell an item's repo, such as "a label that equals a repo's name names that repo". It may not be blank. do-epic doesn't read it yet: until it picks each item's repo from the rule, an issue runs once a label `repo:` records its repo, `do-epic-claude` marks work of Claude's that changes no repo, and `do-epic-human` a task of yours. + +Each repo an epic changes has a project config of its own, as in the rest of this section. `do-epic` fails and names each setting or file that is missing or invalid. + A project runs on the routing ladder this setup ships, `bin/model_routing/routing-ladder.json`: the rungs, least competent first, each a model at an effort, that loop-claude routes workers, subagents and reviews to. To run one project on other rungs, as when an outage or a client's account leaves it fewer models, copy that file to `~/.claude-projects//routing-ladder.json` and cut or change its rungs. Keep one model for each family, such as opus: Claude Code launches a subagent by its family's name. The project's file replaces the shipped ladder whole. loop-claude reads it as each cycle starts, and `subagent-rung` and rule-hook each time they run. The loop never commits it, so commit it in the folder's repo yourself only where it should last. Set the project's models to ones its ladder holds, for example with `LOOP_CLAUDE_WORKER_MODEL` and `LOOP_CLAUDE_JUDGE_MODEL`. A model off the ladder is not routed: every worker runs on that model, at the effort Claude Code's `settings.json` gives. Each cycle of a project on a ladder of its own says so on stderr, naming the file, and says where the worker's model is off it. `loop-claude --help` shows the file's shape. ## Update @@ -181,7 +190,8 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a - `command-patterns [] [--calls ]` prints a table of the shell command patterns a release's archived chunks ran, such as `tea pr create`, for an audit of the release, and with `--calls ` saves each call as a JSON line; `command-patterns --help` states the columns and the line's keys. - `compress-archives` gzips each transcript in loop-claude's archive and each file in the review folders 30 days after it last changed, keeping all of them; Install's crontab line runs it daily. - `deploy-release [--cut] []` moves `~/.claude`, or the checkout named, to the newest release, as Update says; with `--cut`, run in the development clone, it first cuts the next release, as Release says. -- `judge-replay ` replays loop-claude's judge on archived verdicts of the projects run on the model, and prints what each rung a review climbs to misses, and the lowest rung below them that, like each candidate above it, returns them; `judge-replay --help` states the replay. +- `do-epic ` runs the chunks of one Jira epic whose items span repos, or have none, each in its own window of the tmux session `epic-`, on the same cycle, judge and interrupt as loop-claude. The epic's description is its record: goals, audience, scoring and decisions. An open issue runs once a label names its repo, in Jira's backlog order, as many at once as a line such as `**Max parallel chunks:** 2` in the epic's description allows, 1 without it, and as each repo's `parallel-chunks` allows. A chunk whose pull request waits on a gate parks. An open task of yours that blocks another issue gets a window that DMs you once and reads the epic every minute; once you close the task, the issues it blocked may start. do-epic closes the epic only once you confirm every goal. Install โ†’ "Set up a project" says how to set it up; `do-epic --help` states the flags and the contract. +- `judge-replay [--offline]` replays loop-claude's judge on archived verdicts of the projects run on the model. It prints what each rung a review climbs to misses, and the lowest rung below those that returns every verdict, as each candidate above it does. With `--offline` it doesn't call a judge: it counts what a "?" rule, a decision table and a classifier trained on the archive decide of each verdict's kind from the worker's final message, and how many interrupts each misses. `judge-replay --help` states both replays. - `larv-review ` holds larv-review's mechanical rules: the chunk's review folder, the paired ruling's base and vote count, each pass's round count and the opening of its round commit's title, the scoring run's trigger and setup; the comprehension history's format, its latest places and its count of how often seats agree; and each reviewer's prompt header. `larv-review --help` states them. `settings.json` โ†’ `hooks` runs `larv-review reviewer-context` as each subagent starts. A reviewer under `agents/` whose frontmatter has `omitClaudeMd: true` starts without `CLAUDE.md`, and the command hands it that file's "Shared goal", "Prose" and "Vocabulary" sections. - `loop-claude ` runs a project's next chunks, each in its own tmux window, cycle after cycle; `loop-claude --help` states its contract. - `markdown-section ` prints the section of a markdown file under the first heading that holds the fragment, to the next heading of its level or a higher one, instead of a `grep -n` and a guessed `sed -n` range; `markdown-section --help` says what it does where several headings match. @@ -192,8 +202,8 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a - `project-record open ` prints the path of a file holding one of a project's records, for an agent to read and edit, and `project-record save ` stores the edit; for a record kept as a file, that file is the record. `goals-and-audience` opens the goals file the same way. - `rank_todo.py [--check] [todo.md]` scores and ranks a project's `todo.md`, by default the one its config names; `rank_todo.py --help` states the format. - `reading-cost [--reset] [record]` appends a figure for each chunk `loop-claude` finished to a project's `docs/reading-cost.md`, and says whether the figures rise; `reading-cost --help` states the measure. -- `review-replay ` replays a review pass's archived rounds from the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, qualifies: it finds no less of the ๐Ÿ”ด and ๐ŸŸก findings than two replays on the project's rung find, within a margin, and misses no ๐Ÿ”ด that both found. `review-replay --help` states the replay. With `--omit-claude-md` it tests instead whether the pass's reviewer may start without `CLAUDE.md`: it replays each round three times as archived and once without the instruction files, and prints whether the reviewer passes: whether the last replay missed no more findings than the most an unchanged replay missed, with each count and each finding it missed. Each replayed reviewer skips every permission prompt, stability-reviewer has Bash, and an archived prompt may name paths on the host. -- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` โ†’ `hooks` runs it before each Agent, Bash, Edit, Grep, MultiEdit, Read, Skill and Write call, and `rule-hook --help` states the rules and the override. +- `review-replay ` replays a review pass's archived rounds from the projects run on the model, on the code each reviewed, and prints the lowest rung below the project's that, like each rung above it, qualifies: it finds no less of the ๐Ÿ”ด and ๐ŸŸก findings than two replays on the project's rung find, within a margin, and misses no ๐Ÿ”ด that both found. `review-replay --help` states the replay. With `--omit-claude-md`, `--bash-ceiling` or `--read-limit` it tests instead whether the pass's reviewer may start without `CLAUDE.md`, or under the Bash output ceiling loop-claude hands the worker, or a Read limit it doesn't hand yet: it replays each round three times as archived and once with the change, and prints whether the reviewer passes: whether the last replay missed no more findings than the most an unchanged replay missed, with each count and each finding it missed. Each replayed reviewer skips every permission prompt, stability-reviewer has Bash, and an archived prompt may name paths on the host. +- `rule-hook [overrides]` blocks a tool call that breaks a rule a program can check, unless the call overrides the rule; `settings.json` โ†’ `hooks` runs it before each Agent, Bash, Edit, Grep, Monitor, MultiEdit, Read, Skill and Write call, and `rule-hook --help` states the rules and the override. - `run-heavy ` runs a heavy command once it holds one of the machine's heavy slots, one per 4 CPUs, which every session shares; `run-heavy --help` states the slots, and the memory cap it gives a container that sets no limit. - `run-tests [...]` runs this repo's test suite, as Test says. - `score-outcomes [L E I S]` shows what each finished item's chunk measured beside the scores the item was guessed at, to score a new item by. Per chunk it also shows code health's two counter-signals, misread findings and later fixes, and how many of the human's answers reversed a worker's choice; `score-outcomes --help` states the report. @@ -206,23 +216,25 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a ### Packages under `bin/` -`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests. Beside the packages, `bin/`'s root holds only the launchers, the tools of one file with their tests, `test_env.py`, `failure.py` and `setup.cfg`. `failure.py`'s `Failure`, which modules across `bin/` return for a failed step, belongs to no single package. `atlassian/`, `branch_chunks/`, `claude_code/`, `command_lines/`, `forge/`, `git_cli/`, `loop_effects/`, `loop_state/`, `machine_load/`, `model_routing/`, `notifier/`, `plan/`, `plan_store/`, `record_store/`, `release_plan/`, `review_rules/`, `rung_replay/`, `text_files/` and `transcript_archive/` serve more than one tool. No package imports one that imports it back, which `repo_checks/test_package_cycles.py` checks. +`bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests. Beside the packages, `bin/`'s root holds only the launchers, the tools of one file with their tests, `test_env.py`, `failure.py` and `setup.cfg`. `failure.py`'s `Failure`, which modules across `bin/` return for a failed step, belongs to no single package. `agent_cycle/`, `atlassian/`, `branch_chunks/`, `claude_code/`, `command_lines/`, `forge/`, `git_cli/`, `loop_effects/`, `loop_state/`, `machine_load/`, `model_routing/`, `notifier/`, `plan/`, `plan_store/`, `record_store/`, `release_plan/`, `review_rules/`, `rung_replay/`, `text_files/` and `transcript_archive/` serve more than one tool. No package imports one that imports it back, which `repo_checks/test_package_cycles.py` checks. - `adf_codec/` converts markdown to ADF, the rich text of Jira Cloud and Confluence, and back, in a throwaway container running a pinned `@larvit/adf-codec`. It builds the container's image inside a heavy slot. +- `agent_cycle/` runs one cycle (`run.py`): a headless claude worker, then the judge of its transcript. It also holds the interrupt, the tmux window and the routing of the worker's rung. The tool that drives the cycles supplies the prompts and file names of its own workflow (`worker_prompts.py`), and reads which items a worker took in. `loop_claude` drives it, and `do_epic` will; `agent_cycle` doesn't import either. - `atlassian/` holds what the Jira and Confluence backends share (`client.py`): one Atlassian account's credentials, the request both backends send, and each service's error messages. - `branch_chunks/` cuts a default branch into chunks and finds, for each, the items it shipped and the later chunks that fixed its code, for `score_outcomes` and `routing_figures`. It imports `reading_cost`'s record path. - `command_lines/` reads a shell command: its simple commands, and a docker call's subcommand and memory limits, for `command_patterns`, `reading_cost`, `routing_figures`, `rule_hook` and `run_heavy`. - `deploy_release` imports `release_plan`'s `release_standing`. -- `forge/` tells a GitHub origin from a Gitea one and runs that forge's CLI (`commands.py`), and finds a pull request and waits for its checks (`pulls.py`), for `chunk_preflight`, `chunk_pr`, `deploy_release` and `larv_review`. +- `do_epic/` holds `do-epic`, the tool that drives one Jira epic. `main.py` starts the epic's chunks and runs one chunk's cycles in its window, through `agent_cycle`. `scheduler.py` says which issues run now: as many as the epic's cap and each repo's `parallel-chunks` allow, and none in a repo where loop-claude runs chunks. `gates.py` reads once whether a chunk's pull request waits on its checks, an approval or a new review, and `parked.py` keeps each chunk that does in a state file, so it takes no running slot while it waits and goes on waiting after a reboot. `chunk_prompt.py` words what a chunk's worker reads. Through `findings.py` that worker files a finding as an issue of the epic, with `--file`, and appends a decision to its description, with `--record-decision`. `finish.py` ends an epic that holds no open issue: its finishing chunk files a release review of each repo the epic changed, has the human confirm each goal, and with `--close-epic` posts the summary on the epic and closes it. `config.py` reads its machine-wide settings, importing `project_config`'s settings and `model_routing`'s ladder file, and `epic_record.py` reads and writes an epic's description by section, language line and cap line, through `record_store`'s `EpicDescription`. `repos.py` finds a repo's checkout and project config by its name, and clones a missing checkout from the host and owner of the epic's other repos, through `forge`'s `url_beside`. `repo_pick.py` asks claude, by `repo-rule.md`, where an epic item is worked, and records the answer as a label on the issue. `split.py` has claude predict from the code which repos an item changes, and files a sub-task per repo under it where there are several. `ranking.py` orders an epic's items by score over one chain of goals, the epic's, then code health, then the item's repo's, through `plan`'s ranking, and re-ranks the tracker's backlog to that order through `backend/`. `backend/` reads, files and ranks the epic's issues, adds the labels `repo_pick.py` and `split.py` record, posts comments, moves a finished chunk's issue to done, and closes the epic. +- `forge/` tells a GitHub origin from a Gitea one, names a repo beside an origin's and runs that forge's CLI (`commands.py`), and finds a pull request and reads its checks, once or until they settle (`pulls.py`), for `chunk_preflight`, `chunk_pr`, `deploy_release`, `do_epic` and `larv_review`. - `git_cli/` runs git for every tool (`git_repo.py`), and reads the git calls in a shell command (`git_command.py`). -- `judge_replay` imports `loop_claude`'s judge, verdicts, premises and records. +- `judge_replay` imports `agent_cycle`'s judge and verdicts, and `loop_claude`'s premises, records and `ANSWER_DESTINATIONS`. - `loop_claude` runs `chunk-preflight` as a process of its own. - `loop_effects/` holds `Effects`: the processes, terminal, clock and git a tool reaches, injected so its tests can script them. loop-claude, both replay tools and `reviewer-merge-trials`' driver use it. - `loop_state/` holds where loop-claude keeps its files (`paths.py`), which `rung_replay` reads too. - `machine_load/` takes the machine's heavy slots, its shared lock files, for `run_heavy` and `adf_codec`, and reads the memory share that loop-claude waits for. It also gives `mutate_changes` a slot's CPU count. -- `model_routing/` holds the shipped routing ladder (`routing-ladder.json`) and its steps (`ladder.py`), and reads a ladder from that file or a project's own (`ladder_file.py`), for loop-claude, `rule_hook`, both replay tools and `subagent-rung`. It finds a project's config folder through `project_config`. It also holds the list prices, which `routing_figures` and `transcript_archive` read. +- `model_routing/` holds the shipped routing ladder (`routing-ladder.json`) and its steps (`ladder.py`), and reads a ladder from that file or a project's own (`ladder_file.py`), for loop-claude, `rule_hook`, `do_epic`, both replay tools and `subagent-rung`. It finds a project's config folder through `project_config`. It also holds the list prices, which `routing_figures` and `transcript_archive` read. - `notifier/` sends the human a DM through the notifier a project's config names, for `notify` and for loop-claude through `loop_effects`. Its backends are `mattermost.py` and `slack.py`. -- `plan/` is the plan's core, and imports neither `plan_store` nor `release_plan`. It holds an item and the items table's format (`todo_table.py`), releases (`release.py`), the ranking (`ranking.py`, with its score formula in `formula.py` and its blocker order in `blockers.py`), the paths an item's text names (`named_paths.py`) and `citations.py`, which holds `todo.md` and `docs/decisions.md` to each other. `goals_file.py` reads the goals file's sections, goals and goal weights, for the ranking and for `loop_claude`'s premises. +- `plan/` is the plan's core, and imports neither `plan_store` nor `release_plan`. It holds an item and the items table's format (`todo_table.py`), releases (`release.py`), the ranking (`ranking.py`, with its score formula in `formula.py` and its blocker order in `blockers.py`), the paths an item's text names (`named_paths.py`), an item's `Touches:` prediction and where two overlap (`touches.py`), and `citations.py`, which holds `todo.md` and `docs/decisions.md` to each other. `goals_file.py` reads the goals file's sections, goals and goal weights, for the ranking and for `loop_claude`'s premises. - `plan_store/` reads and writes a project's plan through the store the project's config names (`stores.py`). Every backend meets `plans.py`'s `PlanStore`: a markdown file (`todo_md.py`, which also ranks and checks the file in place, with its text edits in `markdown_edits.py`) or a Jira epic (`jira_epic.py`). `scoring.py` reads the goal weights that both backends rank by. `contract.py` holds the tests every backend that writes must pass; `jira_epic.py` doesn't yet, and its own tests stand in for Jira. `fakes.py` holds a plan in memory. - `rank_todo.py`, `settings_filter.py` and `statusline.py` are tools of one file each, tested beside them. - `record_store/` reads and writes a project's records, each through the store the project's config names (`stores.py`): a markdown file, a Confluence Cloud page (`confluence_page.py`), or, for the goals, a Jira epic's description (`jira_description.py`). `contract.py` holds the tests every backend must pass, and `fakes.py` a store in memory that the tests of other tools use. @@ -231,16 +243,16 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a - `rest_client/` sends a JSON request to a REST API and reads its credentials from an env file, for `notifier`, `atlassian` and `record_store`. - `review_rules/` holds the larv-review rules that several tools apply: its passes, its review folder, its round commit titles and the comprehension history's format. - `routing_figures/` measures loop-claude's archive, and each of its projects' default branches, for loop-claude's routing calls. -- `rung_replay/` holds what `judge_replay` and `review_replay` share: the parallel replay runner, the argument reader, the candidate rungs and the exits, and `judge_replay`'s rung search. It reads `loop_claude`'s retry count and models file, and finds that file through `loop_state`'s paths. Its `archive.py` keeps each replay's runs and its report in a `replays/` folder of loop-claude's archive. +- `rung_replay/` holds what `judge_replay` and `review_replay` share: the parallel replay runner, the argument reader, the candidate rungs and the exits, and `judge_replay`'s rung search. It reads `agent_cycle`'s retry count and `loop_claude`'s models file, and finds that file through `loop_state`'s paths. Its `archive.py` keeps each replay's runs and its report in a `replays/` folder of loop-claude's archive. - `test_env.py` serves the tests. -- `text_files/` reads a Markdown file's `##` sections, quotes a passage and joins names into a list, stamps times in UTC, replaces a file in one step, private where it is new, and gzips a file beside its plain name, by which it reads it afterwards. +- `text_files/` reads a Markdown file's `##` sections, quotes a passage and joins names into a list, reads the text piped into a command, stamps times in UTC, replaces a file in one step, private where it is new, and gzips a file beside its plain name, by which it reads it afterwards. - `transcript_archive/` stores loop-claude's transcripts. Its `project_key.py` names each project's archive folder, loop-claude session and models-file entry. ## Test -Test on the host with `bin/run-tests`, which runs pytest through `uvx`, pinned in `bin/run_tests/main.py`. It prints each failing test with its error and pytest's summary line, and saves the whole output to a file whose path it prints. It passes on any test paths or pytest options it is given, such as `bin/run-tests bin/chunk_pr -x`. It needs `uv` on `PATH`, which fetches pytest, pytest-xdist and mypy. pytest-xdist runs the tests in one worker per CPU of a heavy slot (`docs/decisions.md` โ†’ `bin/: the tools support only the latest stable Python, โ€ฆ`). `python3 -m unittest discover -s bin -p 'test_*.py'` runs the same tests one at a time. Test the bench's scripts with `python3 -m unittest discover -s workflow-bench -p 'test_*.py'`. +Test on the host with `bin/run-tests`, which runs pytest through `uvx`, pinned in `bin/run_tests/main.py`. It prints each failing test with its error and pytest's summary line, and saves the whole output to a file whose path it prints. It passes on any test paths or pytest options it is given, such as `bin/run-tests bin/chunk_pr -x`. It needs `uv` on `PATH`, which fetches pytest, pytest-xdist and mypy, a Docker daemon, Linux, and a user other than root; a test that lacks one fails, naming it. pytest-xdist runs the tests in one worker per CPU of a heavy slot (`docs/decisions.md` โ†’ `bin/: the tools support only the latest stable Python, โ€ฆ`). `python3 -m unittest discover -s bin -p 'test_*.py'` runs the same tests one at a time, with the same needs. Test the bench's scripts with `python3 -m unittest discover -s workflow-bench -p 'test_*.py'`. -CI runs `ci.sh` on every pull request, and `main` doesn't merge anything until it passes. The script runs the same tests on the committed HEAD in a pinned Python container; run it under `run-heavy` to reproduce a CI failure, with Docker on `PATH`. +CI runs `ci.sh` on every pull request, and `main` doesn't merge anything until it passes. The script runs the same tests on the committed HEAD in a pinned Python container. Run it under `run-heavy` to reproduce a CI failure, with Docker on `PATH` and its socket at `/var/run/docker.sock`, which the container shares. To mutation-test the lines a branch changes, run `~/.claude/bin/run-heavy ~/.claude/bin/mutate-changes --prefix 'uvx --python "$(command -v python3)" --with pytest==9.2.0 --from "mutmut>=3,<4"'` in its checkout. diff --git a/agents/comprehension-seat.md b/agents/comprehension-seat.md new file mode 100644 index 00000000..6aa3ce4d --- /dev/null +++ b/agents/comprehension-seat.md @@ -0,0 +1,8 @@ +--- +name: comprehension-seat +description: "Use this agent for a seat of larv-review's comprehension check, with the prompt `larv-review prepare-scoring` writes or the paired ruling's brief in skills/comprehension-panel/seat-prompts.md." +tools: Read, Grep, Glob, Bash +omitClaudeMd: true +--- + +You are a simulated reader of a project. Your prompt gives your background, the checkout you read, the files you may not open and the report to write. Read as that reader would, knowing only what the project's files tell you, and follow the prompt exactly. diff --git a/agents/product-owner-reviewer.md b/agents/product-owner-reviewer.md index 48051dc1..17e9ca32 100644 --- a/agents/product-owner-reviewer.md +++ b/agents/product-owner-reviewer.md @@ -47,7 +47,7 @@ Before anything else, **read the technical principles** in the folder your promp ## Your Output Format -Report only a defect a persona hits; a requirement that the change breaks, of the plan in the file your prompt's `Plan:` line names, of the goals in the file your prompt's `Goals and audience:` line names, or of code health (`~/.claude/CLAUDE.md` โ†’ "Shared goal", unless that file voids it); or a breach of a technical principle. Friction that is none of these is not a finding. +Report only a defect a persona hits; a requirement that the change breaks, of the plan in the file your prompt's `Plan:` line names, of the goals in the file your prompt's `Goals and audience:` line names, or of code health (`~/.claude/CLAUDE.md` โ†’ "Shared goal", unless that file voids it); or a breach of a technical principle, or of a coding convention, which counts as one. Friction that is none of these is not a finding. Verdict, severities and rounds: ~/.claude/skills/larv-review/review-findings.md. After the verdict: diff --git a/agents/stability-reviewer.md b/agents/stability-reviewer.md index fc1ee23e..6148799b 100644 --- a/agents/stability-reviewer.md +++ b/agents/stability-reviewer.md @@ -89,6 +89,7 @@ Catch behavior that is unclear or under-specified โ€” "building in ambiguousness - Are consumers idempotent under at-least-once delivery (webhooks, queue retries, double-submit)? Where a duplicate would double-charge or double-send, is there an idempotency key or dedup? - Do DB constraints (NOT NULL, unique, FK) mirror what the code assumes, so bad data cannot arrive via another path? - New column on an existing table: what do existing rows contain โ€” is null/default handled or backfilled? +- Where the change stores a value differently, by escaping, converting or growing it: does the longest value it can now write still fit the column, counted as stored? Does every search, filter and comparison on the column still find what users type? Escaped text takes more room than the input's limit allows for, and a `LIKE` on an escaped column no longer matches an apostrophe. ### 8. Resources & Performance Under Production Data Works-on-fixtures, dies-on-volume is a stability bug. @@ -125,12 +126,14 @@ Every PR gets at least this sweep. - Was a test deleted while the behaviour it pinned remains? Flag it; restore the test. - Were tests and implementation changed together in a way that could mask a regression โ€” e.g. an assertion weakened to make a test pass? Flag it; these should be separate changes. - Does each test earn its keep, as the technical principles' test bullet says? Look at where its expected values come from, what it asserts, what it mocks, what wording it pins, whether another test already pins the behaviour, and whether it sleeps or reads the wall clock. +- Does a test decide at run time whether to check anything, by a conditional skip, early return or catch around its assertions, as the technical principles' test bullet forbids? Flag it. +- Does a test that runs on a shared environment hold the rules for such a test in the technical principles' "Compose, do not entangle" โ†’ "Tests and tooling"? Does the test that covers a seam to a service the team runs mock that service, which the test bullet's "Never mock a service your own team runs" forbids? - Are there untestable side effects that should be refactored? ## Output Format -Report only a defect; a requirement that the change breaks, of the plan in the file your prompt's `Plan:` line names, of the goals in the file your prompt's `Goals and audience:` line names, or of code health (`~/.claude/CLAUDE.md` โ†’ "Shared goal", unless that file voids it); a breach of a technical principle; and the places the prompt hands you. A finding that is none of these is noise, however true: leave it out. +Report only a defect; a requirement that the change breaks, of the plan in the file your prompt's `Plan:` line names, of the goals in the file your prompt's `Goals and audience:` line names, or of code health (`~/.claude/CLAUDE.md` โ†’ "Shared goal", unless that file voids it); a breach of a technical principle, or of a coding convention, which counts as one; and the places the prompt hands you. A finding that is none of these is noise, however true: leave it out. Verdict, severities and rounds: ~/.claude/skills/larv-review/review-findings.md. After the verdict, a short **Coverage & Verification** block: what you ran (tests, static analysis, mutation โ€” and their results), what you could not verify and why, and any files that got a lighter pass โ€” never let silence imply coverage. Then the findings, most severe first, each with: - **Location**: File and line/function diff --git a/bin/adf_codec/test_adf_codec.py b/bin/adf_codec/test_adf_codec.py index b264cc24..c1f79f42 100644 --- a/bin/adf_codec/test_adf_codec.py +++ b/bin/adf_codec/test_adf_codec.py @@ -1,7 +1,6 @@ import contextlib import json import os -import shutil import subprocess import pytest @@ -296,10 +295,10 @@ def test_container_run_passes_an_image_failure_through(tmp_path): assert convert.container_run("to-adf", "text", scripted(down), shim_dir(tmp_path)) == down -@pytest.mark.skipif(shutil.which("docker") is None or subprocess.run(["docker", "info"], capture_output=True, timeout=20).returncode != 0, reason="needs a docker daemon") def test_round_trip_through_the_real_container(): doc = convert.markdown_to_adf("# Title\n\nSome **bold** text.\n") - assert isinstance(doc, dict) and doc["type"] == "doc" and doc["content"][0]["type"] == "heading" + assert isinstance(doc, dict), doc + assert doc["type"] == "doc" and doc["content"][0]["type"] == "heading" markdown = convert.adf_to_markdown(doc) assert isinstance(markdown, str) and markdown.startswith("# Title") and "**bold**" in markdown refused = convert.adf_to_markdown({"type": "nope"}) @@ -324,10 +323,9 @@ JIRA_DESCRIPTION = { } -@pytest.mark.skipif(shutil.which("docker") is None or subprocess.run(["docker", "info"], capture_output=True, timeout=20).returncode != 0, reason="needs a docker daemon") def test_the_lossless_pair_gives_back_a_jira_description_identical_and_the_portable_pair_does_not(): markdown = convert.adf_to_markdown_lossless(JIRA_DESCRIPTION) - assert isinstance(markdown, str) + assert isinstance(markdown, str), markdown assert convert.markdown_to_adf_lossless(markdown) == JIRA_DESCRIPTION assert convert.adfs_to_markdown_lossless([JIRA_DESCRIPTION, JIRA_DESCRIPTION]) == [markdown, markdown] portable = convert.markdown_to_adf(convert.adf_to_markdown(JIRA_DESCRIPTION)) diff --git a/bin/agent_cycle/__init__.py b/bin/agent_cycle/__init__.py new file mode 100644 index 00000000..87650737 --- /dev/null +++ b/bin/agent_cycle/__init__.py @@ -0,0 +1,4 @@ +"""One cycle of a headless claude worker and the judge of its transcript, for every entry point that drives such cycles: loop-claude today, do-epic next (docs/decisions.md โ†’ `do-epic: drives one Jira epic โ€ฆ`). + +run.py runs a cycle through the states in states.py: worker.py runs the worker, judge.py rules on its transcript, interrupt.py puts a question to the human in the tmux window tmux.py names, and routing.py picks the rung the worker runs on next. An entry point hands the cycle what is its own: its prompts and file names (worker_prompts.py) and how it reads the plan items a worker took in (run.py's Cycle.taken_in). No module here imports an entry point's package: loop_claude imports this one, so bin/repo_checks/test_package_cycles.py fails on such an import. A reader finds a module by its name, so the modules stay at one level. docs/decisions.md โ†’ `principles: about seven things per level, โ€ฆ` allows that. +""" diff --git a/bin/agent_cycle/direct_message.py b/bin/agent_cycle/direct_message.py new file mode 100644 index 00000000..32d3da83 --- /dev/null +++ b/bin/agent_cycle/direct_message.py @@ -0,0 +1,8 @@ +from loop_effects.effects import Effects, say + + +def tell(text: str, effects: Effects, directory: str | None = None) -> bool: + """DMs text to the human through the notifier of the project whose checkout holds directory, else the current directory, and returns whether it went out. Where the DM fails, it says why on stderr and goes on: the DM only copies what the terminal shows, so its failure never stops the loop.""" + if failed := effects.dm(text, directory): + say(f"could not DM you: {failed.reason}", effects) + return not failed diff --git a/bin/loop_claude/escalation_mark.py b/bin/agent_cycle/escalation_mark.py similarity index 56% rename from bin/loop_claude/escalation_mark.py rename to bin/agent_cycle/escalation_mark.py index 0df8597a..2e82e48b 100644 --- a/bin/loop_claude/escalation_mark.py +++ b/bin/agent_cycle/escalation_mark.py @@ -1,10 +1,10 @@ """The escalation mark makes one escalation trigger step the worker up one rung, never two (docs/decisions.md โ†’ `routing: a worker whose run ends asking for a higher rung`). -A worker whose run ends asking loop-claude for a higher rung cites the trigger that fired, and the judge quotes the passage where it fired as the verdict's escalation. The mark, Route.escalated_at, is how many transcript entries the session held when an escalation last stepped the worker up, 0 before any. The session's transcript only grows, so the mark holds across cycles, and the archive's Worker: lines keep it for a rerun. +A worker whose run ends asking its driver, such as loop-claude, for a higher rung cites the trigger that fired, and the judge quotes the passage where it fired as the verdict's escalation. The mark, Route.escalated_at, is how many transcript entries the session held when an escalation last stepped the worker up, 0 before any. The session's transcript only grows, so the mark holds across cycles, and the archive's Worker: lines keep it for a rerun. - Stamp. stamped() writes on the judge's verdict how many entries the judge read. - Step. routing.next_route() steps the worker up on an escalation and moves the mark to moved_to(verdict). -- Show. transcript() puts STEP_LINE after the mark, telling the judge that the triggers before that line are spent. QUOTE_AFTER_STEP, in the judge's schema, tells the judge to quote only the worker's words after that line. +- Show. transcript() puts step_line after the mark, telling the judge that the triggers before that line are spent. quote_after_step, in the judge's schema, tells the judge to quote only the worker's words after that line. - Check. checked() keeps an escalation only where it quotes both the transcript the judge was shown and the worker's own lines after the mark, which after_mark() renders, so a passage that bought one step cannot buy another. """ import dataclasses @@ -12,14 +12,25 @@ import typing from claude_code.transcript import assistant_lines, render, Typed from claude_code.entries import AssistantTurn, Entry -from loop_claude.verdicts import Escalating, Verdict -from loop_claude.quotes import verbatim +from agent_cycle.verdicts import Escalating, Verdict +from agent_cycle.quotes import verbatim # Only the judge reads the step line, so it is not a [loop] line. STEP_TAG = "[step]" -STEP_LINE = f"{STEP_TAG} loop-claude stepped the worker up a rung here, so the escalation triggers that fired before this line are spent." -# Reply.escalation's description in cycle/judge.py carries this sentence, so the judge reads STEP_LINE as the mark. -QUOTE_AFTER_STEP = f"Where a {STEP_TAG} line says loop-claude stepped the worker up a rung, quote only the worker's own words after it, and leave this empty where the trigger fired only before it." + + +def asks_higher_rung(driver: str) -> str: + """How a worker's run ends where it wants the rest of its chunk on a higher rung. The worker's prompt tells it to, and the judge's rules name the same words, so both take them from here. driver is the entry point that drives the cycle, such as `do-epic`.""" + return f"asking {driver} for a higher rung" + + +def step_line(driver: str) -> str: + return f"{STEP_TAG} {driver} stepped the worker up a rung here, so the escalation triggers that fired before this line are spent." + + +def quote_after_step(driver: str) -> str: + """Reply.escalation's description in agent_cycle/judge.py carries this sentence, so the judge reads step_line as the mark.""" + return f"Where a {STEP_TAG} line says {driver} stepped the worker up a rung, quote only the worker's own words after it, and leave this empty where the trigger fired only before it." def stamped(verdict: Verdict, entries_read: int) -> Verdict: @@ -32,13 +43,13 @@ def moved_to(verdict: Escalating) -> int | None: return verdict.entries_read if verdict.escalation else None -def transcript(entries: list[Entry], at: int, typed_answer: typing.Callable[[str], Typed | None]) -> str: - """The transcript as the judge sees it. Where the mark, at, is above 0, STEP_LINE comes after that many entries.""" - return "\n\n".join(part for part in (render(entries[:at], typed_answer), STEP_LINE if at else "", render(entries[at:], typed_answer)) if part) +def transcript(entries: list[Entry], at: int, typed_answer: typing.Callable[[str], Typed | None], driver: str) -> str: + """The transcript as the judge sees it. Where the mark, at, is above 0, step_line comes after that many entries.""" + return "\n\n".join(part for part in (render(entries[:at], typed_answer), step_line(driver) if at else "", render(entries[at:], typed_answer)) if part) def after_mark(entries: list[Entry], at: int) -> str: - """All the text and tool calls of the worker's assistant turns after the mark, even where the judge is shown only the transcript's tail. loop-claude's prompts and the human's lines are left out, since they repeat earlier passages, such as the judge's evidence or the worker's question.""" + """All the text and tool calls of the worker's assistant turns after the mark, even where the judge is shown only the transcript's tail. The driver's prompts and the human's lines are left out, since they repeat earlier passages, such as the judge's evidence or the worker's question.""" return "\n\n".join(line for entry in entries[at:] if isinstance(entry, AssistantTurn) for line in assistant_lines(entry)) diff --git a/bin/loop_claude/hand_over.py b/bin/agent_cycle/hand_over.py similarity index 100% rename from bin/loop_claude/hand_over.py rename to bin/agent_cycle/hand_over.py diff --git a/bin/loop_claude/cycle/interrupt.py b/bin/agent_cycle/interrupt.py similarity index 84% rename from bin/loop_claude/cycle/interrupt.py rename to bin/agent_cycle/interrupt.py index 4a3de5fc..e0fcfa7d 100644 --- a/bin/loop_claude/cycle/interrupt.py +++ b/bin/agent_cycle/interrupt.py @@ -5,20 +5,20 @@ import typing import uuid from claude_code.entries import AssistantTurn, Entry -from loop_claude.hand_over import claude_args, ClaudeCommand, hand_over +from agent_cycle.hand_over import claude_args, ClaudeCommand, hand_over from loop_effects.effects import Effects, say from failure import Failure from claude_code.transcript import human_turns, read_transcript from claude_code.reply import Limited -from loop_claude.stop import needs_human, needs_terminal, Stop -from loop_claude.cycle.worker import run_claude +from agent_cycle.stop import needs_human, needs_terminal, Stop +from agent_cycle.worker import run_claude from model_routing.ladder import Rung -from loop_claude.direct_message import tell -from loop_claude import tmux +from agent_cycle.direct_message import tell +from agent_cycle import tmux from claude_code.tokens import tokens_of ANSWER_QUESTION = "answer the question above" -# TALK, what the human types at loop-claude's terminal to open claude for the question, and ASK_ANSWER were picked on 2026-10-17 by a bench of three readers seated as the README's audience, as ~/.claude/CLAUDE.md โ†’ "Working loop" โ†’ "Bench a syntax with the people who write it." has it. +# TALK, what the human types at the entry point's terminal to open claude for the question, and ASK_ANSWER were picked on 2026-10-17 by a bench of three readers seated as the README's audience, as ~/.claude/CLAUDE.md โ†’ "Working loop" โ†’ "Bench a syntax with the people who write it." has it. TALK = "/talk" ASK_ANSWER = f"Answer ({TALK} opens claude, Enter stops the loop): " # While the human hasn't answered, the worker's session is compacted CACHE_MARGIN seconds before its hour-long prompt cache expires: compacting reads the context from the cache, while resuming after it expired writes the whole context again. Below COMPACT_FLOOR tokens of context, that write costs less than compacting. A five-minute cache expires within the margin, so a session that last wrote one isn't compacted. lilleman chose this on 2026-11-11. @@ -29,7 +29,7 @@ COMPACT_FLOOR = 60_000 @dataclasses.dataclass(frozen=True) class WorkerSession: - """The worker's claude session, which /talk opens on loop-claude's terminal.""" + """The worker's claude session, which /talk opens on the entry point's terminal.""" root: str id: uuid.UUID rung: Rung @@ -48,8 +48,8 @@ class Talked: """The human typed in the worker's claude session.""" -def interrupt(question: str, answer_here: bool, session: WorkerSession, window: tmux.Window, effects: Effects) -> str | Talked | Stop: - """Prints question for the human, in window, and returns their answer. While it waits, the window's name ends in tmux.WAITS, and the bell it rings shows in tmux's status bar. The answer: where answer_here, the first line typed at loop-claude's terminal, stripped, or Talked once they typed in the worker's claude session, which TALK opens; else Talked once they typed in that session, which opens at once on the terminal. A Stop where loop-claude is off a terminal, or they type no answer, or the input ends.""" +def interrupt(question: str, answer_here: bool, session: WorkerSession, window: tmux.Window, driver: str, effects: Effects) -> str | Talked | Stop: + """Prints question for the human, in window, and returns their answer. While it waits, the window's name ends in tmux.WAITS, and the bell it rings shows in tmux's status bar. The answer: where answer_here, the first line typed at the terminal of driver, the entry point, stripped, or Talked once they typed in the worker's claude session, which TALK opens; else Talked once they typed in that session, which opens at once on the terminal. A Stop where interrupt runs off a terminal, or they type no answer, or the input ends.""" # The bell ends the question's line, so it rings once the question can be read. say(f"\n{question}{tmux.BELL}", effects) on_terminal = effects.on_terminal() @@ -58,8 +58,8 @@ def interrupt(question: str, answer_here: bool, session: WorkerSession, window: effects.discard_typed() tmux.mark(window, True, effects.tmux) # After the discard, so an answer typed while the DM sends counts; before the terminal check, so a loop that cannot ask still tells the human what it would have asked. - tell(f"loop-claude in {session.root} asks, in the tmux window {window.target}:\n\n{question}", effects) - answer = needs_terminal(ANSWER_QUESTION, on_terminal) or answered(answer_here, session, effects) + tell(f"{driver} in {session.root} asks, in the tmux window {window.target}:\n\n{question}", effects) + answer = needs_terminal(ANSWER_QUESTION, on_terminal, driver) or answered(answer_here, session, effects) # A loop that stops unanswered leaves the mark on its window. if not isinstance(answer, Stop): tmux.mark(window, False, effects.tmux) @@ -120,7 +120,7 @@ def compact(session: WorkerSession, expires: float, effects: Effects) -> None: def typed_answer(effects: Effects) -> str | Stop: - """The line typed at loop-claude's terminal, stripped. A Stop where they type no answer, or the input ends.""" + """The line typed at the entry point's terminal, stripped. A Stop where they type no answer, or the input ends.""" try: text = effects.input(ASK_ANSWER).strip() except EOFError: diff --git a/bin/loop_claude/cycle/judge.py b/bin/agent_cycle/judge.py similarity index 82% rename from bin/loop_claude/cycle/judge.py rename to bin/agent_cycle/judge.py index 6ca54bf4..3591a649 100644 --- a/bin/loop_claude/cycle/judge.py +++ b/bin/agent_cycle/judge.py @@ -3,19 +3,19 @@ import json import re import typing -from loop_claude.cycle.prompts import ANSWER_DESTINATIONS, ANSWERS_RECORDED, typed_answer +from agent_cycle.worker_prompts import answers_recorded, typed_answer from claude_code.reply import claude_reply, Limited from loop_effects.effects import Effects from claude_code.entries import Entry, structured_output -from loop_claude import escalation_mark +from agent_cycle import escalation_mark from failure import Failure -from loop_claude.verdicts import Finished, Interrupt, Judged, QuotedAnswer, Unfinished, Verdict +from agent_cycle.verdicts import Finished, Interrupt, Judged, QuotedAnswer, Unfinished, Verdict from claude_code.transcript import human_answers from transcript_archive.judge_run import JudgeRun -from loop_claude.quotes import plain, verbatim -from loop_claude.cycle.retry import retried +from agent_cycle.quotes import plain, verbatim +from agent_cycle.retry import retried from model_routing.ladder import Rung -from loop_claude.stop import Stop +from agent_cycle.stop import Stop from claude_code.tokens import tokens_of # At 3 to 4 characters a token, 100,000 to 133,000 tokens, which leaves a 200,000-token context room for the reply; the largest session rendered here was 364,000 characters on 2026-10-03. @@ -36,8 +36,8 @@ INTERRUPT_RULES = ( "interrupt=false otherwise. A choice in the repo that no rule reserves for the human is the worker's to make, " "so its taking its own recommendation on one is no reason to interrupt. Where it asks about such a choice and " "recommends an answer, quote the sentence of that rule telling it to take the option it would recommend, even " - "where a [loop] line quoted that sentence for another question. A run that ends asking loop-claude for a higher " - "rung is no reason to interrupt either; one that asks the human's yes to a higher rung is.\n\n" + "where a [loop] line quoted that sentence for another question. A run that ends {asks_higher_rung} " + "is no reason to interrupt either; one that asks the human's yes to a higher rung is.\n\n" "The human may have replied inside the transcript, on a [human] line: judge only what " "is still open, never a question they already answered there. Where a [human] line from earlier in the " "chunk answered the question the worker asks now, interrupt only if that line's own words leave open what " @@ -56,26 +56,35 @@ QUOTE_RULES = ( FINISHED_RULES = ( "The chunk is the piece of work the worker is on, shipped as one pull request.\n\n" "finished=true when the chunk is complete: merged, or nothing remains that the worker could do on " - f"its own. Even then, it is finished only once {ANSWERS_RECORDED}.\n\n" + "its own. Even then, it is finished only once {answers_recorded}.\n\n" "An answer counts as recorded where the premises or the records above already hold it, the human's rules " "(~/.claude/CLAUDE.md, ~/.claude/rules/, the project's review standards, each headed by its path) included. " - f"It also counts where the transcript shows the worker writing it into {ANSWER_DESTINATIONS}, " + "It also counts where the transcript shows the worker writing it into {answer_destinations}, " "but not into one of the human's rules.\n\n" "finished=false when work remains that a further run can " "continue: an open PR, a review not yet run, CI not yet reported, an answer not yet recorded." ) -JUDGE_RULES = "\n\n".join((INTERRUPT_RULES, QUOTE_RULES, FINISHED_RULES)) + + +def judge_rules(answer_destinations: str, driver: str) -> str: + """The rules that end the judge's prompt. answer_destinations names where the worker of driver, the entry point, may record an answer the human typed.""" + return "\n\n".join((INTERRUPT_RULES.format(asks_higher_rung=escalation_mark.asks_higher_rung(driver)), QUOTE_RULES, FINISHED_RULES.format(answers_recorded=answers_recorded(answer_destinations), answer_destinations=answer_destinations))) + + # Open the task and the transcript in the prompt; the premises and the records come before the task, the rules after the transcript. # judge_replay.archived_prompt splits each archived prompt on them, so rewording one makes every run archived before it replay as failed. TASK_HEADING = "The task the worker was given:\n" -TRANSCRIPT_HEADING = "What the worker did and said, what the human typed on [human] lines and what loop-claude told it on [loop] lines, latest last:\n" + + +def transcript_heading(driver: str) -> str: + return f"What the worker did and said, what the human typed on [human] lines and what {driver} told it on [loop] lines, latest last:\n" @dataclasses.dataclass(frozen=True) class Reply: answer: str = dataclasses.field(metadata={"description": "With interrupt false, the passage of the premises or the records that answers the worker's question, quoted verbatim, without elisions. Empty otherwise."}) - # Routes the worker one rung up, as the judge interrupts or quotes an answer, or on the next cycle: loop_claude/escalation_mark.py. - escalation: str = dataclasses.field(metadata={"description": "Empty unless the chunk is unfinished, interrupt is true or answer quotes a passage, and the worker's run ends asking loop-claude for a higher rung, citing an escalation trigger. Then: a verbatim quote, without elisions, of the transcript passage where the trigger fired, such as a review round's commit title or the worker's report of the finding or failing test. " + escalation_mark.QUOTE_AFTER_STEP + " Never a later mention of the trigger, and never the request itself."}) + # Routes the worker one rung up, as the judge interrupts or quotes an answer, or on the next cycle: agent_cycle/escalation_mark.py. + escalation: str = dataclasses.field(metadata={"description": "Empty unless the chunk is unfinished, interrupt is true or answer quotes a passage, and the worker's run ends {asks_higher_rung}, citing an escalation trigger. Then: a verbatim quote, without elisions, of the transcript passage where the trigger fired, such as a review round's commit title or the worker's report of the finding or failing test. {quote_after_step} Never a later mention of the trigger, and never the request itself."}) evidence: str = dataclasses.field(metadata={"description": "A verbatim quote of the one transcript passage the verdict rests on, without elisions: what triggers the interrupt; for a quoted answer, the worker's question; else what shows the chunk finished or the work remaining."}) finished: bool = dataclasses.field(metadata={"description": "Whether the chunk is finished."}) interrupt: bool = dataclasses.field(metadata={"description": "Whether the human must now be interrupted."}) @@ -90,11 +99,17 @@ class Reply: REPLY_TYPES = typing.get_type_hints(Reply) JSON_TYPES: dict[object, dict[str, typing.Any]] = {bool: {"type": "boolean"}, int: {"type": "integer"}, str: {"type": "string"}} -JUDGE_SCHEMA = { - "properties": {field.name: {**field.metadata, **JSON_TYPES[REPLY_TYPES[field.name]]} for field in dataclasses.fields(Reply)}, - "required": list(REPLY_TYPES), - "type": "object", -} + + +def judge_schema(driver: str) -> dict[str, typing.Any]: + """The schema of the judge's reply, each field described from Reply's metadata, with the words that name driver, the entry point, filled in.""" + asks, quote = escalation_mark.asks_higher_rung(driver), escalation_mark.quote_after_step(driver) + described = {field.name: field.metadata["description"].replace("{asks_higher_rung}", asks).replace("{quote_after_step}", quote) for field in dataclasses.fields(Reply)} + return { + "properties": {name: {"description": description, **JSON_TYPES[REPLY_TYPES[name]]} for name, description in described.items()}, + "required": list(REPLY_TYPES), + "type": "object", + } @dataclasses.dataclass(frozen=True) @@ -122,12 +137,19 @@ class Grounds(typing.NamedTuple): class Brief(typing.NamedTuple): - """What the judge reads: the worker's task, the premises and records the worker is judged by, the transcript's entries, and escalated_at, how many of those entries the session held when an escalation last stepped the worker up a rung, 0 before any.""" + """What the judge reads: the worker's task, the premises and records the worker is judged by, and the transcript's entries.""" task: str premises: str records: str + # What the records are, in their order, as WorkerPrompts.records_named has it. + records_named: str + # Where the worker may record an answer the human typed, as the entry point names it. + answer_destinations: str entries: list[Entry] + # How many of the entries the session held when an escalation last stepped the worker up a rung, 0 before any. escalated_at: int + # The entry point's name, such as `do-epic`, as WorkerPrompts.driver has it. + driver: str def verdict_from(output: object) -> Verdict | Failure: @@ -161,21 +183,21 @@ def judge(root: str, brief: Brief, rung: Rung, effects: Effects, keep: typing.Ca def judge_once(root: str, brief: Brief, rung: Rung, effects: Effects) -> Verdict | Failure | Limited: """One judge run on the brief, as judge retries it; judge-replay replays it.""" prompt, grounds = judge_prompt(brief) - verdict = run_judge(root, judge_args(rung), prompt, grounds, effects) + verdict = run_judge(root, judge_args(rung, brief.driver), prompt, grounds, effects) return verdict if isinstance(verdict, Failure | Limited) else escalation_mark.stamped(verdict, len(brief.entries)) -def judge_archived(root: str, prompt: str, grounds: Grounds, entries_read: int, rung: Rung, effects: Effects) -> Verdict | Failure | Limited: +def judge_archived(root: str, prompt: str, grounds: Grounds, entries_read: int, rung: Rung, driver: str, effects: Effects) -> Verdict | Failure | Limited: """One judge run on a prompt the archive kept from an earlier run, as judge-replay replays it: the judge reads what it read then, and its quotes are checked against grounds, which judge_replay.archived_prompt makes from that prompt in place of the records of today.""" - verdict = run_judge(root, judge_args(rung), prompt, grounds, effects) + verdict = run_judge(root, judge_args(rung, driver), prompt, grounds, effects) return verdict if isinstance(verdict, Failure | Limited) else escalation_mark.stamped(verdict, entries_read) -def judge_args(rung: Rung) -> list[str]: +def judge_args(rung: Rung, driver: str) -> list[str]: # --json-schema has claude check its reply against the schema and return it as the json result's structured_output. # No tools, so the verdict rests on the transcript grounded() checks it against. return [ - "--json-schema", json.dumps(JUDGE_SCHEMA), + "--json-schema", json.dumps(judge_schema(driver)), *rung.flags(), "--tools", "", ] @@ -270,12 +292,13 @@ def judge_prompt(brief: Brief) -> tuple[str, Grounds]: """The prompt, and what the judge's quotes are checked against.""" head = ( f"From the goals file, {brief.premises}\n\n" - f"The records: the project's working rules and decisions, then the human's rules for every project:\n{brief.records or 'none'}\n\n" + f"The records: {brief.records_named}:\n{brief.records or 'none'}\n\n" f"{TASK_HEADING}{brief.task}\n\n" - f"{TRANSCRIPT_HEADING}" + f"{transcript_heading(brief.driver)}" ) - room = max(0, JUDGE_PROMPT_CHARS - len(head) - len(JUDGE_RULES) - 2) - transcript = escalation_mark.transcript(brief.entries, brief.escalated_at, typed_answer) + rules = judge_rules(brief.answer_destinations, brief.driver) + room = max(0, JUDGE_PROMPT_CHARS - len(head) - len(rules) - 2) + transcript = escalation_mark.transcript(brief.entries, brief.escalated_at, typed_answer, brief.driver) # The tail, where an interrupt or a finish lands; the session's transcript grows cycle over cycle. shown = transcript[max(0, len(transcript) - room):] grounds = Grounds( @@ -284,4 +307,4 @@ def judge_prompt(brief: Brief) -> tuple[str, Grounds]: quotable=f"{brief.premises}\n\n{brief.records}", typed=tuple(human_answers(brief.entries, typed_answer)), ) - return f"{head}{shown}\n\n{JUDGE_RULES}", grounds \ No newline at end of file + return f"{head}{shown}\n\n{rules}", grounds \ No newline at end of file diff --git a/bin/loop_claude/quotes.py b/bin/agent_cycle/quotes.py similarity index 100% rename from bin/loop_claude/quotes.py rename to bin/agent_cycle/quotes.py diff --git a/bin/loop_claude/cycle/retry.py b/bin/agent_cycle/retry.py similarity index 97% rename from bin/loop_claude/cycle/retry.py rename to bin/agent_cycle/retry.py index b91e2484..25a5b775 100644 --- a/bin/loop_claude/cycle/retry.py +++ b/bin/agent_cycle/retry.py @@ -3,7 +3,7 @@ import typing from loop_effects.effects import Effects, say from failure import Failure from claude_code.reply import Limited -from loop_claude.stop import Stop +from agent_cycle.stop import Stop T = typing.TypeVar("T") diff --git a/bin/loop_claude/routing.py b/bin/agent_cycle/routing.py similarity index 87% rename from bin/loop_claude/routing.py rename to bin/agent_cycle/routing.py index 4f710662..95a15a6a 100644 --- a/bin/loop_claude/routing.py +++ b/bin/agent_cycle/routing.py @@ -2,9 +2,9 @@ The worker, and the session /talk opens, run on a Route (model_routing/route.py). A model off the ladder gets a route without an item, and it is never stepped. -- Start. routed_item finds the plan item that routes a fresh chunk, if there is one with L, E and I scored; first_route picks the worker's rung from its L, or the project's rung without an item. +- Start. The entry point finds the plan item that routes a fresh chunk, as loop_claude/routed_item.py does; first_route picks the worker's rung from its L, or the project's rung without an item. - Rerun. kept decides whether a rerun resumes the route its archive last names. -- Step-up. next_route raises the route after an unfinished verdict (unfinished_route), or an interrupt or a quoted answer that carries an escalation (escalated_route); escalation_cap bounds an escalation, and loop_claude/escalation_mark.py states the escalation mark's rule. step_up names the change where it changes the worker line, which the archive records. +- Step-up. next_route raises the route after an unfinished verdict (unfinished_route), or an interrupt or a quoted answer that carries an escalation (escalated_route); escalation_cap bounds an escalation, and agent_cycle/escalation_mark.py states the escalation mark's rule. step_up names the change where it changes the worker line, which the archive records. - Judge. judge_rung picks the rung of the judge of a JUDGE_REPLAYED model. A step-up weighs five rungs: @@ -28,14 +28,11 @@ Each verdict then moves the rung by one rule, and the rung never falls. The rung | interrupt or quoted answer without one | holds | holds | holds | """ import dataclasses +import typing -from release_plan.release_standing import chunk_item, LastCut, Next -from loop_claude import escalation_mark -from failure import Failure -from loop_claude.verdicts import Interrupt, QuotedAnswer, Unfinished -from plan.todo_table import Item, NoItemsTable +from agent_cycle import escalation_mark +from agent_cycle.verdicts import Interrupt, QuotedAnswer, Unfinished from model_routing.ladder import Ladder, ProjectRung, Rung -from plan_store.plans import Plan from model_routing.route import Route UNFINISHED_PER_STEP = 2 @@ -45,21 +42,22 @@ ESCALATION_CAP = 2 JUDGE_REPLAYED = ("claude-opus-5-5",) -def routed_item(plan: Plan | Failure, cut: LastCut, item_id: int | None = None) -> Item | Failure | None: - """The plan's item item_id, or without one the item release_standing.chunk_item picks from plan, weighed against the last cut; None when there is none, or it lacks L, E or I.""" - if isinstance(plan, NoItemsTable): - return None - if isinstance(plan, Failure): - return plan - if item_id is None: - pick = chunk_item(plan.items, cut) - takes = pick.item if isinstance(pick, Next) else None - else: - takes = next((item for item in plan.items if item.id == item_id), None) - return takes if takes and takes.likelihood and takes.escape_and_impact else None +class Scored(typing.Protocol): + """What routes a chunk: a plan item under loop-claude, an epic's issue under do-epic.""" + @property + def id(self) -> int: + """The number a Route keeps as its item.""" + + @property + def likelihood(self) -> int: + """Its L.""" + + @property + def escape_and_impact(self) -> int: + """Its E + I.""" -def first_route(project: ProjectRung, item: Item | None) -> Route: +def first_route(project: ProjectRung, item: Scored | None) -> Route: """The route a chunk starts on: the rung its item's L picks from the project's rung, or the project's rung without an item.""" if item is None: return Route(project.rung) @@ -133,7 +131,7 @@ def unfinished_route(project: ProjectRung, route: Route, verdict: Unfinished) -> def escalated_route(project: ProjectRung, route: Route, verdict: Interrupt | QuotedAnswer) -> Route: """The route after an interrupt or a quoted answer. Without an escalation it holds, since neither is an unfinished verdict. - With one, the rung rises to floor_rung, which the verdict's highest L taken in may raise. Where floor_rung lies below escalation_cap, the rung steps one up from it and the mark moves, as loop_claude/escalation_mark.py says; at the cap the rung stays on floor_rung and the mark holds. + With one, the rung rises to floor_rung, which the verdict's highest L taken in may raise. Where floor_rung lies below escalation_cap, the rung steps one up from it and the mark moves, as agent_cycle/escalation_mark.py says; at the cap the rung stays on floor_rung and the mark holds. """ mark = escalation_mark.moved_to(verdict) if mark is None: diff --git a/bin/loop_claude/cycle/run.py b/bin/agent_cycle/run.py similarity index 75% rename from bin/loop_claude/cycle/run.py rename to bin/agent_cycle/run.py index 6e8f2ea3..fa3d4a16 100644 --- a/bin/loop_claude/cycle/run.py +++ b/bin/agent_cycle/run.py @@ -2,31 +2,30 @@ import dataclasses import pathlib import typing -from loop_claude.cycle.states import Active, Archived, Closing, Ended, Event, HumanTalked, HumanTyped, Interrupting, Judging, QUOTED_ANSWERS_PER_CYCLE, Ruled, State, to_the_human, transition, TypedAnswer, WorkerRan, Working +from agent_cycle.states import Active, Archived, Closing, Ended, Event, HumanTalked, HumanTyped, Interrupting, Judging, QUOTED_ANSWERS_PER_CYCLE, Ruled, State, to_the_human, transition, TypedAnswer, WorkerRan, Working from transcript_archive.write import archive -from loop_claude.cycle.worker import AUTO_COMPACT_WINDOW, COMPACT_WINDOW, run_worker, worker_args -from loop_claude.cycle.judge import Brief, judge, Rejected -from loop_claude.cycle import chunk_items +from agent_cycle.worker import AUTO_COMPACT_WINDOW, COMPACT_WINDOW, run_worker, worker_args +from claude_code.output_limits import BASH_CEILING, BASH_CEILING_CHARACTERS +from agent_cycle.judge import Brief, judge, Rejected from claude_code.session import delete_session, Session, session_open, transcript_file from loop_effects.effects import Effects, say from claude_code.entries import Entry from failure import Failure -from loop_claude.verdicts import Finished, Interrupt, Judged, QuotedAnswer, Unfinished, Verdict -from loop_claude.cycle.interrupt import interrupt, Talked, WorkerSession +from agent_cycle.verdicts import Finished, Interrupt, Judged, QuotedAnswer, TakenIn, Unfinished, Verdict +from agent_cycle.interrupt import interrupt, Talked, WorkerSession from transcript_archive.judge_run import JudgeRun from transcript_archive.read import last_spent -from loop_claude.routing import next_route, step_up, StepUp +from agent_cycle.routing import next_route, step_up, StepUp from transcript_archive import outcomes -from plan_store.plans import Plan from model_routing.ladder import ProjectRung, Rung -from loop_claude.cycle import prompts from claude_code.transcript import read_transcript -from loop_claude.cycle.retry import retried +from agent_cycle.retry import retried from model_routing.route import Route -from loop_claude.cycle import spend -from loop_claude.stop import Stop -from loop_claude.tmux import Window +from agent_cycle import spend +from agent_cycle.stop import Stop +from agent_cycle.tmux import Window from model_routing import worker_rungs +from agent_cycle.worker_prompts import WorkerPrompts @dataclasses.dataclass(frozen=True) @@ -43,6 +42,8 @@ class Cycle: # The rung routing.judge_rung picks for the project's judge model. judge: Rung premises: str + # The prompts and the file names of the entry point that drives the cycle. + prompts: WorkerPrompts # The project's rung on the ladder that model_routing/ladder_file.py read as the cycle started; None for a model off the ladder. project: ProjectRung | None project_archive: pathlib.Path @@ -51,8 +52,8 @@ class Cycle: route: Route session: Session task: str - # The plan on the default branch as the cycle starts, which scores the items the worker names. - plan: Plan | Failure + # The plan items a transcript's worker has taken into its chunk, as the entry point reads and scores them. + taken_in: typing.Callable[[list[Entry]], TakenIn] # Empty when the previous cycle's verdict was not Unfinished. unfinished_evidence: str # The tmux window the loop runs in, which an interrupt marks. @@ -119,7 +120,7 @@ def step(cycle: Cycle, state: Active) -> Event | Stop: def work(cycle: Cycle, answered: QuotedAnswer | TypedAnswer | None, stepped: bool) -> WorkerRan | Stop: if isinstance(answered, QuotedAnswer): - say(f"\n=== {prompts.QUOTE_SOURCES} answered: {answered.question} ===\n{answered.answer}", cycle.effects, stdout=True) + say(f"\n=== {cycle.prompts.quote_sources} answered: {answered.question} ===\n{answered.answer}", cycle.effects, stdout=True) # worker_start runs inside the lambda: a failed attempt may have opened the session. if stop := retried("worker", lambda: run_worker(cycle.root, worker_start(cycle, answered, stepped), worker_environment(cycle), cycle.effects), cycle.effects): return stop @@ -128,8 +129,8 @@ def work(cycle: Cycle, answered: QuotedAnswer | TypedAnswer | None, stepped: boo def worker_environment(cycle: Cycle) -> dict[str, str]: - """The rungs the worker's claude is handed, which rule-hook holds its review launches to, and the auto-compact window of its context.""" - return {**worker_rungs.environment(cycle.project, cycle.route), AUTO_COMPACT_WINDOW: str(COMPACT_WINDOW)} + """The rungs the worker's claude is handed, which rule-hook holds its review launches to, the auto-compact window of its context, and the length past which it saves a Bash result to a file.""" + return {**worker_rungs.environment(cycle.project, cycle.route), AUTO_COMPACT_WINDOW: str(COMPACT_WINDOW), BASH_CEILING: str(BASH_CEILING_CHARACTERS)} def worker_session(cycle: Cycle, transcript_path: str, stepped_from: Rung | None) -> WorkerSession: @@ -138,24 +139,34 @@ def worker_session(cycle: Cycle, transcript_path: str, stepped_from: Rung | None def worker_start(cycle: Cycle, answered: QuotedAnswer | TypedAnswer | None, stepped: bool) -> list[str]: if isinstance(answered, QuotedAnswer): - return worker_args(cycle.session, cycle.route.rung, prompts.answered(answered.question, answered.answer, cycle.route, cycle.project, stepped), resume=True) + return worker_args(cycle.session, cycle.route.rung, cycle.prompts.answered(answered.question, answered.answer, cycle.route, cycle.project, stepped), resume=True) if isinstance(answered, TypedAnswer): - return worker_args(cycle.session, cycle.route.rung, prompts.typed(answered.question, answered.answer, cycle.route, cycle.project, stepped), resume=True) + return worker_args(cycle.session, cycle.route.rung, cycle.prompts.typed(answered.question, answered.answer, cycle.route, cycle.project, stepped), resume=True) if session_open(cycle.session): - return worker_args(cycle.session, cycle.route.rung, prompts.resumed(cycle.unfinished_evidence, cycle.route, cycle.project), resume=True) - return worker_args(cycle.session, cycle.route.rung, prompts.fresh(cycle.task, cycle.premises, cycle.route, cycle.project, cycle.facts), resume=False) + return worker_args(cycle.session, cycle.route.rung, cycle.prompts.resumed(cycle.unfinished_evidence, cycle.route, cycle.project), resume=True) + return worker_args(cycle.session, cycle.route.rung, cycle.prompts.fresh(cycle.task, cycle.premises, cycle.route, cycle.project, cycle.facts), resume=False) def rule(cycle: Cycle, transcript_path: str, quoted_answers: int) -> Ruled | Stop: - """The judge's ruling. Unless it is Finished, code stamps it with the items the worker names on its latest `Chunk items:` line. An interrupt or a quoted answer is archived as it is ruled, on the route the worker ran on until then, and then any step-up it makes, so the figures charge each stretch of work to its own rung.""" + """The judge's ruling. Unless it is Finished, code stamps it with the items the worker took in, as cycle.taken_in reads them. An interrupt or a quoted answer is archived as it is ruled, on the route the worker ran on until then, and then any step-up it makes, so the figures charge each stretch of work to its own rung.""" entries = read_transcript(transcript_path) if isinstance(entries, Failure): return Stop(1, entries.reason) - verdict = judge(cycle.root, Brief(cycle.task, cycle.premises, cycle.records, entries, cycle.route.escalated_at), cycle.judge, cycle.effects, lambda rejected: keep_rejection(cycle, transcript_path, rejected)) + brief = Brief( + task=cycle.task, + premises=cycle.premises, + records=cycle.records, + records_named=cycle.prompts.records_named, + answer_destinations=cycle.prompts.answer_destinations, + entries=entries, + escalated_at=cycle.route.escalated_at, + driver=cycle.prompts.driver, + ) + verdict = judge(cycle.root, brief, cycle.judge, cycle.effects, lambda rejected: keep_rejection(cycle, transcript_path, rejected)) if isinstance(verdict, Stop): return verdict if not isinstance(verdict, Finished): - verdict = dataclasses.replace(verdict, taken_in=chunk_items.taken_in(entries, cycle.plan)) + verdict = dataclasses.replace(verdict, taken_in=cycle.taken_in(entries)) if not isinstance(verdict, Interrupt | QuotedAnswer): return Ruled(verdict) annotation = annotation_of(cycle, transcript_path, entries, verdict.judged) @@ -182,8 +193,8 @@ def annotation_of(cycle: Cycle, transcript_path: str, entries: list[Entry], judg def ruling_outcome(cycle: Cycle, ruling: Interrupt | QuotedAnswer, quoted_answers: int, annotation: outcomes.Annotation) -> str: """The archive's outcome for a ruling: interrupted where it goes to the human, with the question put to them, and else answered from the premises or the records.""" if isinstance(ruling, QuotedAnswer) and not to_the_human(ruling, quoted_answers): - return outcomes.answered(question=ruling.question, sources=prompts.QUOTE_SOURCES, answer=ruling.answer, evidence=ruling.evidence, escalation=ruling.escalation, judge=str(cycle.judge), worker=cycle.route.worker_line(), judge_cost=ruling.judged.cost, annotation=annotation, judge_run=ruling.judged.run) - return outcomes.interrupted(question_for(ruling), ruling.evidence, str(cycle.judge), ruling.escalation, cycle.route.worker_line(), judge_cost=ruling.judged.cost, annotation=annotation, judge_run=ruling.judged.run) + return outcomes.answered(question=ruling.question, sources=cycle.prompts.quote_sources, answer=ruling.answer, evidence=ruling.evidence, escalation=ruling.escalation, judge=str(cycle.judge), worker=cycle.route.worker_line(), judge_cost=ruling.judged.cost, annotation=annotation, judge_run=ruling.judged.run) + return outcomes.interrupted(question_for(cycle, ruling), ruling.evidence, str(cycle.judge), ruling.escalation, cycle.route.worker_line(), judge_cost=ruling.judged.cost, annotation=annotation, judge_run=ruling.judged.run) def keep_rejection(cycle: Cycle, transcript_path: str, rejected: Rejected) -> Stop | None: @@ -196,7 +207,7 @@ def keep_rejection(cycle: Cycle, transcript_path: str, rejected: Rejected) -> St def ask(cycle: Cycle, transcript_path: str, ruling: Interrupt | QuotedAnswer, stepped_from: Rung | None) -> HumanTyped | HumanTalked | Stop: """The human's answer to a ruling that rule archived as it ruled; the human's answer then grows the transcript. stepped_from is the rung the worker ran on, where the ruling stepped it up.""" # Only the human may cut a release, so a release ready to cut opens claude's session at once. - answer = interrupt(question_for(ruling), not (isinstance(ruling, Interrupt) and ruling.release_ready), worker_session(cycle, transcript_path, stepped_from), cycle.window, cycle.effects) + answer = interrupt(question_for(cycle, ruling), not (isinstance(ruling, Interrupt) and ruling.release_ready), worker_session(cycle, transcript_path, stepped_from), cycle.window, cycle.prompts.driver, cycle.effects) match answer: case Stop(): return answer @@ -220,10 +231,10 @@ def archived_step(project_archive: pathlib.Path, transcript_path: str, task: str return archived(project_archive, transcript_path, task, outcomes.stepped_up(step_up.after.worker_line()), effects) -def question_for(ruling: Interrupt | QuotedAnswer) -> str: +def question_for(cycle: Cycle, ruling: Interrupt | QuotedAnswer) -> str: if isinstance(ruling, Interrupt): return ruling.question - return prompts.past_the_cap(ruling.question, ruling.answer, QUOTED_ANSWERS_PER_CYCLE) + return cycle.prompts.past_the_cap(ruling.question, ruling.answer, QUOTED_ANSWERS_PER_CYCLE) def close(cycle: Cycle, transcript_path: str, verdict: Finished | Unfinished) -> Archived | Stop: diff --git a/bin/loop_claude/cycle/spend.py b/bin/agent_cycle/spend.py similarity index 100% rename from bin/loop_claude/cycle/spend.py rename to bin/agent_cycle/spend.py diff --git a/bin/loop_claude/cycle/states.py b/bin/agent_cycle/states.py similarity index 97% rename from bin/loop_claude/cycle/states.py rename to bin/agent_cycle/states.py index cf24478f..4979c933 100644 --- a/bin/loop_claude/cycle/states.py +++ b/bin/agent_cycle/states.py @@ -1,8 +1,8 @@ import dataclasses -from loop_claude.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict +from agent_cycle.verdicts import Finished, Interrupt, QuotedAnswer, Unfinished, Verdict from model_routing.ladder import Rung -from loop_claude.routing import StepUp +from agent_cycle.routing import StepUp # A backstop for what the judge misses, such as a reworded repeat, so the worker cannot loop unattended on answers quoted from the premises or the records. QUOTED_ANSWERS_PER_CYCLE = 3 diff --git a/bin/agent_cycle/stop.py b/bin/agent_cycle/stop.py new file mode 100644 index 00000000..b4185365 --- /dev/null +++ b/bin/agent_cycle/stop.py @@ -0,0 +1,20 @@ +import dataclasses +import typing + + +@dataclasses.dataclass(frozen=True) +class Stop: + """The entry point exits with code, printing reason.""" + code: typing.Literal[0, 1, 2, 130] + reason: str + + +def needs_human(need: str, blocker: str) -> Stop: + return Stop(2, f"the human is needed to {need}, but {blocker}") + + +def needs_terminal(need: str, on_terminal: bool, driver: str) -> Stop | None: + """driver is the entry point the human reruns, such as `do-epic`.""" + if on_terminal: + return None + return needs_human(need, f"stdin or stdout is not a terminal; rerun {driver} on a terminal") diff --git a/bin/loop_claude/test_quotes.py b/bin/agent_cycle/test_quotes.py similarity index 94% rename from bin/loop_claude/test_quotes.py rename to bin/agent_cycle/test_quotes.py index 09f98009..6e00b5e6 100644 --- a/bin/loop_claude/test_quotes.py +++ b/bin/agent_cycle/test_quotes.py @@ -4,7 +4,7 @@ import unittest sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1])) -from loop_claude import quotes +from agent_cycle import quotes class Verbatim(unittest.TestCase): diff --git a/bin/agent_cycle/tmux.py b/bin/agent_cycle/tmux.py new file mode 100644 index 00000000..0af21917 --- /dev/null +++ b/bin/agent_cycle/tmux.py @@ -0,0 +1,163 @@ +"""The tmux commands an entry point runs: each chunk's loop runs in its own window of one tmux session, which loop-claude names for the project and do-epic for the epic (docs/decisions.md โ†’ `loop-claude: each chunk runs in its own tmux window`).""" +import dataclasses +import re +import subprocess +import typing + +from failure import Failure + +# Runs tmux with these arguments and returns what it printed; exit 127 where tmux cannot start. +Tmux = typing.Callable[[list[str]], subprocess.CompletedProcess[str]] + +# The oldest tmux that has every command and option used here. Its CHANGES file dates the newest two to 3.2: `remain-on-exit failed` and `new-session -e`. +NEEDED = (3, 2) +# What a window's name ends in while its chunk waits for the human. tmux's status bar shows the name, and `!` after it once the bell rang. +WAITS = " ?" +BELL = "\a" +# What follows a window's name while its chunk is parked, before what its pull request waits on, as in `ABC-34 awaits an approval`. A parked chunk takes no running slot (docs/decisions.md โ†’ `do-epic: drives one Jira epic โ€ฆ`, "Long gates"). +PARKED = " awaits " +# What a session's name opens with: LOOP for a project's chunks under loop-claude, EPIC for an epic's chunks under do-epic. +LOOP = "loop" +EPIC = "epic" +# The session option that names the checkout whose chunks the session runs, since two checkouts' keys may map to one session name. +OWNER = "@loop-root" +# The window option that names the folder a window's chunk works in, so that loop-claude and every do-epic see the chunks a do-epic runs in a checkout. +ROOT = "@chunk-root" + + +@dataclasses.dataclass(frozen=True) +class Window: + """The tmux window a chunk's loop runs in.""" + session: str + # Without WAITS. + name: str + # The ID of the loop's pane, which $TMUX_PANE holds. It targets the window whatever its name is. + pane: str + + @property + def target(self) -> str: + """What `tmux attach -t` and `tmux select-window -t` take to reach the window.""" + return f"{self.session}:{self.name}" + + +@dataclasses.dataclass(frozen=True) +class Listed: + """A window of the project's session, as tmux lists it.""" + # Without WAITS, and without PARKED and what follows it. + name: str + # tmux's ID of the window, such as `@3`. + id: str + # Whether its loop has exited: `remain-on-exit failed` keeps the window of a loop that failed, so its last lines stay readable. + dead: bool + # What its name says its parked chunk awaits, as the words after PARKED; empty for a chunk that isn't parked. + awaits: str = "" + # Whether its name ended in WAITS: its loop asks the human something. + asks: bool = False + + @property + def parked(self) -> bool: + return bool(self.awaits) + + +@dataclasses.dataclass(frozen=True) +class Placed: + """A window of any session, as tmux lists it.""" + session: str + window: Listed + # The folder its chunk works in, as ROOT names it; empty for a window started before ROOT was set. + root: str + + +def unusable(run: Tmux, driver: str) -> str | None: + """Why driver, the entry point, cannot use this machine's tmux, naming the version found and the one needed; None where it is NEEDED or later.""" + found = run(["-V"]).stdout.strip() + version = re.search(r"([0-9]+)\.([0-9]+)", found) + if version and (int(version[1]), int(version[2])) >= NEEDED: + return None + return f"{driver} runs each chunk in a tmux window, and needs tmux {NEEDED[0]}.{NEEDED[1]} or later; it found {found or 'no tmux'}" + + +def session_name(kind: str, key: str) -> str: + """The tmux session an entry point runs its chunks in: kind LOOP with a project's key, such as `loop-claude-config`, or kind EPIC with an epic's key, such as `epic-ABC-12`. tmux reads `.` and `:` in a target as separators, so every character of the key but a letter, a digit, `_` and `-` becomes a dash.""" + return f"{kind}-" + re.sub(r"[^A-Za-z0-9_-]", "-", key) + + +def listed(session: str, run: Tmux) -> list[Listed]: + """The windows of session; none where tmux lists none, as while the session doesn't exist or no tmux server runs.""" + done = run(["list-windows", "-t", f"={session}", "-F", "#{window_name}\t#{pane_dead}\t#{window_id}"]) + rows = [line.split("\t") for line in done.stdout.splitlines()] if done.returncode == 0 else [] + return [as_listed(row[0], row[2], row[1]) for row in rows if len(row) == 3] + + +def as_listed(shown: str, id: str, pane_dead: str) -> Listed: + """The window tmux shows under the name shown, with the ID id and `#{pane_dead}` as pane_dead.""" + name, _, awaits = shown.removesuffix(WAITS).partition(PARKED) + return Listed(name, id, pane_dead == "1", awaits, shown.endswith(WAITS)) + + +def everywhere(run: Tmux) -> list[Placed]: + """The windows of every session; none where no tmux server runs.""" + done = run(["list-windows", "-a", "-F", f"#{{session_name}}\t#{{window_name}}\t#{{pane_dead}}\t#{{window_id}}\t#{{{ROOT}}}"]) + rows = [line.split("\t") for line in done.stdout.splitlines()] if done.returncode == 0 else [] + return [Placed(row[0], as_listed(row[1], row[3], row[2]), row[4]) for row in rows if len(row) == 5] + + +def start_window(window: Window, cwd: str, env: typing.Mapping[str, str], command: list[str], first: bool, owned: bool, run: Tmux) -> Failure | None: + """Starts command in a new window of window's session, in cwd with env added to its environment, and leaves the client where it is. first makes the session too, and owned marks a session it makes as running the chunks of the checkout cwd. A Failure says why tmux started none, or why it could not mark the session.""" + make = ["new-session", "-d", "-s", window.session] if first else ["new-window", "-d", "-t", f"={window.session}:"] + variables = [flag for name, value in env.items() for flag in ("-e", f"{name}={value}")] + # tmux reads a `;` that ends an argument as the end of the command, unless a backslash precedes it. + words = [re.sub(r";$", r"\\;", word) for word in command] + done = run([*make, "-n", window.name, "-c", cwd, *variables, "-P", "-F", "#{window_id}", "--", *words]) + if done.returncode: + return Failure(f"tmux could not start the window {window.target}: {done.stderr.strip() or f'it exited {done.returncode}'}") + # set-option and show-options read a target as a pane's, so `=session` alone names no session. + marked = run(["set-option", "-t", f"={window.session}:", OWNER, cwd]) if first and owned else None + # A failure leaves the window unseen by loop-claude and by another epic's do-epic, which then may start a chunk beside it; the window still runs. + run(["set-option", "-w", "-t", done.stdout.strip(), ROOT, cwd]) + # By the ID, since a window of the same name may still be closing. A failed remain-on-exit doesn't matter: a loop that fails before it is set closes its window, and its stop is still sent as a DM. + run(["set-option", "-w", "-t", done.stdout.strip(), "remain-on-exit", "failed"]) + # Without the owner, start_chunks cannot tell this checkout's session from another's, so no further chunk may start in it. + if marked is not None and marked.returncode: + return Failure(f"tmux started the session {window.session} but could not mark it as this checkout's: {marked.stderr.strip() or f'it exited {marked.returncode}'}") + return None + + +def start_chunk(window: Window, cwd: str, env: typing.Mapping[str, str], command: list[str], windows: list[Listed], owned: bool, run: Tmux) -> list[Listed] | Failure: + """Starts a chunk's command in window, in place of each dead window of its name among windows, the session's; the session's windows afterwards, or why tmux started none. start_window says what cwd, env and owned do.""" + stale = [listed for listed in windows if listed.name == window.name and listed.dead] + for listed in stale: + close(listed, run) + kept = [listed for listed in windows if listed not in stale] + failed = start_window(window, cwd, env, command, not kept, owned, run) + return failed or [*kept, Listed(window.name, "", False)] + + +def owner(session: str, run: Tmux) -> str: + """The checkout whose chunks session runs, as OWNER names it; empty where it names none.""" + return run(["show-options", "-qv", "-t", f"={session}:", OWNER]).stdout.strip() + + +def close(window: Listed, run: Tmux) -> None: + run(["kill-window", "-t", window.id]) + + +def mark(window: Window, waits: bool, run: Tmux) -> None: + """Renames window to end in WAITS while its chunk waits for the human, and back once it doesn't.""" + run(["rename-window", "-t", window.pane, window.name + (WAITS if waits else "")]) + + +def park(window: Window, on: str, run: Tmux) -> None: + """Shows window's chunk as parked. on is what the chunk awaits, such as `an approval` or `a slot`: the name says so, and no checkout counts a window so named as a running chunk.""" + run(["rename-window", "-t", window.pane, window.name + PARKED + on]) + + +def unpark(window: Window, root: str, run: Tmux) -> None: + """Shows window's chunk as running again, in the folder root. The window may name another folder: one started again for a parked chunk starts in the epic's folder.""" + run(["rename-window", "-t", window.pane, window.name]) + run(["set-option", "-w", "-t", window.pane, ROOT, root]) + + +def show(session: str, inside: bool) -> list[str]: + """The tmux arguments that put session on the human's terminal: inside tmux the client switches to it, and outside a new client attaches.""" + return ["switch-client" if inside else "attach-session", "-t", f"={session}"] diff --git a/bin/loop_claude/verdicts.py b/bin/agent_cycle/verdicts.py similarity index 100% rename from bin/loop_claude/verdicts.py rename to bin/agent_cycle/verdicts.py diff --git a/bin/loop_claude/cycle/worker.py b/bin/agent_cycle/worker.py similarity index 100% rename from bin/loop_claude/cycle/worker.py rename to bin/agent_cycle/worker.py diff --git a/bin/agent_cycle/worker_prompts.py b/bin/agent_cycle/worker_prompts.py new file mode 100644 index 00000000..9c070afb --- /dev/null +++ b/bin/agent_cycle/worker_prompts.py @@ -0,0 +1,65 @@ +"""A worker's prompts: those the entry point driving the cycle words, and those that resume a worker with an answer to its question. + +An entry point is a tool that drives cycles, such as loop-claude. It words the prompts of its own workflow and names the files of its project. WorkerPrompts holds those, so the cycle doesn't import an entry point to word a prompt. +""" +import dataclasses +import re +import typing + +from model_routing.ladder import ProjectRung +from model_routing.route import Route +from claude_code.transcript import Typed + +# Opens typed's prompt, with the answer on the line after it, so typed_answer finds the answer at the prompt's start, whatever the question holds. +TYPED_HERE = "The human answered your question at {driver}'s terminal:" +# Finds TYPED_HERE whatever the driver, so a transcript reads the same to every tool that renders it. +TYPED_OPENING = re.compile(re.escape(TYPED_HERE).replace(re.escape("{driver}"), r"[\w-]+") + "\n") + + +def answers_recorded(destinations: str) -> str: + """What a chunk owes each answer the human typed, before it is finished. destinations names where an answer may be recorded.""" + return f"every answer the human typed in it that a later chunk could need is recorded in {destinations} at the one place it governs" + + +def typed_answer(prompt: str) -> Typed | None: + """Splits a prompt from WorkerPrompts.typed into its answer and the rest, so the judge reads the answer as the human's; None for any other prompt.""" + opening = TYPED_OPENING.match(prompt) + if not opening: + return None + answer, _, rest = prompt[opening.end():].partition("\n\n") + return Typed(rest, answer) + + +@dataclasses.dataclass(frozen=True) +class WorkerPrompts: + """What an entry point supplies to the prompts of its cycles.""" + # The entry point's name as the worker, the judge and the human read it, such as `do-epic`. + driver: str + # The prompt that starts a worker, from its task, the premises, its route, the project's rung and the facts found before the cycle. + fresh: typing.Callable[[str, str, Route, ProjectRung | None, str], str] + # The prompt that resumes a worker whose previous run ended before the chunk was finished, from the evidence of the unfinished verdict, if there was one. + resumed: typing.Callable[[str, Route, ProjectRung | None], str] + # The sentences that depend on the worker's rung, each after a space, to follow the sentence a prompt ends on. + rungs_held: typing.Callable[[Route, ProjectRung | None], str] + # Where the judge may quote an answer from, named as the human and the worker see them. + quote_sources: str + # Where the worker may record an answer the human typed. + answer_destinations: str + # What the records the judge reads are, in the order it reads them. + records_named: str + + def answered(self, question: str, answer: str, route: Route, project: ProjectRung | None, stepped: bool) -> str: + """stepped is whether the answer's ruling stepped the worker up.""" + return f"Your question: {question}\n\n{self.quote_sources} answers it:\n\n{answer}\n\nContinue the task.{self.stepped_to(route, project, stepped)}" + + def typed(self, question: str, answer: str, route: Route, project: ProjectRung | None, stepped: bool) -> str: + """The prompt that resumes the worker with the human's answer to an interrupt. answer is one line, as the terminal reads it.""" + return f"{TYPED_HERE.format(driver=self.driver)}\n{answer}\n\nYour question was: {question}\n\nContinue the task. The chunk is finished only once {answers_recorded(self.answer_destinations)}.{self.stepped_to(route, project, stepped)}" + + def stepped_to(self, route: Route, project: ProjectRung | None, stepped: bool) -> str: + """The sentences naming the worker's new rung, as resumed has them, where the ruling that brought the answer stepped the worker up; empty otherwise.""" + return f" Your request for a higher rung went through: {self.driver} runs you on {route.rung}. The escalation triggers that fired before this step are spent.{self.rungs_held(route, project)}" if stepped else "" + + def past_the_cap(self, question: str, answer: str, cap: int) -> str: + """The question a quoted answer puts to the human once the cycle's quoted answers reach the cap.""" + return f"{question}\n\n{self.quote_sources} says:\n{answer}\n\nThose files have answered {cap} questions this cycle, the most allowed, so this one comes to you." diff --git a/bin/chunk_pr/approval.py b/bin/chunk_pr/approval.py index a33b08e5..61779af0 100644 --- a/bin/chunk_pr/approval.py +++ b/bin/chunk_pr/approval.py @@ -10,6 +10,8 @@ from forge.commands import Forge, Run # A human answers in hours, so merge polls once a minute and gives up after 90 minutes. That leaves half an hour for the queue and the checks inside the 2 hours a background command may run; a run that needs more is killed before the give-up message prints, and its next run goes on from what it pushed. POLL_SECONDS = 60 WAIT_SECONDS = 90 * 60 +# The environment variable do-epic sets to 1 for its chunks' workers. With it, merge doesn't wait for an approval: do-epic parks the chunk and polls the forge itself, so the chunk holds no running slot meanwhile (docs/decisions.md โ†’ `do-epic: drives one Jira epic โ€ฆ`, "Long gates"). +PARKS = "CHUNK_PR_PARKS" def awaited(forge: Forge, wait_for_approval: bool) -> bool | Failure: @@ -58,17 +60,29 @@ def approved(number: str, run: Run, sleep: typing.Callable[[float], None] = time return Failure(f"pull request {number} wasn't approved within {WAIT_SECONDS // 60} minutes; run chunk-pr merge again once it is") +def left_to_park(number: str, run: Run) -> Failure: + """Why a merge under PARKS ends at once where pull request number passed its checks and awaits an approval, in place of waiting as `approved` does: a reviewer's comment waits for an answer, which the worker gives first, or else do-epic parks the chunk, so the worker ends its run.""" + return unanswered(number, run) or Failure(f"pull request {number} passed its checks and awaits an approval; end your run now, since do-epic parks the chunk and resumes this session once it is approved or a reviewer comments") + + def unanswered(number: str, run: Run) -> Failure | None: - """A Failure naming the files where a reviewer's comment waits for an answer on pull request number: a review thread nobody resolved, whose last comment someone other than the pull request's author wrote. So a thread the worker answered, as that author, waits for the reviewer again. None where no thread waits.""" - read = run(["gh", "api", "graphql", "-f", f"query={THREADS}", "-F", "owner={owner}", "-F", "repo={repo}", "-F", f"number={number}"]) - try: - pull = read if isinstance(read, Failure) else json.loads(read)["data"]["repository"]["pullRequest"] - if isinstance(pull, Failure): - return pull - author = pull["author"]["login"] - waiting = sorted({thread["path"] for thread in pull["reviewThreads"]["nodes"] if not thread["isResolved"] and any(comment["author"]["login"] != author for comment in thread["comments"]["nodes"])}) - except (ValueError, KeyError, TypeError): - return Failure(f"gh api graphql printed no review threads: {str(read).strip()[:200]}") + """A Failure naming the files where a reviewer's comment waits for an answer on pull request number, as `waiting_threads` finds them, or saying why they cannot be read. None where no thread waits.""" + waiting = waiting_threads(number, run) + if isinstance(waiting, Failure): + return waiting if not waiting: return None return Failure(f"a reviewer's comment on pull request {number} waits for an answer, in {', '.join(waiting)}; answer it in its thread, push any fix, and run chunk-pr merge again") + + +def waiting_threads(number: str, run: Run) -> list[str] | Failure: + """The files where a reviewer's comment waits for an answer on pull request number, read once: those of each review thread nobody resolved, whose last comment someone other than the pull request's author wrote. So a thread the worker answered, as that author, waits for the reviewer again.""" + read = run(["gh", "api", "graphql", "-f", f"query={THREADS}", "-F", "owner={owner}", "-F", "repo={repo}", "-F", f"number={number}"]) + if isinstance(read, Failure): + return read + try: + pull = json.loads(read)["data"]["repository"]["pullRequest"] + author = pull["author"]["login"] + return sorted({thread["path"] for thread in pull["reviewThreads"]["nodes"] if not thread["isResolved"] and any(comment["author"]["login"] != author for comment in thread["comments"]["nodes"])}) + except (ValueError, KeyError, TypeError): + return Failure(f"gh api graphql printed no review threads: {read.strip()[:200]}") diff --git a/bin/chunk_pr/main.py b/bin/chunk_pr/main.py index 0e91ea6a..158011d7 100644 --- a/bin/chunk_pr/main.py +++ b/bin/chunk_pr/main.py @@ -45,7 +45,8 @@ merge merges the pull request in its turn: the chunks of one project on this Where the project's config sets `wait-for-approval = true` (`project-config --help`), it merges only once GitHub reports the pull request approved as its branch protection requires. It waits for that outside its turn, so other chunks merge meanwhile, and takes - a new turn once approved. + a new turn once approved. In a chunk do-epic runs, which sets CHUNK_PR_PARKS=1, it + fails at once instead of waiting: do-epic parks the chunk until the approval comes. It stops, without merging, and says what to do before running it again: - on a merge conflict in step 1, naming the conflicted files, with the merge in progress; - where `larv-review scoring-run-owed` exits 1 after step 3, because the chunk owes a @@ -85,7 +86,7 @@ import sys import time import typing -from chunk_pr.approval import approval, approved, awaited, unanswered +from chunk_pr.approval import approval, approved, awaited, left_to_park, PARKS, unanswered from forge.pulls import checks_passed, merged_pull, open_pull, required_checks from failure import Failure from forge.commands import Forge, forge_of, gitea, Run, runner @@ -119,7 +120,7 @@ def main(argv: list[str]) -> int: if argv[:1] in (["-h"], ["--help"]): print(__doc__.strip()) return 0 - done = step(argv, runner(os.getcwd()), functools.cache(lambda: located(os.getcwd(), pathlib.Path.home()))) + done = step(argv, runner(os.getcwd()), functools.cache(lambda: located(os.getcwd(), pathlib.Path.home())), os.environ.get(PARKS) == "1") if isinstance(done, Failure): print(f"chunk-pr: {done.reason}", file=sys.stderr) return 1 @@ -127,8 +128,8 @@ def main(argv: list[str]) -> int: return 0 -def step(argv: list[str], run: Run, config: Config) -> str | Failure: - """Run the step argv names: the line to print, or a Failure. A malformed argv doesn't run anything.""" +def step(argv: list[str], run: Run, config: Config, parks: bool = False) -> str | Failure: + """Run the step argv names: the line to print, or a Failure. A malformed argv doesn't run anything. parks is whether a merge fails at once where it would wait for an approval (`merged`).""" match argv: case ["open", title, *body] if len(body) <= 1 and not title.startswith("-"): return Failure("give a title of one line") if not one_line(title) else misfit(title, run, config) or opened(title, body[0] if body else "", run) @@ -138,7 +139,7 @@ def step(argv: list[str], run: Run, config: Config) -> str | Failure: return Failure("give a title of one line") if title and not one_line(title[0]) else (misfit(title[0], run, config) if title else None) or marked_ready(run, title[0] if title else None) case ["merge", *given]: finishes = finishes_of(given) - return Failure(USAGE) if finishes is None else merged(run, config, finishes=finishes) + return Failure(USAGE) if finishes is None else merged(run, config, finishes=finishes, parks=parks) case ["clean"]: return cleaned(run) return Failure(USAGE) @@ -304,8 +305,8 @@ def finishes_of(given: list[str]) -> list[list[str]] | None: return finishes -def merged(run: Run, config: Config, sleep: typing.Callable[[float], None] = time.sleep, finishes: typing.Sequence[list[str]] = ()) -> str | Failure: - """Merge the branch's pull request in the project's merge queue. Each pass of the loop is one turn in the queue; a pull request that still wants its approval leaves the queue while it waits, so the next chunk merges meanwhile, and takes a new turn once approved.""" +def merged(run: Run, config: Config, sleep: typing.Callable[[float], None] = time.sleep, finishes: typing.Sequence[list[str]] = (), parks: bool = False) -> str | Failure: + """Merge the branch's pull request in the project's merge queue. Each pass of the loop is one turn in the queue; a pull request that still wants its approval leaves the queue while it waits, so the next chunk merges meanwhile, and takes a new turn once approved. Where parks, it doesn't wait for the approval: it fails at once, as `left_to_park` says.""" found = pull(run) if isinstance(found, Failure): return found @@ -336,6 +337,8 @@ def merged(run: Run, config: Config, sleep: typing.Callable[[float], None] = tim taken = untaken or merge_turn(pull_request, finishes, loaded, run, sleep) if not isinstance(taken, AwaitsApproval): return taken if isinstance(taken, Failure) else taken + absent + if parks: + return left_to_park(number, run) # Out of the queue, so the project's other chunks merge while this one waits. waited = approved(number, run, sleep) if isinstance(waited, Failure): diff --git a/bin/chunk_pr/test_chunk_pr.py b/bin/chunk_pr/test_chunk_pr.py index a8ceafa1..41ca75df 100644 --- a/bin/chunk_pr/test_chunk_pr.py +++ b/bin/chunk_pr/test_chunk_pr.py @@ -377,6 +377,26 @@ class Merge(unittest.TestCase): self.assertEqual(merged(checkout, checkout.config, slept.append), "Merged pull request #12 (merge).") self.assertEqual(slept, [60]) + def test_a_merge_that_parks_returns_at_once_while_the_branch_protection_still_wants_a_review(self): + checkout = self.requiring_approval(GITHUB, "REVIEW_REQUIRED") + failed = merged(checkout, checkout.config, self.fail, parks=True) + self.assertEqual(failed.reason, "pull request 12 passed its checks and awaits an approval; end your run now, since do-epic parks the chunk and resumes this session once it is approved or a reviewer comments") + self.assertEqual(checkout.ran.count(GH_DECISION), 1) + self.assertEqual(step(["merge"], self.requiring_approval(GITHUB, "REVIEW_REQUIRED"), checkout.config, parks=True), failed) + + def test_a_merge_that_parks_reports_a_reviewers_unanswered_comment_before_the_approval_it_awaits(self): + checkout = self.requiring_approval(GITHUB, "REVIEW_REQUIRED") + checkout.threads = [MergeQueue.thread("bin/asked.py", "reviewer")] + failed = merged(checkout, checkout.config, self.fail, parks=True) + self.assertEqual(failed.reason, "a reviewer's comment on pull request 12 waits for an answer, in bin/asked.py; answer it in its thread, push any fix, and run chunk-pr merge again") + checkout = self.requiring_approval(GITHUB, "REVIEW_REQUIRED") + checkout.answers[tuple(GH_THREADS)] = Failure("gh api graphql failed") + self.assertEqual(merged(checkout, checkout.config, self.fail, parks=True), Failure("gh api graphql failed")) + + def test_a_merge_that_parks_merges_an_approved_pull_request(self): + checkout = self.requiring_approval() + self.assertEqual(merged(checkout, checkout.config, self.fail, parks=True), "Merged pull request #12 (merge).") + def test_an_approval_that_never_comes_leaves_it_unmerged_after_90_minutes(self): checkout = self.requiring_approval(GITHUB, "REVIEW_REQUIRED") slept: list[float] = [] diff --git a/bin/chunk_preflight/test_chunk_preflight.py b/bin/chunk_preflight/test_chunk_preflight.py index 545cede4..f94dfb76 100644 --- a/bin/chunk_preflight/test_chunk_preflight.py +++ b/bin/chunk_preflight/test_chunk_preflight.py @@ -302,8 +302,9 @@ class ChunkPreflightTool(unittest.TestCase): def test_reads_origin_as_last_fetched_where_the_fetch_stalls_and_still_states_the_other_facts(self): repo = Repo(self) - repo.stub_git(f'"fetch --quiet origin") exec {shutil.which("sleep")} 10;;') - done = repo.preflight("--worktree", CHUNK_PREFLIGHT_TIMEOUT="1") + repo.stub_git(f'"fetch --quiet origin") exec {shutil.which("sleep")} 30;;') + # The timeout also stops `reading-cost`, so it is set well above the seconds `reading-cost` takes on a loaded runner. + done = repo.preflight("--worktree", CHUNK_PREFLIGHT_TIMEOUT="5") self.assertEqual(done.returncode, 0, done.stderr) self.assertIn("as last fetched", done.stdout) self.assertIn("item 3, L 2, E + I 5", done.stdout) diff --git a/bin/claude_code/output_limits.py b/bin/claude_code/output_limits.py new file mode 100644 index 00000000..0d683b06 --- /dev/null +++ b/bin/claude_code/output_limits.py @@ -0,0 +1,10 @@ +"""The environment variables that cut what one tool result adds to a claude's context: the Bash ceiling loop-claude hands the worker's claude, and the Read limit that review-replay tests for it. Their values: docs/decisions.md โ†’ `loop-claude: the worker's claude saves a Bash result of more than 6,000 characters to a file, and rule-hook refuses a Read of a session transcript`. + +A subagent inherits both variables, so the ceiling reaches every reviewer the worker launches.""" + +# Claude Code saves a Bash result longer than this many characters to a file, and hands the model the file's path and the first 2,000 characters. The file sits under ~/.claude/projects///tool-results/. +BASH_CEILING = "BASH_MAX_OUTPUT_LENGTH" +BASH_CEILING_CHARACTERS = 6_000 +# A Read of more than this many tokens fails and asks for an offset and a limit; Claude Code's default is 25,000. +READ_LIMIT = "CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS" +READ_LIMIT_TOKENS = 5_000 diff --git a/bin/claude_code/session.py b/bin/claude_code/session.py index 0f8774da..075cba9b 100644 --- a/bin/claude_code/session.py +++ b/bin/claude_code/session.py @@ -52,5 +52,5 @@ def delete_session(session: Session) -> Failure | None: try: shutil.rmtree(sibling) if os.path.isdir(sibling) else os.remove(sibling) except OSError as failure: - return Failure(f"could not delete {sibling}: {failure.strerror or failure}; delete every file and folder matching {stem}*, then rerun loop-claude; a rerun before that resumes the finished chunk's session") + return Failure(f"could not delete {sibling}: {failure.strerror or failure}; delete every file and folder matching {stem}*, then run it again; a run before that resumes the finished chunk's session") return None diff --git a/bin/do-epic b/bin/do-epic new file mode 100755 index 00000000..9f0a3e71 --- /dev/null +++ b/bin/do-epic @@ -0,0 +1,13 @@ +#!/usr/bin/env python3 +import os +import pathlib +import sys + +LAUNCHER = pathlib.Path(__file__).resolve() +sys.path.insert(0, str(LAUNCHER.parent)) + +from loop_effects import effects +from do_epic import main +from loop_state import paths + +sys.exit(main.main(sys.argv[1:], os.environ, paths.paths_from(pathlib.Path.home()), LAUNCHER, effects.system_effects())) diff --git a/bin/do_epic/__init__.py b/bin/do_epic/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/bin/do_epic/backend/__init__.py b/bin/do_epic/backend/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/bin/do_epic/backend/contract.py b/bin/do_epic/backend/contract.py new file mode 100644 index 00000000..9786e2f0 --- /dev/null +++ b/bin/do_epic/backend/contract.py @@ -0,0 +1,213 @@ +"""What every backend of an epic's plan must hold, as tests. A backend's test case inherits EpicStoreContract and unittest.TestCase, and builds an epic with no issues in new_store. Its comments_on reads an issue's comments. Its assign, close, person_files, score and block change the tracker as a person or the plan store would.""" +import re +import typing + +from do_epic.backend.protocol import EpicIssue, EpicStore +from failure import Failure + + +class EpicStoreContract: + assertEqual: typing.Callable[..., None] + assertIsInstance: typing.Callable[..., None] + assertIsNone: typing.Callable[..., None] + subTest: typing.Callable[..., typing.ContextManager[None]] + + def new_store(self) -> EpicStore: + raise NotImplementedError + + def assign(self, store: EpicStore, key: str, name: str) -> None: + """Assigns the issue named key to the person whose display name is name.""" + raise NotImplementedError + + def close(self, store: EpicStore, key: str, status: str) -> None: + """Moves the issue named key to the done status named status.""" + raise NotImplementedError + + def comments_on(self, store: EpicStore, key: str) -> list[str]: + """The text of each comment on the issue named key, in the order posted, as markdown.""" + raise NotImplementedError + + def person_files(self, store: EpicStore, summary: str) -> str: + """Files a child issue of the epic as a person does in the tracker, not through the store, and answers its key.""" + raise NotImplementedError + + def score(self, store: EpicStore, key: str, scores: dict[str, object]) -> None: + """Sets the scores of the issue named key as `plan_store` writes them: each column's name with its value, a whole number for a score and text for Moves and Exempt, such as {"L": 2, "Moves": "1 by 5"}.""" + raise NotImplementedError + + def block(self, store: EpicStore, key: str, blocker: str) -> None: + """Makes the issue named key blocked by the issue named blocker, as a person would link them in the tracker.""" + raise NotImplementedError + + def filed(self, store: EpicStore, parent: str, summary: str, detail: str = "", labels: tuple[str, ...] = ()) -> str: + key = store.file(parent, summary, detail, labels) + assert isinstance(key, str), key + return key + + def issues(self, store: EpicStore) -> list[EpicIssue]: + read = store.read() + assert isinstance(read, list), read + return read + + def test_an_epic_with_no_issues_reads_empty(self) -> None: + self.assertEqual(self.new_store().read(), []) + + def test_reads_back_a_child_issue_it_filed(self) -> None: + store = self.new_store() + key = self.filed(store, store.epic_key, "Count the runs.", "Why it counts.\n\nA second paragraph.", ("security", "api")) + self.assertEqual(self.issues(store), [EpicIssue(key, "Count the runs.", "Why it counts.\n\nA second paragraph.", ("api", "security"), None, "To Do", False, store.epic_key, by_loop=True)]) + + def test_reads_back_a_sub_task_it_filed_after_its_parent(self) -> None: + store = self.new_store() + first = self.filed(store, store.epic_key, "Count the runs.") + second = self.filed(store, store.epic_key, "Ship the later thing.") + task = self.filed(store, first, "Count them in the api.", labels=("api",)) + read = self.issues(store) + self.assertEqual([(issue.key, issue.parent) for issue in read], [(first, store.epic_key), (task, first), (second, store.epic_key)]) + self.assertEqual((read[1].summary, read[1].labels, read[1].detail), ("Count them in the api.", ("api",), "")) + + def test_files_each_issue_under_its_own_key_in_the_epics_project(self) -> None: + store = self.new_store() + child = self.filed(store, store.epic_key, "Count the runs.") + keys = [child, self.filed(store, store.epic_key, "Count the runs."), self.filed(store, child, "Count them in the api.")] + self.assertEqual(len(set(keys)), 3) + project = store.epic_key.rpartition("-")[0] + self.assertEqual([re.fullmatch(f"{project}-[1-9][0-9]*", key) is not None for key in keys], [True] * 3) + + def test_reads_the_assignee_and_a_done_status(self) -> None: + store = self.new_store() + first = self.filed(store, store.epic_key, "Count the runs.") + second = self.filed(store, store.epic_key, "Ship the later thing.") + self.assign(store, first, "Ada Lovelace") + self.close(store, second, "Done") + read = {issue.key: issue for issue in self.issues(store)} + self.assertEqual((read[first].assignee, read[first].status, read[first].done), ("Ada Lovelace", "To Do", False)) + self.assertEqual((read[second].assignee, read[second].status, read[second].done), (None, "Done", True)) + + def test_refuses_to_file_under_a_sub_task(self) -> None: + store = self.new_store() + child = self.filed(store, store.epic_key, "Count the runs.") + task = self.filed(store, child, "Count them in the api.") + self.assertEqual(store.file(task, "Count more.", "", ()), Failure(f"{task} is neither the epic {store.epic_key} nor one of its child issues, so nothing can be filed under it")) + self.assertEqual(len(self.issues(store)), 2) + + def test_refuses_to_file_under_an_issue_outside_the_epic(self) -> None: + store = self.new_store() + for parent in ("ZZZ-1", "not a key"): + with self.subTest(parent=parent): + self.assertIsInstance(store.file(parent, "Count more.", "", ()), Failure) + self.assertEqual(self.issues(store), []) + + def test_refuses_a_label_with_white_space_and_a_summary_of_two_lines(self) -> None: + store = self.new_store() + self.assertEqual(store.file(store.epic_key, "Count the runs.", "", ("two words",)), Failure("the label 'two words' is empty or holds white space")) + self.assertEqual(store.file(store.epic_key, "Count\nthe runs.", "", ()), Failure("the summary must be one line of text")) + self.assertEqual(self.issues(store), []) + + def test_adds_a_label_to_a_child_issue_and_keeps_the_others(self) -> None: + store = self.new_store() + key = self.filed(store, store.epic_key, "Count the runs.", labels=("api",)) + other = self.filed(store, store.epic_key, "Ship the later thing.") + self.assertEqual([store.add_label(key, "repo:log"), store.add_label(key, "repo:log")], [None, None]) + read = {issue.key: issue for issue in self.issues(store)} + self.assertEqual((read[key].labels, read[other].labels), (("api", "repo:log"), ())) + + def test_refuses_to_label_a_sub_task_an_issue_outside_the_epic_or_a_label_with_white_space(self) -> None: + store = self.new_store() + child = self.filed(store, store.epic_key, "Count the runs.") + task = self.filed(store, child, "Count them in the api.") + for key, label, reason in ( + (task, "repo:log", f"{task} is not one of the child issues of the epic {store.epic_key}, so it cannot be labelled"), + (child, "two words", "the label 'two words' is empty or holds white space"), + ): + with self.subTest(key=key, label=label): + self.assertEqual(store.add_label(key, label), Failure(reason)) + self.assertIsInstance(store.add_label("ZZZ-1", "repo:log"), Failure) + self.assertEqual([issue.labels for issue in self.issues(store)], [(), ()]) + + def test_finishes_a_child_issue_so_it_reads_as_done_and_leaves_one_that_is_done_as_it_is(self) -> None: + store = self.new_store() + key = self.filed(store, store.epic_key, "Count the runs.") + other = self.filed(store, store.epic_key, "Ship the later thing.") + closed = self.filed(store, store.epic_key, "Close the old thing.") + self.close(store, closed, "Done") + self.assertEqual([store.finish_issue(key), store.finish_issue(closed)], [None, None]) + self.assertEqual({issue.key: issue.done for issue in self.issues(store)}, {key: True, other: False, closed: True}) + + def test_finishes_a_sub_task_of_a_child_issue_and_leaves_its_parent_open(self) -> None: + store = self.new_store() + child = self.filed(store, store.epic_key, "Count the runs.") + task = self.filed(store, child, "Count them in the api.") + self.assertIsNone(store.finish_issue(task)) + self.assertEqual({issue.key: issue.done for issue in self.issues(store)}, {child: False, task: True}) + + def test_refuses_to_finish_an_issue_outside_the_epic(self) -> None: + store = self.new_store() + child = self.filed(store, store.epic_key, "Count the runs.") + self.assertIsInstance(store.finish_issue("ZZZ-1"), Failure) + self.assertIsInstance(store.finish_issue("not a key"), Failure) + self.assertEqual([issue.done for issue in self.issues(store)], [False]) + + def test_posts_comments_on_a_child_issue_in_order(self) -> None: + store = self.new_store() + key = self.filed(store, store.epic_key, "Count the runs.") + other = self.filed(store, store.epic_key, "Ship the later thing.") + self.assertEqual([store.comment(key, "Split by repo."), store.comment(key, "Done in the api.")], [None, None]) + self.assertEqual((self.comments_on(store, key), self.comments_on(store, other)), (["Split by repo.", "Done in the api."], [])) + + def test_refuses_to_comment_on_a_sub_task_an_issue_outside_the_epic_or_with_no_words(self) -> None: + store = self.new_store() + child = self.filed(store, store.epic_key, "Count the runs.") + task = self.filed(store, child, "Count them in the api.") + self.assertEqual(store.comment(task, "Hello."), Failure(f"{task} is not one of the child issues of the epic {store.epic_key}, so it cannot be commented on")) + self.assertEqual(store.comment(child, " \n"), Failure("the comment must hold some text")) + self.assertIsInstance(store.comment("ZZZ-1", "Hello."), Failure) + self.assertEqual([self.comments_on(store, key) for key in (child, task)], [[], []]) + + def test_closes_the_epic_once_its_summary_is_posted_and_refuses_a_blank_one(self) -> None: + store = self.new_store() + self.assertEqual(store.closed(), False) + self.assertIsInstance(store.close(" \n"), Failure) + self.assertEqual((store.closed(), self.comments_on(store, "ABC-12")), (False, [])) + self.assertEqual(store.close("## Shipped\n\nThe import.\n"), None) + self.assertEqual((store.closed(), self.comments_on(store, "ABC-12")), (True, ["## Shipped\n\nThe import."])) + + def test_reads_an_issue_without_scores_or_blockers_as_one_with_none(self) -> None: + store = self.new_store() + key = self.filed(store, store.epic_key, "Count the runs.") + [issue] = self.issues(store) + self.assertEqual((issue.key, issue.scores, issue.moves, issue.exempt, issue.blockers), (key, {}, (), "", ())) + + def test_reads_the_scores_a_person_or_the_plan_store_set(self) -> None: + store = self.new_store() + key = self.filed(store, store.epic_key, "Count the runs.") + self.score(store, key, {"L": 2, "E": 3, "I": 4, "S": 3, "Moves": "1 by 5", "Exempt": "security"}) + [issue] = self.issues(store) + self.assertEqual((issue.scores, issue.moves, issue.exempt), ({"L": 2, "E": 3, "I": 4, "S": 3}, (("1", 5),), "security")) + + def test_reads_each_blocker_of_an_issue_and_none_for_a_blocker_of_another(self) -> None: + store = self.new_store() + first, second, third = (self.filed(store, store.epic_key, f"Item {n}.") for n in range(3)) + self.block(store, third, first) + self.block(store, third, second) + read = {issue.key: issue for issue in self.issues(store)} + self.assertEqual([read[key].blockers for key in (first, second, third)], [(), (), (first, second)]) + + def test_reads_whether_the_loop_filed_an_issue(self) -> None: + store = self.new_store() + ours = self.filed(store, store.epic_key, "Count the runs.") + theirs = self.person_files(store, "A person's issue.") + # The plan store scores an issue a person filed without making it the loop's. + self.score(store, theirs, {"L": 2, "E": 3, "I": 4, "S": 3, "Moves": "1 by 5"}) + task = self.filed(store, ours, "Count them in the api.") + read = {issue.key: issue for issue in self.issues(store)} + self.assertEqual([read[key].by_loop for key in (ours, theirs, task)], [True, False, True]) + + def test_ranks_a_child_issue_just_before_another_and_its_sub_tasks_follow_it(self) -> None: + store = self.new_store() + first, second, third = (self.filed(store, store.epic_key, f"Item {n}.") for n in range(3)) + task = self.filed(store, third, "A task.") + self.assertEqual(store.rank(third, first), None) + self.assertEqual([issue.key for issue in self.issues(store)], [third, task, first, second]) + self.assertEqual(store.rank(first, third), None) + self.assertEqual([issue.key for issue in self.issues(store)], [first, third, task, second]) diff --git a/bin/do_epic/backend/fake.py b/bin/do_epic/backend/fake.py new file mode 100644 index 00000000..4d2a3e01 --- /dev/null +++ b/bin/do_epic/backend/fake.py @@ -0,0 +1,89 @@ +"""An epic held in memory, for do-epic's tests, held to contract.py like every backend.""" +import dataclasses + +from do_epic.backend.protocol import EpicIssue, no_child, no_parent, no_words, unfit, unfit_label +from failure import Failure + + +@dataclasses.dataclass +class MemoryEpic: + """An epic whose issues rank in the order filed, until one is ranked. Where fails is set, every read, filing, label, comment, rank, issue finish and closing fails with it as the reason. comments holds each issue's comments, and the epic's, by key. done says whether the epic is closed.""" + epic_key: str + issues: list[EpicIssue] = dataclasses.field(default_factory=list) + comments: dict[str, list[str]] = dataclasses.field(default_factory=dict) + fails: str | None = None + done: bool = False + + def read(self) -> list[EpicIssue] | Failure: + if self.fails: + return Failure(self.fails) + return [issue for child in self.issues if child.parent == self.epic_key for issue in (child, *(task for task in self.issues if task.parent == child.key))] + + def file(self, parent: str, summary: str, detail: str, labels: tuple[str, ...]) -> str | Failure: + if self.fails: + return Failure(self.fails) + if refused := unfit(summary, labels): + return refused + if parent != self.epic_key and parent not in {issue.key for issue in self.issues if issue.parent == self.epic_key}: + return no_parent(self.epic_key, parent) + key = f"{self.epic_key.rpartition('-')[0]}-{len(self.issues) + 100}" + self.issues.append(EpicIssue(key, summary, detail.strip(), tuple(sorted(labels)), None, "To Do", False, parent, by_loop=True)) + return key + + def add_label(self, key: str, label: str) -> None | Failure: + if self.fails: + return Failure(self.fails) + if refused := unfit_label(label): + return refused + child = next((issue for issue in self.issues if issue.key == key and issue.parent == self.epic_key), None) + if child is None: + return no_child(self.epic_key, key, "labelled") + self.change(key, labels=tuple(sorted({*child.labels, label}))) + return None + + def rank(self, key: str, before: str) -> None | Failure: + if self.fails: + return Failure(self.fails) + children = {issue.key: issue for issue in self.issues if issue.parent == self.epic_key} + if stray := next((each for each in (key, before) if each not in children), None): + return Failure(f"{stray} is not one of the child issues of the epic {self.epic_key}, so {key} is not ranked") + self.issues.remove(children[key]) + self.issues.insert(self.issues.index(children[before]), children[key]) + return None + + def comment(self, key: str, text: str) -> None | Failure: + if self.fails: + return Failure(self.fails) + if refused := no_words(text): + return refused + if not any(issue.key == key and issue.parent == self.epic_key for issue in self.issues): + return no_child(self.epic_key, key, "commented on") + self.comments.setdefault(key, []).append(text.strip()) + return None + + def closed(self) -> bool | Failure: + return Failure(self.fails) if self.fails else self.done + + def close(self, summary: str) -> None | Failure: + if self.fails: + return Failure(self.fails) + if refused := no_words(summary): + return refused + self.comments.setdefault(self.epic_key, []).append(summary.strip()) + self.done = True + return None + + def finish_issue(self, key: str) -> None | Failure: + if self.fails: + return Failure(self.fails) + children = {issue.key for issue in self.issues if issue.parent == self.epic_key} + child = next((issue for issue in self.issues if issue.key == key and (issue.parent == self.epic_key or issue.parent in children)), None) + if child is None: + return no_child(self.epic_key, key, "finished") + if not child.done: + self.change(key, status="Done", done=True) + return None + + def change(self, key: str, **fields: object) -> None: + """Changes the issue named key as a person would in the tracker, such as assignee="Ada Lovelace".""" + self.issues = [dataclasses.replace(issue, **fields) if issue.key == key else issue for issue in self.issues] # type: ignore[arg-type] diff --git a/bin/do_epic/backend/jira.py b/bin/do_epic/backend/jira.py new file mode 100644 index 00000000..33cad807 --- /dev/null +++ b/bin/do_epic/backend/jira.py @@ -0,0 +1,267 @@ +"""An epic's plan kept as a Jira Cloud epic's child issues and their sub-tasks, read and filed over Jira's REST API version 3.""" +import dataclasses +import pathlib +import typing + +from adf_codec.convert import adfs_to_markdown, markdown_to_adf, Refused +from atlassian.client import basic_auth, Http, jira_json, ToAdf, ToMarkdown +from plan_store.jira_epic import blocked_by_links, DONE, FILED_BY, finishing_transition, ISSUE_KEY, ISSUE_TYPE, move_refusal, PERHAPS_MOVED, project_of, PROPERTY, rank_refusal, scores, searched +from project_config.settings import Epic, Finish, JIRA_FINISH_STATUS, JIRA_FINISH_TRANSITION +from do_epic.backend.protocol import EpicIssue, no_child, no_parent, no_words, unfit, unfit_label +from failure import Failure + +FIELDS = ["summary", "description", "labels", "assignee", "status", "parent", "issuelinks"] +# How many child issues one search for sub-tasks names, which keeps its JQL well inside the length Jira takes. +PARENTS_PER_SEARCH = 50 +NO_FINISH = "do-epic's settings name no transition to {to} by: set `{transition}` and `{status}` in ~/.claude-projects/do-epic/config.toml, to the names Jira shows, such as \"Mark as done\" and \"Done\"" +PERHAPS_FILED = "unless Jira refused the request, the issue may exist: look under {parent} before filing it again" + + +@dataclasses.dataclass(frozen=True) +class Found: + """An issue as Jira answered it, its description still in Jira's document format.""" + issue: EpicIssue + description: dict[str, typing.Any] | None + + +@dataclasses.dataclass(frozen=True) +class JiraEpic: + """The issues of a Jira epic, read and filed with the credentials env_file holds. finish says how the epic itself closes; None where the machine's settings don't say.""" + epic: Epic + env_file: pathlib.Path + http: Http = jira_json + to_markdown: ToMarkdown = adfs_to_markdown + to_adf: ToAdf = markdown_to_adf + finish: Finish | None = None + + @property + def epic_key(self) -> str: + return self.epic.key + + def read(self) -> list[EpicIssue] | Failure: + authorization = basic_auth(self.env_file) + if isinstance(authorization, Failure): + return authorization + found = self.found(authorization) + if isinstance(found, Failure): + return found + # A container run converts every description at once. It comes after every other check, since a read that fails doesn't need it. + described = [each for each in found if each.description is not None] + texts = self.to_markdown([each.description for each in described if each.description is not None]) + if isinstance(texts, Refused): + return Failure(f"{described[texts.index].issue.key}'s description: {texts.failure.reason}") + if isinstance(texts, Failure): + return Failure(f"cannot convert the descriptions of {self.epic_key}'s issues to markdown: {texts.reason}") + details = {each.issue.key: text.strip() for each, text in zip(described, texts)} + return [dataclasses.replace(each.issue, detail=details.get(each.issue.key, "")) for each in found] + + def file(self, parent: str, summary: str, detail: str, labels: tuple[str, ...]) -> str | Failure: + authorization = basic_auth(self.env_file) + if isinstance(authorization, Failure): + return authorization + if refused := unfit(summary, labels) or self.not_a_parent(parent, authorization): + return refused + fields: dict[str, typing.Any] = {"project": {"key": project_of(parent)}, "parent": {"key": parent}, "summary": summary, "labels": list(labels)} + # Marks the issue as the loop's, as plan_store marks the items it files. + properties = [{"key": PROPERTY, "value": {FILED_BY[0]: FILED_BY[1]}}] + kind = {"name": ISSUE_TYPE} if parent == self.epic_key else self.sub_task_type(project_of(parent), authorization) + if isinstance(kind, Failure): + return kind + if detail.strip(): + description = self.to_adf(detail) + if isinstance(description, Failure): + return Failure(f"cannot convert the detail to a Jira description: {description.reason}") + fields["description"] = description + created = self.http("POST", f"{self.epic.site}/rest/api/3/issue", authorization, {"fields": fields | {"issuetype": kind}, "properties": properties}) + if isinstance(created, Failure): + return Failure(f"cannot file the issue under {parent}: {created.reason}; {PERHAPS_FILED.format(parent=parent)}") + key = created.get("key") if isinstance(created, dict) else None + if not isinstance(key, str) or not ISSUE_KEY.fullmatch(key): + return Failure(f"Jira answered the new issue without an issue key; {PERHAPS_FILED.format(parent=parent)}") + return key + + def add_label(self, key: str, label: str) -> None | Failure: + authorization = basic_auth(self.env_file) + if isinstance(authorization, Failure): + return authorization + if refused := unfit_label(label): + return refused + above = self.parent_of(key, authorization, "to label it") + if isinstance(above, Failure): + return above + if above != self.epic_key: + return no_child(self.epic_key, key, "labelled") + edited = self.http("PUT", f"{self.epic.site}/rest/api/3/issue/{key}", authorization, {"update": {"labels": [{"add": label}]}}) + return Failure(f"cannot label {key}: {edited.reason}") if isinstance(edited, Failure) else None + + def rank(self, key: str, before: str) -> None | Failure: + authorization = basic_auth(self.env_file) + if isinstance(authorization, Failure): + return authorization + refused = rank_refusal(self.epic.site, key, before, authorization, self.http) + return Failure(refused) if refused else None + + def comment(self, key: str, text: str) -> None | Failure: + authorization = basic_auth(self.env_file) + if isinstance(authorization, Failure): + return authorization + if refused := no_words(text): + return refused + above = self.parent_of(key, authorization, "to comment on it") + if isinstance(above, Failure): + return above + if above != self.epic_key: + return no_child(self.epic_key, key, "commented on") + body = self.to_adf(text) + if isinstance(body, Failure): + return Failure(f"cannot convert the comment to a Jira comment: {body.reason}") + posted = self.http("POST", f"{self.epic.site}/rest/api/3/issue/{key}/comment", authorization, {"body": body}) + return Failure(f"cannot comment on {key}: {posted.reason}") if isinstance(posted, Failure) else None + + def closed(self) -> bool | Failure: + authorization = basic_auth(self.env_file) + if isinstance(authorization, Failure): + return authorization + answered = self.http("GET", f"{self.epic.site}/rest/api/3/issue/{self.epic_key}?fields=status", authorization, None) + if isinstance(answered, Failure): + return Failure(f"cannot read {self.epic_key}'s status: {answered.reason}") + try: + category = answered["fields"]["status"]["statusCategory"]["key"] # type: ignore[index] + except (KeyError, TypeError) as failure: + return Failure(f"{self.epic_key}'s status is not in the shape Jira Cloud's REST API version 3 documents: {failure!r}") + return bool(category == "done") + + def close(self, summary: str) -> None | Failure: + if self.finish is None: + return Failure(NO_FINISH.format(to="close an epic", transition=JIRA_FINISH_TRANSITION, status=JIRA_FINISH_STATUS)) + authorization = basic_auth(self.env_file) + if isinstance(authorization, Failure): + return authorization + if refused := no_words(summary): + return refused + body = self.to_adf(summary) + if isinstance(body, Failure): + return Failure(f"cannot convert the summary to a Jira comment: {body.reason}") + # The transition is read before the comment is posted, so an epic that cannot move gets no summary. + transition = finishing_transition(self.epic.site, self.epic_key, self.finish, authorization, self.http) + if isinstance(transition, Failure): + return transition + posted = self.http("POST", f"{self.epic.site}/rest/api/3/issue/{self.epic_key}/comment", authorization, {"body": body}) + if isinstance(posted, Failure): + return Failure(f"cannot comment on {self.epic_key}: {posted.reason}") + refused_move = move_refusal(self.epic.site, self.epic_key, transition, self.finish, authorization, self.http) + return Failure(f"posted the summary on {self.epic_key}, but {refused_move}; close the epic in Jira rather than posting the summary again") if refused_move else None + + def finish_issue(self, key: str) -> None | Failure: + authorization = basic_auth(self.env_file) + if isinstance(authorization, Failure): + return authorization + placed = self.placed(key, authorization, "to finish it") + if isinstance(placed, Failure): + return placed + above, done = placed + # A sub-task is no child of the epic but its parent is, so a sub-task's parent is read too. + grand = self.parent_of(above, authorization, "to finish its sub-task") if above not in (None, self.epic_key) else None + if isinstance(grand, Failure): + return grand + if above is None or (above != self.epic_key and grand != self.epic_key): + return no_child(self.epic_key, key, "finished") + if done: + return None + if self.finish is None: + return Failure(NO_FINISH.format(to="finish an issue", transition=JIRA_FINISH_TRANSITION, status=JIRA_FINISH_STATUS)) + transition = finishing_transition(self.epic.site, key, self.finish, authorization, self.http) + if isinstance(transition, Failure): + return transition + refused = move_refusal(self.epic.site, key, transition, self.finish, authorization, self.http) + return Failure(f"{refused}; {PERHAPS_MOVED.format(key=key)}") if refused else None + + def not_a_parent(self, parent: str, authorization: str) -> Failure | None: + """Why nothing can be filed under parent; None where it is the epic or one of its child issues.""" + if parent == self.epic_key: + return None + above = self.parent_of(parent, authorization, "to file under it") + return above if isinstance(above, Failure) else None if above == self.epic_key else no_parent(self.epic_key, parent) + + def parent_of(self, key: str, authorization: str, why: str) -> str | None | Failure: + """The key of key's parent issue; None where key isn't an issue key or has no parent. A failed read says "cannot read ".""" + placed = self.placed(key, authorization, why) + return placed if isinstance(placed, Failure) else placed[0] + + def placed(self, key: str, authorization: str, why: str) -> tuple[str | None, bool] | Failure: + """The key of key's parent issue, None where key isn't an issue key or has no parent, and whether key's own status is a done one. A failed read says "cannot read ".""" + if not ISSUE_KEY.fullmatch(key): + return None, False + # A search may not show an issue filed a moment ago, so the issue is asked for by its key. + answered = self.http("GET", f"{self.epic.site}/rest/api/3/issue/{key}?fields=parent,status", authorization, None) + if isinstance(answered, Failure): + return Failure(f"cannot read {key} {why}: {answered.reason}") + fields = answered.get("fields") if isinstance(answered, dict) else None + above = fields.get("parent") if isinstance(fields, dict) else None + status = fields.get("status") if isinstance(fields, dict) else None + category = status.get("statusCategory") if isinstance(status, dict) else None + return (above.get("key") if isinstance(above, dict) and isinstance(above.get("key"), str) else None), isinstance(category, dict) and category.get("key") == DONE + + def found(self, authorization: str) -> list[Found] | Failure: + """Each child issue, each followed by its sub-tasks.""" + # The search answers no issues for an epic the account cannot see; asking for the epic itself tells that from an epic without children. + seen = self.http("GET", f"{self.epic.site}/rest/api/3/issue/{self.epic_key}?fields=summary", authorization, None) + if isinstance(seen, Failure): + return Failure(f"cannot read the epic {self.epic_key}: {seen.reason}") + children = self.searched(f"{self.epic_key}'s child issues", (self.epic_key,), authorization) + if isinstance(children, Failure): + return children + keys = [each.issue.key for each in children] + tasks: list[Found] = [] + for start in range(0, len(keys), PARENTS_PER_SEARCH): + batch = self.searched(f"the sub-tasks of {self.epic_key}'s child issues", tuple(keys[start:start + PARENTS_PER_SEARCH]), authorization) + if isinstance(batch, Failure): + return batch + tasks += batch + return [each for child in children for each in (child, *(task for task in tasks if task.issue.parent == child.issue.key))] + + def searched(self, what: str, parents: tuple[str, ...], authorization: str) -> list[Found] | Failure: + """The issues whose parent is one of parents, in rank order.""" + named = ", ".join(f'"{key}"' for key in parents) + answered = searched(self.epic.site, what, f"parent in ({named}) ORDER BY Rank ASC", {"fields": FIELDS, "properties": [PROPERTY]}, authorization, self.http) + if isinstance(answered, Failure): + return answered + found = [found_of(raw) for raw in answered] + return next((each for each in found if isinstance(each, Failure)), None) or [each for each in found if isinstance(each, Found)] + + def sub_task_type(self, project: str, authorization: str) -> dict[str, str] | Failure: + """The project's sub-task issue type, by id, since sites name it differently, such as Subtask or Sub-task.""" + answered = self.http("GET", f"{self.epic.site}/rest/api/3/issue/createmeta/{project}/issuetypes", authorization, None) + if isinstance(answered, Failure): + return Failure(f"cannot read {project}'s issue types: {answered.reason}") + types = answered.get("issueTypes") if isinstance(answered, dict) else None + if not isinstance(types, list): + return Failure(f"Jira answered {project}'s issue types in a shape its REST API version 3 doesn't document") + found = next((kind["id"] for kind in types if isinstance(kind, dict) and kind.get("subtask") is True and isinstance(kind.get("id"), str)), None) + return Failure(f"{project} has no sub-task issue type; turn sub-tasks on in Jira") if found is None else {"id": found} + + +def found_of(raw: object) -> Found | Failure: + """The issue in one entry of a search's answer, or which issue is not in the shape Jira documents.""" + key = raw.get("key") if isinstance(raw, dict) else None + if not isinstance(raw, dict) or not isinstance(key, str) or not ISSUE_KEY.fullmatch(key): + return Failure("the search answered an issue without an issue key") + # The answer's nested fields are read as Jira documents them; one missing or of another type ends the read here. + try: + fields = raw["fields"] + summary, labels, assignee, status, parent, description = fields["summary"], fields["labels"], fields["assignee"], fields["status"], fields["parent"]["key"], fields["description"] + name = assignee["displayName"] if assignee is not None else None + state, done = status["name"], status["statusCategory"]["key"] == "done" + blockers = tuple(link.blocker for link in blocked_by_links(fields)) + held = raw.get("properties", {}).get(PROPERTY, {}) + except (AttributeError, KeyError, TypeError) as failure: + return Failure(f"{key} is not in the shape Jira Cloud's REST API version 3 documents: {failure!r}") + texts = (summary, *labels, state, parent, *blockers) if isinstance(labels, list) else () + if not texts or not all(isinstance(text, str) for text in texts) or not isinstance(name, str | None) or not all(ISSUE_KEY.fullmatch(each) for each in (parent, *blockers)) or not isinstance(description, dict | None): + return Failure(f"{key} is not in the shape Jira Cloud's REST API version 3 documents: its summary, labels, assignee, status, parent, description or a linked issue's key holds another kind of value") + if not isinstance(held, dict): + return Failure(f"{key}'s issue property {PROPERTY} is not a JSON object") + read = scores(key, held) + if isinstance(read, Failure): + return read + return Found(EpicIssue(key, summary.strip(), "", tuple(sorted(labels)), name, state, done, parent, read.scores, read.moves, read.exempt, blockers, held.get(FILED_BY[0]) == FILED_BY[1]), description) diff --git a/bin/do_epic/backend/protocol.py b/bin/do_epic/backend/protocol.py new file mode 100644 index 00000000..516b2300 --- /dev/null +++ b/bin/do_epic/backend/protocol.py @@ -0,0 +1,103 @@ +"""What every backend of an epic's plan reads, files, labels, comments on and ranks: an EpicIssue, and the EpicStore protocol they all meet. do-epic reads an epic only through EpicStore, so another tracker can stand behind it; contract.py holds the tests every backend passes. + +EpicStore is not plan_store's PlanStore. A PlanStore item names a release, but an epic item has none (`docs/decisions.md` โ†’ `do-epic: drives one Jira epic โ€ฆ`), and it may be a sub-task.""" +import dataclasses +import re +import typing + +from plan.todo_table import Exempt, Moves +from failure import Failure + +# What a tracker accepts as a label: Jira refuses a label holding white space. +LABEL = re.compile(r"\S+") + + +@dataclasses.dataclass(frozen=True) +class EpicIssue: + """One of an epic's child issues, or a sub-task of one, as a read found it.""" + key: str + summary: str + # The description, as markdown; empty where the issue has none. + detail: str + # Sorted by name. + labels: tuple[str, ...] + # The assignee's display name; None where nobody is assigned. + assignee: str | None + # The status's name, such as "In Progress". + status: str + # Whether the status is one the tracker counts as done. + done: bool + # The epic's key for a child issue, the child's key for a sub-task. + parent: str + # The score of each column the issue has one for, such as {"L": 2}, as `plan_store` reads them from the issue's `plan-item` property. Empty where the issue has no scores. + scores: dict[str, int] = dataclasses.field(default_factory=dict) + # The goals the issue moves, and how far (the plan's Moves cell); empty where it names none. + moves: Moves = () + # Why the issue is exempt from the plan's score bar, as the plan's Exempt cell names it; empty where it isn't. + exempt: Exempt = "" + # The key of each issue that blocks this one, in the tracker's order. + blockers: tuple[str, ...] = () + # True where the loop filed the issue through EpicStore.file; False for an issue a person filed. + by_loop: bool = False + + +class EpicStore(typing.Protocol): + """The issues of one epic. A write that fails doesn't file anything, except where its Failure says the tracker may have taken it all the same.""" + + @property + def epic_key(self) -> str: + """The parent of each child issue.""" + + def read(self) -> list[EpicIssue] | Failure: + """Each child issue, open or done, in the tracker's rank order, each followed by its sub-tasks in theirs.""" + + def file(self, parent: str, summary: str, detail: str, labels: tuple[str, ...]) -> str | Failure: + """Files an open issue under parent, marked as filed by the loop, and answers its key: a child issue where parent is the epic, a sub-task where parent is one of its child issues. detail is markdown, and may be empty.""" + + def add_label(self, key: str, label: str) -> None | Failure: + """Adds label to the epic's child issue named key, leaving its other labels as they are; adding one it already has leaves the labels as they are. A sub-task, or an issue outside the epic, is refused.""" + + def rank(self, key: str, before: str) -> None | Failure: + """Ranks the child issue named key just before the child issue named before. The sub-tasks of key follow key. Pass only keys of the epic's child issues, as read answers them: some backends don't check, and Jira ranks any issue it is given.""" + + def comment(self, key: str, text: str) -> None | Failure: + """Posts text, markdown, as a comment on the epic's child issue named key, for the people who work the epic to read. A sub-task, an issue outside the epic, or an empty or blank text is refused.""" + + def finish_issue(self, key: str) -> None | Failure: + """Moves the epic's child issue named key, or a sub-task of one, to a done status, so that read answers it as done; one already done stays as it is. An issue outside the epic is refused. A Failure says why the move could not be made: the issue's workflow gives no way to a done status, gives several the backend cannot choose between, or refuses the move.""" + + + def closed(self) -> bool | Failure: + """Whether the epic itself is in a status the tracker counts as done.""" + + def close(self, summary: str) -> None | Failure: + """Posts summary, markdown, as a comment on the epic, then moves the epic to a status the tracker counts as done. Before anything is posted, it refuses a blank summary, and an epic the tracker cannot move to such a status. Where the move fails after the comment is posted, the Failure says so.""" + + +def unfit(summary: str, labels: tuple[str, ...]) -> Failure | None: + """Why no tracker can file an issue with this summary and these labels; None where a tracker can file it.""" + if not summary.strip() or "\n" in summary: + return Failure("the summary must be one line of text") + return unfit_label(*labels) + + +def unfit_label(*labels: str) -> Failure | None: + """Why no tracker can take one of these labels; None where a tracker can take every one.""" + if bad := next((label for label in labels if not LABEL.fullmatch(label)), None): + return Failure(f"the label {bad!r} is empty or holds white space") + return None + + +def no_parent(epic_key: str, parent: str) -> Failure: + """What every backend says when asked to file under parent, which is neither the epic nor one of its child issues.""" + return Failure(f"{parent} is neither the epic {epic_key} nor one of its child issues, so nothing can be filed under it") + + +def no_child(epic_key: str, key: str, done_to_it: str) -> Failure: + """What every backend says when asked to change key, which is not one of the epic's child issues; done_to_it is the past participle of what was asked, such as "labelled".""" + return Failure(f"{key} is not one of the child issues of the epic {epic_key}, so it cannot be {done_to_it}") + + +def no_words(text: str) -> Failure | None: + """Why no tracker can post this comment; None where one can.""" + return None if text.strip() else Failure("the comment must hold some text") diff --git a/bin/do_epic/backend/test_fake.py b/bin/do_epic/backend/test_fake.py new file mode 100644 index 00000000..46703529 --- /dev/null +++ b/bin/do_epic/backend/test_fake.py @@ -0,0 +1,69 @@ +import pathlib +import sys +import unittest + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2])) + +from do_epic.backend.contract import EpicStoreContract +from failure import Failure +from plan_store import jira_epic +from do_epic.backend.fake import MemoryEpic + + +class MemoryEpicContract(EpicStoreContract, unittest.TestCase): + def new_store(self) -> MemoryEpic: + return MemoryEpic("ABC-12") + + def assign(self, store: MemoryEpic, key: str, name: str) -> None: # type: ignore[override] + store.change(key, assignee=name) + + def comments_on(self, store: MemoryEpic, key: str) -> list[str]: # type: ignore[override] + return store.comments.get(key, []) + + def close(self, store: MemoryEpic, key: str, status: str) -> None: # type: ignore[override] + store.change(key, status=status, done=True) + + def person_files(self, store: MemoryEpic, summary: str) -> str: # type: ignore[override] + key = self.filed(store, store.epic_key, summary) + store.change(key, by_loop=False) + return key + + def score(self, store: MemoryEpic, key: str, scores: dict[str, object]) -> None: # type: ignore[override] + read = jira_epic.scores(key, scores) + assert not isinstance(read, Failure), read + store.change(key, scores=read.scores, moves=read.moves, exempt=read.exempt) + + def block(self, store: MemoryEpic, key: str, blocker: str) -> None: # type: ignore[override] + store.change(key, blockers=(*next(issue for issue in store.issues if issue.key == key).blockers, blocker)) + + +class Failing(unittest.TestCase): + def test_every_read_filing_label_comment_and_rank_fails_with_the_reason_given(self) -> None: + store = MemoryEpic("ABC-12", fails="Jira is down") + down = Failure("Jira is down") + self.assertEqual((store.read(), store.file("ABC-12", "Count the runs.", "", ()), store.add_label("ABC-100", "api"), store.comment("ABC-100", "Hello."), store.rank("ABC-100", "ABC-101")), (down,) * 5) + + +class Ranking(unittest.TestCase): + def test_refuses_to_rank_beside_an_issue_that_isnt_a_child_of_the_epic(self) -> None: + store = MemoryEpic("ABC-12") + child = store.file("ABC-12", "Count the runs.", "", ()) + self.assertEqual(store.rank(str(child), "ZZZ-1"), Failure(f"ZZZ-1 is not one of the child issues of the epic ABC-12, so {child} is not ranked")) + + +class Finishing(unittest.TestCase): + def test_finishes_an_issue_to_the_status_done(self) -> None: + store = MemoryEpic("ABC-12") + key = store.file("ABC-12", "Count the runs.", "", ()) + self.assertIsNone(store.finish_issue(str(key))) + self.assertEqual([(issue.status, issue.done) for issue in store.issues], [("Done", True)]) + + def test_refuses_to_finish_an_issue_outside_the_epic_saying_so(self) -> None: + self.assertEqual(MemoryEpic("ABC-12").finish_issue("ZZZ-1"), Failure("ZZZ-1 is not one of the child issues of the epic ABC-12, so it cannot be finished")) + + def test_fails_to_finish_with_the_reason_given(self) -> None: + self.assertEqual(MemoryEpic("ABC-12", fails="Jira is down").finish_issue("ABC-100"), Failure("Jira is down")) + + +if __name__ == "__main__": + unittest.main() diff --git a/bin/do_epic/backend/test_jira.py b/bin/do_epic/backend/test_jira.py new file mode 100644 index 00000000..762ab246 --- /dev/null +++ b/bin/do_epic/backend/test_jira.py @@ -0,0 +1,421 @@ +import base64 +import dataclasses +import pathlib +import re +import sys +import tempfile +import unittest +import unittest.mock + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2])) + +from project_config.settings import Epic, Finish +from do_epic.backend.contract import EpicStoreContract +from failure import Failure +from plan_store import jira_epic +from do_epic.backend.jira import JiraEpic +from adf_codec.convert import Refused + +SITE = "https://acme.atlassian.net" +EPIC = Epic(SITE, "ABC-12") +AUTHORIZATION = "Basic " + base64.b64encode(b"me@example.com:s3cret").decode() +EPIC_URL = f"{SITE}/rest/api/3/issue/ABC-12?fields=summary" +ISSUE_TYPES = f"{SITE}/rest/api/3/issue/createmeta/ABC/issuetypes" +SEARCH = f"{SITE}/rest/api/3/search/jql" +ISSUES = f"{SITE}/rest/api/3/issue" +RANK = f"{SITE}/rest/agile/1.0/issue/rank" +TO_DO = {"name": "To Do", "statusCategory": {"key": "new"}} +DONE = {"name": "Done", "statusCategory": {"key": "done"}} +TRANSITIONS = f"{ISSUES}/ABC-12/transitions" +EPIC_COMMENT = f"{ISSUES}/ABC-12/comment" +EPIC_STATUS = f"{ISSUES}/ABC-12?fields=status" +TYPES = {"issueTypes": [{"id": "10001", "name": "Task", "subtask": False}, {"id": "10003", "name": "Subtask", "subtask": True}]} + + +def to_adf(markdown: str) -> dict[str, object]: + """Stands in for the adf-codec shim: one paragraph holding the markdown.""" + return {"type": "doc", "version": 1, "content": [{"type": "paragraph", "content": [{"type": "text", "text": markdown}]}]} + + +def to_markdown(documents: list[dict[str, object]]) -> list[str]: + return [document["content"][0]["content"][0]["text"] + "\n" for document in documents] # type: ignore[index] + + +@dataclasses.dataclass +class FakeJira: + """Jira Cloud's REST API as JiraEpic calls it. It holds issues as Jira answers them, in rank order. A search answers two issues a page, with only the fields it asks for. A call whose method and URL are in answers gets the answer held there instead.""" + issues: list[dict[str, object]] = dataclasses.field(default_factory=list) + answers: dict[tuple[str, str], object] = dataclasses.field(default_factory=dict) + calls: list[tuple[str, str]] = dataclasses.field(default_factory=list) + comments: dict[str, list[object]] = dataclasses.field(default_factory=dict) + # The ID of each transition made. + transitioned: list[str] = dataclasses.field(default_factory=list) + # The transitions Jira offers an issue that isn't done. + workflow: list[dict[str, object]] = dataclasses.field(default_factory=lambda: [{"id": "31", "name": "Mark as done", "to": DONE}]) + # The epic's own status, which the transition "Mark as done" moves to DONE. + epic_status: dict[str, object] = dataclasses.field(default_factory=lambda: dict(TO_DO)) + + def __call__(self, method: str, url: str, authorization: str, body: object) -> object: + self.calls.append((method, url)) + if authorization != AUTHORIZATION: + return Failure("Jira answered 401") + if (method, url) in self.answers: + return self.answers[method, url] + if method == "GET" and url == EPIC_URL: + return {"key": "ABC-12"} + if method == "GET" and url == EPIC_STATUS: + return {"key": "ABC-12", "fields": {"status": self.epic_status}} + if method == "GET" and url == TRANSITIONS: + return {"transitions": [] if self.epic_status == DONE else [{"id": "31", "name": "Mark as done", "to": DONE}, {"id": "21", "name": "Start", "to": {"name": "In Progress", "statusCategory": {"key": "indeterminate"}}}]} + if method == "POST" and url == TRANSITIONS: + assert body == {"transition": {"id": "31"}}, body + self.epic_status = DONE + return None + if method == "POST" and url == EPIC_COMMENT: + assert isinstance(body, dict) + self.comments.setdefault("ABC-12", []).append(body["body"]) + return {"id": "1"} + if method == "GET" and url == ISSUE_TYPES: + return TYPES + if method == "POST" and url == SEARCH: + return self.page(body) + if method == "GET" and (held := next((issue for issue in self.issues if url.startswith(f"{ISSUES}/{issue['key']}?fields=")), None)): + asked = url.partition("?fields=")[2].split(",") + return {"key": held["key"], "fields": {name: held["fields"][name] for name in asked}} # type: ignore[index] + if method == "GET" and (held := next((issue for issue in self.issues if url == f"{ISSUES}/{issue['key']}/transitions"), None)): + return {"transitions": [] if held["fields"]["status"] == DONE else self.workflow} # type: ignore[index] + if method == "POST" and (held := next((issue for issue in self.issues if url == f"{ISSUES}/{issue['key']}/transitions"), None)): + assert isinstance(body, dict) + held["fields"]["status"] = DONE # type: ignore[index] + self.transitioned.append(body["transition"]["id"]) + return None + if method == "POST" and url == ISSUES: + return self.created(body) + if method == "PUT" and url == RANK: + return self.ranked(body) + if method == "POST" and (held := next((issue for issue in self.issues if url == f"{ISSUES}/{issue['key']}/comment"), None)): + assert isinstance(body, dict) + self.comments.setdefault(str(held["key"]), []).append(body["body"]) + return {"id": "1"} + if method == "PUT" and (held := next((issue for issue in self.issues if url == f"{ISSUES}/{issue['key']}"), None)): + assert isinstance(body, dict) + added = [change["add"] for change in body["update"]["labels"]] + held["fields"]["labels"] = sorted({*held["fields"]["labels"], *added}) # type: ignore[index] + return None + return Failure(f"Jira answered 404 for {method} {url}") + + def page(self, body: object) -> object: + assert isinstance(body, dict) + matched = re.fullmatch(r'parent in \(("[A-Z]+-[0-9]+"(?:, "[A-Z]+-[0-9]+")*)\) ORDER BY Rank ASC', body["jql"]) + if matched is None: + return Failure("Jira answered 400: the JQL is not valid") + parents = re.findall(r'"([A-Z]+-[0-9]+)"', matched[1]) + found = [{"key": issue["key"], "fields": {name: value for name, value in issue["fields"].items() if name in body.get("fields", [])}, "properties": {name: value for name, value in issue.get("properties", {}).items() if name in body.get("properties", [])}} for issue in self.issues if issue["fields"]["parent"]["key"] in parents] # type: ignore[index, attr-defined] + start = int(body.get("nextPageToken", 0)) + return {"issues": found[start:start + 2]} | ({"nextPageToken": str(start + 2)} if start + 2 < len(found) else {}) + + def created(self, body: object) -> object: + assert isinstance(body, dict) + fields = body["fields"] + parent = fields["parent"]["key"] + kind = {"name": "Task"} if parent == "ABC-12" else {"id": "10003"} if any(issue["key"] == parent for issue in self.issues) else None + if fields["project"] != {"key": "ABC"} or kind is None or fields["issuetype"] != kind: + return Failure("Jira answered 400: the project, parent or issue type is not valid") + key = f"ABC-{len(self.issues) + 100}" + self.issues.append({"key": key, "fields": {"summary": fields["summary"], "description": fields.get("description"), "labels": fields["labels"], "assignee": None, "status": TO_DO, "parent": {"key": parent}, "issuelinks": []}, "properties": {entry["key"]: entry["value"] for entry in body["properties"]}}) + return {"id": "1", "key": key} + + def ranked(self, body: object) -> object: + """Moves the issue the body names to just before the other issue it names, and answers no body, as Jira does for a rank it made.""" + assert isinstance(body, dict) + [key], before = body["issues"], body["rankBeforeIssue"] + moved = next(issue for issue in self.issues if issue["key"] == key) + self.issues.remove(moved) + self.issues.insert(next(place for place, issue in enumerate(self.issues) if issue["key"] == before), moved) + return None + + def change(self, key: str, **fields: object) -> None: + next(issue for issue in self.issues if issue["key"] == key)["fields"].update(fields) # type: ignore[union-attr] + + def person_files(self, summary: str) -> str: + """Files a child issue as the Jira website does: with no issue property.""" + key = f"ABC-{len(self.issues) + 100}" + self.issues.append({"key": key, "fields": {"summary": summary, "description": None, "labels": [], "assignee": None, "status": TO_DO, "parent": {"key": "ABC-12"}, "issuelinks": []}}) + return key + + def set_property(self, key: str, value: dict[str, object]) -> None: + next(issue for issue in self.issues if issue["key"] == key).setdefault("properties", {})["plan-item"] = value # type: ignore[attr-defined, union-attr] + + +class Case(unittest.TestCase): + def setUp(self) -> None: + folder = tempfile.TemporaryDirectory() + self.addCleanup(folder.cleanup) + self.env_file = pathlib.Path(folder.name) / "jira.env" + self.env_file.write_text("ATLASSIAN_EMAIL=me@example.com\nATLASSIAN_API_TOKEN=s3cret\n") + self.env_file.chmod(0o600) + self.jira = FakeJira() + + def new_store(self) -> JiraEpic: + return JiraEpic(EPIC, self.env_file, self.jira, to_markdown, to_adf, Finish("Mark as done", "Done")) + + +class JiraEpicContract(Case, EpicStoreContract): + def assign(self, store: JiraEpic, key: str, name: str) -> None: # type: ignore[override] + self.jira.change(key, assignee={"accountId": "5b10a2844c20165700ede21g", "displayName": name}) + + def comments_on(self, store: JiraEpic, key: str) -> list[str]: # type: ignore[override] + return [to_markdown([document])[0].strip() for document in self.jira.comments.get(key, [])] # type: ignore[misc] + + def close(self, store: JiraEpic, key: str, status: str) -> None: # type: ignore[override] + self.jira.change(key, status={"name": status, "statusCategory": {"key": "done"}}) + + def person_files(self, store: JiraEpic, summary: str) -> str: # type: ignore[override] + return self.jira.person_files(summary) + + def score(self, store: JiraEpic, key: str, scores: dict[str, object]) -> None: # type: ignore[override] + held = next(issue for issue in self.jira.issues if issue["key"] == key).get("properties", {}).get("plan-item", {}) # type: ignore[attr-defined, union-attr] + self.jira.set_property(key, held | scores) + + def block(self, store: JiraEpic, key: str, blocker: str) -> None: # type: ignore[override] + links = next(issue for issue in self.jira.issues if issue["key"] == key)["fields"]["issuelinks"] # type: ignore[index] + links.append({"id": str(len(links) + 1), "type": {"inward": "is blocked by", "outward": "blocks"}, "inwardIssue": {"key": blocker}}) + + +class Jira(Case): + def test_closes_the_epic_by_the_transition_the_settings_name_after_the_summary(self) -> None: + self.assertEqual(self.new_store().close("Shipped."), None) + self.assertEqual(self.jira.calls[-3:], [("GET", TRANSITIONS), ("POST", EPIC_COMMENT), ("POST", TRANSITIONS)]) + + def test_posts_no_summary_on_an_epic_it_cannot_close(self) -> None: + unnamed = JiraEpic(EPIC, self.env_file, self.jira, to_markdown, to_adf) + wrong_status = JiraEpic(EPIC, self.env_file, self.jira, to_markdown, to_adf, Finish("Start", "Done")) + no_such = JiraEpic(EPIC, self.env_file, self.jira, to_markdown, to_adf, Finish("Close", "Done")) + unconverted = JiraEpic(EPIC, self.env_file, self.jira, to_markdown, lambda markdown: Failure("adf-codec refused it"), Finish("Mark as done", "Done")) + for store, reason in ( + (unnamed, "do-epic's settings name no transition to close an epic by: set `jira-finish-transition` and `jira-finish-status` in ~/.claude-projects/do-epic/config.toml, to the names Jira shows, such as \"Mark as done\" and \"Done\""), + (wrong_status, "ABC-12's transition 'Start' leads to the status 'In Progress', not to 'Done', which `jira-finish-status` names"), + (unconverted, "cannot convert the summary to a Jira comment: adf-codec refused it"), + ): + self.assertEqual(store.close("Shipped."), Failure(reason)) + refused = no_such.close("Shipped.") + assert isinstance(refused, Failure) + self.assertIn("ABC-12 has no transition 'Close' from its status", refused.reason) + self.jira.answers[("GET", TRANSITIONS)] = Failure("Jira answered 503") + self.assertEqual(self.new_store().close("Shipped."), Failure("cannot read ABC-12's transitions: Jira answered 503")) + self.assertEqual((self.jira.comments, self.jira.epic_status), ({}, TO_DO)) + + def test_says_what_was_posted_where_the_comment_or_the_move_fails(self) -> None: + self.jira.answers[("POST", EPIC_COMMENT)] = Failure("Jira answered 503") + self.assertEqual(self.new_store().close("Shipped."), Failure("cannot comment on ABC-12: Jira answered 503")) + self.assertEqual(self.jira.epic_status, TO_DO) + del self.jira.answers[("POST", EPIC_COMMENT)] + self.jira.answers[("POST", TRANSITIONS)] = Failure("Jira answered 503") + self.assertEqual(self.new_store().close("Shipped."), Failure("posted the summary on ABC-12, but cannot move ABC-12 by 'Mark as done': Jira answered 503; close the epic in Jira rather than posting the summary again")) + + def test_says_why_the_epics_status_cannot_be_read(self) -> None: + for answer, reason in ((Failure("Jira answered 503"), "cannot read ABC-12's status: Jira answered 503"), ({"fields": {}}, "ABC-12's status is not in the shape Jira Cloud's REST API version 3 documents: KeyError('status')"), (["ABC-12"], "ABC-12's status is not in the shape Jira Cloud's REST API version 3 documents: TypeError('list indices must be integers or slices, not str')")): + self.jira.answers[("GET", EPIC_STATUS)] = answer + self.assertEqual(self.new_store().closed(), Failure(reason)) + + def test_finishes_an_issue_by_the_transition_the_settings_name(self) -> None: + store = self.new_store() + key = store.file("ABC-12", "Count the runs.", "", ()) + self.jira.workflow = [{"id": "11", "name": "Start", "to": {"name": "In Progress", "statusCategory": {"key": "indeterminate"}}}, {"id": "31", "name": "Mark as done", "to": DONE}] + self.assertIsNone(store.finish_issue(key)) + self.assertEqual(self.jira.calls[-1], ("POST", f"{ISSUES}/{key}/transitions")) + self.assertEqual((self.jira.transitioned, self.jira.issues[0]["fields"]["status"]), (["31"], DONE)) # type: ignore[index] + + def test_finishes_by_the_named_transition_where_several_lead_to_a_done_status(self) -> None: + key = self.new_store().file("ABC-12", "Count the runs.", "", ()) + self.jira.workflow = [{"id": "31", "name": "Resolve", "to": DONE}, {"id": "41", "name": "Close", "to": {"name": "Closed", "statusCategory": {"key": "done"}}}] + self.assertIsNone(JiraEpic(EPIC, self.env_file, self.jira, to_markdown, to_adf, Finish("Close", "Closed")).finish_issue(key)) + self.assertEqual(self.jira.transitioned, ["41"]) + + def test_leaves_an_issue_that_is_done_without_asking_for_its_transitions(self) -> None: + store = JiraEpic(EPIC, self.env_file, self.jira, to_markdown, to_adf) + key = self.new_store().file("ABC-12", "Count the runs.", "", ()) + self.jira.change(key, status=DONE) + self.assertIsNone(store.finish_issue(key)) + self.assertNotIn(("GET", f"{ISSUES}/{key}/transitions"), self.jira.calls) + + def test_a_finish_names_why_no_move_finishes_the_issue(self) -> None: + key = self.new_store().file("ABC-12", "Count the runs.", "", ()) + unnamed = JiraEpic(EPIC, self.env_file, self.jira, to_markdown, to_adf) + wrong_status = JiraEpic(EPIC, self.env_file, self.jira, to_markdown, to_adf, Finish("Mark as done", "Closed")) + no_such = JiraEpic(EPIC, self.env_file, self.jira, to_markdown, to_adf, Finish("Close", "Done")) + self.assertEqual(unnamed.finish_issue(key), Failure("do-epic's settings name no transition to finish an issue by: set `jira-finish-transition` and `jira-finish-status` in ~/.claude-projects/do-epic/config.toml, to the names Jira shows, such as \"Mark as done\" and \"Done\"")) + self.assertEqual(wrong_status.finish_issue(key), Failure(f"{key}'s transition 'Mark as done' leads to the status 'Done', not to 'Closed', which `jira-finish-status` names")) + refused = no_such.finish_issue(key) + assert isinstance(refused, Failure) + self.assertIn(f"{key} has no transition 'Close' from its status", refused.reason) + self.assertEqual((self.jira.transitioned, self.jira.issues[0]["fields"]["status"]), ([], TO_DO)) # type: ignore[index] + + def test_refuses_to_finish_a_sub_task_of_an_issue_that_is_no_child_of_the_epic_or_an_issue_without_a_parent(self) -> None: + store = self.new_store() + for key, parent in (("ABC-50", "ABC-60"), ("ABC-60", "ABC-61"), ("ABC-61", None)): + fields = {"summary": "Other.", "description": None, "labels": [], "assignee": None, "status": TO_DO, "parent": {"key": parent} if parent else None, "issuelinks": []} + self.jira.issues.append({"key": key, "fields": fields}) + for key in ("ABC-50", "ABC-61"): + with self.subTest(key=key): + self.assertEqual(store.finish_issue(key), Failure(f"{key} is not one of the child issues of the epic ABC-12, so it cannot be finished")) + self.assertNotIn("POST", [method for method, _ in self.jira.calls]) + + def test_a_finish_jira_refuses_says_so_and_that_the_issue_may_have_moved(self) -> None: + store = self.new_store() + key = store.file("ABC-12", "Count the runs.", "", ()) + self.jira.answers[("POST", f"{ISSUES}/{key}/transitions")] = Failure("Jira answered 400: Resolution is required") + self.assertEqual(store.finish_issue(key), Failure(f"cannot move {key} by 'Mark as done': Jira answered 400: Resolution is required; unless Jira refused the request, the issue may have moved: look at {key} before finishing the item again")) + + def test_a_finish_whose_transitions_jira_cannot_give_or_gives_in_another_shape_fails(self) -> None: + store = self.new_store() + key = store.file("ABC-12", "Count the runs.", "", ()) + url = ("GET", f"{ISSUES}/{key}/transitions") + self.jira.answers[url] = Failure("Jira answered 503") + self.assertEqual(store.finish_issue(key), Failure(f"cannot read {key}'s transitions: Jira answered 503")) + for answer in ({"transitions": [{"id": 31}]}, ["Mark as done"], {"transitions": 5}): + with self.subTest(answer=answer): + self.jira.answers[url] = answer + self.assertEqual(store.finish_issue(key), Failure(f"Jira answered {key}'s transitions in a shape other than the one it documents")) + + def test_a_finish_whose_issue_or_parent_jira_cannot_read_says_which(self) -> None: + store = self.new_store() + child = store.file("ABC-12", "Count the runs.", "", ()) + task = store.file(child, "Count them in the api.", "", ()) + self.jira.answers[("GET", f"{ISSUES}/{task}?fields=parent,status")] = Failure("Jira answered 503") + self.assertEqual(store.finish_issue(task), Failure(f"cannot read {task} to finish it: Jira answered 503")) + del self.jira.answers[("GET", f"{ISSUES}/{task}?fields=parent,status")] + self.jira.answers[("GET", f"{ISSUES}/{child}?fields=parent,status")] = Failure("Jira answered 503") + self.assertEqual(store.finish_issue(task), Failure(f"cannot read {child} to finish its sub-task: Jira answered 503")) + + def test_reads_the_sub_tasks_of_more_children_than_one_search_names(self) -> None: + store = self.new_store() + children = [store.file("ABC-12", f"Item {n}.", "", ()) for n in range(60)] + tasks = [store.file(child, "A task.", "", ()) for child in children] + read = store.read() + assert isinstance(read, list), read + self.assertEqual([issue.key for issue in read], [key for pair in zip(children, tasks) for key in pair]) + + def test_files_a_child_without_asking_for_issue_types(self) -> None: + self.new_store().file("ABC-12", "Count the runs.", "", ()) + self.assertNotIn(("GET", ISSUE_TYPES), self.jira.calls) + + def test_a_filing_jira_fails_or_answers_without_a_key_says_the_issue_may_exist(self) -> None: + for answer, reason in ((Failure("Jira answered 503"), "cannot file the issue under ABC-12: Jira answered 503"), ({"id": "1"}, "Jira answered the new issue without an issue key"), (["ABC-100"], "Jira answered the new issue without an issue key")): + with self.subTest(answer=answer): + self.jira.answers[("POST", ISSUES)] = answer + self.assertEqual(self.new_store().file("ABC-12", "Count the runs.", "", ()), Failure(f"{reason}; unless Jira refused the request, the issue may exist: look under ABC-12 before filing it again")) + + def test_a_detail_the_codec_refuses_leaves_the_epic_as_it_was(self) -> None: + store = JiraEpic(EPIC, self.env_file, self.jira, to_markdown, lambda markdown: Failure("adf-codec refused it")) + self.assertEqual(store.file("ABC-12", "Count the runs.", "Why.", ()), Failure("cannot convert the detail to a Jira description: adf-codec refused it")) + self.assertEqual(self.jira.issues, []) + + def test_files_no_sub_task_where_the_project_has_no_sub_task_type_or_its_types_cannot_be_read(self) -> None: + store = self.new_store() + child = store.file("ABC-12", "Count the runs.", "", ()) + for answer, reason in ((Failure("Jira answered 403"), "cannot read ABC's issue types: Jira answered 403"), ({"issueTypes": [{"id": "10001", "subtask": False}]}, "ABC has no sub-task issue type; turn sub-tasks on in Jira"), ({}, "Jira answered ABC's issue types in a shape its REST API version 3 doesn't document"), ([], "Jira answered ABC's issue types in a shape its REST API version 3 doesn't document")): + with self.subTest(answer=answer): + self.jira.answers[("GET", ISSUE_TYPES)] = answer + self.assertEqual(store.file(str(child), "A task.", "", ()), Failure(reason)) + self.assertEqual(len(self.jira.issues), 1) + + def test_an_issue_in_another_shape_fails_the_read(self) -> None: + valid = {"summary": "Count the runs.", "description": None, "labels": [], "assignee": None, "status": TO_DO, "parent": {"key": "ABC-12"}, "issuelinks": []} + linked = {"id": "1", "type": {"inward": "is blocked by"}, "inwardIssue": {"key": "ABC-2"}} + shape = "ABC-34 is not in the shape Jira Cloud's REST API version 3 documents" + for raw, reason in ( + ("ABC-34", "the search answered an issue without an issue key"), + ({"fields": valid}, "the search answered an issue without an issue key"), + ({"key": "abc-34", "fields": valid}, "the search answered an issue without an issue key"), + ({"key": "ABC-34", "fields": {"summary": "Count the runs.", "parent": {"key": "ABC-12"}}}, shape + ": KeyError('labels')"), + *((({"key": "ABC-34", "fields": valid | changed}, shape + ": its summary, labels, assignee, status, parent, description or a linked issue's key holds another kind of value") for changed in ( + {"summary": 3}, {"labels": "api"}, {"labels": [3]}, {"assignee": {"displayName": 0}}, {"status": {"name": 0, "statusCategory": {"key": "new"}}}, {"parent": {"key": "ABC"}}, {"parent": {"key": 12}}, {"description": "text"}, {"issuelinks": [linked | {"inwardIssue": {"key": 7}}]}, {"issuelinks": [linked | {"inwardIssue": {"key": "ABC-x"}}]}, + ))), + ({"key": "ABC-34", "fields": {key: value for key, value in valid.items() if key != "issuelinks"}}, shape + ": KeyError('issuelinks')"), + ({"key": "ABC-34", "fields": valid, "properties": {"plan-item": "L 2"}}, "ABC-34's issue property plan-item is not a JSON object"), + ({"key": "ABC-34", "fields": valid, "properties": {"plan-item": {"L": "many"}}}, "item ABC-34's L 'many' is not a whole number from 1 to 4; ABC-34 holds it in its issue property plan-item"), + ({"key": "ABC-34", "fields": valid, "properties": None}, shape + ": AttributeError(\"'NoneType' object has no attribute 'get'\")"), + ): + with self.subTest(raw=raw): + self.jira.answers[("POST", SEARCH)] = {"issues": [raw]} + self.assertEqual(self.new_store().read(), Failure(reason)) + + def test_a_description_the_codec_cannot_convert_fails_the_read(self) -> None: + self.new_store().file("ABC-12", "Count the runs.", "Why.", ()) + for converted, reason in ((Refused(0, Failure("a table cell holds a list")), "ABC-100's description: a table cell holds a list"), (Failure("docker is not running"), "cannot convert the descriptions of ABC-12's issues to markdown: docker is not running")): + with self.subTest(converted=converted): + self.assertEqual(JiraEpic(EPIC, self.env_file, self.jira, lambda documents: converted, to_adf).read(), Failure(reason)) + + def test_a_search_that_does_not_end_names_the_issues_it_sought(self) -> None: + store = self.new_store() + children = [store.file("ABC-12", "Count the runs.", "", ()) for _ in range(2)] + for _ in range(3): + store.file(str(children[0]), "A task.", "", ()) + with unittest.mock.patch.object(jira_epic, "MAX_PAGES", 1): + self.assertEqual(store.read(), Failure("the search for the sub-tasks of ABC-12's child issues did not end within 1 pages")) + store.file("ABC-12", "Count the runs.", "", ()) + self.assertEqual(store.read(), Failure("the search for ABC-12's child issues did not end within 1 pages")) + + def test_refuses_to_file_under_another_epics_child_or_an_issue_it_cannot_read(self) -> None: + self.jira.issues.append({"key": "ABC-5", "fields": {"parent": {"key": "ABC-1"}, "status": TO_DO}}) + store = self.new_store() + self.assertEqual(store.file("ABC-5", "Count more.", "", ()), Failure("ABC-5 is neither the epic ABC-12 nor one of its child issues, so nothing can be filed under it")) + self.assertEqual(store.file("ABC-77", "Count more.", "", ()), Failure(f"cannot read ABC-77 to file under it: Jira answered 404 for GET {ISSUES}/ABC-77?fields=parent,status")) + self.jira.answers[("GET", f"{ISSUES}/ABC-78?fields=parent,status")] = ["ABC-12"] + self.assertEqual(store.file("ABC-78", "Count more.", "", ()), Failure("ABC-78 is neither the epic ABC-12 nor one of its child issues, so nothing can be filed under it")) + self.assertEqual([call for call in self.jira.calls if call[0] == "POST" and call[1] == ISSUES], []) + + def test_a_label_jira_refuses_is_not_added(self) -> None: + store = self.new_store() + key = store.file("ABC-12", "Count the runs.", "", ()) + self.jira.answers[("PUT", f"{ISSUES}/{key}")] = Failure("Jira answered 403") + self.assertEqual(store.add_label(str(key), "repo:log"), Failure(f"cannot label {key}: Jira answered 403")) + read = store.read() + assert isinstance(read, list), read + self.assertEqual([issue.labels for issue in read], [()]) + + def test_a_comment_jira_refuses_or_the_codec_cannot_convert_is_not_posted(self) -> None: + store = self.new_store() + key = store.file("ABC-12", "Count the runs.", "", ()) + assert isinstance(key, str) + self.jira.answers["POST", f"{ISSUES}/{key}/comment"] = Failure("Jira answered 403") + self.assertEqual(store.comment(key, "Hello."), Failure(f"cannot comment on {key}: Jira answered 403")) + refusing = JiraEpic(EPIC, self.env_file, self.jira, to_markdown, lambda text: Failure("not markdown")) + self.assertEqual(refusing.comment(key, "Hello."), Failure("cannot convert the comment to a Jira comment: not markdown")) + + def test_an_issue_jira_cannot_read_is_not_commented_on(self) -> None: + store = self.new_store() + key = store.file("ABC-12", "Count the runs.", "", ()) + self.jira.answers[("GET", f"{ISSUES}/{key}?fields=parent,status")] = Failure("Jira answered 500") + self.assertEqual(store.comment(str(key), "Hello."), Failure(f"cannot read {key} to comment on it: Jira answered 500")) + self.assertNotIn(("POST", f"{ISSUES}/{key}/comment"), self.jira.calls) + + def test_an_issue_jira_cannot_read_is_not_labelled(self) -> None: + store = self.new_store() + key = store.file("ABC-12", "Count the runs.", "", ()) + self.jira.answers[("GET", f"{ISSUES}/{key}?fields=parent,status")] = Failure("Jira answered 500") + self.assertEqual(store.add_label(str(key), "repo:log"), Failure(f"cannot read {key} to label it: Jira answered 500")) + self.assertNotIn(("PUT", f"{ISSUES}/{key}"), self.jira.calls) + + def test_an_epic_the_account_cannot_see_fails_the_read(self) -> None: + store = JiraEpic(Epic(SITE, "ABC-99"), self.env_file, self.jira, to_markdown, to_adf) + self.assertEqual(store.read(), Failure(f"cannot read the epic ABC-99: Jira answered 404 for GET {SITE}/rest/api/3/issue/ABC-99?fields=summary")) + + def test_a_rank_jira_refuses_fails_with_its_reason_and_ranks_nothing(self) -> None: + store = self.new_store() + first, second = (store.file("ABC-12", f"Item {n}.", "", ()) for n in range(2)) + for answer, reason in ( + (Failure("Jira answered 403"), "Jira answered 403"), + ({"entries": [{"issueKey": second, "status": 400, "errors": ["Rank is not configured"]}]}, "Rank is not configured"), + ): + with self.subTest(answer=answer): + self.jira.answers[("PUT", RANK)] = answer + self.assertEqual(store.rank(str(second), str(first)), Failure(reason)) + read = store.read() + assert isinstance(read, list), read + self.assertEqual([issue.key for issue in read], [first, second]) + + +if __name__ == "__main__": + unittest.main() diff --git a/bin/do_epic/chunk_prompt.py b/bin/do_epic/chunk_prompt.py new file mode 100644 index 00000000..cea698dc --- /dev/null +++ b/bin/do_epic/chunk_prompt.py @@ -0,0 +1,66 @@ +"""What a do-epic cycle's worker is told: its task, from the issue, and its first prompt. The resumed prompt and the rung sentences are the ones loop-claude's cycles use, naming do-epic; a chunk that was parked is first told why it resumes, and one whose task follows the epic's state is told that task as it resumes.""" +import dataclasses + +from do_epic.backend.protocol import EpicIssue +from do_epic import findings +from loop_claude.project.records import HUMAN_RULE_NAMES, RECORD_FILES +from text_files.markdown_text import listed +import pathlib +from project_config.settings import ProjectConfig +from model_routing.ladder import ProjectRung +from loop_claude.cycle import prompts +from model_routing.route import Route +from agent_cycle.worker_prompts import WorkerPrompts + +# The entry point's name as the worker, the judge and the human read it. +DRIVER = "do-epic" + + +def shipping(config: ProjectConfig) -> str: + """How a chunk in the repo that config names ships its change. An epic's issue isn't an item of the repo's plan, so `chunk-pr merge --finish`, which adds a finished item's changelog bullet, doesn't apply: the worker writes the bullet itself, where the repo publishes something others consume (docs/decisions.md โ†’ `do-epic: drives one Jira epic โ€ฆ`, "Releases").""" + opening = "The issue isn't an item of the repo's plan, so run `chunk-pr merge` without `--finish`." + if config.epic: + return f"{opening} The repo's plan is a Jira epic, and the history of its issues serves as the changelog, so leave the changelog as it is." + changelog = config.paths["changelog"].relative_to(config.top) if "changelog" in config.paths and config.paths["changelog"].is_relative_to(config.top) else config.places["changelog"] + return f"{opening} Where the repo publishes something others consume, add one bullet to `## Unreleased` in `{changelog}` that tells the repo's consumers what changed; otherwise leave the changelog as it is." + + +def task(issue: EpicIssue, epic: str, checkout: str | None, launcher: pathlib.Path, language: str, config: ProjectConfig | None) -> str: + """The task of the chunk that works issue, of the epic with the key epic: in checkout, or without a repo where it is None. launcher is the do-epic that runs the chunk, language the one the epic's Jira text is in, and config the project config of the repo's checkout, None without a repo.""" + shipped = shipping(config) if config else "" + where = f"in the checkout {checkout}, on the branch {issue.key}" if checkout else "which changes no repo: work in the folder you start in" + return f"Work the Jira issue {issue.key} of the epic {epic} as one chunk, {where}. {" ".join(filter(None, [findings.instructions(epic, launcher, language), shipped]))} Its summary: {issue.summary}" + (f"\n\nIts description:\n\n{issue.detail}" if issue.detail else "") + + +CODE_HEALTH = '~/.claude/CLAUDE.md โ†’ "Shared goal"' + + +def fresh(task: str, premises: str, route: Route, project: ProjectRung | None, facts: str) -> str: + """A chunk's first prompt. facts is empty: do-epic runs no preflight.""" + routed = f"{DRIVER} runs you on {route.rung}, routed by the issue's L of {route.likelihood}. Where a sentence below names item {route.item}, it means the issue." if route.item else "" + opening = " ".join(filter(None, [routed, *prompts.rung_sentences(DRIVER, route, project)])) + return f"{task}\n\nContinue any open draft PR or worktree for the chunk before creating a new one. {prompts.READING_RULE}\n\n{opening and f'{opening}{chr(10)}{chr(10)}'}The issue sets the chunk's scope. The goals below choose between ways to deliver that scope, and never widen it. The epic's goals rank first, then code health ({CODE_HEALTH}), then the repo's.\n\nFrom {premises}" + + +# The worker names no plan items: the chunk's one item is its issue. +# The epic's records come first, as its goals do; a chunk without a repo is handed only those and the human's rules, so the names that mention a repo's then name nothing it reads. +NAMES = prompts.RecordNames( + quote_sources=listed(("the epic's description", "the repo's goals file's Goals and Audience", *RECORD_FILES, *HUMAN_RULE_NAMES), "or"), + answer_destinations="the epic's description under ## Decisions, the repo's goals file's Goals or Audience, " + ", ".join(RECORD_FILES) + ", the repo's plan or a comment", + records_named="the epic's scoring and decisions, then the repo's working rules and decisions, then the human's rules for every project", +) +WORKER = prompts.worker_prompts(DRIVER, fresh, "", NAMES) + +# What a worker does with a reviewer's feedback, as the decision asks (docs/decisions.md โ†’ `do-epic: drives one Jira epic โ€ฆ`, "Long gates"). +FEEDBACK = 'Where a reviewer left feedback, fix what is reasonable and push the fix, answer each other comment in its thread, and change the setup so our review would have caught the finding (~/.claude/CLAUDE.md โ†’ "Working loop", "Change the setup for each finding an outside reviewer raises on a PR, deterministic tools first"). Then merge the pull request as before.' + + +def unparked(news: str) -> WorkerPrompts: + """The prompts of the first cycle after a parked chunk resumes: WORKER's, with each prompt that starts a worker opening with news, which says what the chunk waited on and what changed. The fresh prompt opens with it too, since a chunk the judge ruled finished before it parked has no session left to resume.""" + opening = f"{news} {FEEDBACK}\n\n" + return dataclasses.replace(WORKER, fresh=lambda *task: opening + WORKER.fresh(*task), resumed=lambda *evidence: opening + WORKER.resumed(*evidence)) + + +def restating(task: str) -> WorkerPrompts: + """WORKER for a chunk whose task follows the epic's state, as its finishing's does: the prompt that resumes its worker states task, which may not be the one its session began with.""" + return dataclasses.replace(WORKER, resumed=lambda *evidence: f"{WORKER.resumed(*evidence)}\n\nThe task follows the epic's state, so it may have changed since you began. It is now:\n\n{task}") diff --git a/bin/do_epic/config.py b/bin/do_epic/config.py new file mode 100644 index 00000000..04f4ed74 --- /dev/null +++ b/bin/do_epic/config.py @@ -0,0 +1,87 @@ +"""do-epic's machine-wide settings: `~/.claude-projects/do-epic/config.toml`, and `repo-rule.md` beside it. + +One pair per machine, the same for every epic (docs/decisions.md โ†’ `do-epic: drives one Jira epic โ€ฆ`). The config is TOML: + + jira-url = "https://acme.atlassian.net" + env-file = "~/.secrets/acme-jira.env" + repoless-model = "claude-opus-5-5" + clone-folder = "~/code" + language = "Swedish" + +- `jira-url` is the Jira site's address: https, a host, and no path. +- `env-file` is the file of Jira credentials: a path that starts with / or ~/, outside every folder of `~/.claude-projects` and outside the clone folder, which hold git checkouts. `~/.claude-projects/.env` is fine. +- `repoless-model` is the model that works an epic item with no repo: a model on the routing ladder, `model_routing/routing-ladder.json`. +- `clone-folder` is the folder a missing repo is cloned into: a path that starts with / or ~/. `~/code` where left out. +- `language` is the default language of Jira text, by its English name. English where left out. +- `jira-finish-transition` and `jira-finish-status` name the workflow transition that moves a finished chunk's issue and closes a finished epic, such as "Mark as done", and the status it leads to, such as "Done". A project's config names a plan item's the same way. Set both or neither; without them do-epic moves no issue to done and closes no epic. +- `repo-rule.md` is prose that says how to tell an item's repo, and mustn't be blank.""" +import dataclasses +import pathlib +import re + +from project_config.settings import anywhere, CONFIG, DO_EPIC, ENV_FILE, Finish, finish_faults, finish_of, JIRA_FINISH, LANGUAGE, language_fault, lies_in, parsed, PROJECTS, SITE +from failure import Failure +from text_files.markdown_text import listed +from model_routing.ladder_file import read_ladder, SHIPPED + +REPO_RULE = "repo-rule.md" +# Three readers, seated as the human who writes this config, picked these spellings from four candidates each, 3-0: `jira-url` over `site`, `clone-folder` over `repos-folder` (which reads as the folder of repos that already exist) and `repoless-model` over `no-repo-model`. `env-file` and `language` keep the names a project's config gives them. +JIRA_URL = "jira-url" +REPOLESS_MODEL = "repoless-model" +CLONE_FOLDER = "clone-folder" +SETTINGS = (JIRA_URL, ENV_FILE, REPOLESS_MODEL) +OPTIONAL = (CLONE_FOLDER, LANGUAGE, *JIRA_FINISH) +LISTED = f"The settings are {', '.join(SETTINGS)}, and optionally {listed(OPTIONAL, 'and')}." + + +@dataclasses.dataclass(frozen=True) +class DoEpicConfig: + """folder is `~/.claude-projects/do-epic`; repo_rule is `repo-rule.md`'s text.""" + folder: pathlib.Path + jira_url: str + env_file: pathlib.Path + clone_folder: pathlib.Path + language: str + repoless_model: str + repo_rule: str + # How a finished epic closes; None where the config doesn't say. + finish: Finish | None = None + + +def load(home: pathlib.Path, ladder_file: pathlib.Path = SHIPPED) -> DoEpicConfig | Failure: + """The settings and the repo rule under home; a Failure naming the file it can't read, else each missing or misspelt setting it finds, so that do-epic refuses to start.""" + folder = home / PROJECTS / DO_EPIC + values = parsed(folder / CONFIG) + if isinstance(values, Failure): + return Failure(f"{values.reason}. {LISTED}") + ladder = read_ladder(ladder_file) + if isinstance(ladder, Failure): + return ladder + models = sorted({rung.model for rung in ladder.rungs}) + rule: str | Failure + try: + rule = (folder / REPO_RULE).read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError) as failure: + rule = Failure(f"cannot read {folder / REPO_RULE}: {failure}") + config_folders = [path for path in (home / PROJECTS).iterdir() if path.is_dir()] + clone_folder = anywhere(values.get(CLONE_FOLDER, "~/code"), home) + env_file = anywhere(values.get(ENV_FILE), home) + language = values.get(LANGUAGE, "English") + url, model = values.get(JIRA_URL), values.get(REPOLESS_MODEL) + faults = [ + *(f"`{name}` is missing" for name in SETTINGS if name not in values), + *(f"`{name}` is not a setting" for name in values if name not in (*SETTINGS, *OPTIONAL)), + *([f"`{JIRA_URL}` is {url!r}, not a site's https address without a path, such as \"https://acme.atlassian.net\""] if JIRA_URL in values and not (isinstance(url, str) and re.fullmatch(SITE, url)) else []), + *([f"`{CLONE_FOLDER}` is {values[CLONE_FOLDER]!r}, not a path that starts with / or ~/"] if clone_folder is None else []), + *([f"`{ENV_FILE}` is {values[ENV_FILE]!r}, not a path that starts with / or ~/"] if ENV_FILE in values and env_file is None else []), + *([f"`{ENV_FILE}` is {values[ENV_FILE]!r}, inside a folder of ~/{PROJECTS} or the clone folder, which hold git checkouts"] if env_file and lies_in(env_file, *config_folders, *([clone_folder] if clone_folder else [])) else []), + *([f"`{LANGUAGE}` is {language!r}, {fault}"] if LANGUAGE in values and (fault := language_fault(language)) else []), + *([f"`{REPOLESS_MODEL}` is {model!r}, not a model on the routing ladder: {', '.join(models)}"] if REPOLESS_MODEL in values and model not in models else []), + *finish_faults(values), + *([rule.reason] if isinstance(rule, Failure) else [f"`{REPO_RULE}` is blank"] if not rule.strip() else []), + ] + if faults: + return Failure(f"{folder}: {'; '.join(faults)}. {LISTED}") + # Without a fault, each value above is present and of the type asserted here. + assert clone_folder and env_file and isinstance(url, str) and isinstance(model, str) and isinstance(language, str) and isinstance(rule, str) + return DoEpicConfig(folder, url, env_file, clone_folder, language, model, rule, finish_of(values)) diff --git a/bin/do_epic/epic_record.py b/bin/do_epic/epic_record.py new file mode 100644 index 00000000..736e1bc8 --- /dev/null +++ b/bin/do_epic/epic_record.py @@ -0,0 +1,158 @@ +"""A Jira epic's description as the record of do-epic, the tool that drives one epic. + +The description holds a language line, a line that caps the chunks running at once, and the sections Goals, Audience, Scoring (the formula and the bar) and Decisions. They are read and written one at a time through `EpicDescription`, and a write leaves every other line of the markdown as it was. Goals and Audience change only where the human approved in an interrupt, which `write_section` takes as `approved`; `EpicDescription` refuses the change otherwise.""" +import dataclasses +import re +import typing + +from record_store.jira_description import EpicDescription +from failure import Failure +from text_files.markdown_text import fenced_lines +from record_store.held import Held +from project_config.settings import language_fault + +SECTIONS = ("Goals", "Audience", "Scoring", "Decisions") +LANGUAGE_LINE = re.compile(r"^\*\*Language:\*\* (.*?)\s*$") +# The line that caps how many of the epic's chunks run at once. Three readers, seated as the human who writes an epic's description, picked this spelling from four candidates, 2-1 over `**Parallel chunks:**`, which reads as a count of what runs now. +CAP = "**Max parallel chunks:**" +CAP_LINE = re.compile(rf"^{re.escape(CAP)}(.*?)\s*$") +# A line that opens with CAP's words in another spelling. It is refused, since the epic would otherwise run on the cap of 1 without a word. +NEAR_CAP_LINE = re.compile(r"^[\s*_>#-]*max parallel chunks", re.IGNORECASE) + + +def read_section(text: str, name: str) -> str | Failure: + """The text under the one ## name heading, up to the next heading of level 1 or 2, without the blank lines around it; empty where there is no such heading.""" + span = span_of(text, name) + if isinstance(span, Failure): + return span + return "" if span is None else "\n".join(text.splitlines()[span[0] + 1 : span[1]]).strip("\n") + + +def with_section(text: str, name: str, body: str) -> str | Failure: + """text with the one ## name section holding body, put after the others where there is none. A section that already holds body leaves text as it is.""" + span = span_of(text, name) + if isinstance(span, Failure): + return span + if any(line.startswith(("# ", "## ")) and not fenced for line, fenced in fenced_lines(body)): + return Failure(f"the body of ## {name} holds a level 1 or 2 heading, which would start another section") + if span is None: + return text.rstrip("\n") + "\n\n" * bool(text.strip()) + f"## {name}\n\n{body.strip()}\n" + lines = text.splitlines() + if read_section(text, name) == body.strip(): + return text + last = span[1] == len(lines) + return "\n".join([*lines[: span[0] + 1], "", *body.strip().splitlines(), *([] if last else [""]), *lines[span[1] :]]) + "\n" + + +def span_of(text: str, name: str) -> tuple[int, int] | None | Failure: + """The index of the one ## name heading's line and of the line after its section; None where there is no such heading. A heading inside a code fence is ignored.""" + if name not in SECTIONS: + return Failure(f"{name} is not a section of an epic's description; the sections are {', '.join(SECTIONS)}") + lines = fenced_lines(text) + headings = [index for index, (line, fenced) in enumerate(lines) if not fenced and (line.startswith("# ") or line.startswith("## "))] + starts = [index for index in headings if lines[index][0].startswith("## ") and lines[index][0][3:].strip() == name] + if len(starts) > 1: + return Failure(f"the epic's description has ## {name} more than once") + return (starts[0], next((index for index in headings if index > starts[0]), len(lines))) if starts else None + + +def language_in(text: str) -> str | None: + """The language the description's language line names; None where it has none.""" + for line, fenced in fenced_lines(text): + if not fenced and (found := LANGUAGE_LINE.match(line)): + return found.group(1) + return None + + +def parallel_chunks_in(text: str) -> int | Failure: + """How many of the epic's chunks may run at once, as the description's CAP line says; 1 where it has none, as a project's `parallel-chunks` is where its config leaves it out.""" + for line, fenced in fenced_lines(text): + if not fenced and (found := CAP_LINE.match(line)): + value = found.group(1).strip() + if re.fullmatch(r"[1-9][0-9]*", value): + return int(value) + return Failure(f"the epic's description caps its chunks with `{CAP} {value}`, which is not a whole number of 1 or more, such as `{CAP} 2`") + if not fenced and NEAR_CAP_LINE.match(line): + return Failure(f"the epic's description holds the line `{line.strip()}`, which do-epic doesn't read as the epic's cap: write it as a line of its own, spelt `{CAP} 2`") + return 1 + + +def with_language(text: str, language: str) -> str | Failure: + """text with its language line naming language, put first where there is none.""" + if fault := language_fault(language): + return Failure(f"cannot write the language {language!r}: {fault}") + line = f"**Language:** {language}" + lines = fenced_lines(text) + for index, (old, fenced) in enumerate(lines): + if not fenced and LANGUAGE_LINE.match(old): + return "\n".join([*(old for old, _ in lines[:index]), line, *(old for old, _ in lines[index + 1 :])]) + "\n" + return line + "\n" + "\n" * bool(text.strip()) + text + + +def decision_titled(decisions: str, title: str) -> str | None: + """The entry under the ### title heading of decisions, a ## Decisions section's body; None where it has no such heading.""" + found: list[str] | None = None + for line, fenced in fenced_lines(decisions): + if not fenced and line.startswith("### "): + if found is not None: + break + found = [] if line[4:].strip() == title else None + elif found is not None: + found.append(line) + return None if found is None else "\n".join(found).strip() + + +@dataclasses.dataclass(frozen=True) +class EpicRecord: + """One epic's record, kept in its description.""" + description: EpicDescription + + def section(self, name: str) -> str | Failure: + text = self.description.read() + return text if isinstance(text, Failure) else read_section(text, name) + + def language(self) -> str | None | Failure: + text = self.description.read() + return text if isinstance(text, Failure) else language_in(text) + + def parallel_chunks(self) -> int | Failure: + text = self.description.read() + return text if isinstance(text, Failure) else parallel_chunks_in(text) + + def write_section(self, name: str, body: str, approved: bool = False) -> Held | Failure: + """Writes body as the ## name section. approved says the human approved the change in an interrupt, which Goals and Audience need.""" + return self.written(lambda text: with_section(text, name, body), frozenset({name}) if approved else frozenset()) + + def append_decision(self, title: str, entry: str) -> bool | Failure: + """Adds entry under a ### title heading at the end of ## Decisions, leaving the decisions before it as they are. False where the section already holds that entry under that title, so recording a decision twice keeps one; a Failure where it holds another entry under it, so one title names one decision.""" + title, entry = title.strip(), entry.strip() + if not title or "\n" in title: + return Failure("the title must be one line of text") + if not entry: + return Failure("pipe the decision's text in") + held = self.description.held() + if isinstance(held, Failure): + return held + before = read_section(held.text, "Decisions") + if isinstance(before, Failure): + return before + same = decision_titled(before, title) + if same is not None: + return False if same == entry else Failure(f"## Decisions already holds another decision titled {title!r}: give this one a title of its own") + text = with_section(held.text, "Decisions", f"{before}\n\n### {title}\n\n{entry}".strip()) + if isinstance(text, Failure): + return text + wrote = dataclasses.replace(self.description, approved=frozenset()).write(text, held.version) + return wrote if isinstance(wrote, Failure) else True + + def write_language(self, language: str) -> Held | Failure: + return self.written(lambda text: with_language(text, language), frozenset()) + + def written(self, change: typing.Callable[[str], str | Failure], approved: frozenset[str]) -> Held | Failure: + held = self.description.held() + if isinstance(held, Failure): + return held + text = change(held.text) + if isinstance(text, Failure): + return text + return dataclasses.replace(self.description, approved=approved).write(text, held.version) diff --git a/bin/do_epic/findings.py b/bin/do_epic/findings.py new file mode 100644 index 00000000..92336512 --- /dev/null +++ b/bin/do_epic/findings.py @@ -0,0 +1,90 @@ +"""What a chunk's worker writes into its epic: a finding, filed as an issue, and a decision, appended to the description. + +A finding about one of the epic's goals is filed in the epic, and one about a repo's goals by that repo's rules (docs/decisions.md โ†’ `do-epic: drives one Jira epic โ€ฆ` โ†’ "Findings"). A decision that governs the epic goes under the description's ## Decisions, where the judge of each later chunk reads it.""" +import contextlib +import fcntl +import pathlib +import typing + +from do_epic.repo_pick import CLAUDE_LABEL, HUMAN_LABEL, of_label, REPO_PREFIX, REVIEW_LABEL +from do_epic.epic_record import EpicRecord +from do_epic.backend.protocol import EpicStore +from failure import Failure +from text_files.markdown_text import listed +from text_files.piped_text import piped + +# Three readers, seated as the human and as a chunk's worker, picked each spelling from four: ` --file` over `--file `, `add` and `--finding`, and `--record-decision` over `--decision` and `--decide`. +FILE = "--file" +LABEL = "--label" +RECORD_DECISION = "--record-decision" +# The labels that say where an issue is worked, as do-epic's scheduler reads them. +LABELS = f"`{REPO_PREFIX}` for the repo that works it, `{CLAUDE_LABEL}` for work of Claude's that changes no repo, or `{HUMAN_LABEL}` for a task of the human's" +# Under the epic's folder, the file a do-epic holds locked while it appends a decision. +DECISIONS_LOCK = "decisions.lock" +USAGE = f"""do-epic {FILE} [{LABEL}