diff --git a/.gitignore b/.gitignore index afdb19a0..ad60162b 100644 --- a/.gitignore +++ b/.gitignore @@ -137,6 +137,9 @@ !bin/todo-item !bin/todo_item/ !bin/todo_item/*.py +!bin/touch-predictions +!bin/touch_predictions/ +!bin/touch_predictions/*.py !bin/transcript_archive/ !bin/transcript_archive/*.py !bin/wait-for diff --git a/README.md b/README.md index b2edd7c6..43a02ffa 100644 --- a/README.md +++ b/README.md @@ -96,6 +96,8 @@ principles = "~/.claude/principles" A project may also set `env-file`, the file of `KEY=value` credentials its tools read, such as `env-file = "~/.claude-projects/log.env"`. It is a path that starts with `/` or `~/`, outside the checkout and the config's folder, so that no git remote receives it. A tool that needs a key fails and names each one the file lacks. No failure message, prompt, log line or archived loop output holds a token from the file. rule-hook refuses a Read or a Grep of the file and a Bash command that names it or searches a folder holding it with `ag`, `grep -r` or `rg`, unless the tool skips the file as hidden or ignored, so a worker doesn't read it in passing; `rule-hook --help` says which calls it checks. A project without the setting has no credentials, and never reads another project's. `project-config env-file` prints the path. +A project may also set `touch-predictions`, a path spelt as the settings above, such as `touch-predictions = "checkout:docs/touch-predictions.md"`. `touch-predictions record` appends each chunk's predicted files and parts there, beside those it touched. A project that leaves it out keeps no such record. + A project may also set `notifier` to `"mattermost"` or `"slack"`, the service that sends loop-claude's DMs. The `env-file` then holds that service's keys. A project that sets none gets no DMs, whatever its env file holds. `project-config notifier` prints the setting. A project's config may set four more things, each optional, as `project-config --help` describes. `language` names the language its owners write in, where that isn't English: the worker writes the project's prose in it, and the prose reviewer judges the prose in it. `commit-title`, `pull-request-title` and `branch` are templates for a project that titles and names these in another way than `CLAUDE.md` → "Git, commits, and PRs" says. `chunk-preflight` names the branch and the worktree by the `branch` template. On a branch that fits it, rule-hook refuses a commit, and `chunk-pr` a pull request, whose title doesn't fit its template; on any other branch, such as a release review's, a title is the summary alone. A tool that reads a commit title, such as a round's `Answer round :`, reads the template's `{summary}`: it tries `commit-title`, then `pull-request-title`, since a squash merge titles the commit as the pull request, then takes the title whole. While a project's config holds a fault, rule-hook refuses every commit there, and the tools that read its titles fail naming the fault. @@ -180,6 +182,7 @@ Install puts only `loop-claude` on `PATH`; call the others by their path, such a - `statusline.py` renders Claude Code's status line; `settings.json` → `statusLine` runs it. - `subagent-rung [--plan] ` prints the Agent tool's model and effort for a subagent's task, picked by its L from the project's model, and with `--plan` never below the project's rung; `subagent-rung --help` states the rule. - `todo-item` reads and changes a project's plan, whatever holds it: `list` prints the items in the plan's order and `show ` one item's row and detail; `add …` files an item under the next free ID, `edit …` sets an item's cells or detail, and `move --release ` sets an item's release; `finish ` removes a finished item and adds the bullet passed with `--changelog` to `CHANGELOG.md`. Each change ranks a markdown plan. `todo-item --help` states the flags. +- `touch-predictions record ...` appends what a chunk's items were predicted to touch, under `Touches:` in their details, beside the files and parts its branch touched, to the record the config names as `touch-predictions`; `touch-predictions --help` shows the spelling and how a part is named. `chunk-preflight` reads the same predictions to name the items that may start chunks at once. - `wait-for [] [--timeout ]` waits until a file a background job writes holds a line matching the pattern, and prints it. `bin/` is the import root: each launcher puts it on `sys.path`, so a module imports another by its path under `bin/`, such as `from claude_code.entries import parse_entry`. Each tool's package holds its tests. Beside the packages, `bin/`'s root holds only the launchers, the tools of one file with their tests, `test_env.py`, `failure.py` and `setup.cfg`. `failure.py`'s `Failure`, which modules across `bin/` return for a failed step, belongs to no single package. `branch_chunks/`, `claude_code/`, `command_lines/`, `git_cli/`, `loop_effects/`, `loop_state/`, `machine_load/`, `model_routing/`, `notifier/`, `plan_store/`, `record_store/`, `release_plan/`, `review_rules/`, `rung_replay/`, `text_files/` and `transcript_archive/` serve more than one tool. diff --git a/bin/chunk_preflight/facts.py b/bin/chunk_preflight/facts.py index cd7cdb4d..cbe1c1df 100644 --- a/bin/chunk_preflight/facts.py +++ b/bin/chunk_preflight/facts.py @@ -1,5 +1,6 @@ """The lines the preflight prints, stated from what it gathered.""" from release_plan.todo_table import bundle_candidates, BUNDLE_MAX_R, Item, ships_alone +from touch_predictions.beside import chunks_beside from chunk_preflight.choices import cut_name, PULLS_LIST from release_plan.release_standing import dated_cut, LastCut, unreleased_entries, UNSEEN_RELEASES from failure import Failure @@ -48,6 +49,7 @@ def item_lines(plan: str, item: ItemFacts) -> list[str]: f"- Worktree: {worktree}", f"- Open pull requests from {item.plan.branch}: {pulls_text(item.pulls, item.plan.branch)}", *bundled(item.plan), + *beside(item.plan), *named_lines(item.named), f"\nItem {first.id}'s tagline and detail:\n\n{quoted(item.plan.text)}", *joining_details(item.plan), @@ -98,6 +100,21 @@ def bundled(plan: Plan) -> list[str]: return [f'- The items of release {version(first.release)} that may join the chunk of item {first.id}, where `~/.claude/CLAUDE.md` → "Working loop" lets them share it:', *listed] +def beside(plan: Plan) -> list[str]: + """The items that may start chunks while first's runs, by touch_predictions' pick; no line where the release's items predict nothing.""" + first = plan.first + found = chunks_beside(plan.items, first) + if found is None: + return [] + none = f"- No item of release {version(first.release)} may start a chunk beside item {first.id}'s" + if isinstance(found, str): + return [f"{none}: {found}."] + refused = f" Not beside it: {'; '.join(found.refused)}." if found.refused else "" + if not found.picked: + return [f"{none}.{refused}"] + return [f"- The items of release {version(first.release)} that may start chunks beside item {first.id}'s, since none is predicted to touch a part that item {first.id} or an item named before it touches: {', '.join(map(str, found.picked))}. An item this chunk takes starts none.{refused}"] + + def named_lines(named: list[NamedPath] | Failure) -> list[str]: if isinstance(named, Failure): return [f"- The files the item names: cannot be listed: {named.reason}"] diff --git a/bin/chunk_preflight/main.py b/bin/chunk_preflight/main.py index 6ab9ad89..7ded8006 100644 --- a/bin/chunk_preflight/main.py +++ b/bin/chunk_preflight/main.py @@ -43,6 +43,12 @@ and prints that the release reviews and the cuts are the human's to call. It pri - the items of that release that a chunk taking the item may take too, each with its L, E + I, S and tagline: those that are not questions, that have R 5 or less, and that no item outside the chunk blocks; none where the item's own R is above 5; +- where an item of that release that doesn't wait holds a prediction, the bullets under a + `Touches:` line in its detail that `touch-predictions --help` describes: the items of the release + that may start chunks while the item's chunk runs. Those are the items that don't wait and are + predicted to touch no part that the item, or an item picked before them, touches too, taken in + the plan's order; a part either side marks (small) doesn't count. Then why each other item may + not. The plan and the records belong in no prediction, since every chunk writes them; - each file or folder git tracks that the item's tagline or detail names in backticks, with a file's lines or a folder's files, then the item's tagline and detail, then the detail of each of the first {JOINING_DETAILS} items that may join, or that one has none; diff --git a/bin/chunk_preflight/test_chunk_preflight.py b/bin/chunk_preflight/test_chunk_preflight.py index 511692cc..d0043e1b 100644 --- a/bin/chunk_preflight/test_chunk_preflight.py +++ b/bin/chunk_preflight/test_chunk_preflight.py @@ -628,5 +628,29 @@ class Running(unittest.TestCase): self.assertIsNone(line_count(folder)) +class BesideTest(unittest.TestCase): + """The fact that names the items that may start chunks beside the next one.""" + # Items 3 and 4 touch other parts of one file, and item 6 touches what item 3 does. + ROWS = "| 4 | 0.2.0 | | | **Do four.** | 1 | 1 | 1 | 1 | 1 | 1 by 1 | 10.0 |\n| 6 | 0.2.0 | | | **Do six.** | 1 | 1 | 1 | 1 | 1 | 1 by 1 | 10.0 |\n" + DETAILS = "\nTouches:\n\n- `src/app.py`: `run`\n\n### 4. Do four.\n\nTouches:\n\n- `src/app.py`: `stop`\n\n### 6. Do six.\n\nTouches:\n\n- `src/app.py`: `run`\n" + + def said(self, todo): + return [line for line in rendered(item_facts(todo)).splitlines() if "beside" in line] + + def test_names_the_items_that_may_start_chunks_beside_it_and_why_the_others_may_not(self): + todo = TODO.replace("| 9 | 0.3.0", self.ROWS + "| 9 | 0.3.0") + self.DETAILS + self.assertEqual(self.said(todo), ["- The items of release 0.2.0 that may start chunks beside item 3's, since none is predicted to touch a part that item 3 or an item named before it touches: 4. An item this chunk takes starts none. Not beside it: item 6, which overlaps item 3 at `src/app.py` `run`."]) + + def test_says_when_none_may(self): + todo = TODO.replace("| 9 | 0.3.0", self.ROWS + "| 9 | 0.3.0") + self.DETAILS.replace("`stop`", "`run`") + self.assertEqual(self.said(todo), ["- No item of release 0.2.0 may start a chunk beside item 3's. Not beside it: item 4, which overlaps item 3 at `src/app.py` `run`; item 6, which overlaps item 3 at `src/app.py` `run`."]) + alone = TODO.replace("| 9 | 0.3.0", self.ROWS + "| 9 | 0.3.0") + "\n### 4. Do four.\n\nTouches:\n\n- `src/app.py`\n" + self.assertEqual(self.said(alone), ["- No item of release 0.2.0 may start a chunk beside item 3's: item 3's detail has no `Touches:` list."]) + + def test_says_nothing_in_a_plan_without_predictions(self): + self.assertEqual(self.said(TODO), []) + + + if __name__ == "__main__": unittest.main() diff --git a/bin/project_config/main.py b/bin/project_config/main.py index e52040b8..56c3cc4e 100644 --- a/bin/project_config/main.py +++ b/bin/project_config/main.py @@ -36,6 +36,10 @@ a path that starts with / or ~/ and lies outside the checkout and the config's f "~/.claude-projects/acme-billing.env". Where it is left out, the project has no credentials file, and a tool that needs one fails naming the setting. It is printed with the others only where it is set. +`touch-predictions` is an optional setting, a path spelt as the settings above: the record where +`touch-predictions record` appends what each chunk was predicted to touch beside what it touched. Left +out, the project keeps no such record. It is printed with the others only where it is set. + `notifier` is an optional setting, "mattermost" or "slack": the service that sends loop-claude's DMs and `notify`'s messages to the human. Setting it requires `env-file`, which holds that service's keys; `notify --help` names them. Left out, the project gets no DMs. It is printed with the others only where it diff --git a/bin/project_config/settings.py b/bin/project_config/settings.py index 9ff1fc65..eb81b8bc 100644 --- a/bin/project_config/settings.py +++ b/bin/project_config/settings.py @@ -37,7 +37,9 @@ LANGUAGE = "language" COMMIT_TITLE = "commit-title" PULL_REQUEST_TITLE = "pull-request-title" BRANCH = "branch" -OPTIONAL = (BRANCH, COMMIT_TITLE, ENV_FILE, LANGUAGE, NOTIFIER, PULL_REQUEST_TITLE, READABILITY_FLOOR, TOOLING_ON_HOST, WAIT_FOR_APPROVAL) +# An optional path setting, spelt as SETTINGS' are: the record of what each chunk was predicted to touch beside what it touched. A project that leaves it out keeps no such record. Three readers, seated as the human who writes the config, picked the name from four candidates. +TOUCH_PREDICTIONS = "touch-predictions" +OPTIONAL = (BRANCH, COMMIT_TITLE, ENV_FILE, LANGUAGE, NOTIFIER, PULL_REQUEST_TITLE, READABILITY_FLOOR, TOOLING_ON_HOST, TOUCH_PREDICTIONS, WAIT_FOR_APPROVAL) # What opens a setting that names a Jira Cloud epic in place of a path: `jira:`, then the epic's address as the browser shows it. The settings below may name an epic; each reads its own part of it. JIRA = "jira:" EPIC_SPELLINGS = { @@ -99,9 +101,9 @@ class ProjectConfig: @property def places(self) -> dict[str, str]: - """Where each of SETTINGS is kept, in SETTINGS' order: a path, or the epic as the config spells it.""" + """Where each of SETTINGS is kept, in SETTINGS' order, then TOUCH_PREDICTIONS where the config sets it: a path, or the epic as the config spells it.""" epics = {"plan": self.epic, "goals-and-audience": self.goals_epic} - return {name: epic.spelling if (epic := epics.get(name)) else str(self.paths[name]) for name in SETTINGS if epics.get(name) or name in self.paths} + return {name: epic.spelling if (epic := epics.get(name)) else str(self.paths[name]) for name in (*SETTINGS, TOUCH_PREDICTIONS) if epics.get(name) or name in self.paths} @property def texts(self) -> dict[str, str]: @@ -138,7 +140,7 @@ def load(top: pathlib.Path, main: pathlib.Path, home: pathlib.Path) -> ProjectCo if len(named) > 1: return Failure(f"{' and '.join(str(file) for file, _ in named)} each name the checkout {main}; only one config may") file, values = named[0] - paths = {name: placed(values[name], top, file.parent, home) for name in SETTINGS if name in values} + paths = {name: placed(values[name], top, file.parent, home) for name in (*SETTINGS, TOUCH_PREDICTIONS) if name in values} epics = {name: epic_of(values.get(name)) for name in EPIC_SPELLINGS} env_file = anywhere(values.get(ENV_FILE), home) misplaced = ENV_FILE in values and (env_file is None or lies_in(env_file, top, main, file.parent)) diff --git a/bin/project_config/test_project_config.py b/bin/project_config/test_project_config.py index 5dbbf102..503bf50d 100644 --- a/bin/project_config/test_project_config.py +++ b/bin/project_config/test_project_config.py @@ -279,6 +279,20 @@ class LoadTest(ConfigCase): assert isinstance(found, Failure), found self.assertIn("`env-file`", found.reason) + def test_touch_predictions_is_an_optional_path_listed_after_the_others_only_where_set(self) -> None: + folder = self.write("acme", CLIENT) + left_out = load(self.checkout, self.checkout, self.home) + assert isinstance(left_out, ProjectConfig) + self.assertNotIn("touch-predictions", left_out.places) + (folder / "config.toml").write_text(CLIENT + 'touch-predictions = "folder:docs/touch-predictions.md"\n', encoding="utf-8") + found = load(self.checkout, self.checkout, self.home) + assert isinstance(found, ProjectConfig) + self.assertEqual(list(found.places)[-2:], ["principles", "touch-predictions"]) + self.assertEqual(found.paths["touch-predictions"], folder / "docs/touch-predictions.md") + self.assertIsNone(found.tracked("touch-predictions")) + (folder / "config.toml").write_text(CLIENT + 'touch-predictions = "docs/touch-predictions.md"\n', encoding="utf-8") + self.assertIn("`touch-predictions` is 'docs/touch-predictions.md', not \"checkout:\"", self.failure()) + def test_wait_for_approval_is_an_optional_boolean_that_is_false_where_left_out(self) -> None: folder = self.write("acme", CLIENT) left_out = load(self.checkout, self.checkout, self.home) @@ -373,7 +387,7 @@ class LoadTest(ConfigCase): def test_the_failure_for_a_misplaced_env_file_and_a_missing_setting_names_both_and_lists_every_setting(self) -> None: folder = self.write("acme", "".join(line + "\n" for line in CLIENT.splitlines() if not line.startswith("plan")) + 'env-file = "checkout:.env"\n') - self.assertEqual(self.failure(), f"{folder / 'config.toml'}: `plan` is missing; `env-file` is 'checkout:.env', not {OUTSIDE}. The settings are {', '.join(SETTINGS)}, and optionally branch, commit-title, env-file, language, notifier, pull-request-title, readability-floor, tooling-on-host and wait-for-approval.") + self.assertEqual(self.failure(), f"{folder / 'config.toml'}: `plan` is missing; `env-file` is 'checkout:.env', not {OUTSIDE}. The settings are {', '.join(SETTINGS)}, and optionally branch, commit-title, env-file, language, notifier, pull-request-title, readability-floor, tooling-on-host, touch-predictions and wait-for-approval.") def test_two_configs_naming_one_checkout_fail_naming_both(self) -> None: self.write("acme", CLIENT) diff --git a/bin/record_store/stores.py b/bin/record_store/stores.py index 317983a0..a8d59435 100644 --- a/bin/record_store/stores.py +++ b/bin/record_store/stores.py @@ -7,7 +7,7 @@ import typing from failure import Failure from git_cli.git_repo import file_at, git, Git -from project_config.settings import ProjectConfig +from project_config.settings import ProjectConfig, TOUCH_PREDICTIONS from text_files.file_writes import replace_file # The settings in a project's config that each name a record. @@ -68,6 +68,13 @@ def store(config: ProjectConfig, record: str) -> RecordStore: return FileStore(config.paths[record], config.tracked(record)) +def touch_predictions_store(config: ProjectConfig) -> RecordStore | None: + """The store of the touch-predictions record, the one record a config may leave out; None where it does.""" + if TOUCH_PREDICTIONS not in config.paths: + return None + return FileStore(config.paths[TOUCH_PREDICTIONS], config.tracked(TOUCH_PREDICTIONS)) + + def goals_store(config: ProjectConfig) -> RecordStore: """The store config names for the goals file, which holds ## Goals and ## Audience: a markdown file, or a Jira epic's description, which the store only reads.""" if config.goals_epic is None: diff --git a/bin/touch-predictions b/bin/touch-predictions new file mode 100755 index 00000000..735480a0 --- /dev/null +++ b/bin/touch-predictions @@ -0,0 +1,9 @@ +#!/usr/bin/env python3 +import pathlib +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) + +from touch_predictions import main + +sys.exit(main.main(sys.argv[1:])) diff --git a/bin/touch_predictions/__init__.py b/bin/touch_predictions/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/bin/touch_predictions/beside.py b/bin/touch_predictions/beside.py new file mode 100644 index 00000000..d5975081 --- /dev/null +++ b/bin/touch_predictions/beside.py @@ -0,0 +1,45 @@ +"""Which of a release's items may start chunks beside the next chunk, by what each is predicted to touch.""" +import dataclasses + +from failure import Failure +from release_plan.todo_table import Item, waiting +from touch_predictions.prediction import LABEL, overlaps, predicted, Touches + + +@dataclasses.dataclass(frozen=True) +class Beside: + """The IDs of the items that may start chunks beside first's, in the plan's order, and why each other item may not.""" + picked: list[int] + refused: list[str] + + +def chunks_beside(found: list[Item], first: Item) -> Beside | str | None: + """The items of first's release that don't wait and overlap neither first nor an item picked before them, taken in the plan's order. + + The plan's order decides, not the largest set: a larger set could start a low-ranked item in place of one ranked above it. An item first's chunk takes starts no chunk of its own. Else why no item may, as a sentence's end; None where no such item's detail holds a prediction, as in a project that writes none.""" + waits = waiting(found, set()) + ready = [item for item in found if item.release == first.release and item.id not in waits] + if not any(LABEL in item.detail.splitlines() for item in ready): + return None + ours = predicted(first.detail) + if not isinstance(ours, dict): + return f"item {first.id}'s {unread(ours)}" + taken: dict[int, Touches] = {first.id: ours} + refused = [] + for item in ready: + theirs = predicted(item.detail) + if item.id == first.id: + continue + if not isinstance(theirs, dict): + refused.append(f"item {item.id}, whose {unread(theirs)}") + continue + shared = next(((id_, places) for id_, touches in taken.items() if (places := overlaps(theirs, touches))), None) + if shared: + refused.append(f"item {item.id}, which overlaps item {shared[0]} at {', '.join(shared[1])}") + else: + taken[item.id] = theirs + return Beside([id_ for id_ in taken if id_ != first.id], refused) + + +def unread(prediction: Failure | None) -> str: + return f"detail has no `{LABEL}` list" if prediction is None else f"`{LABEL}` list cannot be read: {prediction.reason}" diff --git a/bin/touch_predictions/diff_parts.py b/bin/touch_predictions/diff_parts.py new file mode 100644 index 00000000..0758558f --- /dev/null +++ b/bin/touch_predictions/diff_parts.py @@ -0,0 +1,45 @@ +"""The files and parts a diff touched, named as a prediction names them.""" +import re + +from failure import Failure +from git_cli.git_repo import git, Git +from touch_predictions.prediction import Part, Touches +from touch_predictions.file_parts import part_names + +HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@", re.M) + + +def touched(base: str, head: str, cwd: str, skipped: set[str], run: Git = git) -> Touches | Failure: + """What the diff from base to head touched, less the paths in skipped. A file added, deleted or without text hunks is touched whole. A rename reads as a file deleted and one added.""" + listed = run("diff", "--no-renames", "--name-status", "-z", base, head, "--", cwd=cwd) + if isinstance(listed, Failure): + return listed + fields = listed.split("\0") + touches: Touches = {} + for status, path in sorted(zip(fields[::2], fields[1::2]), key=lambda pair: pair[1]): + if path in skipped: + continue + parts = modified(base, head, path, cwd, run) if status == "M" else () + if isinstance(parts, Failure): + return parts + touches[path] = parts + return touches + + +def modified(base: str, head: str, path: str, cwd: str, run: Git) -> tuple[Part, ...] | Failure: + """The parts of path whose lines the diff removed, in base's text, or added, in head's; line ranges of head's text where either text has no names.""" + diff = run("diff", "--no-renames", "-U0", base, head, "--", path, cwd=cwd, errors="replace") + if isinstance(diff, Failure): + return diff + hunks = [(int(old), int(olds or 1), int(new), int(news or 1)) for old, olds, new, news in HUNK.findall(diff)] + names: set[str] | None = set() + for revision, lines in ((base, [line for old, olds, _, _ in hunks for line in range(old, old + olds)]), (head, [line for _, _, new, news in hunks for line in range(new, new + news)])): + text = run("show", f"{revision}:{path}", cwd=cwd, errors="replace") + if isinstance(text, Failure): + return text + found = part_names(path, text, lines) + names = None if names is None or found is None else names | found + if names is None: + # A hunk that only deletes names the line it left, or the first where it left none. + return tuple(Part(f"lines {max(new, 1)}-{max(new + max(news, 1) - 1, 1)}") for _, _, new, news in hunks) + return tuple(Part(name) for name in sorted(names)) diff --git a/bin/touch_predictions/file_parts.py b/bin/touch_predictions/file_parts.py new file mode 100644 index 00000000..62353114 --- /dev/null +++ b/bin/touch_predictions/file_parts.py @@ -0,0 +1,83 @@ +"""The parts of a file that given lines lie in, by the names a prediction uses for them. + +Where each language's boundaries come from: + +- Python, a path ending .py: the standard `ast` module. A part is a function or class, named with the classes and functions around it, as in `Plan.branch`; a top-level assignment, by the name it sets; or `(module)` for any other top-level line. +- Markdown, a path ending .md: the heading lines outside fenced code. A part is the nearest heading above, spelt as its line, as in `## Goals`; `(top)` before the first. +- Any other file, and Python that doesn't parse: no names, so diff_parts falls back to line ranges.""" +import ast +import re +import typing + +MODULE = "(module)" +TOP = "(top)" +HEADING = re.compile(r"#{1,6} \S") +FENCE = re.compile(r" {0,3}(```|~~~)") +DEFINITION = ast.FunctionDef | ast.AsyncFunctionDef | ast.ClassDef + + +def part_names(path: str, text: str, lines: typing.Iterable[int]) -> set[str] | None: + """The part each of lines, counted from 1, lies in; None where the file's kind has no names.""" + if path.endswith(".py"): + return python_parts(text, lines) + if path.endswith(".md"): + return markdown_parts(text, lines) + return None + + +def python_parts(text: str, lines: typing.Iterable[int]) -> set[str] | None: + try: + tree = ast.parse(text) + except (SyntaxError, ValueError): + return None + if not tree.body: + return {MODULE} + # Each top-level statement's first and last line, then each part's with its name. + statements = [(first_line(statement), statement.end_lineno or statement.lineno) for statement in tree.body] + spans = [(*lines_, assigned(statement)) for lines_, statement in zip(statements, tree.body) if not isinstance(statement, DEFINITION)] + spans.extend(nested(tree, "")) + + def name(line: int) -> str: + # A line between two statements, such as a comment, goes with the one below it; one after the last, with the last. + if not any(first <= line <= last for first, last in statements): + line = min((first for first, _ in statements if first > line), default=statements[-1][0]) + return max((span for span in spans if span[0] <= line <= span[1]), key=lambda span: span[0])[2] + + return {name(line) for line in lines} + + +def nested(node: ast.AST, prefix: str) -> list[tuple[int, int, str]]: + """Each function and class under node, at any depth, named from prefix.""" + found = [] + for child in ast.iter_child_nodes(node): + if isinstance(child, DEFINITION): + found.append((first_line(child), child.end_lineno or child.lineno, prefix + child.name)) + found.extend(nested(child, f"{prefix}{child.name}.")) + else: + found.extend(nested(child, prefix)) + return found + + +def first_line(statement: ast.stmt) -> int: + """A statement's first line, which is its first decorator's where it has one.""" + return min([statement.lineno, *(decorator.lineno for decorator in getattr(statement, "decorator_list", []))]) + + +def assigned(statement: ast.stmt) -> str: + """The part a top-level statement that is no function or class makes: the one name it assigns, else the module.""" + targets = statement.targets if isinstance(statement, ast.Assign) else [statement.target] if isinstance(statement, ast.AnnAssign) else [] + return targets[0].id if len(targets) == 1 and isinstance(targets[0], ast.Name) else MODULE + + +def markdown_parts(text: str, lines: typing.Iterable[int]) -> set[str]: + wanted, names = set(lines), set() + heading, fenced = TOP, False + for number, line in enumerate(text.splitlines(), 1): + if FENCE.match(line): + fenced = not fenced + elif not fenced and HEADING.match(line): + heading = line.rstrip() + if number in wanted: + names.add(heading) + # A line past the end, as a deletion at the end names, lies under the last heading. + return names | ({heading} if any(line > len(text.splitlines()) for line in wanted) else set()) diff --git a/bin/touch_predictions/main.py b/bin/touch_predictions/main.py new file mode 100644 index 00000000..1d085b27 --- /dev/null +++ b/bin/touch_predictions/main.py @@ -0,0 +1,109 @@ +"""Record what a chunk's items were predicted to touch beside what its branch touched. + +usage: touch-predictions record ... + +Run it in the chunk's worktree, on the chunk's branch, with every review fix committed and before +`todo-item finish` removes the items; ... are the plan items the chunk takes. It reads each +item's prediction from its detail in the project's plan: the bullets under a `Touches:` line, one +per file, as in + + Touches: + + - `bin/a.py`: `main`, `Plan.branch` (small) + - `README.md`: `## Goals` + - `bin/run.sh`: lines 10-20 + - `docs/new.md` + +A path alone is the whole file, and (small) marks a part the item changes only a little. It names +what the branch touched the same way, from the diff between the branch's merge base with the +default branch and HEAD: + +- in a .py file, by Python's `ast` module: the innermost function or class a changed line lies in, + named with those around it; a top-level assignment by the name it sets; `(module)` for any other + top-level line. A comment between two statements goes with the one below it; +- in a .md file, by the nearest heading above the line, outside fenced code, spelt as its line; + `(top)` before the first heading; +- in any other file, and in Python that doesn't parse, by the line ranges the diff changed, as they + stand at HEAD; +- a file the branch added or deleted, or changed without text hunks, as the path alone. A rename + reads as a file deleted and one added. + +It leaves out the plan and the project's records, which every chunk writes. It appends +`## Chunk , , at ` with both lists to the record the project's config names +as `touch-predictions`, in place of an entry the record already holds for the same items. Commit +the record with the chunk. Where the config leaves the setting out, it says so and exits 0. + +Exit 1, saying why, on any other arguments, when no config names the checkout, the plan or the +record cannot be read or written, the plan holds no item , a `Touches:` list cannot be read, or +git cannot name the merge base or the diff. +""" +import sys +import time + +from git_cli.git_repo import default_ref, git +from failure import Failure +from plan_store.stores import store as plan_store +from touch_predictions.prediction import predicted, together, Touches +from project_config.main import project +from project_config.settings import ProjectConfig, TOUCH_PREDICTIONS +from touch_predictions import record +from record_store.stores import RECORDS, touch_predictions_store +from touch_predictions.diff_parts import touched +from text_files.utc_stamp import utc + + +def main(argv: list[str]) -> int: + if argv[:1] in (["-h"], ["--help"]): + print(__doc__.strip()) + return 0 + if argv[:1] != ["record"] or len(argv) < 2 or not all(id_.isascii() and id_.isdigit() for id_ in argv[1:]): + print("usage: touch-predictions record ...", file=sys.stderr) + return 1 + config = project() + done = config if isinstance(config, Failure) else recorded(config, [int(id_) for id_ in argv[1:]]) + if isinstance(done, Failure): + print(f"touch-predictions: {done.reason}", file=sys.stderr) + return 1 + print(done) + return 0 + + +def recorded(config: ProjectConfig, ids: list[int]) -> str | Failure: + """Appends the entry of the chunk that takes the items ids, and says so; or says that the project keeps no record.""" + store = touch_predictions_store(config) + if store is None: + return f"The project's config sets no `{TOUCH_PREDICTIONS}`, so nothing is recorded." + prediction = predictions(config, ids) + if isinstance(prediction, Failure): + return prediction + cwd = str(config.top) + default = default_ref(cwd) + base = default if isinstance(default, Failure) else git("merge-base", default, "HEAD", cwd=cwd) + head = base if isinstance(base, Failure) else git("rev-parse", "--short", "HEAD", cwd=cwd) + if isinstance(base, Failure) or isinstance(head, Failure): + return head if isinstance(head, Failure) else base + # Every chunk writes the plan and the records, so they tell no two chunks apart. + skipped = {path for name in ("plan", *RECORDS, TOUCH_PREDICTIONS) if (path := config.tracked(name))} + touches = touched(base.strip(), "HEAD", cwd, skipped) + if isinstance(touches, Failure): + return touches + failed = record.append(store, ids, record.entry(ids, utc(time.time()), head.strip(), prediction, touches)) + return failed or f"Recorded what chunk {', '.join(map(str, ids))} was predicted to touch, and what it touched, in {store.label}." + + +def predictions(config: ProjectConfig, ids: list[int]) -> Touches | None | Failure: + """What the items ids predict between them; None where none of them predicts anything.""" + plan = plan_store(config).read() + if isinstance(plan, Failure): + return plan + details = {item.id: item.detail for item in plan.items} + found = [] + for id_ in ids: + if id_ not in details: + return Failure(f"the plan holds no item {id_}") + prediction = predicted(details[id_]) + if isinstance(prediction, Failure): + return Failure(f"item {id_}: {prediction.reason}") + if prediction is not None: + found.append(prediction) + return together(found) if found else None diff --git a/bin/touch_predictions/prediction.py b/bin/touch_predictions/prediction.py new file mode 100644 index 00000000..de1527ab --- /dev/null +++ b/bin/touch_predictions/prediction.py @@ -0,0 +1,101 @@ +"""An item's prediction of the files and parts it will touch, as its detail spells it, and where two predictions overlap. + +The spelling, which three readers seated as the project's audience picked from four candidates: + + Touches: + + - `bin/a.py`: `main`, `Plan.branch` (small) + - `bin/run.sh`: lines 10-20 + - `docs/new.md` + +A path alone is the whole file, and (small) marks a part the item changes only a little. file_parts says how a part is named.""" +import dataclasses +import re + +from failure import Failure + +LABEL = "Touches:" +RANGE = re.compile(r"lines (\d+)-(\d+)") +PART = re.compile(r"(`[^`]+`|lines \d+-\d+)( \(small\))?") +BULLET = re.compile(rf"- `([^`\s]+)`(?:: ({PART.pattern}(?:, {PART.pattern})*))?") + + +@dataclasses.dataclass(frozen=True) +class Part: + """A part of a file, by the name file_parts gives it or as `lines -`; small where the item changes it only a little.""" + name: str + small: bool = False + + +# Each file's parts by its path; no parts where the whole file is touched. +Touches = dict[str, tuple[Part, ...]] + + +def predicted(detail: str) -> Touches | None | Failure: + """The prediction in an item's detail: the bullets under its `Touches:` line, up to the first other text. None where the detail has no such line.""" + lines = detail.splitlines() + if LABEL not in lines: + return None + touches: Touches = {} + for line in lines[lines.index(LABEL) + 1:]: + if not line.strip(): + continue + if not line.startswith("- "): + break + bullet = BULLET.fullmatch(line) + parts = tuple(Part(name.strip("`"), bool(small)) for name, small in PART.findall(bullet.group(2) or "")) if bullet else () + if bullet is None or any(span and span[0] > span[1] for span in map(line_range, parts)): + return Failure(f"the bullet {line!r} under `{LABEL}` is not a path in backticks, then optionally a colon and its parts, each a name in backticks or `lines -`, and optionally ` (small)`") + touches = together([touches, {bullet.group(1): parts}]) + return touches or Failure(f"no bullet follows `{LABEL}`") + + +def rendered(touches: Touches) -> str: + """touches as the bullets predicted reads.""" + return "\n".join(f"- `{path}`" + (": " + ", ".join(shown(part) + " (small)" * part.small for part in parts) if parts else "") for path, parts in touches.items()) + + +def shown(part: Part) -> str: + return part.name if line_range(part) else f"`{part.name}`" + + +def line_range(part: Part) -> tuple[int, int] | None: + found = RANGE.fullmatch(part.name) + return (int(found.group(1)), int(found.group(2))) if found else None + + +def together(several: list[Touches]) -> Touches: + """What several predictions touch between them. A file one of them touches whole is touched whole, and a part is small only where each that names it says so.""" + joined: dict[str, dict[str, bool] | None] = {} + for touches in several: + for path, parts in touches.items(): + held = joined.setdefault(path, {}) + if held is None or not parts: + joined[path] = None + continue + for part in parts: + held[part.name] = held.get(part.name, True) and part.small + return {path: tuple(Part(name, small) for name, small in held.items()) if held else () for path, held in joined.items()} + + +def overlaps(ours: Touches, theirs: Touches) -> list[str]: + """Each place both touch: a file either touches whole, by its path alone, and else each part as ours names it. A part either side marks small is left out: a conflict there is cheap to settle, and a chunk that waits costs hours.""" + found = [] + for path, parts in ours.items(): + if path not in theirs: + continue + if not parts or not theirs[path]: + found.append(f"`{path}`") + continue + found.extend(f"`{path}` {shown(part)}" for part in parts if any(same_part(part, other) and not (part.small or other.small) for other in theirs[path])) + return found + + +def same_part(one: Part, other: Part) -> bool: + """Whether two parts of one file share lines, as far as their names tell: line ranges that share a line, a line range beside a name, equal names, or a class and what it holds.""" + spans = line_range(one), line_range(other) + if spans[0] and spans[1]: + return spans[0][0] <= spans[1][1] and spans[1][0] <= spans[0][1] + if spans[0] or spans[1]: + return True + return one.name == other.name or one.name.startswith(f"{other.name}.") or other.name.startswith(f"{one.name}.") diff --git a/bin/touch_predictions/record.py b/bin/touch_predictions/record.py new file mode 100644 index 00000000..ef274388 --- /dev/null +++ b/bin/touch_predictions/record.py @@ -0,0 +1,27 @@ +"""The touch-predictions record: one entry per chunk, its prediction beside what its diff touched.""" +import re + +from failure import Failure +from record_store.stores import RecordStore +from touch_predictions.prediction import rendered, Touches +from text_files.utc_stamp import STAMPED + +TITLE = "# Touch predictions" + + +def entry(items: list[int], at: str, head: str, predicted: Touches | None, touched: Touches) -> str: + """A chunk's entry: the items it took, the UTC time at, the short commit its diff ends at, and both lists in a prediction's spelling. predicted is None where no item predicted anything.""" + return f"## Chunk {', '.join(map(str, items))}, {at}, at {head}\n\n{listed('Predicted', predicted)}\n\n{listed('Touched', touched)}\n" + + +def listed(label: str, touches: Touches | None) -> str: + return f"{label}:\n\n{rendered(touches)}" if touches else f"{label}: none." + + +def append(store: RecordStore, items: list[int], text: str) -> Failure | None: + """Appends text, the entry of the chunk that took items, in place of each entry the record holds for the same items: a fix after the first entry changes what the chunk touched.""" + held = store.read() + if isinstance(held, Failure): + return held + kept = re.sub(rf"^## Chunk {', '.join(map(str, items))}, {STAMPED},.*?(?=^## |\Z)", "", held, flags=re.M | re.S) + return store.write(f"{kept.rstrip() or TITLE}\n\n{text}") diff --git a/bin/touch_predictions/test_touch_predictions.py b/bin/touch_predictions/test_touch_predictions.py new file mode 100644 index 00000000..a0076d29 --- /dev/null +++ b/bin/touch_predictions/test_touch_predictions.py @@ -0,0 +1,344 @@ +import os +import pathlib +import re +import subprocess +import sys +import tempfile +import unittest + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1])) + +from touch_predictions.beside import Beside, chunks_beside +from touch_predictions import diff_parts, file_parts, record +from failure import Failure +from test_env import isolated_env +from record_store.fakes import MemoryStore +from touch_predictions.prediction import overlaps, Part, predicted, rendered, together +from plan_store.stores import plan_of +from project_config.test_project_config import write_config + +DETAIL = """A planning call. + +Touches: + +- `bin/a.py`: `main`, `Plan.branch` (small) +- `README.md`: `## Goals, and (more): things` +- `bin/run.sh`: lines 10-20 +- `docs/new.md` + +More prose, with `bin/b.py` named. +""" + +PYTHON = """\"\"\"Docstring.\"\"\" +import os + +LIMIT = 3 + + +# A comment that belongs to first. +def first(): + return 1 + + +class Plan: + size: int = 0 + + @property + def branch(self): + def inner(): + return 2 + return inner() + + +if os.name: + pass +""" + +MARKDOWN = """Intro. + +# Title + +Text. + +``` +## Not a heading +``` + +## Goals + +1. One. +""" + + +class PredictedTest(unittest.TestCase): + def test_reads_each_bullet_of_the_touches_list(self) -> None: + self.assertEqual(predicted(DETAIL), { + "bin/a.py": (Part("main"), Part("Plan.branch", small=True)), + "README.md": (Part("## Goals, and (more): things"),), + "bin/run.sh": (Part("lines 10-20"),), + "docs/new.md": (), + }) + + def test_a_detail_without_the_list_predicts_nothing(self) -> None: + self.assertIsNone(predicted("Only prose, with `bin/a.py` named.")) + + def test_a_bullet_it_cannot_read_is_a_failure_that_quotes_it(self) -> None: + for bullet in ("- bin/a.py: main", "- `bin/a.py`: main", "- `bin/a.py`: lines 20-10", "- `bin/a.py`: `main` `other`"): + with self.subTest(bullet): + found = predicted(f"Touches:\n\n{bullet}\n") + assert isinstance(found, Failure) + self.assertIn(bullet, found.reason) + + def test_an_empty_list_is_a_failure(self) -> None: + self.assertIsInstance(predicted("Touches:\n\nProse.\n"), Failure) + + def test_rendered_reads_back(self) -> None: + touches = predicted(DETAIL) + assert isinstance(touches, dict) + self.assertEqual(predicted(f"Touches:\n\n{rendered(touches)}\n"), touches) + + def test_together_joins_items_and_keeps_small_only_where_each_says_so(self) -> None: + joined = together([{"a.py": (Part("f", small=True), Part("g", small=True))}, {"a.py": (Part("f"),), "b.md": ()}, {"b.md": (Part("# T"),)}]) + self.assertEqual(joined, {"a.py": (Part("f"), Part("g", small=True)), "b.md": ()}) + + +class OverlapsTest(unittest.TestCase): + def test_other_files_and_other_parts_do_not_overlap(self) -> None: + self.assertEqual(overlaps({"a.py": (Part("f"),)}, {"a.py": (Part("g"),), "b.py": ()}), []) + + def test_the_same_part_overlaps(self) -> None: + self.assertEqual(overlaps({"a.py": (Part("f"), Part("g"))}, {"a.py": (Part("g"),)}), ["`a.py` `g`"]) + + def test_a_class_overlaps_its_method_but_not_a_name_it_only_starts(self) -> None: + self.assertEqual(overlaps({"a.py": (Part("Plan"),)}, {"a.py": (Part("Plan.branch"), Part("Planned"))}), ["`a.py` `Plan`"]) + + def test_a_whole_file_overlaps_any_part_of_it(self) -> None: + self.assertEqual(overlaps({"a.py": ()}, {"a.py": (Part("g", small=True),)}), ["`a.py`"]) + + def test_line_ranges_overlap_where_they_share_a_line(self) -> None: + self.assertEqual(overlaps({"x.sh": (Part("lines 1-10"),)}, {"x.sh": (Part("lines 10-12"),)}), ["`x.sh` lines 1-10"]) + self.assertEqual(overlaps({"x.sh": (Part("lines 1-9"),)}, {"x.sh": (Part("lines 10-12"),)}), []) + + def test_a_line_range_overlaps_a_named_part_since_nothing_tells_them_apart(self) -> None: + self.assertEqual(overlaps({"a.py": (Part("lines 1-9"),)}, {"a.py": (Part("f"),)}), ["`a.py` lines 1-9"]) + + def test_a_small_change_on_either_side_is_taken(self) -> None: + self.assertEqual(overlaps({"a.py": (Part("f", small=True),)}, {"a.py": (Part("f"),)}), []) + self.assertEqual(overlaps({"a.py": (Part("f"),)}, {"a.py": (Part("f", small=True),)}), []) + + +class FilePartsTest(unittest.TestCase): + def names(self, path: str, text: str, *lines: int) -> set[str] | None: + return file_parts.part_names(path, text, lines) + + def test_python_names_the_innermost_function_or_class(self) -> None: + self.assertEqual(self.names("a.py", PYTHON, 9), {"first"}) + self.assertEqual(self.names("a.py", PYTHON, 13), {"Plan"}) + self.assertEqual(self.names("a.py", PYTHON, 15, 18), {"Plan.branch", "Plan.branch.inner"}) + + def test_python_names_a_top_level_assignment_and_calls_the_rest_the_module(self) -> None: + self.assertEqual(self.names("a.py", PYTHON, 4), {"LIMIT"}) + self.assertEqual(self.names("a.py", PYTHON, 1, 2, 22), {"(module)"}) + + def test_python_gives_a_line_between_statements_to_the_next_one(self) -> None: + self.assertEqual(self.names("a.py", PYTHON, 7), {"first"}) + self.assertEqual(self.names("a.py", PYTHON, 99), {"(module)"}) + self.assertEqual(self.names("a.py", "", 1), {"(module)"}) + + def test_python_that_does_not_parse_has_no_names(self) -> None: + self.assertIsNone(self.names("a.py", "def (:\n", 1)) + + def test_markdown_names_the_heading_above_and_skips_fenced_lines(self) -> None: + self.assertEqual(self.names("a.md", MARKDOWN, 1), {"(top)"}) + self.assertEqual(self.names("a.md", MARKDOWN, 3, 5, 8), {"# Title"}) + self.assertEqual(self.names("a.md", MARKDOWN, 13), {"## Goals"}) + + def test_any_other_file_has_no_names(self) -> None: + self.assertIsNone(self.names("run.sh", "echo\n", 1)) + + +def plan(*items: tuple[int, str, str, str]) -> list: + """A plan's items, each given as its ID, release, the IDs that block it and the bullets of its prediction, none for an item without one.""" + rows = "".join(f"| {id_} | {release} | {blocked} | | **Do {id_}.** | 1 | 1 | 1 | 1 | 1 | 1 by 1 | 10.0 |\n" for id_, release, blocked, _ in items) + details = "".join(f"### {id_}. Do {id_}.\n\nTouches:\n\n{bullets}\n\n" for id_, _, _, bullets in items if bullets) + found = plan_of(f"# todo\n\n## Scoring\n\n`Score = 100 * V / (2**S + 8)`\n\n`Bar = 0.1`\n\n`Next ID = 99`\n\n## Items\n\n| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |\n|---|---|---|---|---|---|---|---|---|---|---|---|\n{rows}\n## Details\n\n{details}") + assert not isinstance(found, Failure), found + return found.items + + +class ChunksBesideTest(unittest.TestCase): + def test_picks_in_the_plans_order_each_item_that_overlaps_none_before_it(self) -> None: + items = plan((1, "0.2.0", "", "- `a.py`: `f`"), (2, "0.2.0", "", "- `a.py`: `g`\n- `b.py`"), (3, "0.2.0", "", "- `b.py`: `h`"), (4, "0.2.0", "", "- `a.py`: `f`"), (5, "0.2.0", "", "- `c.py`")) + self.assertEqual(chunks_beside(items, items[0]), Beside([2, 5], ["item 3, which overlaps item 2 at `b.py`", "item 4, which overlaps item 1 at `a.py` `f`"])) + + def test_leaves_out_an_item_that_waits_or_names_a_later_release(self) -> None: + items = plan((1, "0.2.0", "", "- `a.py`"), (2, "0.2.0", "1", "- `b.py`"), (3, "0.3.0", "", "- `c.py`")) + self.assertEqual(chunks_beside(items, items[0]), Beside([], [])) + + def test_refuses_an_item_without_a_prediction_it_can_read(self) -> None: + items = plan((1, "0.2.0", "", "- `a.py`"), (2, "0.2.0", "", ""), (3, "0.2.0", "", "- b.py")) + found = chunks_beside(items, items[0]) + assert isinstance(found, Beside) + self.assertEqual(found.picked, []) + self.assertEqual(found.refused[0], "item 2, whose detail has no `Touches:` list") + self.assertTrue(found.refused[1].startswith("item 3, whose `Touches:` list cannot be read: the bullet '- b.py'")) + + def test_no_item_starts_beside_one_that_predicts_nothing(self) -> None: + items = plan((1, "0.2.0", "", ""), (2, "0.2.0", "", "- `b.py`")) + self.assertEqual(chunks_beside(items, items[0]), "item 1's detail has no `Touches:` list") + + def test_says_nothing_where_no_item_predicts(self) -> None: + items = plan((1, "0.2.0", "", ""), (2, "0.2.0", "", ""), (3, "0.3.0", "", "- `c.py`")) + self.assertIsNone(chunks_beside(items, items[0])) + + +class Repo: + """A git checkout in a temporary folder.""" + + def __init__(self, folder: str) -> None: + self.folder = folder + self.run("init", "-q", "-b", "main") + + def run(self, *args: str) -> str: + return subprocess.run(["git", "-c", "user.name=t", "-c", "user.email=t@t", *args], cwd=self.folder, check=True, capture_output=True, text=True).stdout.strip() + + def commit(self, files: dict[str, str | bytes | None]) -> str: + for name, text in files.items(): + path = pathlib.Path(self.folder, name) + if text is None: + path.unlink() + elif isinstance(text, bytes): + path.write_bytes(text) + else: + path.write_text(text, encoding="utf-8") + self.run("add", "-A") + self.run("commit", "-qm", "x") + return self.run("rev-parse", "HEAD") + + +class TouchedTest(unittest.TestCase): + def setUp(self) -> None: + folder = tempfile.TemporaryDirectory() + self.addCleanup(folder.cleanup) + self.repo = Repo(folder.name) + self.base = self.repo.commit({"a.py": PYTHON, "a.md": MARKDOWN, "run.sh": "one\ntwo\nthree\nfour\n", "gone.py": "x = 1\n", "todo.md": "plan\n", "blob.bin": b"\x00\x01"}) + + def touched(self, files: dict[str, str | bytes | None], *skipped: str) -> object: + return diff_parts.touched(self.base, self.repo.commit(files), self.repo.folder, set(skipped)) + + def test_names_the_parts_of_changed_lines_on_both_sides(self) -> None: + changed = PYTHON.replace("\treturn 1\n", "\treturn 3\n").replace("LIMIT = 3\n", "").replace("\tpass\n", "\tpass\n\n\ndef last():\n\tpass\n") + self.assertEqual(self.touched({"a.py": changed, "a.md": MARKDOWN + "2. Two.\n"}), { + "a.md": (Part("## Goals"),), + "a.py": (Part("LIMIT"), Part("first"), Part("last")), + }) + + def test_an_added_or_deleted_file_is_touched_whole(self) -> None: + self.assertEqual(self.touched({"new.py": "y = 2\n", "gone.py": None}), {"gone.py": (), "new.py": ()}) + + def test_a_file_without_names_falls_back_to_the_new_sides_line_ranges(self) -> None: + self.assertEqual(self.touched({"run.sh": "one\n2\n3\nfour\nfive\n"}), {"run.sh": (Part("lines 2-3"), Part("lines 5-5"))}) + + def test_a_deletion_falls_back_to_the_line_it_left(self) -> None: + self.assertEqual(self.touched({"run.sh": "one\nfour\n"}), {"run.sh": (Part("lines 1-1"),)}) + + def test_a_binary_file_is_touched_whole(self) -> None: + self.assertEqual(self.touched({"blob.bin": b"\x00\x02"}), {"blob.bin": ()}) + + def test_python_that_stops_parsing_falls_back_to_line_ranges(self) -> None: + self.assertEqual(self.touched({"gone.py": "def (:\n"}), {"gone.py": (Part("lines 1-1"),)}) + + def test_skipped_paths_do_not_count(self) -> None: + self.assertEqual(self.touched({"todo.md": "plan\nmore\n", "gone.py": "x = 2\n"}, "todo.md"), {"gone.py": (Part("x"),)}) + + def test_an_unknown_base_is_a_failure(self) -> None: + self.assertIsInstance(diff_parts.touched("nope", "HEAD", self.repo.folder, set()), Failure) + + +class EntryTest(unittest.TestCase): + def test_appends_the_prediction_beside_the_touched_parts(self) -> None: + store = MemoryStore("# Touch predictions\n\n## Chunk 1, 2026-01-01T00:00:00Z, at aaaaaaa\n\nPredicted: none.\n\nTouched: none.\n") + failed = record.append(store, [373, 374], record.entry([373, 374], "2026-11-15T10:00:00Z", "abc1234", {"a.py": (Part("f", small=True),)}, {"a.py": (Part("f"), Part("g")), "b.md": ()})) + self.assertIsNone(failed) + self.assertEqual(store.read(), "# Touch predictions\n\n## Chunk 1, 2026-01-01T00:00:00Z, at aaaaaaa\n\nPredicted: none.\n\nTouched: none.\n\n## Chunk 373, 374, 2026-11-15T10:00:00Z, at abc1234\n\nPredicted:\n\n- `a.py`: `f` (small)\n\nTouched:\n\n- `a.py`: `f`, `g`\n- `b.md`\n") + + def test_starts_an_empty_record_with_its_title(self) -> None: + store = MemoryStore("") + record.append(store, [5], record.entry([5], "2026-11-15T10:00:00Z", "abc1234", None, {})) + self.assertEqual(store.read(), "# Touch predictions\n\n## Chunk 5, 2026-11-15T10:00:00Z, at abc1234\n\nPredicted: none.\n\nTouched: none.\n") + + def test_replaces_the_entry_of_the_same_items_and_keeps_the_others(self) -> None: + store = MemoryStore("") + for items, head in (([5], "aaaaaaa"), ([5, 6], "bbbbbbb"), ([7], "ccccccc"), ([5], "ddddddd")): + record.append(store, items, record.entry(items, "2026-11-15T10:00:00Z", head, None, {})) + self.assertEqual([line.split(" at ")[1] for line in store.text.splitlines() if line.startswith("## ")], ["bbbbbbb", "ccccccc", "ddddddd"]) + self.assertEqual(store.text.count("\n\n\n"), 0) + + def test_a_record_that_cannot_be_read_is_a_failure(self) -> None: + self.assertEqual(record.append(MemoryStore(fails="no disk"), [5], "x"), Failure("no disk")) + + +class RecordCommandTest(unittest.TestCase): + """`touch-predictions record`, run as a program in a configured checkout on a chunk's branch.""" + APP = "def run():\n\treturn 1\n\n\ndef stop():\n\treturn 2\n" + TODO = "# todo\n\n## Scoring\n\n`Score = 100 * V / (2**S + 8)`\n\n`Bar = 0.1`\n\n`Next ID = 9`\n\n## Items\n\n| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |\n|---|---|---|---|---|---|---|---|---|---|---|---|\n| 3 | 0.2.0 | | | **Do three.** | 1 | 1 | 1 | 1 | 1 | 1 by 1 | 10.0 |\n| 4 | 0.2.0 | | | **Do four.** | 1 | 1 | 1 | 1 | 1 | 1 by 1 | 10.0 |\n\n## Details\n\n### 3. Do three.\n\nTouches:\n\n- `app.py`: `run`\n\n### 4. Do four.\n\nNo prediction.\n" + + def setUp(self) -> None: + made = tempfile.TemporaryDirectory() + self.addCleanup(made.cleanup) + self.home = pathlib.Path(made.name, "home") + self.checkout = pathlib.Path(made.name, "proj") + self.checkout.mkdir() + self.repo = Repo(str(self.checkout)) + self.config = write_config(self.home, self.checkout) / "config.toml" + self.repo.commit({"app.py": self.APP, "todo.md": self.TODO}) + self.repo.run("checkout", "-q", "-b", "item-3") + self.repo.commit({"app.py": self.APP.replace("return 1", "return 3").replace("return 2", "return 4"), "todo.md": self.TODO + "\nMore.\n", "CHANGELOG.md": "- New.\n"}) + + def record(self, *args: str) -> subprocess.CompletedProcess[str]: + tool = pathlib.Path(__file__).resolve().parents[1] / "touch-predictions" + return subprocess.run([sys.executable, str(tool), *args], cwd=self.checkout, env={**isolated_env(), "HOME": str(self.home), "GIT_CONFIG_GLOBAL": os.devnull}, capture_output=True, text=True) + + def set_record(self) -> pathlib.Path: + self.config.write_text(self.config.read_text(encoding="utf-8") + 'touch-predictions = "checkout:docs/touch-predictions.md"\n', encoding="utf-8") + return self.checkout / "docs/touch-predictions.md" + + def test_appends_the_items_prediction_beside_what_the_branch_touched_less_the_plan_and_records(self) -> None: + path = self.set_record() + ran = self.record("record", "3", "4") + self.assertEqual((ran.returncode, ran.stderr), (0, "")) + self.assertEqual(ran.stdout, f"Recorded what chunk 3, 4 was predicted to touch, and what it touched, in {path}.\n") + head = self.repo.run("rev-parse", "--short", "HEAD") + entry = re.fullmatch(rf"# Touch predictions\n\n## Chunk 3, 4, \d{{4}}-\d\d-\d\dT\d\d:\d\d:\d\dZ, at {head}\n\n(.*)", path.read_text(encoding="utf-8"), re.S) + assert entry is not None + self.assertEqual(entry.group(1), "Predicted:\n\n- `app.py`: `run`\n\nTouched:\n\n- `app.py`: `run`, `stop`\n") + + def test_a_second_run_replaces_the_entry_and_leaves_the_record_out_of_what_was_touched(self) -> None: + path = self.set_record() + self.record("record", "3") + self.repo.commit({}) + self.assertEqual(self.record("record", "3").returncode, 0) + text = path.read_text(encoding="utf-8") + self.assertEqual(text.count("## Chunk 3, "), 1) + self.assertNotIn("touch-predictions.md", text) + + def test_records_nothing_where_the_config_leaves_the_setting_out(self) -> None: + ran = self.record("record", "3") + self.assertEqual((ran.returncode, ran.stdout), (0, "The project's config sets no `touch-predictions`, so nothing is recorded.\n")) + self.assertFalse((self.checkout / "docs").exists()) + + def test_fails_on_an_item_the_plan_lacks_a_list_it_cannot_read_and_other_arguments(self) -> None: + path = self.set_record() + for args, said in ((("record", "8"), "touch-predictions: the plan holds no item 8\n"), (("record",), "usage: touch-predictions record ...\n"), (("record", "x"), "usage: touch-predictions record ...\n"), ((), "usage: touch-predictions record ...\n")): + with self.subTest(args): + ran = self.record(*args) + self.assertEqual((ran.returncode, ran.stdout, ran.stderr), (1, "", said)) + (self.checkout / "todo.md").write_text(self.TODO.replace("- `app.py`: `run`", "- app.py"), encoding="utf-8") + ran = self.record("record", "3") + self.assertEqual(ran.returncode, 1) + self.assertTrue(ran.stderr.startswith("touch-predictions: item 3: the bullet '- app.py'"), ran.stderr) + self.assertFalse(path.exists()) + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/decisions.md b/docs/decisions.md index 7ebe08ca..a5664413 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -38,6 +38,7 @@ Each project principle states one rule that the decisions below apply; their ent - routing: the worker changes rung only as a chunk starts, after every second unfinished verdict, when it names a chunk item that picks a higher rung, or when it cites an escalation trigger the judge finds - tests: each test earns its keep as the technical principles' test bullet says, and the worker takes a test's expected values from its item, never from the code - the repo may name bender, bender-infra and ai-sandbox +- todo.md: an item of the next release predicts the files and parts it will touch under `Touches:` in its detail; chunk-preflight takes, in the plan's order, the items that overlap none taken before them as the chunks that may run at once, and the optional `touch-predictions` record keeps each chunk's prediction beside what it touched - todo.md: one table of items, scored by the formula it states, held to its bar and ranked by `bin/rank_todo.py`, which ranks each blocker above the items it blocks; each achieve goal weighs by its position, code health first and four times as heavy while the comprehension panel's overall is under the project's readability floor or unmeasured, then the goals file's goals, and a maintain goal weighs 0.5 per point of the confidence an item's test, gate leg or probe adds to it; the table's order is the plan's only order - todo.md: only a session or subagent on the project's rung or above scores or reshapes the plan; one below it drafts new items and changes to filed ones for a subagent at or above that rung to apply and score, and removes finished ones - todo.md: tools and agents read and change the plan as it stands only through `bin/plan_store` and `todo-item`; a tool that reads what a past commit held, ranks the file or checks its rank still parses `todo.md` @@ -652,6 +653,20 @@ Valid while the worker writes the tests that larv-review's reviewers hold to the 2026-10-04, lilleman, answering the principles audit of PR #215. Serves goal 4, which keeps credentials, transcripts and memory files off the remote; the names of the host, its infrastructure project and the sandbox VM are none of those. Valid while the repo names them without their addresses or credentials. +## todo.md: an item of the next release predicts the files and parts it will touch under `Touches:` in its detail; chunk-preflight takes, in the plan's order, the items that overlap none taken before them as the chunks that may run at once, and the optional `touch-predictions` record keeps each chunk's prediction beside what it touched + +2026-11-15, lilleman's call of 2026-11-13 in item 373, which split item 361. Serves goal 1.5, held to goal 1.4: several chunks of one project run at once only where they are unlikely to touch the same lines, since a conflict costs a rebase, a gate rerun and often a review round. A session on the project's rung or above writes each prediction, as `rules/todo-items.md` says, and `bin/touch_predictions/` reads it. The choices the call left open: + +- **The spelling.** A `Touches:` line, then one bullet per file: the path in backticks, then its parts, each a name in backticks or `lines -`, with `(small)` after a part the item changes only a little. A path alone is the whole file. Three readers seated as the README's audience picked the bulleted list over one line and over a table, since backticks hold a heading's commas, colons and parentheses. Two of the three put `(small)` after the part and outside the backticks, and two left line ranges unquoted. They split three ways on the config key and `touch-predictions` was each other reader's second choice. +- **A part's boundaries.** Python's `ast` module names the innermost function or class, a top-level assignment by the name it sets, and `(module)` for the rest. A Markdown file's parts are its headings outside fenced code. Any other file, and Python that doesn't parse, falls back to line ranges. A table row is no part of its own: the item named one, but the tables chunks collide on are the plan's and the changelog's, which don't count. +- **What doesn't count.** The plan and every record the config names, the changelog among them, since every chunk writes them: item 374 writes the plan and the changelog at merge time, and the other records are appended to. `touch-predictions record` leaves them out of what a chunk touched, and the rule keeps them out of predictions. +- **The pick.** `chunk-preflight` takes the items of the next release that don't wait, in the plan's order, each where it overlaps neither the next item nor an item taken before it. The call asked for the set that keeps the most chunks running. The plan's order comes first all the same: the largest set could start a low-ranked item in place of one ranked above it, and with more ready items than the parallel limit both fill the limit. Item 375 takes as many of the picked items as its limit allows. +- **An overlap that is taken.** Two items may share a part where either marks it `(small)`. The call takes an overlap only for a small change to the same part, and a few lines re-apply onto whatever the other chunk did to it. +- **Which items share a chunk** stays the worker's choice, by the review passes they fire, so the pick names items, and an item the next chunk takes starts no chunk of its own. +- **The record is optional.** No path setting has a default, and a mandatory one would stop every project until its owner edits the config (goal 1.6). A project that leaves `touch-predictions` out is still picked for, and keeps no record. The worker runs `touch-predictions record` before it removes the chunk's items (`skills/larv-review/SKILL.md` §5), since the prediction leaves the plan with the item, and a plan in Jira keeps no history to read it back from. + +Valid while item 375 runs chunks by this pick. The later item that measures how often the predictions miss revisits the pick's order and the `(small)` rule. + ## todo.md: one table of items, scored by the formula it states, held to its bar and ranked by `bin/rank_todo.py`, which ranks each blocker above the items it blocks; each achieve goal weighs by its position, code health first and four times as heavy while the comprehension panel's overall is under the project's readability floor or unmeasured, then the goals file's goals, and a maintain goal weighs 0.5 per point of the confidence an item's test, gate leg or probe adds to it; the table's order is the plan's only order 2026-10-02, lilleman. Serves goal 1: findings were filed faster than chunks closed them, so the plan grew without telling what mattered. Each item is scored as `~/.claude/rules/todo-items.md` says; this repo's formula counts value highest and size least. The format is mandatory in every project, as the premises are (goal 1.3); the formula and bar are each project's own. A script ranks the table so a re-rank after each addition or edit is deterministic and cheap. Valid while one human sets the bar and reads the ranking, and while the items marked `principle` stay few enough to rank. diff --git a/rules/todo-items.md b/rules/todo-items.md index d1da96e1..872941f1 100644 --- a/rules/todo-items.md +++ b/rules/todo-items.md @@ -30,3 +30,4 @@ paths: - **The Exempt cell keeps an item whatever its score:** `security` for a security finding, `defect` for a 🔴 finding or an item that restores a maintain goal the project breaks today, `question` for a question only the human can answer, which ranks first in its release, `decision` for an item `docs/decisions.md` cites, `owner` for an item the human told you to keep, and, where no other reason fits, `principle` for a 🟡 finding against a file in the config's principles folder, whose item's detail names that principle. Every item `docs/decisions.md` cites carries one of these reasons. - **A maintain goal vetoes an item that breaks it,** whatever the item scores: redesign the item so it holds the goal, or drop it. - **When you remove an item that no chunk built, title the commit `Drop item : `,** or `Drop items 12, 14 and 15: ` for several. Such an item was deleted under the bar, vetoed, superseded or moot; a finished item is not one. Without the title, `score-outcomes` credits the item with the work of the chunk that removed it, where that chunk changed more than `todo.md`. +- **Predict in its detail what each item of the next release will touch.** When you plan the next release, or file or move an item into it, give the item's detail a `Touches:` line, then one bullet per file the item will change. Each bullet names the file's parts the item will change: a function, a class, a section heading, or line ranges where the file has neither. Read the code first; a prediction from the item's text alone misses. `~/.claude/bin/touch-predictions --help` shows the spelling and how a part is named. Mark a part `(small)` where the item changes only a few lines of it. Leave out the plan and the project's records, which every chunk writes. `chunk-preflight` reads the predictions to name the items that may start chunks at once, and `touch-predictions record` keeps each beside what its chunk touched. diff --git a/skills/larv-review/SKILL.md b/skills/larv-review/SKILL.md index 521335ba..ffe7f0ef 100644 --- a/skills/larv-review/SKILL.md +++ b/skills/larv-review/SKILL.md @@ -201,7 +201,7 @@ Launch prose-reviewer with what the diff touched and every section this review c 1. Read the finished PR against the goals file's goals and code health, unless that file voids it: does anything in it work against one? It breaks a maintain goal → do not merge; redesign the change or drop its item. It works against an achieve goal → do not merge; ask the human, stating the conflict. 2. Run the comprehension check (§3) where it runs; it runs once. When `origin/` moves after that, return to §4, and keep the ruling and scoring run already recorded. -3. Remove the chunk's items from `todo.md` per `~/.claude/CLAUDE.md` → "Working loop" in the branch, commit and push that; the commit re-enters no pass. +3. Run `~/.claude/bin/touch-predictions record ...` in the worktree, naming the chunk's items: it records what they were predicted to touch beside what the branch touched, where the project's config names that record. Then remove the chunk's items from `todo.md` per `~/.claude/CLAUDE.md` → "Working loop" in the branch. Commit and push both; the commit re-enters no pass. 4. Aligned and every pass done → **merge** with `~/.claude/bin/chunk-pr merge`, run in the background with its `timeout` set to 7200000 ms, since its wait can outlast the Bash tool's default. It waits for the pull request's checks to pass, and for a human's approval where the project's config sets `wait-for-approval` (`chunk-pr --help`), then takes the first merge method the repository allows on its forge. A repo whose `AGENTS.md` names another method merges that way with the forge's CLI instead, once the pull request's checks pass, `~/.claude/bin/larv-review scoring-run-owed` exits 0 and, where `~/.claude/bin/project-config wait-for-approval` prints `true`, GitHub reports the pull request approved. A repo withholds this in its `AGENTS.md` → leave the PR ready and unmerged, and say which line withheld it. 5. Merged → run `~/.claude/bin/chunk-pr clean` in the chunk's worktree, and put a failure it prints in the summary. 6. Stop. The next chunk gets a fresh session.