230 lines
14 KiB
Python
230 lines
14 KiB
Python
"""Show what finished chunks measured beside the scores their items were guessed at, to score a new item by.
|
|
|
|
usage: score-outcomes [L E I S]
|
|
|
|
Run it inside a project checkout or one of its worktrees. It fetches origin, when the checkout has
|
|
that remote, and reads origin's default branch, or else the local main or master; a failed fetch
|
|
reads origin as last fetched and says so on stderr.
|
|
|
|
Git history is the record. The default branch's first-parent line is cut into chunks by one rule: a
|
|
merge commit is a chunk of its own, and any other commit joins the run of the newer commit above it when
|
|
both share a commit time, else starts a chunk. A chunk whose change removed items from todo.md, and
|
|
changed more than todo.md and the records below, shipped those items. A commit title in the chunk that
|
|
opens `Drop item 69: <why>` or `Drop items 69, 70 and 71: <why>` takes those items out, since the chunk
|
|
did not build them. Each item's guess is the scores it held in the todo.md the chunk started from: L, E,
|
|
I, R and S. An item scored before L, E and I replaced C holds R and S, and C too where its table also
|
|
had A, G and Goals. The chunk measures:
|
|
|
|
- lines changed: lines added plus lines deleted against the first parent, leaving out todo.md and
|
|
the records tools append to, docs/comprehension-history.md and docs/reading-cost.md;
|
|
- files: the files those lines are in;
|
|
- review rounds past each pass's first: among the branch's `Answer <pass> round <n>:` commit
|
|
titles, each pass's rounds as `larv-review round-count` counts them, less one, summed;
|
|
- 🔴 and 🟡 findings: summed over those titles, each opening `<k> 🔴, <m> 🟡` after the colon,
|
|
such as `Answer stability round 1: 1 🔴, 2 🟡 (1 misread); flag a deleted test`. Unknown where a
|
|
round title lacks the counts, or the branch has no round;
|
|
- misread findings: how many of those 🔴 and 🟡 findings a reviewer marked `[misread]`, the
|
|
`(<j> misread)` after the 🟡 count, summed over the titles. Unknown where a round title lacks
|
|
that count, as every title written before the count does, or the branch has no round;
|
|
- later fixes: the later chunks that fixed code it shipped, each counted once. A later chunk fixed
|
|
it where a change of the later chunk removes or rewrites lines, outside todo.md and the records,
|
|
that `git blame` traces to a commit this chunk brought to the default branch. The change is the
|
|
later chunk's whole change where it ships a todo.md item whose Exempt cell is `defect` or
|
|
`security`, or one of its round commits that may answer a misread 🔴. A title counts misread 🔴
|
|
and 🟡 findings together, so this takes each round commit whose title counts both a 🔴 and a
|
|
misread finding. A fixing chunk that also ships other items counts all of its change, so the
|
|
count can run high; a fix that ships no such item, and answers no such round, goes uncounted. A
|
|
recent chunk has had less time to gather any.
|
|
|
|
A chunk is marked `by rebase` where a commit in it has a committer time apart from its author time,
|
|
since a rebase replays a pull request's commits at once, and `never replayed` where none does, since
|
|
a pushed or fast-forwarded commit may be part of a pull request. A merge's titles are its own and
|
|
its branch's commits'; any other chunk's are its own. Two pull requests merged in the same second
|
|
read as one chunk. Rebase drops a round's commit that changed no file, so a rebase or push chunk
|
|
gives its rounds and findings as lower bounds, `at least`, and the report ends saying all this.
|
|
|
|
Given L, E and I, on the levels ~/.claude/rules/todo-items.md defines, and S, it works out R from
|
|
L, E and I, as rank_todo.py does, and lists the 5 finished items whose guesses sit closest, closest
|
|
first: the least mean squared difference over the scores both hold, so R and S alone for an item
|
|
scored before L, E and I, the newest first among equals. Each line gives what the item's chunk
|
|
measured, which items shipped with it, and its tagline. Where the item's tagline or detail names
|
|
tracked files or folders in backticks, each by its path from the repository root such as
|
|
`bin/rank_todo.py`, the line also gives their size, after the later fixes: how many files they
|
|
are, and how many lines they held before the chunk. A named folder counts the files it holds. This is the
|
|
size of the code the item said it would change; the chunk's change may reach beyond it. Where the
|
|
todo.md a chunk started from holds a ## Details section that cannot be read, it says so on stderr
|
|
and sizes that chunk's items by their taglines alone.
|
|
|
|
Then the project's anchors, the shipped items its docs/score-anchors.md names for each level of L,
|
|
E and I, read from the checkout's working tree, each line as above, or that no chunk shipped it. A
|
|
file that cannot be read, a row that does not hold a level on its scale and a list of IDs, or a
|
|
level named twice leaves the anchors out, saying why.
|
|
|
|
Then the median of the outcome R and S each predict at each value they were guessed at: R against 🔴
|
|
findings, S against lines changed. And how well L, E and I each order chunks by review rounds past
|
|
each pass's first and by 🔴 findings: Spearman's rank correlation, from -1 to +1, where +1 orders
|
|
them exactly; none where every chunk held the same level, or the same outcome. It counts only chunks
|
|
that shipped one item, since a chunk measures all its items at once. It leaves out the chunks never
|
|
replayed, which may hold only part of a pull request. It reports a score only once 10 of the chunks
|
|
it counts hold it and its outcomes. Where more than half of the chunks it counts for L hold no 🔴, L's
|
|
line gives its rank correlation by review rounds alone. A second line gives the share of chunks with a 🔴 at each L.
|
|
|
|
Then, newest first, each finished chunk's 🔴 and 🟡 findings with its misread findings, for every
|
|
chunk whose round titles count them: code health's counter-signal in
|
|
~/.claude/CLAUDE.md → "Shared goal".
|
|
|
|
Then, newest first, each finished chunk's later fixes, naming the fixing chunks, for every chunk
|
|
that has any, then how many have none: the defects that reappear in what already shipped, the
|
|
other half of that counter-signal.
|
|
|
|
Then, newest first, how many of the human's typed answers each finished chunk's judge quoted as
|
|
reversing a choice the worker took on its own recommendation, for every chunk loop-claude finished
|
|
that has any, then how many have none: how often a choice the worker took on its own recommendation
|
|
fails to hold. It reads loop-claude's archive, and counts only chunks whose verdicts record a
|
|
Config line, since earlier judges didn't look for reversals.
|
|
|
|
Exit 1, saying why, on bad arguments, when git cannot start, finds no checkout or reads a shallow
|
|
clone, or the default branch cannot be read; or, after the rest of the report, when the anchors or
|
|
loop-claude's archive cannot be read.
|
|
"""
|
|
import functools
|
|
import os
|
|
import pathlib
|
|
import re
|
|
import sys
|
|
|
|
from score_outcomes.anchors import anchor_lines, Anchors, ANCHORS_PATH, parse_anchors
|
|
from branch_chunks.read import branch_chunks, unsupported
|
|
from git_cli.git_repo import default_branch, full_history, git, top
|
|
from failure import Failure
|
|
from score_outcomes.results import file_lines, fixed_later, item_text, misreads, nearest, render, report, Result, results_of
|
|
from plan.todo_table import item_details, LEVELS, risk, TOP_SCORE
|
|
from text_files.markdown_text import listed
|
|
from plan.named_code import named_files
|
|
from project_config.main import project, title_templates
|
|
from transcript_archive.location import project_archive
|
|
from score_outcomes.reversals import reversal_lines
|
|
from record_store.stores import store
|
|
|
|
# The scores score-outcomes takes, in order.
|
|
GIVEN = ("L", "E", "I", "S")
|
|
# How the report reads the chunks that left no merge commit, which history bounds only by their shared commit time.
|
|
UNMERGED = [
|
|
"Chunks that left no merge commit:",
|
|
"",
|
|
"- Each is the run of commits on the default branch that share one commit time, since a rebase merge replays a pull request's commits at once. Its guesses come from the todo.md before its first commit.",
|
|
"- Two pull requests merged in the same second read as one chunk, so a pull request that shipped no item adds its lines, files and rounds to the chunk it shares a time with.",
|
|
"- A run whose commits all kept their author time was never replayed: it was pushed or fast-forwarded as committed. It may hold only part of its pull request, so the medians and the rank correlations leave it out.",
|
|
"- Rebase drops a round's commit that changed no file, so their rounds and findings are at least what history shows, and the medians count the rebased ones as shown.",
|
|
]
|
|
|
|
|
|
def main(argv: list[str]) -> int:
|
|
if argv[:1] in (["-h"], ["--help"]):
|
|
print(__doc__.strip())
|
|
return 0
|
|
if argv and (len(argv) != len(GIVEN) or not all(re.fullmatch(r"[1-9][0-9]?", arg) and int(arg) <= LEVELS.get(name, TOP_SCORE) for name, arg in zip(GIVEN, argv))):
|
|
scales = listed([f"{name} from 1 to {LEVELS.get(name, TOP_SCORE)}" for name in GIVEN], "and")
|
|
print(f"score-outcomes: cannot read the arguments {' '.join(argv)}; usage: score-outcomes [L E I S], {scales}", file=sys.stderr)
|
|
return 1
|
|
found = default_branch(stale_ok=True)
|
|
if isinstance(found, Failure):
|
|
return failed(found)
|
|
default, note = found
|
|
if note:
|
|
print(f"score-outcomes: {note}", file=sys.stderr)
|
|
results = read_results(default)
|
|
if isinstance(results, Failure):
|
|
return failed(results)
|
|
anchored = read_anchors()
|
|
reversal = read_reversals()
|
|
print(report_text(results, anchored, dict(zip(GIVEN, map(int, argv))) if argv else None, reversal))
|
|
unread = [found for found in (anchored, reversal) if isinstance(found, Failure)]
|
|
for failure in unread:
|
|
failed(failure)
|
|
return 1 if unread else 0
|
|
|
|
|
|
def report_text(results: list[Result], anchored: Anchors | Failure, given: dict[str, int] | None, reversal: list[str] | Failure) -> str:
|
|
"""The report: the finished items nearest the given scores, where any are given, then the anchors, then the medians, then each chunk's misread findings, then its later fixes, then its reversals, as reversal_lines gives them."""
|
|
sections = []
|
|
if given is not None:
|
|
r = risk(given["L"], given["E"], given["I"])
|
|
near = [render(result) for result in nearest(results, given | {"R": r})] or ["- None: no chunk on the default branch shipped an item from todo.md."]
|
|
sections.append([f"Finished items nearest L {given['L']}, E {given['E']}, I {given['I']}, so R {r}, and S {given['S']}, closest first:", "", *near])
|
|
sections.append([f"Anchors in {ANCHORS_PATH}, each level's shipped items:", "", *anchor_lines(anchored, results)])
|
|
sections.append(["Median outcome at each guessed score, among chunks that shipped one item, never-replayed ones left out:", "", *report(results)])
|
|
sections.append(["🔴 and 🟡 findings, with how many a reviewer marked `[misread]`, per finished chunk whose round titles count them, newest first:", "", *misreads(results)])
|
|
sections.append(["Later fixes: the chunks that fixed code each finished chunk shipped, newest first:", "", *fixed_later(results)])
|
|
sections.append(["Reversals: the human's typed answers that reversed a choice the worker took on its own recommendation, per chunk loop-claude finished, newest first:", "", *([f"- None: {reversal.reason}"] if isinstance(reversal, Failure) else reversal)])
|
|
if any(result.landing != "merge" for result in results):
|
|
sections.append(UNMERGED)
|
|
return "\n\n".join("\n".join(section) for section in sections)
|
|
|
|
|
|
def failed(failure: Failure) -> int:
|
|
print(f"score-outcomes: {failure.reason}", file=sys.stderr)
|
|
return 1
|
|
|
|
|
|
def read_anchors() -> Anchors | Failure:
|
|
"""The project's anchors as they stand in the store its config names, so a branch that edits them shows its edit; none where it keeps none."""
|
|
config = project()
|
|
if isinstance(config, Failure):
|
|
return config
|
|
text = store(config, "score-anchors").read()
|
|
return text if isinstance(text, Failure) else parse_anchors(text)
|
|
|
|
|
|
def read_reversals() -> list[str] | Failure:
|
|
"""The reversals in loop-claude's archive of this project, which the main checkout's path keys, whichever worktree this runs in."""
|
|
common = git("rev-parse", "--path-format=absolute", "--git-common-dir")
|
|
if isinstance(common, Failure):
|
|
return common
|
|
return reversal_lines(project_archive(str(pathlib.Path(common.strip()).parent), pathlib.Path.home(), os.environ))
|
|
|
|
|
|
def read_results(default: str) -> list[Result] | Failure:
|
|
"""What each chunk on the default branch's first-parent line measured."""
|
|
if partial := full_history():
|
|
return partial
|
|
if why := unsupported(None, pathlib.Path.home()):
|
|
return Failure(f"this project is not supported yet: {why}")
|
|
root = top()
|
|
if isinstance(root, Failure):
|
|
return root
|
|
run = functools.partial(git, cwd=root)
|
|
templates = title_templates(None, pathlib.Path.home())
|
|
if isinstance(templates, Failure):
|
|
return templates
|
|
chunks = branch_chunks(default, run, templates)
|
|
if isinstance(chunks, Failure):
|
|
return chunks
|
|
empty_tree = run("hash-object", "-t", "tree", os.devnull)
|
|
if isinstance(empty_tree, Failure):
|
|
return empty_tree
|
|
results: list[Result] = []
|
|
for chunk in chunks:
|
|
if chunk.before is None or not chunk.shipped:
|
|
continue
|
|
numstat = run("diff", "--numstat", chunk.before, chunk.tip)
|
|
if isinstance(numstat, Failure):
|
|
return numstat
|
|
plan = run("show", f"{chunk.before}:todo.md")
|
|
if isinstance(plan, Failure):
|
|
return plan
|
|
found = item_details(plan)
|
|
if isinstance(found, Failure):
|
|
print(f"score-outcomes: the todo.md before {chunk.label} cannot be read for its details, so its items are sized by their taglines alone: {found.reason}", file=sys.stderr)
|
|
details = {} if isinstance(found, Failure) else found
|
|
tracked = run("ls-tree", "-r", "-z", "--name-only", chunk.before)
|
|
if isinstance(tracked, Failure):
|
|
return tracked
|
|
# Count only the files the items name, since counting every file's lines for every chunk doubles the run.
|
|
named = list(dict.fromkeys(path for item in chunk.shipped for path in named_files(item_text(item, details), tracked.split("\0")[:-1])))
|
|
tree = run("--literal-pathspecs", "diff", "--numstat", "-z", empty_tree.strip(), chunk.before, "--", *named) if named else ""
|
|
if isinstance(tree, Failure):
|
|
return tree
|
|
results += results_of(chunk, numstat, details, file_lines(tree))
|
|
return results
|