Files
claude-config/bin/score_outcomes/main.py
T

230 lines
14 KiB
Python

"""Show what finished chunks measured beside the scores their items were guessed at, to score a new item by.
usage: score-outcomes [L E I S]
Run it inside a project checkout or one of its worktrees. It fetches origin, when the checkout has
that remote, and reads origin's default branch, or else the local main or master; a failed fetch
reads origin as last fetched and says so on stderr.
Git history is the record. The default branch's first-parent line is cut into chunks by one rule: a
merge commit is a chunk of its own, and any other commit joins the run of the newer commit above it when
both share a commit time, else starts a chunk. A chunk whose change removed items from todo.md, and
changed more than todo.md and the records below, shipped those items. A commit title in the chunk that
opens `Drop item 69: <why>` or `Drop items 69, 70 and 71: <why>` takes those items out, since the chunk
did not build them. Each item's guess is the scores it held in the todo.md the chunk started from: L, E,
I, R and S. An item scored before L, E and I replaced C holds R and S, and C too where its table also
had A, G and Goals. The chunk measures:
- lines changed: lines added plus lines deleted against the first parent, leaving out todo.md and
the records tools append to, docs/comprehension-history.md and docs/reading-cost.md;
- files: the files those lines are in;
- review rounds past each pass's first: among the branch's `Answer <pass> round <n>:` commit
titles, each pass's rounds as `larv-review round-count` counts them, less one, summed;
- 🔴 and 🟡 findings: summed over those titles, each opening `<k> 🔴, <m> 🟡` after the colon,
such as `Answer stability round 1: 1 🔴, 2 🟡 (1 misread); flag a deleted test`. Unknown where a
round title lacks the counts, or the branch has no round;
- misread findings: how many of those 🔴 and 🟡 findings a reviewer marked `[misread]`, the
`(<j> misread)` after the 🟡 count, summed over the titles. Unknown where a round title lacks
that count, as every title written before the count does, or the branch has no round;
- later fixes: the later chunks that fixed code it shipped, each counted once. A later chunk fixed
it where a change of the later chunk removes or rewrites lines, outside todo.md and the records,
that `git blame` traces to a commit this chunk brought to the default branch. The change is the
later chunk's whole change where it ships a todo.md item whose Exempt cell is `defect` or
`security`, or one of its round commits that may answer a misread 🔴. A title counts misread 🔴
and 🟡 findings together, so this takes each round commit whose title counts both a 🔴 and a
misread finding. A fixing chunk that also ships other items counts all of its change, so the
count can run high; a fix that ships no such item, and answers no such round, goes uncounted. A
recent chunk has had less time to gather any.
A chunk is marked `by rebase` where a commit in it has a committer time apart from its author time,
since a rebase replays a pull request's commits at once, and `never replayed` where none does, since
a pushed or fast-forwarded commit may be part of a pull request. A merge's titles are its own and
its branch's commits'; any other chunk's are its own. Two pull requests merged in the same second
read as one chunk. Rebase drops a round's commit that changed no file, so a rebase or push chunk
gives its rounds and findings as lower bounds, `at least`, and the report ends saying all this.
Given L, E and I, on the levels ~/.claude/rules/todo-items.md defines, and S, it works out R from
L, E and I, as rank_todo.py does, and lists the 5 finished items whose guesses sit closest, closest
first: the least mean squared difference over the scores both hold, so R and S alone for an item
scored before L, E and I, the newest first among equals. Each line gives what the item's chunk
measured, which items shipped with it, and its tagline. Where the item's tagline or detail names
tracked files or folders in backticks, each by its path from the repository root such as
`bin/rank_todo.py`, the line also gives their size, after the later fixes: how many files they
are, and how many lines they held before the chunk. A named folder counts the files it holds. This is the
size of the code the item said it would change; the chunk's change may reach beyond it. Where the
todo.md a chunk started from holds a ## Details section that cannot be read, it says so on stderr
and sizes that chunk's items by their taglines alone.
Then the project's anchors, the shipped items its docs/score-anchors.md names for each level of L,
E and I, read from the checkout's working tree, each line as above, or that no chunk shipped it. A
file that cannot be read, a row that does not hold a level on its scale and a list of IDs, or a
level named twice leaves the anchors out, saying why.
Then the median of the outcome R and S each predict at each value they were guessed at: R against 🔴
findings, S against lines changed. And how well L, E and I each order chunks by review rounds past
each pass's first and by 🔴 findings: Spearman's rank correlation, from -1 to +1, where +1 orders
them exactly; none where every chunk held the same level, or the same outcome. It counts only chunks
that shipped one item, since a chunk measures all its items at once. It leaves out the chunks never
replayed, which may hold only part of a pull request. It reports a score only once 10 of the chunks
it counts hold it and its outcomes. Where more than half of the chunks it counts for L hold no 🔴, L's
line gives its rank correlation by review rounds alone. A second line gives the share of chunks with a 🔴 at each L.
Then, newest first, each finished chunk's 🔴 and 🟡 findings with its misread findings, for every
chunk whose round titles count them: code health's counter-signal in
~/.claude/CLAUDE.md → "Shared goal".
Then, newest first, each finished chunk's later fixes, naming the fixing chunks, for every chunk
that has any, then how many have none: the defects that reappear in what already shipped, the
other half of that counter-signal.
Then, newest first, how many of the human's typed answers each finished chunk's judge quoted as
reversing a choice the worker took on its own recommendation, for every chunk loop-claude finished
that has any, then how many have none: how often a choice the worker took on its own recommendation
fails to hold. It reads loop-claude's archive, and counts only chunks whose verdicts record a
Config line, since earlier judges didn't look for reversals.
Exit 1, saying why, on bad arguments, when git cannot start, finds no checkout or reads a shallow
clone, or the default branch cannot be read; or, after the rest of the report, when the anchors or
loop-claude's archive cannot be read.
"""
import functools
import os
import pathlib
import re
import sys
from score_outcomes.anchors import anchor_lines, Anchors, ANCHORS_PATH, parse_anchors
from branch_chunks.read import branch_chunks, unsupported
from git_cli.git_repo import default_branch, full_history, git, top
from failure import Failure
from score_outcomes.results import file_lines, fixed_later, item_text, misreads, nearest, render, report, Result, results_of
from plan.todo_table import item_details, LEVELS, risk, TOP_SCORE
from text_files.markdown_text import listed
from plan.named_code import named_files
from project_config.main import project, title_templates
from transcript_archive.location import project_archive
from score_outcomes.reversals import reversal_lines
from record_store.stores import store
# The scores score-outcomes takes, in order.
GIVEN = ("L", "E", "I", "S")
# How the report reads the chunks that left no merge commit, which history bounds only by their shared commit time.
UNMERGED = [
"Chunks that left no merge commit:",
"",
"- Each is the run of commits on the default branch that share one commit time, since a rebase merge replays a pull request's commits at once. Its guesses come from the todo.md before its first commit.",
"- Two pull requests merged in the same second read as one chunk, so a pull request that shipped no item adds its lines, files and rounds to the chunk it shares a time with.",
"- A run whose commits all kept their author time was never replayed: it was pushed or fast-forwarded as committed. It may hold only part of its pull request, so the medians and the rank correlations leave it out.",
"- Rebase drops a round's commit that changed no file, so their rounds and findings are at least what history shows, and the medians count the rebased ones as shown.",
]
def main(argv: list[str]) -> int:
if argv[:1] in (["-h"], ["--help"]):
print(__doc__.strip())
return 0
if argv and (len(argv) != len(GIVEN) or not all(re.fullmatch(r"[1-9][0-9]?", arg) and int(arg) <= LEVELS.get(name, TOP_SCORE) for name, arg in zip(GIVEN, argv))):
scales = listed([f"{name} from 1 to {LEVELS.get(name, TOP_SCORE)}" for name in GIVEN], "and")
print(f"score-outcomes: cannot read the arguments {' '.join(argv)}; usage: score-outcomes [L E I S], {scales}", file=sys.stderr)
return 1
found = default_branch(stale_ok=True)
if isinstance(found, Failure):
return failed(found)
default, note = found
if note:
print(f"score-outcomes: {note}", file=sys.stderr)
results = read_results(default)
if isinstance(results, Failure):
return failed(results)
anchored = read_anchors()
reversal = read_reversals()
print(report_text(results, anchored, dict(zip(GIVEN, map(int, argv))) if argv else None, reversal))
unread = [found for found in (anchored, reversal) if isinstance(found, Failure)]
for failure in unread:
failed(failure)
return 1 if unread else 0
def report_text(results: list[Result], anchored: Anchors | Failure, given: dict[str, int] | None, reversal: list[str] | Failure) -> str:
"""The report: the finished items nearest the given scores, where any are given, then the anchors, then the medians, then each chunk's misread findings, then its later fixes, then its reversals, as reversal_lines gives them."""
sections = []
if given is not None:
r = risk(given["L"], given["E"], given["I"])
near = [render(result) for result in nearest(results, given | {"R": r})] or ["- None: no chunk on the default branch shipped an item from todo.md."]
sections.append([f"Finished items nearest L {given['L']}, E {given['E']}, I {given['I']}, so R {r}, and S {given['S']}, closest first:", "", *near])
sections.append([f"Anchors in {ANCHORS_PATH}, each level's shipped items:", "", *anchor_lines(anchored, results)])
sections.append(["Median outcome at each guessed score, among chunks that shipped one item, never-replayed ones left out:", "", *report(results)])
sections.append(["🔴 and 🟡 findings, with how many a reviewer marked `[misread]`, per finished chunk whose round titles count them, newest first:", "", *misreads(results)])
sections.append(["Later fixes: the chunks that fixed code each finished chunk shipped, newest first:", "", *fixed_later(results)])
sections.append(["Reversals: the human's typed answers that reversed a choice the worker took on its own recommendation, per chunk loop-claude finished, newest first:", "", *([f"- None: {reversal.reason}"] if isinstance(reversal, Failure) else reversal)])
if any(result.landing != "merge" for result in results):
sections.append(UNMERGED)
return "\n\n".join("\n".join(section) for section in sections)
def failed(failure: Failure) -> int:
print(f"score-outcomes: {failure.reason}", file=sys.stderr)
return 1
def read_anchors() -> Anchors | Failure:
"""The project's anchors as they stand in the store its config names, so a branch that edits them shows its edit; none where it keeps none."""
config = project()
if isinstance(config, Failure):
return config
text = store(config, "score-anchors").read()
return text if isinstance(text, Failure) else parse_anchors(text)
def read_reversals() -> list[str] | Failure:
"""The reversals in loop-claude's archive of this project, which the main checkout's path keys, whichever worktree this runs in."""
common = git("rev-parse", "--path-format=absolute", "--git-common-dir")
if isinstance(common, Failure):
return common
return reversal_lines(project_archive(str(pathlib.Path(common.strip()).parent), pathlib.Path.home(), os.environ))
def read_results(default: str) -> list[Result] | Failure:
"""What each chunk on the default branch's first-parent line measured."""
if partial := full_history():
return partial
if why := unsupported(None, pathlib.Path.home()):
return Failure(f"this project is not supported yet: {why}")
root = top()
if isinstance(root, Failure):
return root
run = functools.partial(git, cwd=root)
templates = title_templates(None, pathlib.Path.home())
if isinstance(templates, Failure):
return templates
chunks = branch_chunks(default, run, templates)
if isinstance(chunks, Failure):
return chunks
empty_tree = run("hash-object", "-t", "tree", os.devnull)
if isinstance(empty_tree, Failure):
return empty_tree
results: list[Result] = []
for chunk in chunks:
if chunk.before is None or not chunk.shipped:
continue
numstat = run("diff", "--numstat", chunk.before, chunk.tip)
if isinstance(numstat, Failure):
return numstat
plan = run("show", f"{chunk.before}:todo.md")
if isinstance(plan, Failure):
return plan
found = item_details(plan)
if isinstance(found, Failure):
print(f"score-outcomes: the todo.md before {chunk.label} cannot be read for its details, so its items are sized by their taglines alone: {found.reason}", file=sys.stderr)
details = {} if isinstance(found, Failure) else found
tracked = run("ls-tree", "-r", "-z", "--name-only", chunk.before)
if isinstance(tracked, Failure):
return tracked
# Count only the files the items name, since counting every file's lines for every chunk doubles the run.
named = list(dict.fromkeys(path for item in chunk.shipped for path in named_files(item_text(item, details), tracked.split("\0")[:-1])))
tree = run("--literal-pathspecs", "diff", "--numstat", "-z", empty_tree.strip(), chunk.before, "--", *named) if named else ""
if isinstance(tree, Failure):
return tree
results += results_of(chunk, numstat, details, file_lines(tree))
return results