Answer stability round 1: 0 🔴, 1 🟡 (0 misread); print each round's opening line, not a label read off it, and sort the allowlist entries
This commit is contained in:
@@ -1,21 +1,20 @@
|
||||
# Stability skip count
|
||||
|
||||
This count asks one question: would skipping stability round 2, where round 1's findings had a mechanical answer, have skipped a round that blocked?
|
||||
This count asks one question: would skipping stability round 2 have skipped a round that blocked, where round 1's findings each asked only that a named mutant be killed or a named failing test pass?
|
||||
It ran on 2026-11-12, over loop-claude's archive in `~/.claude/loop-claude-transcripts/`. `docs/decisions.md` → `review: a fix for a 🔴 reruns the pass that found it…` records what was decided from it, and `skills/larv-review/SKILL.md` §3 → "Re-entry" → "Mechanical answers" holds the rule.
|
||||
|
||||
## Terms
|
||||
|
||||
- **Report:** a stability-reviewer run in the archive, with the round number its launch names.
|
||||
- **Round before:** the previous report in the same archived cycle.
|
||||
- **Report before:** the previous report in the same archived cycle.
|
||||
- **Mechanical:** a finding that asks only that a named mutant be killed or a named failing test pass.
|
||||
|
||||
## Method
|
||||
|
||||
`python3 -I stability-skip-count/count.py` prints the totals, then each report of round 2 or later whose round before reported a 🔴 and whose every 🔴 and 🟡 line matches mutant or test-failure wording. The match casts wide; reading each printed round by hand decides which are mechanical. The counts grow as the archive does.
|
||||
`python3 -I stability-skip-count/count.py` prints the totals, then each report of round 2 or later whose report before held a 🔴 and had every 🔴 and 🟡 line match mutant or test-failure wording, with that report's opening line and the report before's findings. The match casts wide; reading each printed round by hand decides which are mechanical. It reads only reports begun before its `UNTIL`, so a rerun on this machine prints the same counts. It needs the archive, which only the machine that ran the cycles holds.
|
||||
|
||||
## Results
|
||||
|
||||
- 326 reports of round 2 or later, from 13 projects; 81 open with BLOCK.
|
||||
- 5 rounds before matched; the round 2 after each of the 5 approved, so no BLOCK falls in a round the skip could skip.
|
||||
- 5 reports before matched, and the round 2 after each of the 5 approved, so no BLOCK falls in a round the skip could skip.
|
||||
- Read by hand, 1 of the 5 is mechanical: fejkdata, 2026-11-04, a pin test that fails. The other 4 matched on wording alone: a test's structure, a workflow condition, a failing type check and a missed timing target.
|
||||
- A rule counting only the 🔴s would have skipped PR #503's round 2 in this repo, on 2026-11-12. Round 1's only 🔴 was a red suite, and round 2 blocked on a 🟡 whose answer was wrong, which no test or mutant showed. So the rule counts the 🟡s too.
|
||||
|
||||
@@ -3,12 +3,16 @@ import collections
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
ARCHIVE = pathlib.Path.home() / ".claude/loop-claude-transcripts"
|
||||
# The count recorded in README.md and docs/decisions.md read the reports that began before this; later ones would change it.
|
||||
UNTIL = "2026-11-12T10:50:00Z"
|
||||
# A line that opens a 🔴 or 🟡 finding, after any list, number or heading marks.
|
||||
FINDING = re.compile(r"^\s*(?:[-*]\s*|\d+\.\s*|#+\s*)*(?:\*\*)?(🔴|🟡)")
|
||||
# Wording of a finding that names a surviving mutant or a failing test. It casts wide; the hand reading decides.
|
||||
MECHANICAL = re.compile(r"surviv|mutant|fail(s|ing|ed)? (test|on)|test (fails|failed|failing)|tests? (still )?fail|\bred\b|suite (is red|fails)", re.I)
|
||||
# A report whose verdict line, at the start of a line, says BLOCK. A verdict stated mid-sentence isn't counted, so each printed round shows its opening line too.
|
||||
BLOCK = re.compile(r"^\W*(Verdict:?\s*)?\**BLOCK", re.M)
|
||||
|
||||
|
||||
@@ -48,16 +52,19 @@ def rounds() -> list[dict]:
|
||||
"at": entries[0].get("timestamp", "") if entries else "",
|
||||
"round": int(number.group(1)) if number else 0,
|
||||
"block": bool(BLOCK.search(last)),
|
||||
"opens": last.strip().split("\n")[0][:120],
|
||||
"findings": [line.strip() for line in last.splitlines() if FINDING.match(line)],
|
||||
})
|
||||
return found
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Print the totals, then each round of 2 or later whose round before reported a 🔴 and whose every 🔴 and 🟡 line reads as mechanical."""
|
||||
every = rounds()
|
||||
"""Print the totals, then each report of round 2 or later where the report before it, in the same cycle, held a 🔴 and every 🔴 and 🟡 line of that report before reads as mechanical."""
|
||||
if not ARCHIVE.is_dir():
|
||||
sys.exit(f"No archive at {ARCHIVE}: the count reads loop-claude's transcripts on the machine that ran them.")
|
||||
every = [run for run in rounds() if run["at"] < UNTIL]
|
||||
later = [run for run in every if run["round"] >= 2]
|
||||
print(f"{len(later)} reports of round 2 or later, {sum(run['block'] for run in later)} opening with BLOCK, from {len({run['project'] for run in every})} projects")
|
||||
print(f"{len(later)} reports of round 2 or later, {sum(run['block'] for run in later)} opening with BLOCK, from {len({run['project'] for run in later})} projects")
|
||||
by_cycle = collections.defaultdict(list)
|
||||
for run in every:
|
||||
by_cycle[(run["project"], run["cycle"])].append(run)
|
||||
@@ -66,7 +73,7 @@ def main() -> None:
|
||||
for before, run in zip(runs, runs[1:]):
|
||||
reds = [line for line in before["findings"] if "🔴" in line[:12]]
|
||||
if run["round"] >= 2 and reds and all(MECHANICAL.search(line) for line in before["findings"]):
|
||||
print(f"\n{run['project']} {run['at'][:16]} round {run['round']}: {'BLOCK' if run['block'] else 'approved'}")
|
||||
print(f"\n{run['project']} {run['at'][:16]} round {run['round']}, opening: {run['opens']}")
|
||||
for line in before["findings"]:
|
||||
print(f" before: {line[:300]}")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user