Files
claude-config/docs/scoring-and-goals-research.md

6.9 KiB
Raw Permalink Blame History

Scoring and goals research

The evidence behind how todo.md scores an item and how a README states its goals. Read it only when changing that method; daily work needs only rules/todo-items.md and rules/goals-and-audience.md. docs/decisions.md → todo.md: … records what was chosen from it. Gathered on 2026-10-15 by web research; a source marked † was read only as an abstract or a secondary summary.

Risk is likelihood, escape and impact, added on log scales

  • Multiplying 1–10 labels misleads. FMEA's risk priority number, S × O × D, gives one product for different risks (2×6×5 = 3×4×5), because ordinal codes carry order, not size (Stevens 1946, Science 103:677). AIAG-VDA replaced it in 2019 with a lookup that ranks severity first†. Risk matrices compress ranges and can rank worse than chance when likelihood and impact move oppositely (Cox 2008, Risk Analysis, doi:10.1111/j.1539-6924.2008.01030.x).
  • Give each level a real quantity, about ×10 per step, and add the levels. Adding log-scale levels multiplies the underlying probability and cost, which is what risk is (Kaplan & Garrick 1981, Risk Analysis, doi:10.1111/j.1539-6924.1981.tb01350.x†).
  • Take the worst impact dimension, not a sum. NIST defines impact level as "the assessed worst-case potential impact" (csrc.nist.gov/glossary/term/impact_level). CVSS v4 buckets impacts by their High values (first.org/cvss/v4-0/specification-document).
  • Impact dimensions: confidentiality, integrity and availability (CVSS v4); reach, the people and systems a failure spreads to (NIST; CVSS "subsequent systems"; Hyrum's law, hyrumslaw.com); reversibility, one-way versus two-way doors (Bezos, 2015 shareholder letter); duration, until users stop feeling pain (Google Cloud blog, 2019).
  • Count detection once, as its own axis. FMEA rates detection apart from severity†. DORA's change failure rate measures likelihood, not impact (dora.dev/guides/dora-metrics).

What predicts that a change brings a defect

Every study below predicts from the diff or history, none from a description, so for a planned item these are priors.

  1. Size, the strongest. Added lines alone beat CC2Vec and DeepJIT on over 310,000 changes (Zeng et al. 2021, ISSTA, doi:10.1145/3460319.3464819). Relative churn predicts well (Nagappan & Ball 2005, ICSE†).
  2. Spread across subsystems, directories and files (Kamei et al. 2013, TSE, doi:10.1109/TSE.2012.70†; Hassan 2009, ICSE, change entropy†).
  3. Defect history of the touched code: about a tenth of files held 73–95% of faults (Kim et al. 2007, ICSE†).
  4. Familiarity and ownership (Bird et al. 2011, FSE†; contested in open source, Foucault et al. 2014†).
  5. Review coverage and participation (McIntosh et al. 2016, EMSE).
  6. Live config or infrastructure as code. About 70% of Google's outages come from changes to a live system (sre.google/sre-book/introduction). Syntax and configuration defects were 36.5–62.7% of IaC defects at four organisations (Rahman et al. 2018, arXiv:1809.07937).
  7. New technology or AI-written code, with mixed evidence (Pearce et al. 2022, S&P†; Perry et al. 2023, CCS†; Spracklen et al. 2025, USENIX Security).

Complexity metrics add almost no information once size is accounted for (El Emam et al. 2001, TSE, doi:10.1109/32.935855†). So a separate difficulty score beside size, spread and novelty measures nothing new.

Value per cost, not a sum

  • Order by value over cost. Weight over duration minimises total weighted waiting time on one machine (Smith's rule, 1956†); WSJF is that rule with cost of delay as the weight (Reinertsen, The Principles of Product Development Flow, 2009). An additive score has no such derivation, and its order changes with the unit of size.
  • WSJF counts reducing risk as value (framework.scaledagile.com/wsjf).
  • No source grounds a fixed score bar. DSDM fills a timebox by priority up to its capacity† (MoSCoW, DSDM v4.2, 2006).
  • Pairwise comparison is the best-supported method (Karlsson, Wohlin & Regnell 1998, IST, doi:10.1016/S0950-5849(97)00053-0; Bukhsh et al. 2020, Computer Standards & Interfaces, doi:10.1016/j.csi.2019.103389).
  • LLM scorers: absolute ratings bunch near the middle or top (arXiv:2509.24678†); pairwise prompting beat pointwise by over 10% on document ranking (Qin et al., arXiv:2306.17563); worked examples cut story-point error about 59% (Tawosi et al., arXiv:2403.08430†); swapping the order of two candidates can flip a judgement (Wang et al., arXiv:2305.17926).

Goals and audience

Almost none of this is tested on software projects; the firmer evidence is goal-setting in general.

  • Specific, difficult goals beat vague ones (Locke & Latham 2002†; Epton et al. 2017, 141 RCT papers, d = .34†). A narrow goal neglects what it leaves out (Ordóñez et al. 2009, "Goals gone wild"†), and a metric made a target gets gamed (Manheim & Garrabrant 2018, arXiv:1803.04585).
  • Maintain goals differ from achieve goals. KAOS separates goals that hold a state from goals that reach one (van Lamsweerde 2001, RE'01). Anthropic's constitution keeps its hard limits outside its weighing (anthropic.com/constitution).
  • Weights from a ranking: against weights people actually gave, sum-reciprocal weights did best and rank-order centroid worst of the rank rules (Burk & Nehring 2023, Decision Analysis†).
  • Written guidance can fail to change behaviour. Repository context files did not generally raise agents' task success and added over 20% inference cost (arXiv:2602.11988†), so a project's goals need a test that they still decide cases.
  • Personas have uneven, weakly evidenced use (Nielsen & Hansen 2014, CHI†; arXiv:2403.15917).

Calibrating scores against outcomes

  • Log each guess with an outcome window fixed in advance, such as a 🔴 in review or a later fix within N days, and compare guesses with what happened. The Brier score measures it (Brier 1950, doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2†); Murphy (1973†) splits it into reliability (do L3 items fail about as often as L3 says?) and resolution (do higher scores fail more?).
  • When a level misses by a factor of ten, fix its anchor, not the formula, as reference-class forecasting corrects estimates against similar past cases (Flyvbjerg 2006†; Lovallo & Kahneman 2003†).
  • Anchor each level to observable cases, two or three already built (Smith & Kendall 1963, "unambiguous anchors for rating scales"†), and keep the axes' weights equal (Dawes 1979†).
  • Expect overconfidence: people's "90%" ranges hold about 11 times in 20 (Hubbard, How to Measure Anything†).
  • Routing confounds outcomes: a high score buys more review, which finds more. Meta randomised review on a slice of risky diffs to measure it (arXiv:2410.06351†).

This repo's own measurements

score-outcomes on 2026-10-15: the median lines changed by a chunk at guessed S 3, 4, 5, 6 and 7 were 16, 78, 160.5, 511.5 and 1149, roughly ×2.5 per step.