Answer prose round 1: 0 🔴, 11 🟡 (0 misread); correct the rank, release review, round cap and judge-quote claims, explain terms where a newcomer meets them, drop repeats and unused glossary words, and fix the banned negations
CI / tests (pull_request) Successful in 2m14s
CI / tests (pull_request) Successful in 2m14s
This commit was merged in pull request #458.
This commit is contained in:
@@ -34,7 +34,7 @@ People in tech who want to build software factories, work with AI tools more eff
|
||||
- **Colours mean roles:** amber the human, blue the worker, violet the judge, teal the reviewers, green the merge.
|
||||
- **Each page fits one landscape A4 sheet.** After an edit, rebuild the PDF and check the page count: a page that spills shows up as an extra page.
|
||||
- **Draw arrows after the section backgrounds,** or a section box hides them.
|
||||
- **Pages 2 to 17 each carry a "Why: sources …" line,** linking to the numbered rows of the "Why it works this way" pages. Each row is tagged Research, Book, Vendor docs, Benchmark, Own measurement, Experience or Reasoning.
|
||||
- **Pages 2 to 15 and 17 each carry a "Why: sources …" line,** linking to the numbered rows of the "Why it works this way" pages. Each row is tagged Research, Book, Vendor docs, Benchmark, Own measurement, Experience or Reasoning.
|
||||
|
||||
## Where the figures come from
|
||||
|
||||
|
||||
+25
-32
@@ -4,7 +4,7 @@
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Bricolage+Grotesque:opsz,wght@12..96,600;12..96,800&family=Atkinson+Hyperlegible:ital,wght@0,400;0,700;1,400&family=JetBrains+Mono:wght@400;600&display=swap">
|
||||
<style>
|
||||
/* Layout: an A4 booklet. Part 1 is one picture per page with a headline; part 2 holds the detail. Each role keeps one colour on every page. */
|
||||
/* Layout: one landscape A4 sheet per page. Each role keeps one colour on every page. */
|
||||
:root {
|
||||
--bg: #f6f7f9;
|
||||
--paper: #ffffff;
|
||||
@@ -284,7 +284,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<text class="s" x="372" y="498">next chunk, with a</text><text class="s" x="372" y="516" style="fill:var(--worker);font-weight:700">fresh context</text>
|
||||
<rect class="box" x="578" y="452" width="200" height="80" rx="7"/>
|
||||
<circle class="f-grey" cx="594" cy="471" r="6"/><text class="t" x="606" y="476" style="font-size:15px">Keep going</text>
|
||||
<text class="s" x="588" y="498">unfinished, or the</text><text class="s" x="588" y="516">records answer it</text>
|
||||
<text class="s" x="588" y="498">unfinished, or the</text><text class="s" x="588" y="516">docs answer it</text>
|
||||
<rect class="you" x="794" y="452" width="224" height="80" rx="7"/>
|
||||
<circle class="f-you" cx="810" cy="471" r="6"/><text class="t" x="822" y="476" style="font-size:15px">Human's call</text>
|
||||
<text class="s" x="804" y="498">only a human can</text><text class="s" x="804" y="516">decide; they answer</text>
|
||||
@@ -306,8 +306,8 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<section class="page" id="full">
|
||||
<div class="eyebrow">The flow, every piece</div>
|
||||
<h2>The same three sections, opened up</h2>
|
||||
<p class="lead">Same layout as the last page. Each big box there is the group of steps in the same place here.</p>
|
||||
<div class="diagram" role="img" aria-label="The full flow. The human writes goals and audience in the README; the plan in todo.md feeds A. A, Start, deterministic: prepare, pick the work, pick a model. When the release is done, the loop stops and the human cuts the release. B, Work, the worker: build, review, gate, goals check, readability panel, merge; findings and failed checks go back to build; a stuck worker moves to a stronger model; a planning chunk is drafted by the worker and approved by the human; the release review adds architecture and principles over the whole project and a before-and-after readability ruling; new items go into the plan. C, Check, the judge: finished goes back to A with a fresh context; unfinished, already settled and human's call go back to B in the same session; every 20 cycles routing figures go to the human.">
|
||||
<p class="lead">Same layout as the previous page. Each big box there is the group of steps in the same place here.</p>
|
||||
<div class="diagram" role="img" aria-label="The full flow. The human writes goals and audience in the README; the plan in todo.md feeds A. A, Start, deterministic: prepare, pick the work, pick a model. When the release is done, the loop stops and the human cuts the release. B, Work, the worker: build, review, gate, goals check, readability panel, merge; findings and failed checks go back to build; a stuck worker moves to a stronger model; a planning chunk is drafted by the worker and approved by the human; the release review adds architecture and principles over the whole project and a before-and-after readability ruling; new items go into the plan. C, Check, the judge: finished goes back to A with a fresh context; unfinished, already settled and human's call go back to B in the same session; every 20 cycles model-use figures go to the human.">
|
||||
<svg viewBox="0 0 1200 640">
|
||||
<defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path class="head" d="M0,0 L10,5 L0,10 z"/></marker></defs>
|
||||
|
||||
@@ -393,7 +393,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<text class="s" x="540" y="498">work is left;</text><text class="s" x="540" y="516">6 in a row → human</text>
|
||||
<rect class="judge" x="698" y="452" width="152" height="80" rx="7"/>
|
||||
<circle class="f-judge" cx="714" cy="471" r="6"/><text class="t" x="726" y="476" style="font-size:15px">Already settled</text>
|
||||
<text class="s" x="708" y="498">the records</text><text class="s" x="708" y="516">answer it</text>
|
||||
<text class="s" x="708" y="498">the repo's docs</text><text class="s" x="708" y="516">answer it</text>
|
||||
<rect class="you" x="866" y="452" width="152" height="80" rx="7"/>
|
||||
<circle class="f-you" cx="882" cy="471" r="6"/><text class="t" x="894" y="476" style="font-size:15px">Human's call</text>
|
||||
<text class="s" x="876" y="498">terminal, inbox</text><text class="s" x="876" y="516">or /talk</text>
|
||||
@@ -402,7 +402,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<path class="edge" d="M774,552 L774,538" marker-end="url(#ah)"/>
|
||||
<path class="edge" d="M942,552 L942,538" marker-end="url(#ah)"/>
|
||||
<rect class="you" x="866" y="572" width="318" height="50" rx="7"/>
|
||||
<text class="t" x="878" y="593" style="font-size:15px">Every 20 cycles</text><text class="s" x="878" y="612">routing figures → human decides model changes</text>
|
||||
<text class="t" x="878" y="593" style="font-size:15px">Every 20 cycles</text><text class="s" x="878" y="612">model-use figures → human decides model changes</text>
|
||||
|
||||
<!-- drawn last, so the section backgrounds never cover them -->
|
||||
<path class="edge" d="M155,166 L155,190" marker-end="url(#ah)"/>
|
||||
@@ -508,17 +508,13 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<div style="display:grid;gap:6px">
|
||||
<h3>Rank the plan, then pick the work</h3>
|
||||
<ul class="tight">
|
||||
<li><b>Rank:</b> <code>rank_todo.py</code> scores every item from its L, E, I, size and the goals it moves, and sorts the plan. A commit that changes <code>todo.md</code> waits until it is ranked.</li>
|
||||
<li><b>Rank:</b> <code>rank_todo.py</code> scores every item by the goals it moves and its size, and sorts the plan. A program refuses a commit that leaves <code>todo.md</code> unranked.</li>
|
||||
<li>An item <b>waits</b> while it is a question for the human, or while an item that blocks it is still in the plan.</li>
|
||||
<li>The chunk takes the first item that doesn't wait, plus every item of the same release that needs the same reviewers.</li>
|
||||
<li>An item with risk R above 5 ships alone. Up to ten tiny items can share a chunk.</li>
|
||||
<li>A risky item (page 12) ships alone. Up to ten tiny items can share a chunk.</li>
|
||||
</ul>
|
||||
<h3>Pick model</h3>
|
||||
<ul class="tight">
|
||||
<li><b>Worker:</b> by the item's L. L 1–2 one rung down, L 3 the project's model, L 4 one up.</li>
|
||||
<li><b>Reviewers:</b> the highest of the worker's model, the project's, and a step up for a high E + I. Readers and security: the project's model. The judge: never below it.</li>
|
||||
<li>The worker steps up after every second unfinished verdict, or when stuck. At most two rungs without asking.</li>
|
||||
</ul>
|
||||
<p>By the item's scores: page 13.</p>
|
||||
</div>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
@@ -595,7 +591,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<section class="page" id="reviewers">
|
||||
<div class="eyebrow">Zoom in: B · Work → Review</div>
|
||||
<h2>The reviewers</h2>
|
||||
<p class="lead">Each reviewer has one narrow job: a merged reviewer loses one pass's findings. They run in waves, each once the one before is done. Wave 1 runs only in the release review.</p>
|
||||
<p class="lead">Each reviewer has one narrow job. When two were merged into one, it missed findings (source 14). They run in waves, each once the one before is done. Wave 1 runs only in the release review.</p>
|
||||
<div class="diagram" role="img" aria-label="Wave 1: architecture. Wave 2: principles, security, overrides. Wave 3: stability, product owner, prose. Each wave starts when the one before it is done. Wave 1 runs only in the release review.">
|
||||
<svg viewBox="0 0 960 150">
|
||||
<defs><marker id="ah4" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path class="head" d="M0,0 L10,5 L0,10 z"/></marker></defs>
|
||||
@@ -664,12 +660,12 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<text class="s" x="768" y="98">last fixes merge</text>
|
||||
<text class="s" x="768" y="118">once gate is green</text>
|
||||
<rect x="20" y="160" width="920" height="30" rx="15" style="fill: var(--bg)"/>
|
||||
<text class="lbl" x="480" y="180" text-anchor="middle" style="font-weight:700">At most 2 rounds per reviewer per chunk. A program blocks a third.</text>
|
||||
<text class="lbl" x="480" y="180" text-anchor="middle" style="font-weight:700">At most 2 rounds per reviewer per chunk, 5 in the release review. A program blocks one more.</text>
|
||||
</svg>
|
||||
</div>
|
||||
<ul class="tight">
|
||||
<li><b>Outside this change?</b> The finding becomes a new item in <code>todo.md</code>, not a fix here.</li>
|
||||
<li><b>Only red fixes re-run reviewers.</b> A fix for yellow, green or a question re-runs nothing.</li>
|
||||
<li><b>Only red fixes re-run reviewers.</b> A fix for yellow, green or a question doesn't re-run any reviewer.</li>
|
||||
<li><b>A decline is written down</b>, in a code comment or in <code>docs/decisions.md</code>, so the next review does not raise it again.</li>
|
||||
</ul>
|
||||
<p class="why">Why: sources <a href="#src-15">15</a>, <a href="#src-16">16</a></p>
|
||||
@@ -716,8 +712,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<div class="role review"><b>Scoring run</b><p>Four readers give a 1–10 score. Runs when a release's items run out, or when an item aims at the 7.0 target.</p></div>
|
||||
<div class="role judge"><b>Paired ruling</b><p>Two readers compare before and after: better, same or worse? Runs in each release review.</p></div>
|
||||
</div>
|
||||
<p class="note">Readability blocks no merge. It steers the plan: code health ranks first among every project's goals.</p>
|
||||
<p class="why">Why: sources <a href="#src-22">22</a>, <a href="#src-23">23</a>, <a href="#src-24">24</a></p>
|
||||
<p class="note">Readability doesn't block a merge. It steers the plan: code health ranks first among every project's goals. <span class="why">Why: sources <a href="#src-22">22</a>, <a href="#src-23">23</a>, <a href="#src-24">24</a></span></p>
|
||||
</section>
|
||||
|
||||
<section class="page" id="zoom-c">
|
||||
@@ -731,7 +726,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<p>A question those files already answer. It quotes the answer word for word, and the worker goes on.</p>
|
||||
<p>A quote not found word for word goes to the human instead.</p></div>
|
||||
<div class="role judge"><b>Checks on its reply</b>
|
||||
<p>A reply in the wrong shape, or with a made-up quote, counts as a failure and is retried. Three failures in a row stop the loop.</p></div>
|
||||
<p>A reply in the wrong shape, or quoting the log wrongly, counts as a failure and is retried. Three failures in a row stop the loop.</p></div>
|
||||
</div>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
@@ -739,7 +734,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<tbody>
|
||||
<tr><td><span class="dot" style="display:inline-block;width:10px;height:10px;background:var(--green)"></span> Finished</td><td>The PR is merged, and every answer the human gave is written down.</td><td>Back to A. A new session, with a fresh, clean context.</td></tr>
|
||||
<tr><td><span class="dot" style="display:inline-block;width:10px;height:10px;background:var(--grey)"></span> Unfinished</td><td>Work is left.</td><td>Back to B. The same session goes on, told what is left.</td></tr>
|
||||
<tr><td><span class="dot" style="display:inline-block;width:10px;height:10px;background:var(--judge)"></span> Already settled</td><td>The worker asked something the records answer.</td><td>Back to B, with the quote.</td></tr>
|
||||
<tr><td><span class="dot" style="display:inline-block;width:10px;height:10px;background:var(--judge)"></span> Already settled</td><td>The worker asked something the repo's docs already answer.</td><td>Back to B, with the quote.</td></tr>
|
||||
<tr><td><span class="dot" style="display:inline-block;width:10px;height:10px;background:var(--you)"></span> Human's call</td><td>Only the human can decide.</td><td>The human types an answer, or opens <code>/talk</code>. Back to B with it.</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -871,7 +866,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
</div>
|
||||
<div class="two">
|
||||
<ul class="tight">
|
||||
<li><b>Release review:</b> the architect and the principles auditor read the whole project, up to five rounds each, and fix what they find.</li>
|
||||
<li><b>Release review:</b> the architect and the principles auditor read the whole project, up to five rounds each. The worker fixes what they find.</li>
|
||||
<li>Then the readers compare before and after.</li>
|
||||
</ul>
|
||||
<ul class="tight">
|
||||
@@ -937,7 +932,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<table>
|
||||
<thead><tr><th>Rail</th><th>What it does</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Sandbox</td><td>Runs only on a dedicated machine, such as a VM, with no access to production. Claude Code skips permission prompts there.</td></tr>
|
||||
<tr><td>Sandbox</td><td>Run it only on a dedicated machine, such as a VM with no access to production: Claude Code skips permission prompts there.</td></tr>
|
||||
<tr><td><code>rule-hook</code></td><td>Runs before each Agent, Bash, Edit, MultiEdit, Read, Skill and Write call. Blocks e.g. multi-line commit messages, unpinned images, containers without memory limits, a third review round. Some rules allow a written override; the Overrides reviewer judges each one.</td></tr>
|
||||
<tr><td><code>run-heavy</code></td><td>Heavy commands (builds, mutation tests, full test runs) wait for a shared slot: one per 4 CPUs on the machine.</td></tr>
|
||||
<tr><td>Memory</td><td>Every container gets a memory limit. A new worker starts only while a quarter of the machine's memory is free.</td></tr>
|
||||
@@ -961,10 +956,10 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
|
||||
<section class="page" id="sources">
|
||||
<h2>Why it works this way</h2>
|
||||
<p class="lead">Each choice rests on research, a book, our own measurements, or trial and error. The numbers match the "Why" line at the bottom of each page.</p>
|
||||
<p class="lead">Each choice rests on research, a book, our own measurements, or trial and error. The numbers match the "Why" line on each page.</p>
|
||||
<h3 class="group">The loop</h3>
|
||||
<div class="tablewrap"><table class="src-t"><colgroup><col style="width:4%"><col style="width:24%"><col style="width:72%"></colgroup><tbody>
|
||||
<tr id="src-1"><td>1</td><td><b>A fresh, clean context every chunk</b></td><td><span class="kind x">Reasoning</span> Every turn re-sends the whole context, and compaction loses detail. The repo holds the memory between chunks, so a new chunk loses little.<br><span class="kind m">Own measurement</span> Simulated on 90 archived chunks, compacting at about 170k tokens would save about $78, 21% of the workers' spend. 36 full rewrites of a context after a cache gap cost $61, up to 785k tokens each. Within a chunk the session goes on: a fresh context for review would save $39, which compaction already saves as part of its $78.<br><span class="kind r">Research</span> Models use facts in the middle of a long context worst: Liu et al., "Lost in the Middle", TACL 2024. Answers degrade as input grows: Hong et al., "Context Rot", Chroma Research, 2025.</td></tr>
|
||||
<tr id="src-1"><td>1</td><td><b>A fresh, clean context every chunk</b></td><td><span class="kind x">Reasoning</span> Every turn re-sends the whole context, and compaction loses detail. The repo holds the memory between chunks, so a new chunk loses little.<br><span class="kind m">Own measurement</span> Simulated on 90 archived chunks, compacting at about 170k tokens would save about $78, 21% of the workers' spend. 36 times the cache had expired and the whole context was sent again, costing $61, up to 785k tokens each. Within a chunk the session goes on: a fresh context for review would save $39, but compaction already saves that as part of its $78.<br><span class="kind r">Research</span> Models use facts in the middle of a long context worst: Liu et al., "Lost in the Middle", TACL 2024. Answers degrade as input grows: Hong et al., "Context Rot", Chroma Research, 2025.</td></tr>
|
||||
<tr id="src-2"><td>2</td><td><b>Several items per chunk; risky items alone</b></td><td><span class="kind m">Own measurement</span> Over 165 chunks, one item cost a median 1.30M tokens and two items 1.33M. Review costs about the same per chunk, so sharing it is cheap. At risk R 6 the median rose to 2.7M, and one in ten passed 8.8M, so R above 5 ships alone.</td></tr>
|
||||
<tr id="src-3"><td>3</td><td><b>The worker makes its own small choices</b></td><td><span class="kind m">Own measurement</span> 92 of 255 interrupts already held the worker's recommendation. A chunk with an interrupt took a median 7.1 hours, against 0.44 hours without one. So the worker takes its recommendation on any choice no rule reserves for the human, and records it.</td></tr>
|
||||
<tr id="src-4"><td>4</td><td><b>Every answer is written down</b></td><td><span class="kind x">Reasoning</span> A finished chunk's session is deleted. An answer kept nowhere else would be asked again in a later chunk. One goal of the setup is to never ask the same thing twice.</td></tr>
|
||||
@@ -979,7 +974,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<h3 class="group">The plan</h3>
|
||||
<div class="tablewrap"><table class="src-t"><colgroup><col style="width:4%"><col style="width:24%"><col style="width:72%"></colgroup><tbody>
|
||||
<tr id="src-9"><td>9</td><td><b>Goals with a signal and a counter-signal; achieve vs. maintain</b></td><td><span class="kind r">Research</span> Specific, hard goals beat vague ones: Locke & Latham 2002. A narrow goal makes people neglect the rest: Ordóñez et al. 2009, "Goals gone wild". A measure made a target gets gamed: Manheim & Garrabrant 2018 (arXiv:1803.04585). Goals that keep a state differ from goals that reach one: van Lamsweerde 2001 (KAOS).<br><span class="kind r">Research</span> Instruction files alone did not raise agents' success (arXiv:2602.11988). So each goal names a signal that shows it still decides cases.</td></tr>
|
||||
<tr id="src-10"><td>10</td><td><b>Risk = L + E + I, each level about ×10</b></td><td><span class="kind r">Research</span> Multiplying 1–10 labels misleads: Stevens 1946; Cox 2008, <i>Risk Analysis</i>. Adding levels on a ×10 scale multiplies the real chance and cost: Kaplan & Garrick 1981. Impact is the worst case, not a sum: NIST; CVSS v4.</td></tr>
|
||||
<tr id="src-10"><td>10</td><td><b>Risk adds L, E and I, each level about ×10</b></td><td><span class="kind r">Research</span> Multiplying 1–10 labels misleads: Stevens 1946; Cox 2008, <i>Risk Analysis</i>. Adding levels on a ×10 scale multiplies the real chance and cost: Kaplan & Garrick 1981. Impact is the worst case, not a sum: NIST; CVSS v4.</td></tr>
|
||||
<tr id="src-11"><td>11</td><td><b>What makes L high: size, spread, something new, live config</b></td><td><span class="kind r">Research</span> Size predicts a defect best: Zeng et al. 2021, ISSTA. So does spread across files: Kamei et al. 2013. About 70% of Google's outages come from changes to a live system: Google SRE book. Complexity metrics add little once size is known: El Emam et al. 2001.</td></tr>
|
||||
<tr id="src-12"><td>12</td><td><b>Order the plan by value over size</b></td><td><span class="kind b">Book</span> Doing the highest value per unit of work first minimises total waiting: Smith's rule, 1956; Reinertsen, <i>The Principles of Product Development Flow</i>, 2009 (WSJF).</td></tr>
|
||||
<tr id="src-13"><td>13</td><td><b>Check every score against finished chunks</b></td><td><span class="kind r">Research</span> Compare guesses with outcomes: Brier 1950. Correct estimates against similar past cases: Flyvbjerg 2006. Anchor each level to real examples: Smith & Kendall 1963. People are overconfident: Hubbard, <i>How to Measure Anything</i>.<br><span class="kind m">Own measurement</span> <code>score-outcomes</code> shows each finished item's guessed scores beside what its chunk measured.</td></tr>
|
||||
@@ -992,7 +987,7 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<tr id="src-14"><td>14</td><td><b>Narrow, separate reviewers</b></td><td><span class="kind x">Experience</span> Early on, a reviewer with too broad a responsibility found fewer issues.<br><span class="kind m">Own measurement</span> On 2026-11-09, merged reviewers were replayed on archived rounds. Stability, product owner and prose as one reviewer cost half as much, but found 15 of 34 prose findings again, against 22–23 for the prose reviewer. Architect and principles auditor as one found 5 of 17 principles findings, against 10, and lost both red ones. Even stability and product owner alone, as one, found 4–5 of 16 product-owner findings, against 7–10. In every merge, one absorbed pass lost findings, at a third to half the cost. Method and figures: <code>reviewer-merge-trials/</code> in the setup's repo. Merging reviewers that check the same kind of thing did hold up: one principles auditor found as much as several.</td></tr>
|
||||
<tr id="src-15"><td>15</td><td><b>At most two rounds per reviewer</b></td><td><span class="kind m">Own measurement</span> Since 2026-10-21, across all projects: round 1 ran 357 times and found 59 red findings. Round 2 ran about 200 times and found 12, mostly in round 1's fixes. Rounds 3 and later ran about 110 times and found none, for about $125. Asks for more rounds were 24 of 59 interrupts.</td></tr>
|
||||
<tr id="src-16"><td>16</td><td><b>Reviewers report only real problems</b></td><td><span class="kind r">Vendor docs</span> Anthropic's Claude Code best practices: "A reviewer prompted to find gaps will usually report some, even when the work is sound."<br><span class="kind m">Own measurement</span> 320 of the setup repo's first 876 commits, by 2026-10-13, answered a review round. So stability and product owner report only defects, broken requirements or broken principles.</td></tr>
|
||||
<tr id="src-17"><td>17</td><td><b>Architect and principles only in the release review</b></td><td><span class="kind m">Own measurement</span> The principles audit was the costliest reviewer after the worker, and filed findings unrelated to the chunk. Once a release, over the whole project, catches the same drift.</td></tr>
|
||||
<tr id="src-17"><td>17</td><td><b>Architect and principles only in the release review</b></td><td><span class="kind m">Own measurement</span> The principles audit was the costliest reviewer after the worker, and filed findings unrelated to the chunk. Run once per release over the whole project, it catches the same drift.</td></tr>
|
||||
<tr id="src-18"><td>18</td><td><b>Security: <code>/security-review</code>, not a hand-written reviewer</b></td><td><span class="kind x">Reasoning</span> Its tuned exclusion list and confidence floor would be reproduced worse by hand.<br><span class="kind m">Own measurement</span> Run beside the deep scan, it returned a near-identical report, so the deep scan replaces it when both would run.</td></tr>
|
||||
<tr id="src-19"><td>19</td><td><b>Mutation testing instead of coverage</b></td><td><span class="kind r">Research</span> Coverage barely predicts fault detection: Inozemtseva & Holmes, ICSE 2014. Surviving mutants do predict real faults: Just et al., FSE 2014. Google runs mutation testing in code review: Petrović et al., arXiv:2102.11378.</td></tr>
|
||||
<tr id="src-20"><td>20</td><td><b>Test first, with expected values from the item</b></td><td><span class="kind r">Research</span> Tests that copy the code's current output pass on a bug: Charoosaei et al., arXiv:2610.11835 (preprint); Mathews & Nagappan, arXiv:2412.14137. Agents add mocks more often than people: Hora & Robbes, MSR 2026.<br><span class="kind b">Book</span> <i>Software Engineering at Google</i>, ch. 11, 12, 14; Beck, "Test Desiderata"; Khorikov, <i>Unit Testing</i>, ch. 4.</td></tr>
|
||||
@@ -1004,13 +999,13 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<h3 class="group">Readability</h3>
|
||||
<div class="tablewrap"><table class="src-t"><colgroup><col style="width:4%"><col style="width:24%"><col style="width:72%"></colgroup><tbody>
|
||||
<tr id="src-22"><td>22</td><td><b>Code that fits in your head: about seven things per level</b></td><td><span class="kind b">Book</span> Seemann, <i>Code That Fits in Your Head</i>, 2021. Ousterhout, <i>A Philosophy of Software Design</i> (deep modules).<br><span class="kind r">Research</span> Working memory holds about four chunks: Cowan 2001. A list found by name is no memory limit: Nielsen Norman Group. No study supports a fixed limit on function length: Jones 2023.</td></tr>
|
||||
<tr id="src-23"><td>23</td><td><b>Reader panel: before-and-after rulings, never blocking</b></td><td><span class="kind m">Own measurement</span> Two independent four-reader runs moved up to 0.4 on near-identical code (PR #224). Readers comparing before and after should cancel that noise. A check of that is still pending: it needs 10 commits each scored twice.<br><span class="kind r">Research</span> No published study validates simulated reader panels yet. So the panel steers the plan and blocks no merge.</td></tr>
|
||||
<tr id="src-23"><td>23</td><td><b>Reader panel: before-and-after rulings, never blocking</b></td><td><span class="kind m">Own measurement</span> Two independent four-reader runs moved up to 0.4 on near-identical code (PR #224). Readers comparing before and after should cancel that noise. A check of that is still pending: it needs 10 commits each scored twice.<br><span class="kind r">Research</span> No published study validates simulated reader panels yet. So the panel steers the plan and doesn't block a merge.</td></tr>
|
||||
<tr id="src-24"><td>24</td><td><b>Reading cost: the median of five chunks</b></td><td><span class="kind m">Own measurement</span> One chunk read 2.99M tokens before its first commit, against a typical 80k–370k. An average let that one chunk decide the trend alone, so a rise is the last five chunks' median above the five before.</td></tr>
|
||||
</tbody></table></div>
|
||||
<h3 class="group">Models</h3>
|
||||
<div class="tablewrap"><table class="src-t"><colgroup><col style="width:4%"><col style="width:24%"><col style="width:72%"></colgroup><tbody>
|
||||
<tr id="src-25"><td>25</td><td><b>The model ladder, and no Haiku</b></td><td><span class="kind r">Benchmark</span> The rungs follow the Artificial Analysis Intelligence Index, except Fable 5.1 at max: the index scores it below Opus 5.5 at high, and the human put it on top for the hardest tasks.<br><span class="kind m">Own measurement</span> A chunk on Haiku 5.5 ran 2 h 17 min and wrote 3.1M tokens. The chunk before it, on Sonnet 5.5, took 29 min and 111k. Chunks on Haiku took 7.8 extra review rounds, against 2.8 on Sonnet. So Haiku left the ladder.</td></tr>
|
||||
<tr id="src-26"><td>26</td><td><b>Judge and reviewers never run below the project's model</b></td><td><span class="kind m">Own measurement</span> A replay of 39 judge verdicts, 31 of which the reference judge reproduced: on those 31, Opus 5.5 at medium missed none, and Sonnet 5.5 mislabelled 3 interrupts. A replay of review rounds: Sonnet 5.5 missed 44–55% of what the project's model found.</td></tr>
|
||||
<tr id="src-26"><td>26</td><td><b>Judge and reviewers never run below the project's model</b></td><td><span class="kind m">Own measurement</span> 39 judge verdicts were replayed. The reference judge reproduced 31; on those, Opus 5.5 at medium missed none, and Sonnet 5.5 mislabelled 3 interrupts. A replay of review rounds: Sonnet 5.5 missed 44–55% of what the project's model found.</td></tr>
|
||||
</tbody></table></div>
|
||||
<h3 class="group">The machine</h3>
|
||||
<div class="tablewrap"><table class="src-t"><colgroup><col style="width:4%"><col style="width:24%"><col style="width:72%"></colgroup><tbody>
|
||||
@@ -1025,17 +1020,15 @@ dl.gloss dt { font-weight: 700; } dl.gloss dd { margin: 0; color: var(--muted);
|
||||
<dl class="gloss">
|
||||
<dt>Chunk</dt><dd>One unit of work, shipped as one pull request. One or more plan items.</dd>
|
||||
<dt>Cycle</dt><dd>One worker run plus the judge's verdict on it.</dd>
|
||||
<dt>Item</dt><dd>One row of <code>todo.md</code>, with a bold one-sentence goal and its scores.</dd>
|
||||
<dt>Item</dt><dd>One row of <code>todo.md</code>: a bold one-sentence summary of the change, and its scores.</dd>
|
||||
<dt>Worker</dt><dd>The headless Claude Code session that does the chunk.</dd>
|
||||
<dt>Judge</dt><dd>The short AI run that reads each worker run and gives a verdict.</dd>
|
||||
<dt>Interrupt</dt><dd>A question only you can answer. Shown in the terminal and in <code>loop-inbox</code>.</dd>
|
||||
<dt>Pass</dt><dd>One kind of review, such as Stability. It runs in rounds.</dd>
|
||||
<dt>Round</dt><dd>One review by one reviewer. At most two per pass per chunk.</dd>
|
||||
<dt>Round</dt><dd>One review by one reviewer. At most two per pass per chunk, five in the release review.</dd>
|
||||
<dt>Gate</dt><dd>The checks a PR must pass to merge: tests, CI, plan format.</dd>
|
||||
<dt>Rung</dt><dd>A model at an effort level, such as "Opus 5.5 at high".</dd>
|
||||
<dt>Premises</dt><dd>The README's <code>## Goals</code> and <code>## Audience</code>. Nothing runs without them.</dd>
|
||||
<dt>Release review</dt><dd>A release's last chunk: whole-project review before you cut it.</dd>
|
||||
<dt>Seat</dt><dd>One simulated reader on the comprehension panel.</dd>
|
||||
<dt>Worktree</dt><dd>A separate git checkout per chunk, so your main checkout stays untouched.</dd>
|
||||
</dl>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user