Answer prose round 1: 0 🔴, 3 🟡 (0 misread); make the turns' and reads' figures one reading each

This commit is contained in:
2026-11-22 21:34:09 +01:00
parent e47e71167b
commit 98754352e5
+3 -3
View File
@@ -850,10 +850,10 @@ Measured on `fejkdata` (Go, 5 bugs planted in the files `larv-review places` nam
| Output tokens | 25% | 12% |
| Cache-read tokens | 41% | 17% |
| Turns | 31%, or 3.0 turns | 14%, or 1.8 turns |
| Reads before the first open of the bug's file | 4.0 reads | 0.6 reads |
| Reads and searches before the first open of the bug's file | 4.0 reads | 0.6 reads |
Neither count sees a change the tokens can't. The bugs' median turns average 14.5 and their median reads 3.9, so the mean's spread is 12% of the turns and 15% of the reads, in line with the tokens' 12% and 17%. The All row's 4.0 reads is more than the reads themselves. Reads are small whole numbers, from 0 to 9, and 9 of the 50 runs never opened the bug's file, so a read is given as a difference, never a share. The tokens' own figures moved by up to a fifth against the first calibration's above (25 and 41 against 31 and 50 at the All row), so each 95th percentile is good to about that much.
- **One finished chunk**: `fejkdata`'s PR #490, base `33b7fa80`, head `5b9c2a14`, which the comprehension panel ruled better on Navigation, Locality and Shape and the same on Self-sufficiency. The bench planted 5 bugs and ran 38 of 50 runs before the machine restarted. The first 3 runs per bug and side, 30 runs, are the ones counted, and all passed. The mean of the bugs' changes was +4.6% in output tokens, +8.9% in cache-read tokens and +3.8% in turns, with the bugs' own changes in cache-read tokens from -41% to +47%. The two token figures sit inside the spread at 5 × 3, 17% and 22%; the spread of the turns wasn't measured. The mean of the wall times' changes, +105%, lies above the 86% of that row, from one bug whose head runs took 6.8 times as long at the median. Nobody read its streams to find out why. The head cost slightly more tokens where the panel finds it easier to read, and the wall time can't be judged, so the bench neither confirmed nor contradicted the panel.
Neither count sees a change the tokens can't. The bugs' median turns average 14.5 and their median reads 3.9, so 1.8 turns is 12% of a bug's turns and 0.6 reads is 15% of its reads, in line with the tokens' 12% and 17%. The All row's spread, 4.0 reads, is more than a bug's typical 3.9 reads. Reads are small whole numbers, from 0 to 9, so the change in reads is given in reads, not as a percentage. The 9 of the 50 runs that never opened the bug's file are left out of the reads' medians. The tokens' own figures moved by up to a fifth against the first calibration's above (25 and 41 against 31 and 50 at the All row), so each 95th percentile may be off by about a fifth.
- **One finished chunk**: `fejkdata`'s PR #490, base `33b7fa80`, head `5b9c2a14`, which the comprehension panel ruled better on Navigation, Locality and Shape and the same on Self-sufficiency. The bench planted 5 bugs and ran 38 of 50 runs before the machine restarted. The first 3 runs per bug and side, 30 runs, are the ones counted, and all passed. The mean of the bugs' changes was +4.6% in output tokens, +8.9% in cache-read tokens and +3.8% in turns, with the bugs' own changes in cache-read tokens from -41% to +47%. The two token figures sit inside the spread at 5 × 3, 17% and 22%. The turns' +3.8% sits inside their spread at 5 × 5, 14%, and the spread at 5 × 3 can only be wider. The mean of the wall times' changes, +105%, lies above the 86% of that row, from one bug whose head runs took 6.8 times as long at the median. Nobody read its streams to find out why. The head cost slightly more tokens where the panel finds it easier to read, and the wall time can't be judged, so the bench neither confirmed nor contradicted the panel.
- **Cost**: 5 bugs × 6 runs took 3.9 hours and 34M cache-read, 1.2M cache-write and 0.3M output tokens; a run takes about 3.5 minutes, and planting 16 to 25 minutes. A 5 × 5 bench is about 3.3 hours and 28M cache-read tokens. Gremlins ran out of 8 GB at the base of PR #490, so the help's example takes 16 GB.
## tests: CI runs the suite through `ci.sh` on every pull request, in a pinned Python container, and `main`'s branch protection holds a merge until it passes