From e5022d0ef3e298fc4f984ca38a7c2d7388f5228f Mon Sep 17 00:00:00 2001 From: Lilleman Date: Sun, 22 Nov 2026 16:55:39 +0100 Subject: [PATCH] File item 514: make stability-reviewer check that a test isolates git from the developer's environment --- todo.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/todo.md b/todo.md index 9d734661..54f0b8db 100644 --- a/todo.md +++ b/todo.md @@ -6,7 +6,7 @@ `Bar = 0.1` -`Next ID = 514` +`Next ID = 515` ## Items @@ -90,6 +90,7 @@ | 181 | 0.23.0 | | principle | **Have the test of a quoted answer's step-up assert the stepped rung's facts as literals, not through `prompts.rungs_held`.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 0.9 | | 193 | 0.23.0 | | | **Narrow `numbered`'s mark to `Mark` by a typed lookup in place of `typing.cast`, so a mark added to the regex but not the lookup fails at run time, and one added to the lookup but not `Mark` fails mypy.** | 1 | 1 | 3 | 3 | 1 | 1.1 by 2 | 0.9 | | 194 | 0.23.0 | | | **Make `test_waits_out_a_usage_limit_which_ends_a_run_of_failures` expect a usage-limit wait independent of the wall-clock time that passes during the run.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 0.9 | +| 514 | 0.23.0 | | | **Make stability-reviewer check that a test which runs git, or a script that does, isolates it from the developer's git environment.** | 1 | 2 | 2 | 3 | 1 | 1.1 by 2 | 0.9 | | 298 | 0.23.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 0.9 | | 265 | 0.23.0 | | | **Archive each wait that costs a chunk wall time, for a usage limit, low memory, a failed worker start or a heavy slot, with its start and end stamps, so goal 1.5's hours split waiting from work.** | 3 | 2 | 2 | 5 | 4 | 1.5 by 3 | 0.8 | | 475 | 0.23.0 | | | **Have stability-reviewer report, as a finding, each lookup that an input from outside the repo can make fail without an error naming what was missing.** | 1 | 1 | 2 | 2 | 2 | 1.1 by 2 | 0.8 | @@ -619,6 +620,12 @@ Found by review of PR #317. `bin/plan/goals_file.py`'s `numbered` writes `typing Found by review of PR #317. The black-box test, `bin/loop_claude/test_black_box.py:609`, not the unit test of the same name in `bin/agent_cycle/test_retry.py`, which runs on a fake clock, sets `resets_at` from the test's `time.time()` and expects the wait, rounded to tens, to be 3720. When more than about 5 s pass between that read and the loop's, the wait rounds to 3710 and the test fails with no bug. Item 102 covers only replacing the `time.sleep` swap; this item changes the expected value, for example by asserting the wait against `resets_at + 120` minus the time the loop read, or a bound on it. +### 514. Make stability-reviewer check that a test which runs git, or a script that does, isolates it from the developer's git environment. + +An outside reviewer caught what our review missed: a test that ran git, through the script under test, inherited the environment of whoever ran it. The stability reviewer found one half (the developer's global config: `commit.gpgsign=true` made every test error), and the forge's reviewer the other (a `GIT_DIR` set by a hook or an outer git command points git at another repository). + +Add one check to the stability reviewer (`agents/stability-reviewer.md`), or to the testing principle it cites: a test that runs git, or runs a script that does, sets `GIT_CONFIG_GLOBAL=/dev/null` and `GIT_CONFIG_NOSYSTEM=1`, and clears `GIT_DIR`, `GIT_WORK_TREE`, `GIT_INDEX_FILE`, `GIT_OBJECT_DIRECTORY` and `GIT_ALTERNATE_OBJECT_DIRECTORIES`. A finding names the test and the variables it inherits. + ### 298. Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind. 153 `Judge cost:` entries total $109.76, a median of about $0.76 a verdict at every rung, and the cost follows the size of the records, not the transcript: about 76k characters in log and 188k in claude-config. In 224 of 251 recent verdicts (89%) the evidence sits in the worker's final message. A missed interrupt is the costly error, so the bar is zero changed interrupts and zero new rejections across the replay. The judge is about 4.5% of dollars, so the saving is small; each cycle also gets faster.