File item 514: make stability-reviewer check that a test isolates git from the developer's environment
CI / tests (pull_request) Successful in 5m50s
CI / tests (pull_request) Successful in 5m50s
This commit is contained in:
@@ -6,7 +6,7 @@
|
||||
|
||||
`Bar = 0.1`
|
||||
|
||||
`Next ID = 514`
|
||||
`Next ID = 515`
|
||||
|
||||
## Items
|
||||
|
||||
@@ -90,6 +90,7 @@
|
||||
| 181 | 0.23.0 | | principle | **Have the test of a quoted answer's step-up assert the stepped rung's facts as literals, not through `prompts.rungs_held`.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 0.9 |
|
||||
| 193 | 0.23.0 | | | **Narrow `numbered`'s mark to `Mark` by a typed lookup in place of `typing.cast`, so a mark added to the regex but not the lookup fails at run time, and one added to the lookup but not `Mark` fails mypy.** | 1 | 1 | 3 | 3 | 1 | 1.1 by 2 | 0.9 |
|
||||
| 194 | 0.23.0 | | | **Make `test_waits_out_a_usage_limit_which_ends_a_run_of_failures` expect a usage-limit wait independent of the wall-clock time that passes during the run.** | 1 | 1 | 1 | 1 | 1 | 1.1 by 2 | 0.9 |
|
||||
| 514 | 0.23.0 | | | **Make stability-reviewer check that a test which runs git, or a script that does, isolates it from the developer's git environment.** | 1 | 2 | 2 | 3 | 1 | 1.1 by 2 | 0.9 |
|
||||
| 298 | 0.23.0 | | | **Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.** | 3 | 3 | 3 | 7 | 4 | 1.4 by 2, 1.5 by 1 | 0.9 |
|
||||
| 265 | 0.23.0 | | | **Archive each wait that costs a chunk wall time, for a usage limit, low memory, a failed worker start or a heavy slot, with its start and end stamps, so goal 1.5's hours split waiting from work.** | 3 | 2 | 2 | 5 | 4 | 1.5 by 3 | 0.8 |
|
||||
| 475 | 0.23.0 | | | **Have stability-reviewer report, as a finding, each lookup that an input from outside the repo can make fail without an error naming what was missing.** | 1 | 1 | 2 | 2 | 2 | 1.1 by 2 | 0.8 |
|
||||
@@ -619,6 +620,12 @@ Found by review of PR #317. `bin/plan/goals_file.py`'s `numbered` writes `typing
|
||||
|
||||
Found by review of PR #317. The black-box test, `bin/loop_claude/test_black_box.py:609`, not the unit test of the same name in `bin/agent_cycle/test_retry.py`, which runs on a fake clock, sets `resets_at` from the test's `time.time()` and expects the wait, rounded to tens, to be 3720. When more than about 5 s pass between that read and the loop's, the wait rounds to 3710 and the test fails with no bug. Item 102 covers only replacing the `time.sleep` swap; this item changes the expected value, for example by asserting the wait against `resets_at + 120` minus the time the loop read, or a bound on it.
|
||||
|
||||
### 514. Make stability-reviewer check that a test which runs git, or a script that does, isolates it from the developer's git environment.
|
||||
|
||||
An outside reviewer caught what our review missed: a test that ran git, through the script under test, inherited the environment of whoever ran it. The stability reviewer found one half (the developer's global config: `commit.gpgsign=true` made every test error), and the forge's reviewer the other (a `GIT_DIR` set by a hook or an outer git command points git at another repository).
|
||||
|
||||
Add one check to the stability reviewer (`agents/stability-reviewer.md`), or to the testing principle it cites: a test that runs git, or runs a script that does, sets `GIT_CONFIG_GLOBAL=/dev/null` and `GIT_CONFIG_NOSYSTEM=1`, and clears `GIT_DIR`, `GIT_WORK_TREE`, `GIT_INDEX_FILE`, `GIT_OBJECT_DIRECTORY` and `GIT_ALTERNATE_OBJECT_DIRECTORIES`. A finding names the test and the variables it inherits.
|
||||
|
||||
### 298. Send the judge only the decision entries and the transcript tail a verdict needs, and keep the cut only where `judge-replay` changes no verdict kind.
|
||||
|
||||
153 `Judge cost:` entries total $109.76, a median of about $0.76 a verdict at every rung, and the cost follows the size of the records, not the transcript: about 76k characters in log and 188k in claude-config. In 224 of 251 recent verdicts (89%) the evidence sits in the worker's final message. A missed interrupt is the costly error, so the bar is zero changed interrupts and zero new rejections across the replay. The judge is about 4.5% of dollars, so the saving is small; each cycle also gets faster.
|
||||
|
||||
Reference in New Issue
Block a user