Add task-bench, which has Haiku fix planted bugs at a base and a head and prints what each side took #683

Merged
lilleman merged 11 commits from item-501 into main 2026-11-21 03:46:52 +01:00
Owner

Adds task-bench: it plants bugs the tests catch in the Go files the comprehension seats found hardest, has Haiku fix each in a container that reaches only the API, at the base and at the head, and prints how often the fix passes with its median tokens, turns, wall time and reads.

Adds `task-bench`: it plants bugs the tests catch in the Go files the comprehension seats found hardest, has Haiku fix each in a container that reaches only the API, at the base and at the head, and prints how often the fix passes with its median tokens, turns, wall time and reads.
lilleman added 1 commit 2026-11-21 01:05:55 +01:00
lilleman added 2 commits 2026-11-21 01:39:18 +01:00
lilleman added 1 commit 2026-11-21 02:04:42 +01:00
lilleman added 3 commits 2026-11-21 02:41:42 +01:00
lilleman added 1 commit 2026-11-21 03:09:40 +01:00
Test a line printed before any test's mark
CI / tests (pull_request) Successful in 3m51s
0c17559efe
lilleman marked the pull request as ready for review 2026-11-21 03:40:15 +01:00
lilleman added 2 commits 2026-11-21 03:40:16 +01:00
lilleman added 1 commit 2026-11-21 03:40:24 +01:00
Finish item 501 in the plan
CI / tests (pull_request) Successful in 6m11s
d813fdad5a
lilleman merged commit d813fdad5a into main 2026-11-21 03:46:51 +01:00
lilleman deleted branch item-501 2026-11-21 03:46:52 +01:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lilleman/claude-config#683