193 lines
9.7 KiB
Markdown
193 lines
9.7 KiB
Markdown
# Claude Code session watchdog
|
|
|
|
Resumes Claude Code **background** sessions that died on a usage limit, an API
|
|
outage, or a network failure.
|
|
|
|
## Why it lives outside Claude Code
|
|
|
|
Claude Code has no auto-resume: on a usage limit it blocks further requests
|
|
until the reset time and the process exits. Nothing scheduled *inside* a
|
|
session (a 15-minute self-check, a `/loop`, a scheduled task) can rescue it,
|
|
because there is no process left to fire the schedule. Recovery has to come
|
|
from outside, so this is a cron job.
|
|
|
|
It also costs nothing while everything is healthy — it reads state files on
|
|
disk rather than waking sessions up. An in-session poll would re-send the
|
|
session's full context on every tick, which draws down the very limit it is
|
|
trying to survive.
|
|
|
|
## How it decides
|
|
|
|
Every 15 minutes it runs `claude agents --json` and, for each **background**
|
|
session, resumes it only when all of these hold:
|
|
|
|
1. **The process is gone** — no `pid`, or the pid no longer exists.
|
|
2. **`~/.claude/jobs/<id>/state.json` says `"state": "blocked"`.**
|
|
3. **`detail` names a recoverable cause** — any quota Claude Code reports as
|
|
`You've hit your <limit>`: session, weekly, Opus, Sonnet, Fable 5, fast,
|
|
usage, rate, and the ones that cost money rather than time (monthly or
|
|
individual spend limit, usage credit limit, out of usage credits or extra
|
|
usage). Plus API errors, overload, and network/connection failures.
|
|
4. **`detail` does not name a human** — "input needed", "waiting for",
|
|
permission, approval, or awaiting your direction/decision. A session parked
|
|
on your decision must stay parked, and this veto wins even when the same
|
|
detail also names a limit. It also covers entitlements only an admin can
|
|
change, which no amount of waiting clears: a seat type that excludes usage
|
|
credits, an allocation disabled by your admin, a group limit set to $0, a
|
|
service disabled for your org.
|
|
5. **Any stated reset time has passed.** `detail` carries it verbatim, e.g.
|
|
`You've hit your session limit · resets 7pm (Europe/Stockholm)`, so the
|
|
watchdog waits for that moment instead of guessing. Day-less times are
|
|
anchored to the `state.json` mtime; weekday forms like `resets Mon 12:00am`
|
|
are handled too. The result is clamped to the limit window — 5h for a
|
|
session or Opus limit, 7 days for a weekly one — because `state.json` is
|
|
sometimes rewritten *after* the reset already passed, which would otherwise
|
|
push the parsed time-of-day a full day into the future and strand the
|
|
session. Blocks with no stated reset (API outage, network) are retried on
|
|
the next tick, with no wait.
|
|
|
|
A **spend** block states no reset because none exists: it clears when you
|
|
raise the cap, when the month rolls over, or when the subscription window
|
|
puts the session back on included quota. So it is neither retried every tick
|
|
nor parked for 5h — it is retried hourly until one of those happens.
|
|
6. **Attempt caps are not exhausted** — see the circuit breakers below.
|
|
|
|
It resumes with `claude --bg --resume <sessionId> --name "<original name>"`
|
|
from the session's own recorded working directory, so the conversation, its
|
|
context and its name all carry over. Oldest block is resumed first.
|
|
|
|
The resume registers a **new** job id, leaving the original as a dead husk that
|
|
is permanently `blocked` yet still listed as active. The watchdog therefore
|
|
runs `claude stop <old id>` after a successful resume. Without that the husk
|
|
would be picked up again on a later tick and resumed into a *second* session
|
|
working the same conversation — two agents racing on the same git worktree.
|
|
|
|
### Circuit breakers
|
|
|
|
Three limits stop a broken session, or a still-ongoing outage, from spawning an
|
|
endless chain of blocked jobs:
|
|
|
|
| Limit | Default | Meaning |
|
|
| ----------------- | ------- | ------------------------------------------------ |
|
|
| `--max-per-run` | 0 | Resumes per tick; 0 means every eligible session |
|
|
| `--max-per-day` | 50 | Resumes across all sessions in a rolling 24h |
|
|
| `--max-attempts` | 1 | Resumes of any one session, ever |
|
|
|
|
`--max-attempts 1` is not a one-shot recovery: if the resumed session stalls
|
|
again it is a *new* session id and qualifies on its own, so an outage spanning
|
|
several reset windows still gets retried. What the cap prevents is one
|
|
conversation being resumed into several concurrent copies. A resume that fails
|
|
to launch at all records no attempt and is retried on the next tick.
|
|
|
|
`--max-per-day` is the one guard that cannot be derived from anything else. A
|
|
usage-limit block is self-pacing, because the stated reset time is respected —
|
|
but a *network or API* block states no reset, so it is retried every tick, and
|
|
each retry re-sends the session's whole context. During a multi-hour outage
|
|
that is the one path that can burn a lot of tokens for nothing. Set
|
|
`--max-per-run` to a small number instead if you would rather stagger.
|
|
|
|
A *spend* block is the other open-ended one: if you never raise the cap it
|
|
retries hourly until the month turns over, one job per session per hour. Those
|
|
retries are rejected before the model sees them, so they cost no tokens — but
|
|
they do spend the daily budget, so raise `--max-per-day` if you routinely have
|
|
several sessions blocked at once.
|
|
|
|
## What it deliberately does not do
|
|
|
|
- **Interactive sessions** — you are sitting there; you can press enter.
|
|
- **Sessions waiting on a human** (condition 4 above).
|
|
- **Sessions you stopped** (`"state": "stopped"`) or that finished (`"done"`).
|
|
- **Anything whose working directory has disappeared** (deleted worktree) —
|
|
logged and skipped.
|
|
|
|
## Install
|
|
|
|
Needs Claude Code (for `claude agents --json` and `claude --bg --resume`) and
|
|
Python 3.9+ — standard library only, 3.14+ to run the tests.
|
|
|
|
Clone anywhere — `~/.claude/session-watchdog` just keeps it travelling with the
|
|
rest of `~/.claude`. Then add one cron entry with `crontab -e`:
|
|
|
|
```bash
|
|
git clone https://gitea.larvit.se/larvit/claude-session-watchdog.git ~/.claude/session-watchdog
|
|
```
|
|
|
|
```
|
|
7,22,37,52 * * * * $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1
|
|
```
|
|
|
|
`:07 :22 :37 :52` keeps it off the `:00` crowd. Nothing wraps it — no `PATH`
|
|
prefix, no `flock`, no `chmod`, because the script handles both of the things a
|
|
scheduler would otherwise have to:
|
|
|
|
- **Finding `claude`** — `PATH` first, then `~/.local/bin`, `/usr/local/bin`,
|
|
`/opt/homebrew/bin`. If it finds nothing it says so and exits non-zero, rather
|
|
than logging `nothing to resume` and looking healthy while resuming nothing,
|
|
forever. `--claude-bin` still wins if you want a specific install.
|
|
- **Not overlapping itself** — it takes an exclusive lock on
|
|
`~/.claude/session-watchdog/.lock` and exits if another run holds it. Two
|
|
concurrent runs would read `ledger.json` before either writes it and resume
|
|
the same session twice, into two agents racing on one git worktree.
|
|
|
|
So any scheduler that fires every 15 minutes will do: cron, launchd with
|
|
`StartInterval 900`, a systemd timer, Task Scheduler.
|
|
|
|
Use absolute paths rather than `$HOME` if your cron does not export `HOME`
|
|
(busybox crond does). On Alpine, user crontabs live in `/etc/crontabs/<user>`
|
|
and `crontab -e` writes there via `/var/spool/cron/crontabs`, so nothing needs
|
|
root.
|
|
|
|
## Verify and observe
|
|
|
|
```bash
|
|
# Does it still classify every block correctly?
|
|
python3 ~/.claude/session-watchdog/test-resume-stalled-sessions.py
|
|
|
|
# What would it do right now? Resumes nothing.
|
|
~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run
|
|
|
|
# The same, with cron's bare environment instead of your shell's.
|
|
env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run
|
|
|
|
crontab -l # is it scheduled?
|
|
tail -f ~/.claude/session-watchdog/watchdog.log # what it has done
|
|
```
|
|
|
|
One `nothing to resume` line per tick, so the log grows by roughly 8 KB a day.
|
|
Truncate it whenever you like — nothing reads it back.
|
|
|
|
A stalled session that has been resumed shows up as a new job in
|
|
`claude agents`; `claude logs <id>` shows it picking the work back up.
|
|
|
|
## Configuration
|
|
|
|
Every option is a CLI flag with an environment-variable fallback — nothing is
|
|
hardcoded. To make one permanent, add it to the crontab line.
|
|
|
|
| Flag | Env var | Default |
|
|
| ----------------- | ------------------------- | -------------------- |
|
|
| `--claude-bin` | `CLAUDE_BIN` | `claude` |
|
|
| `--claude-home` | `CLAUDE_HOME` | `~/.claude` |
|
|
| `--max-attempts` | `WATCHDOG_MAX_ATTEMPTS` | `1` |
|
|
| `--max-per-day` | `WATCHDOG_MAX_PER_DAY` | `50` |
|
|
| `--max-per-run` | `WATCHDOG_MAX_PER_RUN` | `0` (all) |
|
|
| `--prompt` | `WATCHDOG_PROMPT` | see `--help` |
|
|
| `--dry-run` | — | off |
|
|
|
|
## Files
|
|
|
|
| Path | Role |
|
|
| ------------------------------------------ | ------------------------------------------ |
|
|
| `resume-stalled-sessions.py` | The watchdog — the only file it needs |
|
|
| `test-resume-stalled-sessions.py` | Its tests, on real Claude Code block texts |
|
|
| `~/.claude/session-watchdog/ledger.json` | Attempt counts and resume timestamps |
|
|
| `~/.claude/session-watchdog/.lock` | Held for the duration of a run |
|
|
|
|
Both live under `--claude-home`, wherever the clone itself is. `ledger.json` is
|
|
generated: delete it to forget all history, or edit the `attempts` map to give a
|
|
specific session another chance.
|
|
|
|
## Uninstall
|
|
|
|
Drop the line from `crontab -e` and delete the clone.
|