# Claude Code session watchdog

Resumes Claude Code **background** sessions that died on a usage limit, an API
outage, or a network failure.

## Why it lives outside Claude Code

Claude Code has no auto-resume: on a usage limit it blocks further requests
until the reset time and the process exits. Nothing scheduled *inside* a
session (a 15-minute self-check, a `/loop`, a scheduled task) can rescue it,
because there is no process left to fire the schedule. Recovery has to come
from outside, so this is a cron job.

It also costs nothing while everything is healthy — it reads state files on
disk rather than waking sessions up. An in-session poll would re-send the
session's full context on every tick, which draws down the very limit it is
trying to survive.

## How it decides

Every 15 minutes it runs `claude agents --json` and, for each **background**
session, resumes it only when all of these hold:

1. **The process is gone** — no `pid`, or the pid no longer exists.
2. **`~/.claude/jobs/<id>/state.json` says `"state": "blocked"`.**
3. **`detail` names a recoverable cause** — any quota Claude Code reports as
   `You've hit your <limit>`: session, weekly, Opus, Sonnet, Fable 5, fast,
   usage, rate, and the ones that cost money rather than time (monthly or
   individual spend limit, usage credit limit, out of usage credits or extra
   usage). Plus API errors, overload, and network/connection failures.
4. **`detail` does not name a human** — "input needed", "waiting for",
   permission, approval, or awaiting your direction/decision. A session parked
   on your decision must stay parked, and this veto wins even when the same
   detail also names a limit. It also covers entitlements only an admin can
   change, which no amount of waiting clears: a seat type that excludes usage
   credits, an allocation disabled by your admin, a group limit set to $0, a
   service disabled for your org.
5. **Any stated reset time has passed.** `detail` carries it verbatim, e.g.
   `You've hit your session limit · resets 7pm (Europe/Stockholm)`, so the
   watchdog waits for that moment instead of guessing. Day-less times are
   anchored to the `state.json` mtime; weekday forms like `resets Mon 12:00am`
   are handled too. The result is clamped to the limit window — 5h for a
   session or Opus limit, 7 days for a weekly one — because `state.json` is
   sometimes rewritten *after* the reset already passed, which would otherwise
   push the parsed time-of-day a full day into the future and strand the
   session. Blocks with no stated reset (API outage, network) are retried on
   the next tick, with no wait.

   A **spend** block states no reset because none exists: it clears when you
   raise the cap, when the month rolls over, or when the subscription window
   puts the session back on included quota. So it is neither retried every tick
   nor parked for 5h — it is retried hourly until one of those happens.
6. **Attempt caps are not exhausted** — see the circuit breakers below.

It resumes with `claude --bg --resume <sessionId> --name "<original name>"`
from the session's own recorded working directory, so the conversation, its
context and its name all carry over. Oldest block is resumed first.

The resume registers a **new** job id, leaving the original as a dead husk that
is permanently `blocked` yet still listed as active. The watchdog therefore
runs `claude stop <old id>` after a successful resume. Without that the husk
would be picked up again on a later tick and resumed into a *second* session
working the same conversation — two agents racing on the same git worktree.

### Circuit breakers

Three limits stop a broken session, or a still-ongoing outage, from spawning an
endless chain of blocked jobs:

| Limit             | Default | Meaning                                          |
| ----------------- | ------- | ------------------------------------------------ |
| `--max-per-run`   | 0       | Resumes per tick; 0 means every eligible session |
| `--max-per-day`   | 50      | Resumes across all sessions in a rolling 24h     |
| `--max-attempts`  | 1       | Resumes of any one session, ever                 |

`--max-attempts 1` is not a one-shot recovery: if the resumed session stalls
again it is a *new* session id and qualifies on its own, so an outage spanning
several reset windows still gets retried. What the cap prevents is one
conversation being resumed into several concurrent copies. A resume that fails
to launch at all records no attempt and is retried on the next tick.

`--max-per-day` is the one guard that cannot be derived from anything else. A
usage-limit block is self-pacing, because the stated reset time is respected —
but a *network or API* block states no reset, so it is retried every tick, and
each retry re-sends the session's whole context. During a multi-hour outage
that is the one path that can burn a lot of tokens for nothing. Set
`--max-per-run` to a small number instead if you would rather stagger.

A *spend* block is the other open-ended one: if you never raise the cap it
retries hourly until the month turns over, one job per session per hour. Those
retries are rejected before the model sees them, so they cost no tokens — but
they do spend the daily budget, so raise `--max-per-day` if you routinely have
several sessions blocked at once.

## What it deliberately does not do

- **Interactive sessions** — you are sitting there; you can press enter.
- **Sessions waiting on a human** (condition 4 above).
- **Sessions you stopped** (`"state": "stopped"`) or that finished (`"done"`).
- **Anything whose working directory has disappeared** (deleted worktree) —
  logged and skipped.

## Install

Needs Claude Code (for `claude agents --json` and `claude --bg --resume`) and
Python 3.9+ — standard library only, 3.14+ to run the tests.

Clone anywhere — `~/.claude/session-watchdog` just keeps it travelling with the
rest of `~/.claude`. Then add one cron entry with `crontab -e`:

```bash
git clone https://gitea.larvit.se/larvit/claude-session-watchdog.git ~/.claude/session-watchdog
```

```
7,22,37,52 * * * * $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1
```

`:07 :22 :37 :52` keeps it off the `:00` crowd. Nothing wraps it — no `PATH`
prefix, no `flock`, no `chmod`, because the script handles both of the things a
scheduler would otherwise have to:

- **Finding `claude`** — `PATH` first, then `~/.local/bin`, `/usr/local/bin`,
  `/opt/homebrew/bin`. If it finds nothing it says so and exits non-zero, rather
  than logging `nothing to resume` and looking healthy while resuming nothing,
  forever. `--claude-bin` still wins if you want a specific install.
- **Not overlapping itself** — it takes an exclusive lock on
  `~/.claude/session-watchdog/.lock` and exits if another run holds it. Two
  concurrent runs would read `ledger.json` before either writes it and resume
  the same session twice, into two agents racing on one git worktree.

So any scheduler that fires every 15 minutes will do: cron, launchd with
`StartInterval 900`, a systemd timer, Task Scheduler.

Use absolute paths rather than `$HOME` if your cron does not export `HOME`
(busybox crond does). On Alpine, user crontabs live in `/etc/crontabs/<user>`
and `crontab -e` writes there via `/var/spool/cron/crontabs`, so nothing needs
root.

## Verify and observe

```bash
# Does it still classify every block correctly?
python3 ~/.claude/session-watchdog/test-resume-stalled-sessions.py

# What would it do right now? Resumes nothing.
~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run

# The same, with cron's bare environment instead of your shell's.
env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run

crontab -l                                       # is it scheduled?
tail -f ~/.claude/session-watchdog/watchdog.log  # what it has done
```

One `nothing to resume` line per tick, so the log grows by roughly 8 KB a day.
Truncate it whenever you like — nothing reads it back.

A stalled session that has been resumed shows up as a new job in
`claude agents`; `claude logs <id>` shows it picking the work back up.

## Configuration

Every option is a CLI flag with an environment-variable fallback — nothing is
hardcoded. To make one permanent, add it to the crontab line.

| Flag              | Env var                   | Default              |
| ----------------- | ------------------------- | -------------------- |
| `--claude-bin`    | `CLAUDE_BIN`              | `claude`             |
| `--claude-home`   | `CLAUDE_HOME`             | `~/.claude`          |
| `--max-attempts`  | `WATCHDOG_MAX_ATTEMPTS`   | `1`                  |
| `--max-per-day`   | `WATCHDOG_MAX_PER_DAY`    | `50`                 |
| `--max-per-run`   | `WATCHDOG_MAX_PER_RUN`    | `0` (all)            |
| `--prompt`        | `WATCHDOG_PROMPT`         | see `--help`         |
| `--dry-run`       | —                         | off                  |

## Files

| Path                                       | Role                                       |
| ------------------------------------------ | ------------------------------------------ |
| `resume-stalled-sessions.py`               | The watchdog — the only file it needs      |
| `test-resume-stalled-sessions.py`          | Its tests, on real Claude Code block texts |
| `~/.claude/session-watchdog/ledger.json`   | Attempt counts and resume timestamps       |
| `~/.claude/session-watchdog/.lock`         | Held for the duration of a run             |

Both live under `--claude-home`, wherever the clone itself is. `ledger.json` is
generated: delete it to forget all history, or edit the `attempts` map to give a
specific session another chance.

## Uninstall

Drop the line from `crontab -e` and delete the clone.