claude-session-watchdog/README.md

# Claude Code session watchdog

Resumes Claude Code **background** sessions that died on a usage limit, an API
outage, or a network failure.

## Why it lives outside Claude Code

Claude Code has no auto-resume: on a usage limit it blocks further requests
until the reset time and the process exits. Nothing scheduled *inside* a
session (a 15-minute self-check, a `/loop`, a scheduled task) can rescue it,
because there is no process left to fire the schedule. Recovery has to come
from outside, so this is a systemd user timer.

It also costs nothing while everything is healthy — it reads state files on
disk rather than waking sessions up. An in-session poll would re-send the
session's full context on every tick, which draws down the very limit it is
trying to survive.

## How it decides

Every 15 minutes it runs `claude agents --json` and, for each **background**
session, resumes it only when all of these hold:

1. **The process is gone** — no `pid`, or the pid no longer exists.
2. **`~/.claude/jobs/<id>/state.json` says `"state": "blocked"`.**
3. **`detail` names a recoverable cause** — session/weekly/Opus limit, usage or
   rate limit, API error, overload, or a network/connection failure.
4. **`detail` does not name a human** — "input needed", "waiting for",
   permission, approval, or awaiting your direction/decision. A session parked
   on your decision must stay parked, and this veto wins even when the same
   detail also names a limit.
5. **Any stated reset time has passed.** `detail` carries it verbatim, e.g.
   `You've hit your session limit · resets 7pm (Europe/Stockholm)`, so the
   watchdog waits for that moment instead of guessing. Day-less times are
   anchored to the `state.json` mtime; weekday forms like `resets Mon 12:00am`
   are handled too. The result is clamped to the limit window — 5h for a
   session or Opus limit, 7 days for a weekly one — because `state.json` is
   sometimes rewritten *after* the reset already passed, which would otherwise
   push the parsed time-of-day a full day into the future and strand the
   session. Blocks with no stated reset (API outage, network) are retried on
   the next tick, with no wait.
6. **Attempt caps are not exhausted** — see the circuit breakers below.

It resumes with `claude --bg --resume <sessionId> --name "<original name>"`
from the session's own recorded working directory, so the conversation, its
context and its name all carry over. Oldest block is resumed first.

The resume registers a **new** job id, leaving the original as a dead husk that
is permanently `blocked` yet still listed as active. The watchdog therefore
runs `claude stop <old id>` after a successful resume. Without that the husk
would be picked up again on a later tick and resumed into a *second* session
working the same conversation — two agents racing on the same git worktree.

### Circuit breakers

Three limits stop a broken session, or a still-ongoing outage, from spawning an
endless chain of blocked jobs:

| Limit             | Default | Meaning                                          |
| ----------------- | ------- | ------------------------------------------------ |
| `--max-per-run`   | 0       | Resumes per tick; 0 means every eligible session |
| `--max-per-day`   | 50      | Resumes across all sessions in a rolling 24h     |
| `--max-attempts`  | 1       | Resumes of any one session, ever                 |

`--max-attempts 1` is not a one-shot recovery: if the resumed session stalls
again it is a *new* session id and qualifies on its own, so an outage spanning
several reset windows still gets retried. What the cap prevents is one
conversation being resumed into several concurrent copies. A resume that fails
to launch at all records no attempt and is retried on the next tick.

`--max-per-day` is the one guard that cannot be derived from anything else. A
usage-limit block is self-pacing, because the stated reset time is respected —
but a *network or API* block states no reset, so it is retried every tick, and
each retry re-sends the session's whole context. During a multi-hour outage
that is the one path that can burn a lot of tokens for nothing. Set
`--max-per-run` to a small number instead if you would rather stagger.

## What it deliberately does not do

- **Interactive sessions** — you are sitting there; you can press enter.
- **Sessions waiting on a human** (condition 4 above).
- **Sessions you stopped** (`"state": "stopped"`) or that finished (`"done"`).
- **Anything whose working directory has disappeared** (deleted worktree) —
  logged and skipped.

## Install

Needs Claude Code (for `claude agents --json` and `claude --bg --resume`) and
Python 3.9+ — standard library only.

Clone into `~/.claude/session-watchdog`. The unit file expects it there, and the
watchdog then travels with the rest of `~/.claude`:

```bash
git clone https://gitea.larvit.se/larvit/claude-session-watchdog.git ~/.claude/session-watchdog
chmod +x ~/.claude/session-watchdog/resume-stalled-sessions.py
mkdir -p ~/.config/systemd/user
ln -sf ~/.claude/session-watchdog/claude-session-watchdog.service ~/.config/systemd/user/
ln -sf ~/.claude/session-watchdog/claude-session-watchdog.timer   ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now claude-session-watchdog.timer
```

Cloned somewhere else? Point `ExecStart=` at it — that path is the only thing
tying the two together. `ledger.json` is written under `--claude-home`
regardless of where the script lives.

To keep it running while you are logged out (background jobs outlive your
shell, so this is usually what you want):

```bash
sudo loginctl enable-linger "$USER"
```

### cron (Alpine, macOS, anything without systemd)

The script itself is portable — only the scheduling is not. Add a cron entry
with `crontab -e`:

```
7,22,37,52 * * * * PATH=/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin /usr/bin/flock -n $HOME/.claude/session-watchdog/.lock $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1
```

Both prefixes matter, and neither is cosmetic:

- **`PATH`** — cron's default omits `/usr/local/bin`, which is where `claude`
  usually is. Without it the watchdog fails *silently*: it logs `nothing to
  resume` and looks perfectly healthy while resuming nothing, forever. Use
  `CLAUDE_BIN=/full/path/to/claude` instead if you prefer. Verify with
  `env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run`
  — that reproduces cron's environment, so it fails the same way cron would.
- **`flock`** — systemd's `Type=oneshot` refuses to run a second copy while the
  first is going; cron happily overlaps them. Two concurrent runs read the same
  `ledger.json` before either writes it, so both resume the same session. On a
  host without `flock` (macOS), use a launchd agent with `StartInterval` set to
  `900` instead — launchd also refuses to overlap a run with itself.

Use absolute paths rather than `$HOME` if your cron does not export `HOME`
(busybox crond does). On Alpine, user crontabs live in `/etc/crontabs/<user>`
and `crontab -e` writes there via `/var/spool/cron/crontabs`, so nothing needs
root.

## Verify and observe

```bash
# What would it do right now? Resumes nothing.
~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run

# Timer health and next fire time
systemctl --user list-timers claude-session-watchdog.timer

# What it has done
journalctl --user -u claude-session-watchdog.service --since today
```

Under cron there is no journal, so the redirect in the crontab entry is the
record:

```bash
crontab -l                                     # is it scheduled?
tail -f ~/.claude/session-watchdog/watchdog.log  # what it has done
```

One `nothing to resume` line per tick, so the log grows by roughly 8 KB a day.
Truncate it whenever you like — nothing reads it back.

A stalled session that has been resumed shows up as a new job in
`claude agents`; `claude logs <id>` shows it picking the work back up.

## Configuration

Every option is a CLI flag with an environment-variable fallback — nothing is
hardcoded. To make one permanent, put it in the unit's `[Service]` block, or on
the crontab line next to `PATH`.

| Flag              | Env var                   | Default              |
| ----------------- | ------------------------- | -------------------- |
| `--claude-bin`    | `CLAUDE_BIN`              | `claude`             |
| `--claude-home`   | `CLAUDE_HOME`             | `~/.claude`          |
| `--max-attempts`  | `WATCHDOG_MAX_ATTEMPTS`   | `1`                  |
| `--max-per-day`   | `WATCHDOG_MAX_PER_DAY`    | `50`                 |
| `--max-per-run`   | `WATCHDOG_MAX_PER_RUN`    | `0` (all)            |
| `--prompt`        | `WATCHDOG_PROMPT`         | see `--help`         |
| `--dry-run`       | —                         | off                  |

## Files

| Path                                | Role                                              |
| ----------------------------------- | ------------------------------------------------- |
| `resume-stalled-sessions.py`        | The watchdog                                      |
| `claude-session-watchdog.service`   | systemd unit that runs it once                    |
| `claude-session-watchdog.timer`     | Fires at `:07 :22 :37 :52` (off the `:00` crowd)  |
| `~/.claude/session-watchdog/ledger.json` | Per-session attempt counts and resume timestamps |

`ledger.json` is generated. Delete it to forget all history; edit the
`attempts` map to give a specific session another chance.

## Uninstall

```bash
systemctl --user disable --now claude-session-watchdog.timer
rm ~/.config/systemd/user/claude-session-watchdog.{service,timer}
systemctl --user daemon-reload
```