claude-session-watchdog/README.md

Claude Code session watchdog

Resumes Claude Code background sessions that died on a usage limit, an API outage, or a network failure.

Why it lives outside Claude Code

Claude Code has no auto-resume: on a usage limit it blocks further requests until the reset time and the process exits. Nothing scheduled inside a session (a 15-minute self-check, a /loop, a scheduled task) can rescue it, because there is no process left to fire the schedule. Recovery has to come from outside, so this is a systemd user timer.

It also costs nothing while everything is healthy — it reads state files on disk rather than waking sessions up. An in-session poll would re-send the session's full context on every tick, which draws down the very limit it is trying to survive.

How it decides

Every 15 minutes it runs claude agents --json and, for each background session, resumes it only when all of these hold:

1. The process is gone — no pid, or the pid no longer exists.
2. ~/.claude/jobs/<id>/state.json says "state": "blocked".
3. detail names a recoverable cause — session/weekly/Opus limit, usage or rate limit, API error, overload, or a network/connection failure.
4. detail does not name a human — "input needed", "waiting for", permission, approval, or awaiting your direction/decision. A session parked on your decision must stay parked, and this veto wins even when the same detail also names a limit.
5. Any stated reset time has passed. detail carries it verbatim, e.g. You've hit your session limit · resets 7pm (Europe/Stockholm), so the watchdog waits for that moment instead of guessing. Day-less times are anchored to the state.json mtime; weekday forms like resets Mon 12:00am are handled too. The result is clamped to the limit window — 5h for a session or Opus limit, 7 days for a weekly one — because state.json is sometimes rewritten after the reset already passed, which would otherwise push the parsed time-of-day a full day into the future and strand the session. Blocks with no stated reset (API outage, network) are retried on the next tick, with no wait.
6. Attempt caps are not exhausted — see the circuit breakers below.

It resumes with claude --bg --resume <sessionId> --name "<original name>" from the session's own recorded working directory, so the conversation, its context and its name all carry over. Oldest block is resumed first.

The resume registers a new job id, leaving the original as a dead husk that is permanently blocked yet still listed as active. The watchdog therefore runs claude stop <old id> after a successful resume. Without that the husk would be picked up again on a later tick and resumed into a second session working the same conversation — two agents racing on the same git worktree.

Circuit breakers

Three limits stop a broken session, or a still-ongoing outage, from spawning an endless chain of blocked jobs:

Limit	Default	Meaning
--max-per-run	0	Resumes per tick; 0 means every eligible session
--max-per-day	50	Resumes across all sessions in a rolling 24h
--max-attempts	1	Resumes of any one session, ever

--max-attempts 1 is not a one-shot recovery: if the resumed session stalls again it is a new session id and qualifies on its own, so an outage spanning several reset windows still gets retried. What the cap prevents is one conversation being resumed into several concurrent copies. A resume that fails to launch at all records no attempt and is retried on the next tick.

--max-per-day is the one guard that cannot be derived from anything else. A usage-limit block is self-pacing, because the stated reset time is respected — but a network or API block states no reset, so it is retried every tick, and each retry re-sends the session's whole context. During a multi-hour outage that is the one path that can burn a lot of tokens for nothing. Set --max-per-run to a small number instead if you would rather stagger.

What it deliberately does not do

• Interactive sessions — you are sitting there; you can press enter.
• Sessions waiting on a human (condition 4 above).
• Sessions you stopped ("state": "stopped") or that finished ("done").
• Anything whose working directory has disappeared (deleted worktree) — logged and skipped.

Install

Needs Claude Code (for claude agents --json and claude --bg --resume) and Python 3.9+ — standard library only.

Clone into ~/.claude/session-watchdog. The unit file expects it there, and the watchdog then travels with the rest of ~/.claude:

git clone https://gitea.larvit.se/larvit/claude-session-watchdog.git ~/.claude/session-watchdog
chmod +x ~/.claude/session-watchdog/resume-stalled-sessions.py
mkdir -p ~/.config/systemd/user
ln -sf ~/.claude/session-watchdog/claude-session-watchdog.service ~/.config/systemd/user/
ln -sf ~/.claude/session-watchdog/claude-session-watchdog.timer   ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now claude-session-watchdog.timer

Cloned somewhere else? Point ExecStart= at it — that path is the only thing tying the two together. ledger.json is written under --claude-home regardless of where the script lives.

To keep it running while you are logged out (background jobs outlive your shell, so this is usually what you want):

sudo loginctl enable-linger "$USER"

cron (Alpine, macOS, anything without systemd)

The script itself is portable — only the scheduling is not. Add a cron entry with crontab -e:

7,22,37,52 * * * * PATH=/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin /usr/bin/flock -n $HOME/.claude/session-watchdog/.lock $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1

Both prefixes matter, and neither is cosmetic:

• PATH — cron's default omits /usr/local/bin, which is where claude usually is. Without it the watchdog fails silently: it logs nothing to resume and looks perfectly healthy while resuming nothing, forever. Use CLAUDE_BIN=/full/path/to/claude instead if you prefer. Verify with env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run — that reproduces cron's environment, so it fails the same way cron would.
• flock — systemd's Type=oneshot refuses to run a second copy while the first is going; cron happily overlaps them. Two concurrent runs read the same ledger.json before either writes it, so both resume the same session. On a host without flock (macOS), use a launchd agent with StartInterval set to 900 instead — launchd also refuses to overlap a run with itself.

Use absolute paths rather than $HOME if your cron does not export HOME (busybox crond does). On Alpine, user crontabs live in /etc/crontabs/<user> and crontab -e writes there via /var/spool/cron/crontabs, so nothing needs root.

Verify and observe

# What would it do right now? Resumes nothing.
~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run

# Timer health and next fire time
systemctl --user list-timers claude-session-watchdog.timer

# What it has done
journalctl --user -u claude-session-watchdog.service --since today

Under cron there is no journal, so the redirect in the crontab entry is the record:

crontab -l                                     # is it scheduled?
tail -f ~/.claude/session-watchdog/watchdog.log  # what it has done

One nothing to resume line per tick, so the log grows by roughly 8 KB a day. Truncate it whenever you like — nothing reads it back.

A stalled session that has been resumed shows up as a new job in claude agents; claude logs <id> shows it picking the work back up.

Configuration

Every option is a CLI flag with an environment-variable fallback — nothing is hardcoded. To make one permanent, put it in the unit's [Service] block, or on the crontab line next to PATH.

Flag	Env var	Default
--claude-bin	CLAUDE_BIN	claude
--claude-home	CLAUDE_HOME	~/.claude
--max-attempts	WATCHDOG_MAX_ATTEMPTS	1
--max-per-day	WATCHDOG_MAX_PER_DAY	50
--max-per-run	WATCHDOG_MAX_PER_RUN	0 (all)
--prompt	WATCHDOG_PROMPT	see --help
--dry-run	—	off

Files

Path	Role
resume-stalled-sessions.py	The watchdog
claude-session-watchdog.service	systemd unit that runs it once
claude-session-watchdog.timer	Fires at :07 :22 :37 :52 (off the :00 crowd)
~/.claude/session-watchdog/ledger.json	Per-session attempt counts and resume timestamps

ledger.json is generated. Delete it to forget all history; edit the attempts map to give a specific session another chance.

Uninstall

systemctl --user disable --now claude-session-watchdog.timer
rm ~/.config/systemd/user/claude-session-watchdog.{service,timer}
systemctl --user daemon-reload