Initial import of the session watchdog
This commit is contained in:
@@ -0,0 +1,204 @@
|
||||
# Claude Code session watchdog
|
||||
|
||||
Resumes Claude Code **background** sessions that died on a usage limit, an API
|
||||
outage, or a network failure.
|
||||
|
||||
## Why it lives outside Claude Code
|
||||
|
||||
Claude Code has no auto-resume: on a usage limit it blocks further requests
|
||||
until the reset time and the process exits. Nothing scheduled *inside* a
|
||||
session (a 15-minute self-check, a `/loop`, a scheduled task) can rescue it,
|
||||
because there is no process left to fire the schedule. Recovery has to come
|
||||
from outside, so this is a systemd user timer.
|
||||
|
||||
It also costs nothing while everything is healthy — it reads state files on
|
||||
disk rather than waking sessions up. An in-session poll would re-send the
|
||||
session's full context on every tick, which draws down the very limit it is
|
||||
trying to survive.
|
||||
|
||||
## How it decides
|
||||
|
||||
Every 15 minutes it runs `claude agents --json` and, for each **background**
|
||||
session, resumes it only when all of these hold:
|
||||
|
||||
1. **The process is gone** — no `pid`, or the pid no longer exists.
|
||||
2. **`~/.claude/jobs/<id>/state.json` says `"state": "blocked"`.**
|
||||
3. **`detail` names a recoverable cause** — session/weekly/Opus limit, usage or
|
||||
rate limit, API error, overload, or a network/connection failure.
|
||||
4. **`detail` does not name a human** — "input needed", "waiting for",
|
||||
permission, approval, or awaiting your direction/decision. A session parked
|
||||
on your decision must stay parked, and this veto wins even when the same
|
||||
detail also names a limit.
|
||||
5. **Any stated reset time has passed.** `detail` carries it verbatim, e.g.
|
||||
`You've hit your session limit · resets 7pm (Europe/Stockholm)`, so the
|
||||
watchdog waits for that moment instead of guessing. Day-less times are
|
||||
anchored to the `state.json` mtime; weekday forms like `resets Mon 12:00am`
|
||||
are handled too. The result is clamped to the limit window — 5h for a
|
||||
session or Opus limit, 7 days for a weekly one — because `state.json` is
|
||||
sometimes rewritten *after* the reset already passed, which would otherwise
|
||||
push the parsed time-of-day a full day into the future and strand the
|
||||
session. Blocks with no stated reset (API outage, network) are retried on
|
||||
the next tick, with no wait.
|
||||
6. **Attempt caps are not exhausted** — see the circuit breakers below.
|
||||
|
||||
It resumes with `claude --bg --resume <sessionId> --name "<original name>"`
|
||||
from the session's own recorded working directory, so the conversation, its
|
||||
context and its name all carry over. Oldest block is resumed first.
|
||||
|
||||
The resume registers a **new** job id, leaving the original as a dead husk that
|
||||
is permanently `blocked` yet still listed as active. The watchdog therefore
|
||||
runs `claude stop <old id>` after a successful resume. Without that the husk
|
||||
would be picked up again on a later tick and resumed into a *second* session
|
||||
working the same conversation — two agents racing on the same git worktree.
|
||||
|
||||
### Circuit breakers
|
||||
|
||||
Three limits stop a broken session, or a still-ongoing outage, from spawning an
|
||||
endless chain of blocked jobs:
|
||||
|
||||
| Limit | Default | Meaning |
|
||||
| ----------------- | ------- | ------------------------------------------------ |
|
||||
| `--max-per-run` | 0 | Resumes per tick; 0 means every eligible session |
|
||||
| `--max-per-day` | 50 | Resumes across all sessions in a rolling 24h |
|
||||
| `--max-attempts` | 1 | Resumes of any one session, ever |
|
||||
|
||||
`--max-attempts 1` is not a one-shot recovery: if the resumed session stalls
|
||||
again it is a *new* session id and qualifies on its own, so an outage spanning
|
||||
several reset windows still gets retried. What the cap prevents is one
|
||||
conversation being resumed into several concurrent copies. A resume that fails
|
||||
to launch at all records no attempt and is retried on the next tick.
|
||||
|
||||
`--max-per-day` is the one guard that cannot be derived from anything else. A
|
||||
usage-limit block is self-pacing, because the stated reset time is respected —
|
||||
but a *network or API* block states no reset, so it is retried every tick, and
|
||||
each retry re-sends the session's whole context. During a multi-hour outage
|
||||
that is the one path that can burn a lot of tokens for nothing. Set
|
||||
`--max-per-run` to a small number instead if you would rather stagger.
|
||||
|
||||
## What it deliberately does not do
|
||||
|
||||
- **Interactive sessions** — you are sitting there; you can press enter.
|
||||
- **Sessions waiting on a human** (condition 4 above).
|
||||
- **Sessions you stopped** (`"state": "stopped"`) or that finished (`"done"`).
|
||||
- **Anything whose working directory has disappeared** (deleted worktree) —
|
||||
logged and skipped.
|
||||
|
||||
## Install
|
||||
|
||||
Needs Claude Code (for `claude agents --json` and `claude --bg --resume`) and
|
||||
Python 3.9+ — standard library only.
|
||||
|
||||
Clone into `~/.claude/session-watchdog`. The unit file expects it there, and the
|
||||
watchdog then travels with the rest of `~/.claude`:
|
||||
|
||||
```bash
|
||||
git clone https://gitea.larvit.se/larvit/claude-session-watchdog.git ~/.claude/session-watchdog
|
||||
chmod +x ~/.claude/session-watchdog/resume-stalled-sessions.py
|
||||
mkdir -p ~/.config/systemd/user
|
||||
ln -sf ~/.claude/session-watchdog/claude-session-watchdog.service ~/.config/systemd/user/
|
||||
ln -sf ~/.claude/session-watchdog/claude-session-watchdog.timer ~/.config/systemd/user/
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now claude-session-watchdog.timer
|
||||
```
|
||||
|
||||
Cloned somewhere else? Point `ExecStart=` at it — that path is the only thing
|
||||
tying the two together. `ledger.json` is written under `--claude-home`
|
||||
regardless of where the script lives.
|
||||
|
||||
To keep it running while you are logged out (background jobs outlive your
|
||||
shell, so this is usually what you want):
|
||||
|
||||
```bash
|
||||
sudo loginctl enable-linger "$USER"
|
||||
```
|
||||
|
||||
### cron (Alpine, macOS, anything without systemd)
|
||||
|
||||
The script itself is portable — only the scheduling is not. Add a cron entry
|
||||
with `crontab -e`:
|
||||
|
||||
```
|
||||
7,22,37,52 * * * * PATH=/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin /usr/bin/flock -n $HOME/.claude/session-watchdog/.lock $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1
|
||||
```
|
||||
|
||||
Both prefixes matter, and neither is cosmetic:
|
||||
|
||||
- **`PATH`** — cron's default omits `/usr/local/bin`, which is where `claude`
|
||||
usually is. Without it the watchdog fails *silently*: it logs `nothing to
|
||||
resume` and looks perfectly healthy while resuming nothing, forever. Use
|
||||
`CLAUDE_BIN=/full/path/to/claude` instead if you prefer. Verify with
|
||||
`env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run`
|
||||
— that reproduces cron's environment, so it fails the same way cron would.
|
||||
- **`flock`** — systemd's `Type=oneshot` refuses to run a second copy while the
|
||||
first is going; cron happily overlaps them. Two concurrent runs read the same
|
||||
`ledger.json` before either writes it, so both resume the same session. On a
|
||||
host without `flock` (macOS), use a launchd agent with `StartInterval` set to
|
||||
`900` instead — launchd also refuses to overlap a run with itself.
|
||||
|
||||
Use absolute paths rather than `$HOME` if your cron does not export `HOME`
|
||||
(busybox crond does). On Alpine, user crontabs live in `/etc/crontabs/<user>`
|
||||
and `crontab -e` writes there via `/var/spool/cron/crontabs`, so nothing needs
|
||||
root.
|
||||
|
||||
## Verify and observe
|
||||
|
||||
```bash
|
||||
# What would it do right now? Resumes nothing.
|
||||
~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run
|
||||
|
||||
# Timer health and next fire time
|
||||
systemctl --user list-timers claude-session-watchdog.timer
|
||||
|
||||
# What it has done
|
||||
journalctl --user -u claude-session-watchdog.service --since today
|
||||
```
|
||||
|
||||
Under cron there is no journal, so the redirect in the crontab entry is the
|
||||
record:
|
||||
|
||||
```bash
|
||||
crontab -l # is it scheduled?
|
||||
tail -f ~/.claude/session-watchdog/watchdog.log # what it has done
|
||||
```
|
||||
|
||||
One `nothing to resume` line per tick, so the log grows by roughly 8 KB a day.
|
||||
Truncate it whenever you like — nothing reads it back.
|
||||
|
||||
A stalled session that has been resumed shows up as a new job in
|
||||
`claude agents`; `claude logs <id>` shows it picking the work back up.
|
||||
|
||||
## Configuration
|
||||
|
||||
Every option is a CLI flag with an environment-variable fallback — nothing is
|
||||
hardcoded. To make one permanent, put it in the unit's `[Service]` block, or on
|
||||
the crontab line next to `PATH`.
|
||||
|
||||
| Flag | Env var | Default |
|
||||
| ----------------- | ------------------------- | -------------------- |
|
||||
| `--claude-bin` | `CLAUDE_BIN` | `claude` |
|
||||
| `--claude-home` | `CLAUDE_HOME` | `~/.claude` |
|
||||
| `--max-attempts` | `WATCHDOG_MAX_ATTEMPTS` | `1` |
|
||||
| `--max-per-day` | `WATCHDOG_MAX_PER_DAY` | `50` |
|
||||
| `--max-per-run` | `WATCHDOG_MAX_PER_RUN` | `0` (all) |
|
||||
| `--prompt` | `WATCHDOG_PROMPT` | see `--help` |
|
||||
| `--dry-run` | — | off |
|
||||
|
||||
## Files
|
||||
|
||||
| Path | Role |
|
||||
| ----------------------------------- | ------------------------------------------------- |
|
||||
| `resume-stalled-sessions.py` | The watchdog |
|
||||
| `claude-session-watchdog.service` | systemd unit that runs it once |
|
||||
| `claude-session-watchdog.timer` | Fires at `:07 :22 :37 :52` (off the `:00` crowd) |
|
||||
| `~/.claude/session-watchdog/ledger.json` | Per-session attempt counts and resume timestamps |
|
||||
|
||||
`ledger.json` is generated. Delete it to forget all history; edit the
|
||||
`attempts` map to give a specific session another chance.
|
||||
|
||||
## Uninstall
|
||||
|
||||
```bash
|
||||
systemctl --user disable --now claude-session-watchdog.timer
|
||||
rm ~/.config/systemd/user/claude-session-watchdog.{service,timer}
|
||||
systemctl --user daemon-reload
|
||||
```
|
||||
Reference in New Issue
Block a user