Drop the systemd units; the script now locks itself and finds claude
This commit is contained in:
@@ -9,7 +9,7 @@ Claude Code has no auto-resume: on a usage limit it blocks further requests
|
||||
until the reset time and the process exits. Nothing scheduled *inside* a
|
||||
session (a 15-minute self-check, a `/loop`, a scheduled task) can rescue it,
|
||||
because there is no process left to fire the schedule. Recovery has to come
|
||||
from outside, so this is a systemd user timer.
|
||||
from outside, so this is a cron job.
|
||||
|
||||
It also costs nothing while everything is healthy — it reads state files on
|
||||
disk rather than waking sessions up. An in-session poll would re-send the
|
||||
@@ -88,52 +88,32 @@ that is the one path that can burn a lot of tokens for nothing. Set
|
||||
Needs Claude Code (for `claude agents --json` and `claude --bg --resume`) and
|
||||
Python 3.9+ — standard library only.
|
||||
|
||||
Clone into `~/.claude/session-watchdog`. The unit file expects it there, and the
|
||||
watchdog then travels with the rest of `~/.claude`:
|
||||
Clone anywhere — `~/.claude/session-watchdog` just keeps it travelling with the
|
||||
rest of `~/.claude`. Then add one cron entry with `crontab -e`:
|
||||
|
||||
```bash
|
||||
git clone https://gitea.larvit.se/larvit/claude-session-watchdog.git ~/.claude/session-watchdog
|
||||
chmod +x ~/.claude/session-watchdog/resume-stalled-sessions.py
|
||||
mkdir -p ~/.config/systemd/user
|
||||
ln -sf ~/.claude/session-watchdog/claude-session-watchdog.service ~/.config/systemd/user/
|
||||
ln -sf ~/.claude/session-watchdog/claude-session-watchdog.timer ~/.config/systemd/user/
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now claude-session-watchdog.timer
|
||||
```
|
||||
|
||||
Cloned somewhere else? Point `ExecStart=` at it — that path is the only thing
|
||||
tying the two together. `ledger.json` is written under `--claude-home`
|
||||
regardless of where the script lives.
|
||||
|
||||
To keep it running while you are logged out (background jobs outlive your
|
||||
shell, so this is usually what you want):
|
||||
|
||||
```bash
|
||||
sudo loginctl enable-linger "$USER"
|
||||
```
|
||||
7,22,37,52 * * * * $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1
|
||||
```
|
||||
|
||||
### cron (Alpine, macOS, anything without systemd)
|
||||
`:07 :22 :37 :52` keeps it off the `:00` crowd. Nothing wraps it — no `PATH`
|
||||
prefix, no `flock`, no `chmod`, because the script handles both of the things a
|
||||
scheduler would otherwise have to:
|
||||
|
||||
The script itself is portable — only the scheduling is not. Add a cron entry
|
||||
with `crontab -e`:
|
||||
- **Finding `claude`** — `PATH` first, then `~/.local/bin`, `/usr/local/bin`,
|
||||
`/opt/homebrew/bin`. If it finds nothing it says so and exits non-zero, rather
|
||||
than logging `nothing to resume` and looking healthy while resuming nothing,
|
||||
forever. `--claude-bin` still wins if you want a specific install.
|
||||
- **Not overlapping itself** — it takes an exclusive lock on
|
||||
`~/.claude/session-watchdog/.lock` and exits if another run holds it. Two
|
||||
concurrent runs would read `ledger.json` before either writes it and resume
|
||||
the same session twice, into two agents racing on one git worktree.
|
||||
|
||||
```
|
||||
7,22,37,52 * * * * PATH=/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin /usr/bin/flock -n $HOME/.claude/session-watchdog/.lock $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1
|
||||
```
|
||||
|
||||
Both prefixes matter, and neither is cosmetic:
|
||||
|
||||
- **`PATH`** — cron's default omits `/usr/local/bin`, which is where `claude`
|
||||
usually is. Without it the watchdog fails *silently*: it logs `nothing to
|
||||
resume` and looks perfectly healthy while resuming nothing, forever. Use
|
||||
`CLAUDE_BIN=/full/path/to/claude` instead if you prefer. Verify with
|
||||
`env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run`
|
||||
— that reproduces cron's environment, so it fails the same way cron would.
|
||||
- **`flock`** — systemd's `Type=oneshot` refuses to run a second copy while the
|
||||
first is going; cron happily overlaps them. Two concurrent runs read the same
|
||||
`ledger.json` before either writes it, so both resume the same session. On a
|
||||
host without `flock` (macOS), use a launchd agent with `StartInterval` set to
|
||||
`900` instead — launchd also refuses to overlap a run with itself.
|
||||
So any scheduler that fires every 15 minutes will do: cron, launchd with
|
||||
`StartInterval 900`, a systemd timer, Task Scheduler.
|
||||
|
||||
Use absolute paths rather than `$HOME` if your cron does not export `HOME`
|
||||
(busybox crond does). On Alpine, user crontabs live in `/etc/crontabs/<user>`
|
||||
@@ -146,18 +126,10 @@ root.
|
||||
# What would it do right now? Resumes nothing.
|
||||
~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run
|
||||
|
||||
# Timer health and next fire time
|
||||
systemctl --user list-timers claude-session-watchdog.timer
|
||||
# The same, with cron's bare environment instead of your shell's.
|
||||
env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run
|
||||
|
||||
# What it has done
|
||||
journalctl --user -u claude-session-watchdog.service --since today
|
||||
```
|
||||
|
||||
Under cron there is no journal, so the redirect in the crontab entry is the
|
||||
record:
|
||||
|
||||
```bash
|
||||
crontab -l # is it scheduled?
|
||||
crontab -l # is it scheduled?
|
||||
tail -f ~/.claude/session-watchdog/watchdog.log # what it has done
|
||||
```
|
||||
|
||||
@@ -170,8 +142,7 @@ A stalled session that has been resumed shows up as a new job in
|
||||
## Configuration
|
||||
|
||||
Every option is a CLI flag with an environment-variable fallback — nothing is
|
||||
hardcoded. To make one permanent, put it in the unit's `[Service]` block, or on
|
||||
the crontab line next to `PATH`.
|
||||
hardcoded. To make one permanent, add it to the crontab line.
|
||||
|
||||
| Flag | Env var | Default |
|
||||
| ----------------- | ------------------------- | -------------------- |
|
||||
@@ -185,20 +156,16 @@ the crontab line next to `PATH`.
|
||||
|
||||
## Files
|
||||
|
||||
| Path | Role |
|
||||
| ----------------------------------- | ------------------------------------------------- |
|
||||
| `resume-stalled-sessions.py` | The watchdog |
|
||||
| `claude-session-watchdog.service` | systemd unit that runs it once |
|
||||
| `claude-session-watchdog.timer` | Fires at `:07 :22 :37 :52` (off the `:00` crowd) |
|
||||
| `~/.claude/session-watchdog/ledger.json` | Per-session attempt counts and resume timestamps |
|
||||
| Path | Role |
|
||||
| ------------------------------------------ | ------------------------------------------ |
|
||||
| `resume-stalled-sessions.py` | The watchdog — the only file it needs |
|
||||
| `~/.claude/session-watchdog/ledger.json` | Attempt counts and resume timestamps |
|
||||
| `~/.claude/session-watchdog/.lock` | Held for the duration of a run |
|
||||
|
||||
`ledger.json` is generated. Delete it to forget all history; edit the
|
||||
`attempts` map to give a specific session another chance.
|
||||
Both live under `--claude-home`, wherever the clone itself is. `ledger.json` is
|
||||
generated: delete it to forget all history, or edit the `attempts` map to give a
|
||||
specific session another chance.
|
||||
|
||||
## Uninstall
|
||||
|
||||
```bash
|
||||
systemctl --user disable --now claude-session-watchdog.timer
|
||||
rm ~/.config/systemd/user/claude-session-watchdog.{service,timer}
|
||||
systemctl --user daemon-reload
|
||||
```
|
||||
Drop the line from `crontab -e` and delete the clone.
|
||||
|
||||
@@ -1,7 +0,0 @@
|
||||
[Unit]
|
||||
Description=Resume Claude Code background sessions stalled on usage limits or outages
|
||||
|
||||
[Service]
|
||||
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
|
||||
ExecStart=%h/.claude/session-watchdog/resume-stalled-sessions.py
|
||||
Type=oneshot
|
||||
@@ -1,10 +0,0 @@
|
||||
[Unit]
|
||||
Description=Check every 15 minutes for stalled Claude Code background sessions
|
||||
|
||||
[Timer]
|
||||
AccuracySec=1min
|
||||
OnCalendar=*:07/15
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -8,15 +8,20 @@ cause has cleared.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
# Where `claude` installs itself, and what a scheduler's minimal PATH omits.
|
||||
CLAUDE_FALLBACK_DIRS = ("~/.local/bin", "/usr/local/bin", "/opt/homebrew/bin")
|
||||
|
||||
DEFAULT_PROMPT = (
|
||||
"Continue where you left off. Your previous run was interrupted by a "
|
||||
"transient failure (usage limit, API outage, or network), which has cleared."
|
||||
@@ -64,8 +69,36 @@ def log(message):
|
||||
print(f"{datetime.now().astimezone().isoformat(timespec='seconds')} {message}", flush=True)
|
||||
|
||||
|
||||
def resolve_claude(name):
|
||||
"""Absolute path to the Claude Code binary, or None if it cannot be found."""
|
||||
found = shutil.which(name)
|
||||
if found:
|
||||
return found
|
||||
for directory in CLAUDE_FALLBACK_DIRS:
|
||||
candidate = Path(directory).expanduser() / name
|
||||
if candidate.is_file() and os.access(candidate, os.X_OK):
|
||||
return str(candidate)
|
||||
return None
|
||||
|
||||
|
||||
def single_instance(path):
|
||||
"""Exclusive lock for this run, or None when another run already holds it.
|
||||
|
||||
Overlapping runs both read the ledger before either writes it, so both
|
||||
resume the same session — two agents racing on one git worktree.
|
||||
"""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
handle = path.open("w")
|
||||
try:
|
||||
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except OSError:
|
||||
handle.close()
|
||||
return None
|
||||
return handle
|
||||
|
||||
|
||||
def active_sessions(claude_bin):
|
||||
"""Sessions Claude Code still considers active (completed ones are excluded)."""
|
||||
"""Sessions Claude Code still considers active, or None if they cannot be read."""
|
||||
try:
|
||||
done = subprocess.run(
|
||||
[claude_bin, "agents", "--json"],
|
||||
@@ -73,12 +106,12 @@ def active_sessions(claude_bin):
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError) as err:
|
||||
log(f"error: could not list sessions: {err}")
|
||||
return []
|
||||
return None
|
||||
try:
|
||||
return json.loads(done.stdout)
|
||||
except json.JSONDecodeError as err:
|
||||
log(f"error: unparseable session list: {err}")
|
||||
return []
|
||||
return None
|
||||
|
||||
|
||||
def pid_alive(pid):
|
||||
@@ -259,8 +292,22 @@ def main():
|
||||
parser.add_argument("--prompt", default=os.environ.get("WATCHDOG_PROMPT", DEFAULT_PROMPT))
|
||||
args = parser.parse_args()
|
||||
|
||||
claude_bin = resolve_claude(args.claude_bin)
|
||||
if not claude_bin:
|
||||
log(f"error: '{args.claude_bin}' is not in PATH or "
|
||||
f"{', '.join(CLAUDE_FALLBACK_DIRS)} — resuming nothing")
|
||||
return 1
|
||||
|
||||
state_dir = args.claude_home / "session-watchdog"
|
||||
lock = None
|
||||
if not args.dry_run:
|
||||
lock = single_instance(state_dir / ".lock") # released when main returns
|
||||
if lock is None:
|
||||
log("another run is still going, skipping")
|
||||
return 0
|
||||
|
||||
now = datetime.now().astimezone()
|
||||
ledger_path = args.claude_home / "session-watchdog" / "ledger.json"
|
||||
ledger_path = state_dir / "ledger.json"
|
||||
ledger = load_ledger(ledger_path)
|
||||
|
||||
recent = [at for at in ledger["resumes"] if now.timestamp() - at < 86400]
|
||||
@@ -271,7 +318,10 @@ def main():
|
||||
log(f"circuit breaker: {len(recent)} resumes in the last 24h (max {args.max_per_day})")
|
||||
return 0
|
||||
|
||||
sessions = active_sessions(args.claude_bin)
|
||||
sessions = active_sessions(claude_bin)
|
||||
if sessions is None:
|
||||
return 1
|
||||
|
||||
resumed = 0
|
||||
for job in candidates(sessions, args.claude_home / "jobs", now):
|
||||
if resumed >= budget:
|
||||
@@ -279,7 +329,7 @@ def main():
|
||||
tried = ledger["attempts"].get(job["session_id"], 0)
|
||||
if tried >= args.max_attempts:
|
||||
continue
|
||||
if not resume(job, args.claude_bin, args.prompt, args.dry_run):
|
||||
if not resume(job, claude_bin, args.prompt, args.dry_run):
|
||||
continue
|
||||
ledger["attempts"][job["session_id"]] = tried + 1
|
||||
ledger["resumes"] = recent + [now.timestamp()]
|
||||
|
||||
Reference in New Issue
Block a user