diff --git a/README.md b/README.md index b6781b7..d6424e2 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ Claude Code has no auto-resume: on a usage limit it blocks further requests until the reset time and the process exits. Nothing scheduled *inside* a session (a 15-minute self-check, a `/loop`, a scheduled task) can rescue it, because there is no process left to fire the schedule. Recovery has to come -from outside, so this is a systemd user timer. +from outside, so this is a cron job. It also costs nothing while everything is healthy — it reads state files on disk rather than waking sessions up. An in-session poll would re-send the @@ -88,52 +88,32 @@ that is the one path that can burn a lot of tokens for nothing. Set Needs Claude Code (for `claude agents --json` and `claude --bg --resume`) and Python 3.9+ — standard library only. -Clone into `~/.claude/session-watchdog`. The unit file expects it there, and the -watchdog then travels with the rest of `~/.claude`: +Clone anywhere — `~/.claude/session-watchdog` just keeps it travelling with the +rest of `~/.claude`. Then add one cron entry with `crontab -e`: ```bash git clone https://gitea.larvit.se/larvit/claude-session-watchdog.git ~/.claude/session-watchdog -chmod +x ~/.claude/session-watchdog/resume-stalled-sessions.py -mkdir -p ~/.config/systemd/user -ln -sf ~/.claude/session-watchdog/claude-session-watchdog.service ~/.config/systemd/user/ -ln -sf ~/.claude/session-watchdog/claude-session-watchdog.timer ~/.config/systemd/user/ -systemctl --user daemon-reload -systemctl --user enable --now claude-session-watchdog.timer ``` -Cloned somewhere else? Point `ExecStart=` at it — that path is the only thing -tying the two together. `ledger.json` is written under `--claude-home` -regardless of where the script lives. - -To keep it running while you are logged out (background jobs outlive your -shell, so this is usually what you want): - -```bash -sudo loginctl enable-linger "$USER" +``` +7,22,37,52 * * * * $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1 ``` -### cron (Alpine, macOS, anything without systemd) +`:07 :22 :37 :52` keeps it off the `:00` crowd. Nothing wraps it — no `PATH` +prefix, no `flock`, no `chmod`, because the script handles both of the things a +scheduler would otherwise have to: -The script itself is portable — only the scheduling is not. Add a cron entry -with `crontab -e`: +- **Finding `claude`** — `PATH` first, then `~/.local/bin`, `/usr/local/bin`, + `/opt/homebrew/bin`. If it finds nothing it says so and exits non-zero, rather + than logging `nothing to resume` and looking healthy while resuming nothing, + forever. `--claude-bin` still wins if you want a specific install. +- **Not overlapping itself** — it takes an exclusive lock on + `~/.claude/session-watchdog/.lock` and exits if another run holds it. Two + concurrent runs would read `ledger.json` before either writes it and resume + the same session twice, into two agents racing on one git worktree. -``` -7,22,37,52 * * * * PATH=/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin /usr/bin/flock -n $HOME/.claude/session-watchdog/.lock $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1 -``` - -Both prefixes matter, and neither is cosmetic: - -- **`PATH`** — cron's default omits `/usr/local/bin`, which is where `claude` - usually is. Without it the watchdog fails *silently*: it logs `nothing to - resume` and looks perfectly healthy while resuming nothing, forever. Use - `CLAUDE_BIN=/full/path/to/claude` instead if you prefer. Verify with - `env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run` - — that reproduces cron's environment, so it fails the same way cron would. -- **`flock`** — systemd's `Type=oneshot` refuses to run a second copy while the - first is going; cron happily overlaps them. Two concurrent runs read the same - `ledger.json` before either writes it, so both resume the same session. On a - host without `flock` (macOS), use a launchd agent with `StartInterval` set to - `900` instead — launchd also refuses to overlap a run with itself. +So any scheduler that fires every 15 minutes will do: cron, launchd with +`StartInterval 900`, a systemd timer, Task Scheduler. Use absolute paths rather than `$HOME` if your cron does not export `HOME` (busybox crond does). On Alpine, user crontabs live in `/etc/crontabs/` @@ -146,18 +126,10 @@ root. # What would it do right now? Resumes nothing. ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run -# Timer health and next fire time -systemctl --user list-timers claude-session-watchdog.timer +# The same, with cron's bare environment instead of your shell's. +env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run -# What it has done -journalctl --user -u claude-session-watchdog.service --since today -``` - -Under cron there is no journal, so the redirect in the crontab entry is the -record: - -```bash -crontab -l # is it scheduled? +crontab -l # is it scheduled? tail -f ~/.claude/session-watchdog/watchdog.log # what it has done ``` @@ -170,8 +142,7 @@ A stalled session that has been resumed shows up as a new job in ## Configuration Every option is a CLI flag with an environment-variable fallback — nothing is -hardcoded. To make one permanent, put it in the unit's `[Service]` block, or on -the crontab line next to `PATH`. +hardcoded. To make one permanent, add it to the crontab line. | Flag | Env var | Default | | ----------------- | ------------------------- | -------------------- | @@ -185,20 +156,16 @@ the crontab line next to `PATH`. ## Files -| Path | Role | -| ----------------------------------- | ------------------------------------------------- | -| `resume-stalled-sessions.py` | The watchdog | -| `claude-session-watchdog.service` | systemd unit that runs it once | -| `claude-session-watchdog.timer` | Fires at `:07 :22 :37 :52` (off the `:00` crowd) | -| `~/.claude/session-watchdog/ledger.json` | Per-session attempt counts and resume timestamps | +| Path | Role | +| ------------------------------------------ | ------------------------------------------ | +| `resume-stalled-sessions.py` | The watchdog — the only file it needs | +| `~/.claude/session-watchdog/ledger.json` | Attempt counts and resume timestamps | +| `~/.claude/session-watchdog/.lock` | Held for the duration of a run | -`ledger.json` is generated. Delete it to forget all history; edit the -`attempts` map to give a specific session another chance. +Both live under `--claude-home`, wherever the clone itself is. `ledger.json` is +generated: delete it to forget all history, or edit the `attempts` map to give a +specific session another chance. ## Uninstall -```bash -systemctl --user disable --now claude-session-watchdog.timer -rm ~/.config/systemd/user/claude-session-watchdog.{service,timer} -systemctl --user daemon-reload -``` +Drop the line from `crontab -e` and delete the clone. diff --git a/claude-session-watchdog.service b/claude-session-watchdog.service deleted file mode 100644 index 8ee3d11..0000000 --- a/claude-session-watchdog.service +++ /dev/null @@ -1,7 +0,0 @@ -[Unit] -Description=Resume Claude Code background sessions stalled on usage limits or outages - -[Service] -Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin -ExecStart=%h/.claude/session-watchdog/resume-stalled-sessions.py -Type=oneshot diff --git a/claude-session-watchdog.timer b/claude-session-watchdog.timer deleted file mode 100644 index 3b2f9ac..0000000 --- a/claude-session-watchdog.timer +++ /dev/null @@ -1,10 +0,0 @@ -[Unit] -Description=Check every 15 minutes for stalled Claude Code background sessions - -[Timer] -AccuracySec=1min -OnCalendar=*:07/15 -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/resume-stalled-sessions.py b/resume-stalled-sessions.py index ffd15e8..f1cc350 100755 --- a/resume-stalled-sessions.py +++ b/resume-stalled-sessions.py @@ -8,15 +8,20 @@ cause has cleared. """ import argparse +import fcntl import json import os import re +import shutil import subprocess import sys from datetime import datetime, timedelta from pathlib import Path from zoneinfo import ZoneInfo +# Where `claude` installs itself, and what a scheduler's minimal PATH omits. +CLAUDE_FALLBACK_DIRS = ("~/.local/bin", "/usr/local/bin", "/opt/homebrew/bin") + DEFAULT_PROMPT = ( "Continue where you left off. Your previous run was interrupted by a " "transient failure (usage limit, API outage, or network), which has cleared." @@ -64,8 +69,36 @@ def log(message): print(f"{datetime.now().astimezone().isoformat(timespec='seconds')} {message}", flush=True) +def resolve_claude(name): + """Absolute path to the Claude Code binary, or None if it cannot be found.""" + found = shutil.which(name) + if found: + return found + for directory in CLAUDE_FALLBACK_DIRS: + candidate = Path(directory).expanduser() / name + if candidate.is_file() and os.access(candidate, os.X_OK): + return str(candidate) + return None + + +def single_instance(path): + """Exclusive lock for this run, or None when another run already holds it. + + Overlapping runs both read the ledger before either writes it, so both + resume the same session — two agents racing on one git worktree. + """ + path.parent.mkdir(parents=True, exist_ok=True) + handle = path.open("w") + try: + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + handle.close() + return None + return handle + + def active_sessions(claude_bin): - """Sessions Claude Code still considers active (completed ones are excluded).""" + """Sessions Claude Code still considers active, or None if they cannot be read.""" try: done = subprocess.run( [claude_bin, "agents", "--json"], @@ -73,12 +106,12 @@ def active_sessions(claude_bin): ) except (OSError, subprocess.SubprocessError) as err: log(f"error: could not list sessions: {err}") - return [] + return None try: return json.loads(done.stdout) except json.JSONDecodeError as err: log(f"error: unparseable session list: {err}") - return [] + return None def pid_alive(pid): @@ -259,8 +292,22 @@ def main(): parser.add_argument("--prompt", default=os.environ.get("WATCHDOG_PROMPT", DEFAULT_PROMPT)) args = parser.parse_args() + claude_bin = resolve_claude(args.claude_bin) + if not claude_bin: + log(f"error: '{args.claude_bin}' is not in PATH or " + f"{', '.join(CLAUDE_FALLBACK_DIRS)} — resuming nothing") + return 1 + + state_dir = args.claude_home / "session-watchdog" + lock = None + if not args.dry_run: + lock = single_instance(state_dir / ".lock") # released when main returns + if lock is None: + log("another run is still going, skipping") + return 0 + now = datetime.now().astimezone() - ledger_path = args.claude_home / "session-watchdog" / "ledger.json" + ledger_path = state_dir / "ledger.json" ledger = load_ledger(ledger_path) recent = [at for at in ledger["resumes"] if now.timestamp() - at < 86400] @@ -271,7 +318,10 @@ def main(): log(f"circuit breaker: {len(recent)} resumes in the last 24h (max {args.max_per_day})") return 0 - sessions = active_sessions(args.claude_bin) + sessions = active_sessions(claude_bin) + if sessions is None: + return 1 + resumed = 0 for job in candidates(sessions, args.claude_home / "jobs", now): if resumed >= budget: @@ -279,7 +329,7 @@ def main(): tried = ledger["attempts"].get(job["session_id"], 0) if tried >= args.max_attempts: continue - if not resume(job, args.claude_bin, args.prompt, args.dry_run): + if not resume(job, claude_bin, args.prompt, args.dry_run): continue ledger["attempts"][job["session_id"]] = tried + 1 ledger["resumes"] = recent + [now.timestamp()]