Drop the systemd units; the script now locks itself and finds claude

This commit is contained in:
2026-07-29 22:57:13 +02:00
parent 891b28d28c
commit 6e5bfdfb2c
4 changed files with 87 additions and 87 deletions
+31 -64
View File
@@ -9,7 +9,7 @@ Claude Code has no auto-resume: on a usage limit it blocks further requests
until the reset time and the process exits. Nothing scheduled *inside* a until the reset time and the process exits. Nothing scheduled *inside* a
session (a 15-minute self-check, a `/loop`, a scheduled task) can rescue it, session (a 15-minute self-check, a `/loop`, a scheduled task) can rescue it,
because there is no process left to fire the schedule. Recovery has to come because there is no process left to fire the schedule. Recovery has to come
from outside, so this is a systemd user timer. from outside, so this is a cron job.
It also costs nothing while everything is healthy — it reads state files on It also costs nothing while everything is healthy — it reads state files on
disk rather than waking sessions up. An in-session poll would re-send the disk rather than waking sessions up. An in-session poll would re-send the
@@ -88,52 +88,32 @@ that is the one path that can burn a lot of tokens for nothing. Set
Needs Claude Code (for `claude agents --json` and `claude --bg --resume`) and Needs Claude Code (for `claude agents --json` and `claude --bg --resume`) and
Python 3.9+ — standard library only. Python 3.9+ — standard library only.
Clone into `~/.claude/session-watchdog`. The unit file expects it there, and the Clone anywhere — `~/.claude/session-watchdog` just keeps it travelling with the
watchdog then travels with the rest of `~/.claude`: rest of `~/.claude`. Then add one cron entry with `crontab -e`:
```bash ```bash
git clone https://gitea.larvit.se/larvit/claude-session-watchdog.git ~/.claude/session-watchdog git clone https://gitea.larvit.se/larvit/claude-session-watchdog.git ~/.claude/session-watchdog
chmod +x ~/.claude/session-watchdog/resume-stalled-sessions.py
mkdir -p ~/.config/systemd/user
ln -sf ~/.claude/session-watchdog/claude-session-watchdog.service ~/.config/systemd/user/
ln -sf ~/.claude/session-watchdog/claude-session-watchdog.timer ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now claude-session-watchdog.timer
``` ```
Cloned somewhere else? Point `ExecStart=` at it — that path is the only thing ```
tying the two together. `ledger.json` is written under `--claude-home` 7,22,37,52 * * * * $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1
regardless of where the script lives.
To keep it running while you are logged out (background jobs outlive your
shell, so this is usually what you want):
```bash
sudo loginctl enable-linger "$USER"
``` ```
### cron (Alpine, macOS, anything without systemd) `:07 :22 :37 :52` keeps it off the `:00` crowd. Nothing wraps it — no `PATH`
prefix, no `flock`, no `chmod`, because the script handles both of the things a
scheduler would otherwise have to:
The script itself is portable — only the scheduling is not. Add a cron entry - **Finding `claude`** — `PATH` first, then `~/.local/bin`, `/usr/local/bin`,
with `crontab -e`: `/opt/homebrew/bin`. If it finds nothing it says so and exits non-zero, rather
than logging `nothing to resume` and looking healthy while resuming nothing,
forever. `--claude-bin` still wins if you want a specific install.
- **Not overlapping itself** — it takes an exclusive lock on
`~/.claude/session-watchdog/.lock` and exits if another run holds it. Two
concurrent runs would read `ledger.json` before either writes it and resume
the same session twice, into two agents racing on one git worktree.
``` So any scheduler that fires every 15 minutes will do: cron, launchd with
7,22,37,52 * * * * PATH=/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin /usr/bin/flock -n $HOME/.claude/session-watchdog/.lock $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1 `StartInterval 900`, a systemd timer, Task Scheduler.
```
Both prefixes matter, and neither is cosmetic:
- **`PATH`** — cron's default omits `/usr/local/bin`, which is where `claude`
usually is. Without it the watchdog fails *silently*: it logs `nothing to
resume` and looks perfectly healthy while resuming nothing, forever. Use
`CLAUDE_BIN=/full/path/to/claude` instead if you prefer. Verify with
`env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run`
— that reproduces cron's environment, so it fails the same way cron would.
- **`flock`** — systemd's `Type=oneshot` refuses to run a second copy while the
first is going; cron happily overlaps them. Two concurrent runs read the same
`ledger.json` before either writes it, so both resume the same session. On a
host without `flock` (macOS), use a launchd agent with `StartInterval` set to
`900` instead — launchd also refuses to overlap a run with itself.
Use absolute paths rather than `$HOME` if your cron does not export `HOME` Use absolute paths rather than `$HOME` if your cron does not export `HOME`
(busybox crond does). On Alpine, user crontabs live in `/etc/crontabs/<user>` (busybox crond does). On Alpine, user crontabs live in `/etc/crontabs/<user>`
@@ -146,18 +126,10 @@ root.
# What would it do right now? Resumes nothing. # What would it do right now? Resumes nothing.
~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run
# Timer health and next fire time # The same, with cron's bare environment instead of your shell's.
systemctl --user list-timers claude-session-watchdog.timer env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run
# What it has done crontab -l # is it scheduled?
journalctl --user -u claude-session-watchdog.service --since today
```
Under cron there is no journal, so the redirect in the crontab entry is the
record:
```bash
crontab -l # is it scheduled?
tail -f ~/.claude/session-watchdog/watchdog.log # what it has done tail -f ~/.claude/session-watchdog/watchdog.log # what it has done
``` ```
@@ -170,8 +142,7 @@ A stalled session that has been resumed shows up as a new job in
## Configuration ## Configuration
Every option is a CLI flag with an environment-variable fallback — nothing is Every option is a CLI flag with an environment-variable fallback — nothing is
hardcoded. To make one permanent, put it in the unit's `[Service]` block, or on hardcoded. To make one permanent, add it to the crontab line.
the crontab line next to `PATH`.
| Flag | Env var | Default | | Flag | Env var | Default |
| ----------------- | ------------------------- | -------------------- | | ----------------- | ------------------------- | -------------------- |
@@ -185,20 +156,16 @@ the crontab line next to `PATH`.
## Files ## Files
| Path | Role | | Path | Role |
| ----------------------------------- | ------------------------------------------------- | | ------------------------------------------ | ------------------------------------------ |
| `resume-stalled-sessions.py` | The watchdog | | `resume-stalled-sessions.py` | The watchdog — the only file it needs |
| `claude-session-watchdog.service` | systemd unit that runs it once | | `~/.claude/session-watchdog/ledger.json` | Attempt counts and resume timestamps |
| `claude-session-watchdog.timer` | Fires at `:07 :22 :37 :52` (off the `:00` crowd) | | `~/.claude/session-watchdog/.lock` | Held for the duration of a run |
| `~/.claude/session-watchdog/ledger.json` | Per-session attempt counts and resume timestamps |
`ledger.json` is generated. Delete it to forget all history; edit the Both live under `--claude-home`, wherever the clone itself is. `ledger.json` is
`attempts` map to give a specific session another chance. generated: delete it to forget all history, or edit the `attempts` map to give a
specific session another chance.
## Uninstall ## Uninstall
```bash Drop the line from `crontab -e` and delete the clone.
systemctl --user disable --now claude-session-watchdog.timer
rm ~/.config/systemd/user/claude-session-watchdog.{service,timer}
systemctl --user daemon-reload
```
-7
View File
@@ -1,7 +0,0 @@
[Unit]
Description=Resume Claude Code background sessions stalled on usage limits or outages
[Service]
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
ExecStart=%h/.claude/session-watchdog/resume-stalled-sessions.py
Type=oneshot
-10
View File
@@ -1,10 +0,0 @@
[Unit]
Description=Check every 15 minutes for stalled Claude Code background sessions
[Timer]
AccuracySec=1min
OnCalendar=*:07/15
Persistent=true
[Install]
WantedBy=timers.target
+56 -6
View File
@@ -8,15 +8,20 @@ cause has cleared.
""" """
import argparse import argparse
import fcntl
import json import json
import os import os
import re import re
import shutil
import subprocess import subprocess
import sys import sys
from datetime import datetime, timedelta from datetime import datetime, timedelta
from pathlib import Path from pathlib import Path
from zoneinfo import ZoneInfo from zoneinfo import ZoneInfo
# Where `claude` installs itself, and what a scheduler's minimal PATH omits.
CLAUDE_FALLBACK_DIRS = ("~/.local/bin", "/usr/local/bin", "/opt/homebrew/bin")
DEFAULT_PROMPT = ( DEFAULT_PROMPT = (
"Continue where you left off. Your previous run was interrupted by a " "Continue where you left off. Your previous run was interrupted by a "
"transient failure (usage limit, API outage, or network), which has cleared." "transient failure (usage limit, API outage, or network), which has cleared."
@@ -64,8 +69,36 @@ def log(message):
print(f"{datetime.now().astimezone().isoformat(timespec='seconds')} {message}", flush=True) print(f"{datetime.now().astimezone().isoformat(timespec='seconds')} {message}", flush=True)
def resolve_claude(name):
"""Absolute path to the Claude Code binary, or None if it cannot be found."""
found = shutil.which(name)
if found:
return found
for directory in CLAUDE_FALLBACK_DIRS:
candidate = Path(directory).expanduser() / name
if candidate.is_file() and os.access(candidate, os.X_OK):
return str(candidate)
return None
def single_instance(path):
"""Exclusive lock for this run, or None when another run already holds it.
Overlapping runs both read the ledger before either writes it, so both
resume the same session — two agents racing on one git worktree.
"""
path.parent.mkdir(parents=True, exist_ok=True)
handle = path.open("w")
try:
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
handle.close()
return None
return handle
def active_sessions(claude_bin): def active_sessions(claude_bin):
"""Sessions Claude Code still considers active (completed ones are excluded).""" """Sessions Claude Code still considers active, or None if they cannot be read."""
try: try:
done = subprocess.run( done = subprocess.run(
[claude_bin, "agents", "--json"], [claude_bin, "agents", "--json"],
@@ -73,12 +106,12 @@ def active_sessions(claude_bin):
) )
except (OSError, subprocess.SubprocessError) as err: except (OSError, subprocess.SubprocessError) as err:
log(f"error: could not list sessions: {err}") log(f"error: could not list sessions: {err}")
return [] return None
try: try:
return json.loads(done.stdout) return json.loads(done.stdout)
except json.JSONDecodeError as err: except json.JSONDecodeError as err:
log(f"error: unparseable session list: {err}") log(f"error: unparseable session list: {err}")
return [] return None
def pid_alive(pid): def pid_alive(pid):
@@ -259,8 +292,22 @@ def main():
parser.add_argument("--prompt", default=os.environ.get("WATCHDOG_PROMPT", DEFAULT_PROMPT)) parser.add_argument("--prompt", default=os.environ.get("WATCHDOG_PROMPT", DEFAULT_PROMPT))
args = parser.parse_args() args = parser.parse_args()
claude_bin = resolve_claude(args.claude_bin)
if not claude_bin:
log(f"error: '{args.claude_bin}' is not in PATH or "
f"{', '.join(CLAUDE_FALLBACK_DIRS)} — resuming nothing")
return 1
state_dir = args.claude_home / "session-watchdog"
lock = None
if not args.dry_run:
lock = single_instance(state_dir / ".lock") # released when main returns
if lock is None:
log("another run is still going, skipping")
return 0
now = datetime.now().astimezone() now = datetime.now().astimezone()
ledger_path = args.claude_home / "session-watchdog" / "ledger.json" ledger_path = state_dir / "ledger.json"
ledger = load_ledger(ledger_path) ledger = load_ledger(ledger_path)
recent = [at for at in ledger["resumes"] if now.timestamp() - at < 86400] recent = [at for at in ledger["resumes"] if now.timestamp() - at < 86400]
@@ -271,7 +318,10 @@ def main():
log(f"circuit breaker: {len(recent)} resumes in the last 24h (max {args.max_per_day})") log(f"circuit breaker: {len(recent)} resumes in the last 24h (max {args.max_per_day})")
return 0 return 0
sessions = active_sessions(args.claude_bin) sessions = active_sessions(claude_bin)
if sessions is None:
return 1
resumed = 0 resumed = 0
for job in candidates(sessions, args.claude_home / "jobs", now): for job in candidates(sessions, args.claude_home / "jobs", now):
if resumed >= budget: if resumed >= budget:
@@ -279,7 +329,7 @@ def main():
tried = ledger["attempts"].get(job["session_id"], 0) tried = ledger["attempts"].get(job["session_id"], 0)
if tried >= args.max_attempts: if tried >= args.max_attempts:
continue continue
if not resume(job, args.claude_bin, args.prompt, args.dry_run): if not resume(job, claude_bin, args.prompt, args.dry_run):
continue continue
ledger["attempts"][job["session_id"]] = tried + 1 ledger["attempts"][job["session_id"]] = tried + 1
ledger["resumes"] = recent + [now.timestamp()] ledger["resumes"] = recent + [now.timestamp()]