Drop the systemd units; the script now locks itself and finds claude
This commit is contained in:
@@ -9,7 +9,7 @@ Claude Code has no auto-resume: on a usage limit it blocks further requests
|
|||||||
until the reset time and the process exits. Nothing scheduled *inside* a
|
until the reset time and the process exits. Nothing scheduled *inside* a
|
||||||
session (a 15-minute self-check, a `/loop`, a scheduled task) can rescue it,
|
session (a 15-minute self-check, a `/loop`, a scheduled task) can rescue it,
|
||||||
because there is no process left to fire the schedule. Recovery has to come
|
because there is no process left to fire the schedule. Recovery has to come
|
||||||
from outside, so this is a systemd user timer.
|
from outside, so this is a cron job.
|
||||||
|
|
||||||
It also costs nothing while everything is healthy — it reads state files on
|
It also costs nothing while everything is healthy — it reads state files on
|
||||||
disk rather than waking sessions up. An in-session poll would re-send the
|
disk rather than waking sessions up. An in-session poll would re-send the
|
||||||
@@ -88,52 +88,32 @@ that is the one path that can burn a lot of tokens for nothing. Set
|
|||||||
Needs Claude Code (for `claude agents --json` and `claude --bg --resume`) and
|
Needs Claude Code (for `claude agents --json` and `claude --bg --resume`) and
|
||||||
Python 3.9+ — standard library only.
|
Python 3.9+ — standard library only.
|
||||||
|
|
||||||
Clone into `~/.claude/session-watchdog`. The unit file expects it there, and the
|
Clone anywhere — `~/.claude/session-watchdog` just keeps it travelling with the
|
||||||
watchdog then travels with the rest of `~/.claude`:
|
rest of `~/.claude`. Then add one cron entry with `crontab -e`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://gitea.larvit.se/larvit/claude-session-watchdog.git ~/.claude/session-watchdog
|
git clone https://gitea.larvit.se/larvit/claude-session-watchdog.git ~/.claude/session-watchdog
|
||||||
chmod +x ~/.claude/session-watchdog/resume-stalled-sessions.py
|
|
||||||
mkdir -p ~/.config/systemd/user
|
|
||||||
ln -sf ~/.claude/session-watchdog/claude-session-watchdog.service ~/.config/systemd/user/
|
|
||||||
ln -sf ~/.claude/session-watchdog/claude-session-watchdog.timer ~/.config/systemd/user/
|
|
||||||
systemctl --user daemon-reload
|
|
||||||
systemctl --user enable --now claude-session-watchdog.timer
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Cloned somewhere else? Point `ExecStart=` at it — that path is the only thing
|
```
|
||||||
tying the two together. `ledger.json` is written under `--claude-home`
|
7,22,37,52 * * * * $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1
|
||||||
regardless of where the script lives.
|
|
||||||
|
|
||||||
To keep it running while you are logged out (background jobs outlive your
|
|
||||||
shell, so this is usually what you want):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo loginctl enable-linger "$USER"
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### cron (Alpine, macOS, anything without systemd)
|
`:07 :22 :37 :52` keeps it off the `:00` crowd. Nothing wraps it — no `PATH`
|
||||||
|
prefix, no `flock`, no `chmod`, because the script handles both of the things a
|
||||||
|
scheduler would otherwise have to:
|
||||||
|
|
||||||
The script itself is portable — only the scheduling is not. Add a cron entry
|
- **Finding `claude`** — `PATH` first, then `~/.local/bin`, `/usr/local/bin`,
|
||||||
with `crontab -e`:
|
`/opt/homebrew/bin`. If it finds nothing it says so and exits non-zero, rather
|
||||||
|
than logging `nothing to resume` and looking healthy while resuming nothing,
|
||||||
|
forever. `--claude-bin` still wins if you want a specific install.
|
||||||
|
- **Not overlapping itself** — it takes an exclusive lock on
|
||||||
|
`~/.claude/session-watchdog/.lock` and exits if another run holds it. Two
|
||||||
|
concurrent runs would read `ledger.json` before either writes it and resume
|
||||||
|
the same session twice, into two agents racing on one git worktree.
|
||||||
|
|
||||||
```
|
So any scheduler that fires every 15 minutes will do: cron, launchd with
|
||||||
7,22,37,52 * * * * PATH=/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin /usr/bin/flock -n $HOME/.claude/session-watchdog/.lock $HOME/.claude/session-watchdog/resume-stalled-sessions.py >> $HOME/.claude/session-watchdog/watchdog.log 2>&1
|
`StartInterval 900`, a systemd timer, Task Scheduler.
|
||||||
```
|
|
||||||
|
|
||||||
Both prefixes matter, and neither is cosmetic:
|
|
||||||
|
|
||||||
- **`PATH`** — cron's default omits `/usr/local/bin`, which is where `claude`
|
|
||||||
usually is. Without it the watchdog fails *silently*: it logs `nothing to
|
|
||||||
resume` and looks perfectly healthy while resuming nothing, forever. Use
|
|
||||||
`CLAUDE_BIN=/full/path/to/claude` instead if you prefer. Verify with
|
|
||||||
`env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run`
|
|
||||||
— that reproduces cron's environment, so it fails the same way cron would.
|
|
||||||
- **`flock`** — systemd's `Type=oneshot` refuses to run a second copy while the
|
|
||||||
first is going; cron happily overlaps them. Two concurrent runs read the same
|
|
||||||
`ledger.json` before either writes it, so both resume the same session. On a
|
|
||||||
host without `flock` (macOS), use a launchd agent with `StartInterval` set to
|
|
||||||
`900` instead — launchd also refuses to overlap a run with itself.
|
|
||||||
|
|
||||||
Use absolute paths rather than `$HOME` if your cron does not export `HOME`
|
Use absolute paths rather than `$HOME` if your cron does not export `HOME`
|
||||||
(busybox crond does). On Alpine, user crontabs live in `/etc/crontabs/<user>`
|
(busybox crond does). On Alpine, user crontabs live in `/etc/crontabs/<user>`
|
||||||
@@ -146,18 +126,10 @@ root.
|
|||||||
# What would it do right now? Resumes nothing.
|
# What would it do right now? Resumes nothing.
|
||||||
~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run
|
~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run
|
||||||
|
|
||||||
# Timer health and next fire time
|
# The same, with cron's bare environment instead of your shell's.
|
||||||
systemctl --user list-timers claude-session-watchdog.timer
|
env -i HOME=$HOME PATH=/usr/bin:/bin ~/.claude/session-watchdog/resume-stalled-sessions.py --dry-run
|
||||||
|
|
||||||
# What it has done
|
crontab -l # is it scheduled?
|
||||||
journalctl --user -u claude-session-watchdog.service --since today
|
|
||||||
```
|
|
||||||
|
|
||||||
Under cron there is no journal, so the redirect in the crontab entry is the
|
|
||||||
record:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
crontab -l # is it scheduled?
|
|
||||||
tail -f ~/.claude/session-watchdog/watchdog.log # what it has done
|
tail -f ~/.claude/session-watchdog/watchdog.log # what it has done
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -170,8 +142,7 @@ A stalled session that has been resumed shows up as a new job in
|
|||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
Every option is a CLI flag with an environment-variable fallback — nothing is
|
Every option is a CLI flag with an environment-variable fallback — nothing is
|
||||||
hardcoded. To make one permanent, put it in the unit's `[Service]` block, or on
|
hardcoded. To make one permanent, add it to the crontab line.
|
||||||
the crontab line next to `PATH`.
|
|
||||||
|
|
||||||
| Flag | Env var | Default |
|
| Flag | Env var | Default |
|
||||||
| ----------------- | ------------------------- | -------------------- |
|
| ----------------- | ------------------------- | -------------------- |
|
||||||
@@ -185,20 +156,16 @@ the crontab line next to `PATH`.
|
|||||||
|
|
||||||
## Files
|
## Files
|
||||||
|
|
||||||
| Path | Role |
|
| Path | Role |
|
||||||
| ----------------------------------- | ------------------------------------------------- |
|
| ------------------------------------------ | ------------------------------------------ |
|
||||||
| `resume-stalled-sessions.py` | The watchdog |
|
| `resume-stalled-sessions.py` | The watchdog — the only file it needs |
|
||||||
| `claude-session-watchdog.service` | systemd unit that runs it once |
|
| `~/.claude/session-watchdog/ledger.json` | Attempt counts and resume timestamps |
|
||||||
| `claude-session-watchdog.timer` | Fires at `:07 :22 :37 :52` (off the `:00` crowd) |
|
| `~/.claude/session-watchdog/.lock` | Held for the duration of a run |
|
||||||
| `~/.claude/session-watchdog/ledger.json` | Per-session attempt counts and resume timestamps |
|
|
||||||
|
|
||||||
`ledger.json` is generated. Delete it to forget all history; edit the
|
Both live under `--claude-home`, wherever the clone itself is. `ledger.json` is
|
||||||
`attempts` map to give a specific session another chance.
|
generated: delete it to forget all history, or edit the `attempts` map to give a
|
||||||
|
specific session another chance.
|
||||||
|
|
||||||
## Uninstall
|
## Uninstall
|
||||||
|
|
||||||
```bash
|
Drop the line from `crontab -e` and delete the clone.
|
||||||
systemctl --user disable --now claude-session-watchdog.timer
|
|
||||||
rm ~/.config/systemd/user/claude-session-watchdog.{service,timer}
|
|
||||||
systemctl --user daemon-reload
|
|
||||||
```
|
|
||||||
|
|||||||
@@ -1,7 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=Resume Claude Code background sessions stalled on usage limits or outages
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
|
|
||||||
ExecStart=%h/.claude/session-watchdog/resume-stalled-sessions.py
|
|
||||||
Type=oneshot
|
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=Check every 15 minutes for stalled Claude Code background sessions
|
|
||||||
|
|
||||||
[Timer]
|
|
||||||
AccuracySec=1min
|
|
||||||
OnCalendar=*:07/15
|
|
||||||
Persistent=true
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=timers.target
|
|
||||||
@@ -8,15 +8,20 @@ cause has cleared.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import fcntl
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from zoneinfo import ZoneInfo
|
from zoneinfo import ZoneInfo
|
||||||
|
|
||||||
|
# Where `claude` installs itself, and what a scheduler's minimal PATH omits.
|
||||||
|
CLAUDE_FALLBACK_DIRS = ("~/.local/bin", "/usr/local/bin", "/opt/homebrew/bin")
|
||||||
|
|
||||||
DEFAULT_PROMPT = (
|
DEFAULT_PROMPT = (
|
||||||
"Continue where you left off. Your previous run was interrupted by a "
|
"Continue where you left off. Your previous run was interrupted by a "
|
||||||
"transient failure (usage limit, API outage, or network), which has cleared."
|
"transient failure (usage limit, API outage, or network), which has cleared."
|
||||||
@@ -64,8 +69,36 @@ def log(message):
|
|||||||
print(f"{datetime.now().astimezone().isoformat(timespec='seconds')} {message}", flush=True)
|
print(f"{datetime.now().astimezone().isoformat(timespec='seconds')} {message}", flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_claude(name):
|
||||||
|
"""Absolute path to the Claude Code binary, or None if it cannot be found."""
|
||||||
|
found = shutil.which(name)
|
||||||
|
if found:
|
||||||
|
return found
|
||||||
|
for directory in CLAUDE_FALLBACK_DIRS:
|
||||||
|
candidate = Path(directory).expanduser() / name
|
||||||
|
if candidate.is_file() and os.access(candidate, os.X_OK):
|
||||||
|
return str(candidate)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def single_instance(path):
|
||||||
|
"""Exclusive lock for this run, or None when another run already holds it.
|
||||||
|
|
||||||
|
Overlapping runs both read the ledger before either writes it, so both
|
||||||
|
resume the same session — two agents racing on one git worktree.
|
||||||
|
"""
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
handle = path.open("w")
|
||||||
|
try:
|
||||||
|
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||||
|
except OSError:
|
||||||
|
handle.close()
|
||||||
|
return None
|
||||||
|
return handle
|
||||||
|
|
||||||
|
|
||||||
def active_sessions(claude_bin):
|
def active_sessions(claude_bin):
|
||||||
"""Sessions Claude Code still considers active (completed ones are excluded)."""
|
"""Sessions Claude Code still considers active, or None if they cannot be read."""
|
||||||
try:
|
try:
|
||||||
done = subprocess.run(
|
done = subprocess.run(
|
||||||
[claude_bin, "agents", "--json"],
|
[claude_bin, "agents", "--json"],
|
||||||
@@ -73,12 +106,12 @@ def active_sessions(claude_bin):
|
|||||||
)
|
)
|
||||||
except (OSError, subprocess.SubprocessError) as err:
|
except (OSError, subprocess.SubprocessError) as err:
|
||||||
log(f"error: could not list sessions: {err}")
|
log(f"error: could not list sessions: {err}")
|
||||||
return []
|
return None
|
||||||
try:
|
try:
|
||||||
return json.loads(done.stdout)
|
return json.loads(done.stdout)
|
||||||
except json.JSONDecodeError as err:
|
except json.JSONDecodeError as err:
|
||||||
log(f"error: unparseable session list: {err}")
|
log(f"error: unparseable session list: {err}")
|
||||||
return []
|
return None
|
||||||
|
|
||||||
|
|
||||||
def pid_alive(pid):
|
def pid_alive(pid):
|
||||||
@@ -259,8 +292,22 @@ def main():
|
|||||||
parser.add_argument("--prompt", default=os.environ.get("WATCHDOG_PROMPT", DEFAULT_PROMPT))
|
parser.add_argument("--prompt", default=os.environ.get("WATCHDOG_PROMPT", DEFAULT_PROMPT))
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
claude_bin = resolve_claude(args.claude_bin)
|
||||||
|
if not claude_bin:
|
||||||
|
log(f"error: '{args.claude_bin}' is not in PATH or "
|
||||||
|
f"{', '.join(CLAUDE_FALLBACK_DIRS)} — resuming nothing")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
state_dir = args.claude_home / "session-watchdog"
|
||||||
|
lock = None
|
||||||
|
if not args.dry_run:
|
||||||
|
lock = single_instance(state_dir / ".lock") # released when main returns
|
||||||
|
if lock is None:
|
||||||
|
log("another run is still going, skipping")
|
||||||
|
return 0
|
||||||
|
|
||||||
now = datetime.now().astimezone()
|
now = datetime.now().astimezone()
|
||||||
ledger_path = args.claude_home / "session-watchdog" / "ledger.json"
|
ledger_path = state_dir / "ledger.json"
|
||||||
ledger = load_ledger(ledger_path)
|
ledger = load_ledger(ledger_path)
|
||||||
|
|
||||||
recent = [at for at in ledger["resumes"] if now.timestamp() - at < 86400]
|
recent = [at for at in ledger["resumes"] if now.timestamp() - at < 86400]
|
||||||
@@ -271,7 +318,10 @@ def main():
|
|||||||
log(f"circuit breaker: {len(recent)} resumes in the last 24h (max {args.max_per_day})")
|
log(f"circuit breaker: {len(recent)} resumes in the last 24h (max {args.max_per_day})")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
sessions = active_sessions(args.claude_bin)
|
sessions = active_sessions(claude_bin)
|
||||||
|
if sessions is None:
|
||||||
|
return 1
|
||||||
|
|
||||||
resumed = 0
|
resumed = 0
|
||||||
for job in candidates(sessions, args.claude_home / "jobs", now):
|
for job in candidates(sessions, args.claude_home / "jobs", now):
|
||||||
if resumed >= budget:
|
if resumed >= budget:
|
||||||
@@ -279,7 +329,7 @@ def main():
|
|||||||
tried = ledger["attempts"].get(job["session_id"], 0)
|
tried = ledger["attempts"].get(job["session_id"], 0)
|
||||||
if tried >= args.max_attempts:
|
if tried >= args.max_attempts:
|
||||||
continue
|
continue
|
||||||
if not resume(job, args.claude_bin, args.prompt, args.dry_run):
|
if not resume(job, claude_bin, args.prompt, args.dry_run):
|
||||||
continue
|
continue
|
||||||
ledger["attempts"][job["session_id"]] = tried + 1
|
ledger["attempts"][job["session_id"]] = tried + 1
|
||||||
ledger["resumes"] = recent + [now.timestamp()]
|
ledger["resumes"] = recent + [now.timestamp()]
|
||||||
|
|||||||
Reference in New Issue
Block a user