Hermes-Skills/devops/clash-docker-workflow/references/mihomo-watchdog.md

# Mihomo Watchdog — Prevent Proxy Crashes from Killing Downstream Services

## Problem

mihomo periodically crashes or becomes unresponsive. When this happens, Hermes's Telegram adapter loses connectivity, hits 10 retries, sets a fatal error, and the `_platform_reconnect_watcher` may silently die — leaving Telegram permanently offline even after mihomo recovers.

## Solution: Watchdog Script

A lightweight watchdog that monitors mihomo's health and restarts it before downstream services notice.

### ⚠️ CRITICAL BUG: systemd cgroup kills mihomo on watchdog exit

**Root cause**: When the watchdog runs as a systemd user service (`Type=oneshot`) and calls `startup.sh restart mihomo`, the mihomo process becomes a child of the watchdog's service cgroup. When the watchdog service exits, systemd kills all processes in the cgroup — **including the just-started mihomo**.

**Death loop**: watchdog starts mihomo → watchdog exits → systemd kills mihomo → next watchdog cycle finds mihomo dead → repeat every 2 minutes.

**Symptoms**:
- mihomo log shows `Mihomo shutting down` 3-6 seconds after each startup
- watchdog.log shows repeated "mihomo 进程不存在，正在重启" entries
- mihomo PID file is stale (process already dead)
- Ports 7890/9090 never stay up

**Diagnosis**:
```bash
# Check if watchdog is restarting in a loop
tail -20 ~/startup/logs/watchdog.log
# Check mihomo uptime pattern (should be minutes/hours, not seconds)
grep "Mihomo shutting down" ~/startup/logs/mihomo.log | tail -10
# Check if watchdog timer is active
systemctl --user status mihomo-watchdog.timer
```

**Fix — Option A (recommended): mihomo as dedicated systemd service**

mihomo MUST have its own systemd service so it runs in its own cgroup, independent of the watchdog. Full recipe: `skill_view(name="clash-docker-workflow", file_path="references/systemd-setup.md")`.

Then update the watchdog to call `systemctl --user restart mihomo` instead of `startup.sh restart mihomo`:
**⚠️ Do NOT check proxy connectivity in the watchdog.** Proxy node issues (upstream provider downtime, node rotation) are NOT mihomo's fault — restarting mihomo won't fix them. Only check process existence + API responsiveness. Proxy connectivity checks cause unnecessary restarts that break stable connections (user lesson: 2026-06-23).

```bash
#!/bin/bash
# mihomo watchdog — uses systemctl, NO proxy connectivity check
HEALTH_URL="http://127.0.0.1:9090/version"
LOG="/home/openclaw/startup/logs/watchdog.log"

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG"; }

# 1. Process alive?
if ! pgrep -f "mihomo -d" > /dev/null 2>&1; then
    log "mihomo 进程不存在，正在重启..."
    systemctl --user restart mihomo >> "$LOG" 2>&1; sleep 3
fi

# 2. API responsive?
if ! curl -s --max-time 3 "$HEALTH_URL" > /dev/null 2>&1; then
    log "mihomo API 无响应，正在重启..."
    systemctl --user restart mihomo >> "$LOG" 2>&1
fi
```

**Fix — Option B: Use `systemd-run` to escape cgroup** (if mihomo can't be a service)

```bash
# In the watchdog script, replace:
#   $STARTUP restart mihomo
# With:
systemd-run --user --scope --slice=mihomo.slice bash -c "cd ~/clash && exec ./mihomo -d config"
```

**NEVER do this**: Run `startup.sh restart mihomo` from inside a systemd oneshot service. The spawned process inherits the service's cgroup and gets killed on exit.

---

## Deployed Script: `~/startup/mihomo-watchdog.sh`

Uses `systemctl --user restart mihomo` (requires mihomo as a dedicated systemd service). **No proxy connectivity check** — proxy node issues ≠ mihomo issues.

```bash
#!/bin/bash
# mihomo watchdog — 每 2 分钟检查一次，挂了自动拉起
# ⚠️ 必须配合 mihomo 专用 systemd service 使用（不能用 startup.sh）
HEALTH_URL="http://127.0.0.1:9090/version"
LOG="/home/openclaw/startup/logs/watchdog.log"

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG"; }

# 1. Process alive?
if ! pgrep -f "mihomo -d" > /dev/null 2>&1; then
    log "mihomo 进程不存在，正在重启..."
    systemctl --user restart mihomo >> "$LOG" 2>&1; sleep 3
fi

# 2. API responsive?
if ! curl -s --max-time 3 "$HEALTH_URL" > /dev/null 2>&1; then
    log "mihomo API 无响应，正在重启..."
    systemctl --user restart mihomo >> "$LOG" 2>&1
fi
```

**systemd timer** (runs every 2 minutes, 30s after boot):
```ini
# ~/.config/systemd/user/mihomo-watchdog.timer
[Timer]
OnBootSec=30s
OnUnitActiveSec=2min
AccuracySec=10s
[Install]
WantedBy=timers.target
```

```ini
# ~/.config/systemd/user/mihomo-watchdog.service
[Service]
Type=oneshot
ExecStart=/home/openclaw/startup/mihomo-watchdog.sh
```

```bash
systemctl --user daemon-reload
systemctl --user enable --now mihomo-watchdog.timer
```

Silent when healthy (no log output). Only writes to `watchdog.log` on failure/restart.

### Alternative Script: `~/clash/watchdog.sh` (standalone)

```bash
#!/bin/bash
# Mihomo watchdog — checks proxy health every 60s, restarts if dead
# Run via systemd timer or cron

MIHOMO_API="http://127.0.0.1:9090"
MIHOMO_BIN="$HOME/clash/mihomo"
MIHOMO_DIR="$HOME/clash/config"
LOG="$HOME/clash/watchdog.log"
MAX_LOG_SIZE=1048576  # 1MB

# Rotate log if too big
if [ -f "$LOG" ] && [ "$(stat -c%s "$LOG" 2>/dev/null || echo 0)" -gt "$MAX_LOG_SIZE" ]; then
    mv "$LOG" "${LOG}.old"
fi

check_health() {
    # Method 1: API health check
    if curl -s --max-time 5 "$MIHOMO_API/version" >/dev/null 2>&1; then
        return 0
    fi
    # Method 2: Check if process is alive
    if pgrep -f "mihomo -d" >/dev/null 2>&1; then
        # Process exists but API unresponsive — stuck
        return 1
    fi
    # Process dead
    return 1
}

restart_mihomo() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo unhealthy, restarting..." >> "$LOG"
    # Kill stale processes
    pkill -f "mihomo -d" 2>/dev/null
    sleep 2
    # Start fresh
    cd "$MIHOMO_DIR" && exec "$MIHOMO_BIN" -d "$MIHOMO_DIR" &
    sleep 5
    if curl -s --max-time 5 "$MIHOMO_API/version" >/dev/null 2>&1; then
        echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo restarted successfully" >> "$LOG"
    else
        echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo restart FAILED" >> "$LOG"
    fi
}

if ! check_health; then
    restart_mihomo
fi
```

### Deployment Options

**Option A: Cron (simplest)**
```bash
# Every 2 minutes
*/2 * * * * bash ~/clash/watchdog.sh
```

**Option B: Hermes cronjob (delivers alert on failure)**
```python
cronjob(action='create', schedule='2m', name='mihomo-watchdog',
        prompt='Run: bash ~/clash/watchdog.sh. Only report if restart was needed.',
        deliver='local', enabled_toolsets=['terminal'])
```

**Option C: systemd timer (most robust)**
```ini
# /etc/systemd/system/mihomo-watchdog.timer
[Unit]
Description=Mihomo watchdog timer

[Timer]
OnBootSec=30
OnUnitActiveSec=120

[Install]
WantedBy=timers.target
```

```ini
# /etc/systemd/system/mihomo-watchdog.service
[Unit]
Description=Mihomo watchdog

[Service]
Type=oneshot
ExecStart=/home/openclaw/clash/watchdog.sh
User=openclaw
```

## Why Not Just Restart the Gateway?

Restarting the gateway:
- Kills all active agent sessions
- Interrupts running cron jobs
- Takes 10-20 seconds to fully initialize
- Doesn't fix the underlying mihomo instability

A mihomo watchdog:
- Restarts only the proxy (1-2 seconds)
- No impact on gateway sessions or cron jobs
- Prevents the cascade before it starts
- Is the correct layer to fix (proxy stability, not gateway resilience)