Hermes-Skills/devops/clash-docker-workflow/references/mihomo-watchdog.md

Mihomo Watchdog — Prevent Proxy Crashes from Killing Downstream Services

Problem

mihomo periodically crashes or becomes unresponsive. When this happens, Hermes's Telegram adapter loses connectivity, hits 10 retries, sets a fatal error, and the _platform_reconnect_watcher may silently die — leaving Telegram permanently offline even after mihomo recovers.

Solution: Watchdog Script

A lightweight watchdog that monitors mihomo's health and restarts it before downstream services notice.

⚠️ CRITICAL BUG: systemd cgroup kills mihomo on watchdog exit

Root cause: When the watchdog runs as a systemd user service (Type=oneshot) and calls startup.sh restart mihomo, the mihomo process becomes a child of the watchdog's service cgroup. When the watchdog service exits, systemd kills all processes in the cgroup — including the just-started mihomo.

Death loop: watchdog starts mihomo → watchdog exits → systemd kills mihomo → next watchdog cycle finds mihomo dead → repeat every 2 minutes.

Symptoms:

• mihomo log shows Mihomo shutting down 3-6 seconds after each startup
• watchdog.log shows repeated "mihomo 进程不存在，正在重启" entries
• mihomo PID file is stale (process already dead)
• Ports 7890/9090 never stay up

Diagnosis:

# Check if watchdog is restarting in a loop
tail -20 ~/startup/logs/watchdog.log
# Check mihomo uptime pattern (should be minutes/hours, not seconds)
grep "Mihomo shutting down" ~/startup/logs/mihomo.log | tail -10
# Check if watchdog timer is active
systemctl --user status mihomo-watchdog.timer

Fix — Option A (recommended): mihomo as dedicated systemd service

mihomo MUST have its own systemd service so it runs in its own cgroup, independent of the watchdog. Full recipe: skill_view(name="clash-docker-workflow", file_path="references/systemd-setup.md").

Then update the watchdog to call systemctl --user restart mihomo instead of startup.sh restart mihomo: ⚠️ Do NOT check proxy connectivity in the watchdog. Proxy node issues (upstream provider downtime, node rotation) are NOT mihomo's fault — restarting mihomo won't fix them. Only check process existence + API responsiveness. Proxy connectivity checks cause unnecessary restarts that break stable connections (user lesson: 2026-06-23).

#!/bin/bash
# mihomo watchdog — uses systemctl, NO proxy connectivity check
HEALTH_URL="http://127.0.0.1:9090/version"
LOG="/home/openclaw/startup/logs/watchdog.log"

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG"; }

# 1. Process alive?
if ! pgrep -f "mihomo -d" > /dev/null 2>&1; then
    log "mihomo 进程不存在，正在重启..."
    systemctl --user restart mihomo >> "$LOG" 2>&1; sleep 3
fi

# 2. API responsive?
if ! curl -s --max-time 3 "$HEALTH_URL" > /dev/null 2>&1; then
    log "mihomo API 无响应，正在重启..."
    systemctl --user restart mihomo >> "$LOG" 2>&1
fi

Fix — Option B: Use systemd-run to escape cgroup (if mihomo can't be a service)

# In the watchdog script, replace:
#   $STARTUP restart mihomo
# With:
systemd-run --user --scope --slice=mihomo.slice bash -c "cd ~/clash && exec ./mihomo -d config"

NEVER do this: Run startup.sh restart mihomo from inside a systemd oneshot service. The spawned process inherits the service's cgroup and gets killed on exit.

---

Deployed Script: ~/startup/mihomo-watchdog.sh

Uses systemctl --user restart mihomo (requires mihomo as a dedicated systemd service). No proxy connectivity check — proxy node issues ≠ mihomo issues.

#!/bin/bash
# mihomo watchdog — 每 2 分钟检查一次，挂了自动拉起
# ⚠️ 必须配合 mihomo 专用 systemd service 使用（不能用 startup.sh）
HEALTH_URL="http://127.0.0.1:9090/version"
LOG="/home/openclaw/startup/logs/watchdog.log"

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG"; }

# 1. Process alive?
if ! pgrep -f "mihomo -d" > /dev/null 2>&1; then
    log "mihomo 进程不存在，正在重启..."
    systemctl --user restart mihomo >> "$LOG" 2>&1; sleep 3
fi

# 2. API responsive?
if ! curl -s --max-time 3 "$HEALTH_URL" > /dev/null 2>&1; then
    log "mihomo API 无响应，正在重启..."
    systemctl --user restart mihomo >> "$LOG" 2>&1
fi

systemd timer (runs every 2 minutes, 30s after boot):

# ~/.config/systemd/user/mihomo-watchdog.timer
[Timer]
OnBootSec=30s
OnUnitActiveSec=2min
AccuracySec=10s
[Install]
WantedBy=timers.target

# ~/.config/systemd/user/mihomo-watchdog.service
[Service]
Type=oneshot
ExecStart=/home/openclaw/startup/mihomo-watchdog.sh

systemctl --user daemon-reload
systemctl --user enable --now mihomo-watchdog.timer

Silent when healthy (no log output). Only writes to watchdog.log on failure/restart.

Alternative Script: ~/clash/watchdog.sh (standalone)

#!/bin/bash
# Mihomo watchdog — checks proxy health every 60s, restarts if dead
# Run via systemd timer or cron

MIHOMO_API="http://127.0.0.1:9090"
MIHOMO_BIN="$HOME/clash/mihomo"
MIHOMO_DIR="$HOME/clash/config"
LOG="$HOME/clash/watchdog.log"
MAX_LOG_SIZE=1048576  # 1MB

# Rotate log if too big
if [ -f "$LOG" ] && [ "$(stat -c%s "$LOG" 2>/dev/null || echo 0)" -gt "$MAX_LOG_SIZE" ]; then
    mv "$LOG" "${LOG}.old"
fi

check_health() {
    # Method 1: API health check
    if curl -s --max-time 5 "$MIHOMO_API/version" >/dev/null 2>&1; then
        return 0
    fi
    # Method 2: Check if process is alive
    if pgrep -f "mihomo -d" >/dev/null 2>&1; then
        # Process exists but API unresponsive — stuck
        return 1
    fi
    # Process dead
    return 1
}

restart_mihomo() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo unhealthy, restarting..." >> "$LOG"
    # Kill stale processes
    pkill -f "mihomo -d" 2>/dev/null
    sleep 2
    # Start fresh
    cd "$MIHOMO_DIR" && exec "$MIHOMO_BIN" -d "$MIHOMO_DIR" &
    sleep 5
    if curl -s --max-time 5 "$MIHOMO_API/version" >/dev/null 2>&1; then
        echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo restarted successfully" >> "$LOG"
    else
        echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo restart FAILED" >> "$LOG"
    fi
}

if ! check_health; then
    restart_mihomo
fi

Deployment Options

Option A: Cron (simplest)

# Every 2 minutes
*/2 * * * * bash ~/clash/watchdog.sh

Option B: Hermes cronjob (delivers alert on failure)

cronjob(action='create', schedule='2m', name='mihomo-watchdog',
        prompt='Run: bash ~/clash/watchdog.sh. Only report if restart was needed.',
        deliver='local', enabled_toolsets=['terminal'])

Option C: systemd timer (most robust)

# /etc/systemd/system/mihomo-watchdog.timer
[Unit]
Description=Mihomo watchdog timer

[Timer]
OnBootSec=30
OnUnitActiveSec=120

[Install]
WantedBy=timers.target

# /etc/systemd/system/mihomo-watchdog.service
[Unit]
Description=Mihomo watchdog

[Service]
Type=oneshot
ExecStart=/home/openclaw/clash/watchdog.sh
User=openclaw

Why Not Just Restart the Gateway?

Restarting the gateway:

• Kills all active agent sessions
• Interrupts running cron jobs
• Takes 10-20 seconds to fully initialize
• Doesn't fix the underlying mihomo instability

A mihomo watchdog:

• Restarts only the proxy (1-2 seconds)
• No impact on gateway sessions or cron jobs
• Prevents the cascade before it starts
• Is the correct layer to fix (proxy stability, not gateway resilience)