Initial commit: Hermes Agent skills collection

- Trading skills (OKX, dividend, lottery, quantitative) - Creative skills (ASCII art, diagrams, video) - Development skills (GitHub, debugging, TDD) - Research skills (arXiv, blog monitoring) - Productivity skills (email, documents, notes) - MCP integration skills - Custom user skills
2026-07-05 02:31:15 -04:00
commit 6770bc9b9d
908 changed files with 239614 additions and 0 deletions
@@ -0,0 +1,229 @@
+# Mihomo Watchdog — Prevent Proxy Crashes from Killing Downstream Services
+
+## Problem
+
+mihomo periodically crashes or becomes unresponsive. When this happens, Hermes's Telegram adapter loses connectivity, hits 10 retries, sets a fatal error, and the `_platform_reconnect_watcher` may silently die — leaving Telegram permanently offline even after mihomo recovers.
+
+## Solution: Watchdog Script
+
+A lightweight watchdog that monitors mihomo's health and restarts it before downstream services notice.
+
+### ⚠️ CRITICAL BUG: systemd cgroup kills mihomo on watchdog exit
+
+**Root cause**: When the watchdog runs as a systemd user service (`Type=oneshot`) and calls `startup.sh restart mihomo`, the mihomo process becomes a child of the watchdog's service cgroup. When the watchdog service exits, systemd kills all processes in the cgroup — **including the just-started mihomo**.
+
+**Death loop**: watchdog starts mihomo → watchdog exits → systemd kills mihomo → next watchdog cycle finds mihomo dead → repeat every 2 minutes.
+
+**Symptoms**:
+- mihomo log shows `Mihomo shutting down` 3-6 seconds after each startup
+- watchdog.log shows repeated "mihomo 进程不存在，正在重启" entries
+- mihomo PID file is stale (process already dead)
+- Ports 7890/9090 never stay up
+
+**Diagnosis**:
+```bash
+# Check if watchdog is restarting in a loop
+tail -20 ~/startup/logs/watchdog.log
+# Check mihomo uptime pattern (should be minutes/hours, not seconds)
+grep "Mihomo shutting down" ~/startup/logs/mihomo.log | tail -10
+# Check if watchdog timer is active
+systemctl --user status mihomo-watchdog.timer
+```
+
+**Fix — Option A (recommended): mihomo as dedicated systemd service**
+
+mihomo MUST have its own systemd service so it runs in its own cgroup, independent of the watchdog. Full recipe: `skill_view(name="clash-docker-workflow", file_path="references/systemd-setup.md")`.
+
+Then update the watchdog to call `systemctl --user restart mihomo` instead of `startup.sh restart mihomo`:
+**⚠️ Do NOT check proxy connectivity in the watchdog.** Proxy node issues (upstream provider downtime, node rotation) are NOT mihomo's fault — restarting mihomo won't fix them. Only check process existence + API responsiveness. Proxy connectivity checks cause unnecessary restarts that break stable connections (user lesson: 2026-06-23).
+
+```bash
+#!/bin/bash
+# mihomo watchdog — uses systemctl, NO proxy connectivity check
+HEALTH_URL="http://127.0.0.1:9090/version"
+LOG="/home/openclaw/startup/logs/watchdog.log"
+
+log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG"; }
+
+# 1. Process alive?
+if ! pgrep -f "mihomo -d" > /dev/null 2>&1; then
+    log "mihomo 进程不存在，正在重启..."
+    systemctl --user restart mihomo >> "$LOG" 2>&1; sleep 3
+fi
+
+# 2. API responsive?
+if ! curl -s --max-time 3 "$HEALTH_URL" > /dev/null 2>&1; then
+    log "mihomo API 无响应，正在重启..."
+    systemctl --user restart mihomo >> "$LOG" 2>&1
+fi
+```
+
+**Fix — Option B: Use `systemd-run` to escape cgroup** (if mihomo can't be a service)
+
+```bash
+# In the watchdog script, replace:
+#   $STARTUP restart mihomo
+# With:
+systemd-run --user --scope --slice=mihomo.slice bash -c "cd ~/clash && exec ./mihomo -d config"
+```
+
+**NEVER do this**: Run `startup.sh restart mihomo` from inside a systemd oneshot service. The spawned process inherits the service's cgroup and gets killed on exit.
+
+---
+
+## Deployed Script: `~/startup/mihomo-watchdog.sh`
+
+Uses `systemctl --user restart mihomo` (requires mihomo as a dedicated systemd service). **No proxy connectivity check** — proxy node issues ≠ mihomo issues.
+
+```bash
+#!/bin/bash
+# mihomo watchdog — 每 2 分钟检查一次，挂了自动拉起
+# ⚠️ 必须配合 mihomo 专用 systemd service 使用（不能用 startup.sh）
+HEALTH_URL="http://127.0.0.1:9090/version"
+LOG="/home/openclaw/startup/logs/watchdog.log"
+
+log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG"; }
+
+# 1. Process alive?
+if ! pgrep -f "mihomo -d" > /dev/null 2>&1; then
+    log "mihomo 进程不存在，正在重启..."
+    systemctl --user restart mihomo >> "$LOG" 2>&1; sleep 3
+fi
+
+# 2. API responsive?
+if ! curl -s --max-time 3 "$HEALTH_URL" > /dev/null 2>&1; then
+    log "mihomo API 无响应，正在重启..."
+    systemctl --user restart mihomo >> "$LOG" 2>&1
+fi
+```
+
+**systemd timer** (runs every 2 minutes, 30s after boot):
+```ini
+# ~/.config/systemd/user/mihomo-watchdog.timer
+[Timer]
+OnBootSec=30s
+OnUnitActiveSec=2min
+AccuracySec=10s
+[Install]
+WantedBy=timers.target
+```
+
+```ini
+# ~/.config/systemd/user/mihomo-watchdog.service
+[Service]
+Type=oneshot
+ExecStart=/home/openclaw/startup/mihomo-watchdog.sh
+```
+
+```bash
+systemctl --user daemon-reload
+systemctl --user enable --now mihomo-watchdog.timer
+```
+
+Silent when healthy (no log output). Only writes to `watchdog.log` on failure/restart.
+
+### Alternative Script: `~/clash/watchdog.sh` (standalone)
+
+```bash
+#!/bin/bash
+# Mihomo watchdog — checks proxy health every 60s, restarts if dead
+# Run via systemd timer or cron
+
+MIHOMO_API="http://127.0.0.1:9090"
+MIHOMO_BIN="$HOME/clash/mihomo"
+MIHOMO_DIR="$HOME/clash/config"
+LOG="$HOME/clash/watchdog.log"
+MAX_LOG_SIZE=1048576  # 1MB
+
+# Rotate log if too big
+if [ -f "$LOG" ] && [ "$(stat -c%s "$LOG" 2>/dev/null || echo 0)" -gt "$MAX_LOG_SIZE" ]; then
+    mv "$LOG" "${LOG}.old"
+fi
+
+check_health() {
+    # Method 1: API health check
+    if curl -s --max-time 5 "$MIHOMO_API/version" >/dev/null 2>&1; then
+        return 0
+    fi
+    # Method 2: Check if process is alive
+    if pgrep -f "mihomo -d" >/dev/null 2>&1; then
+        # Process exists but API unresponsive — stuck
+        return 1
+    fi
+    # Process dead
+    return 1
+}
+
+restart_mihomo() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo unhealthy, restarting..." >> "$LOG"
+    # Kill stale processes
+    pkill -f "mihomo -d" 2>/dev/null
+    sleep 2
+    # Start fresh
+    cd "$MIHOMO_DIR" && exec "$MIHOMO_BIN" -d "$MIHOMO_DIR" &
+    sleep 5
+    if curl -s --max-time 5 "$MIHOMO_API/version" >/dev/null 2>&1; then
+        echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo restarted successfully" >> "$LOG"
+    else
+        echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo restart FAILED" >> "$LOG"
+    fi
+}
+
+if ! check_health; then
+    restart_mihomo
+fi
+```
+
+### Deployment Options
+
+**Option A: Cron (simplest)**
+```bash
+# Every 2 minutes
+*/2 * * * * bash ~/clash/watchdog.sh
+```
+
+**Option B: Hermes cronjob (delivers alert on failure)**
+```python
+cronjob(action='create', schedule='2m', name='mihomo-watchdog',
+        prompt='Run: bash ~/clash/watchdog.sh. Only report if restart was needed.',
+        deliver='local', enabled_toolsets=['terminal'])
+```
+
+**Option C: systemd timer (most robust)**
+```ini
+# /etc/systemd/system/mihomo-watchdog.timer
+[Unit]
+Description=Mihomo watchdog timer
+
+[Timer]
+OnBootSec=30
+OnUnitActiveSec=120
+
+[Install]
+WantedBy=timers.target
+```
+
+```ini
+# /etc/systemd/system/mihomo-watchdog.service
+[Unit]
+Description=Mihomo watchdog
+
+[Service]
+Type=oneshot
+ExecStart=/home/openclaw/clash/watchdog.sh
+User=openclaw
+```
+
+## Why Not Just Restart the Gateway?
+
+Restarting the gateway:
+- Kills all active agent sessions
+- Interrupts running cron jobs
+- Takes 10-20 seconds to fully initialize
+- Doesn't fix the underlying mihomo instability
+
+A mihomo watchdog:
+- Restarts only the proxy (1-2 seconds)
+- No impact on gateway sessions or cron jobs
+- Prevents the cascade before it starts
+- Is the correct layer to fix (proxy stability, not gateway resilience)