Initial commit: Hermes Agent skills collection
- Trading skills (OKX, dividend, lottery, quantitative) - Creative skills (ASCII art, diagrams, video) - Development skills (GitHub, debugging, TDD) - Research skills (arXiv, blog monitoring) - Productivity skills (email, documents, notes) - MCP integration skills - Custom user skills
This commit is contained in:
Hermes Skills Manager
@@ -0,0 +1,107 @@
|
||||
# Hermes Telegram Proxy Configuration
|
||||
|
||||
How to route Hermes Agent's Telegram gateway through a local proxy (Clash/Mihomo).
|
||||
|
||||
## Why needed
|
||||
Telegram API (`api.telegram.org`) is blocked in China. The Hermes gateway needs a proxy to connect.
|
||||
|
||||
## How the Telegram Adapter Reads Proxy
|
||||
|
||||
The Telegram adapter reads proxy from environment variables at the network layer.
|
||||
|
||||
Source code:
|
||||
- `gateway/platforms/telegram_network.py` → `_resolve_proxy_url()` → calls `resolve_proxy_url("TELEGRAM_PROXY", ...)` from `gateway/platforms/base.py`
|
||||
- `gateway/platforms/telegram.py` line 2068 → same `resolve_proxy_url()` call
|
||||
|
||||
`resolve_proxy_url()` checks (in order):
|
||||
1. `TELEGRAM_PROXY` env var (Telegram-specific, highest priority)
|
||||
2. `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` env vars (and lowercase variants)
|
||||
3. macOS system proxy auto-detection
|
||||
|
||||
**Key: `telegram.proxy_url` in config.yaml IS bridged to `TELEGRAM_PROXY` env var** by `load_gateway_config()` at startup. So both config.yaml and direct env vars work.
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### Option A — `telegram.proxy_url` in config.yaml (simpler, recommended)
|
||||
|
||||
```bash
|
||||
hermes config set telegram.proxy_url http://127.0.0.1:7890
|
||||
hermes gateway restart
|
||||
```
|
||||
|
||||
Verified working — `load_gateway_config()` reads `telegram.proxy_url` and sets `TELEGRAM_PROXY` env var.
|
||||
|
||||
### Option B — System env vars in systemd service file
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
import pathlib
|
||||
p = pathlib.Path.home() / '.config/systemd/user/hermes-gateway.service'
|
||||
lines = p.read_text().splitlines()
|
||||
# Remove any existing proxy lines first
|
||||
lines = [l for l in lines if 'HTTPS_PROXY' not in l and 'HTTP_PROXY' not in l]
|
||||
out = []
|
||||
for l in lines:
|
||||
out.append(l)
|
||||
# Insert after HERMES_HOME line
|
||||
if 'HERMES_HOME' in l and l.strip().startswith('Environment'):
|
||||
out.append('Environment=\"HTTPS_PROXY=http://127.0.0.1:7890\"')
|
||||
out.append('Environment=\"HTTP_PROXY=http://127.0.0.1:7890\"')
|
||||
p.write_text('\n'.join(out) + '\n')
|
||||
"
|
||||
|
||||
systemctl --user daemon-reload
|
||||
hermes gateway restart
|
||||
```
|
||||
|
||||
Or use the quick-fix script: `bash ~/clash/fix-proxy.sh` → option 6 → option 1.
|
||||
|
||||
### Option C — Dual defense (recommended)
|
||||
|
||||
Set BOTH config.yaml bridge AND systemd env vars. This ensures proxy works whether the config bridge runs or not:
|
||||
|
||||
```bash
|
||||
# 1. Config bridge
|
||||
hermes config set telegram.proxy_url http://127.0.0.1:7890
|
||||
|
||||
# 2. Systemd env vars (see Option B above)
|
||||
|
||||
# 3. Restart
|
||||
hermes gateway restart
|
||||
```
|
||||
|
||||
## Applying changes
|
||||
```bash
|
||||
hermes gateway restart
|
||||
# ⚠️ Cannot restart from inside the gateway — instruct user to run from separate shell
|
||||
```
|
||||
|
||||
## Verification
|
||||
```bash
|
||||
# Check config
|
||||
grep proxy_url ~/.hermes/config.yaml
|
||||
|
||||
# Check process env vars
|
||||
PID=$(pgrep -f "hermes_cli.main gateway" | head -1)
|
||||
tr '\0' '\n' < /proc/$PID/environ | grep -i proxy
|
||||
# Expected: TELEGRAM_PROXY=http://127.0.0.1:7890 (from config bridge)
|
||||
# HTTPS_PROXY=http://127.0.0.1:7890 (from systemd, if set)
|
||||
|
||||
# Check logs
|
||||
journalctl --user -u hermes-gateway -n 50 | grep -i telegram
|
||||
# Good: "connected" state
|
||||
# Bad: "connection failed" / "Timed out"
|
||||
|
||||
# Check gateway state
|
||||
cat ~/.hermes/gateway_state.json | python3 -m json.tool | grep -A3 telegram
|
||||
```
|
||||
|
||||
## Supported proxy types
|
||||
- HTTP proxy: `http://127.0.0.1:7890`
|
||||
- SOCKS5 proxy: `socks5://127.0.0.1:1080`
|
||||
|
||||
## Notes
|
||||
- The proxy applies to both the main Telegram connection and the fallback IP transport.
|
||||
- Mihomo/Clash mixed-port (7890) supports both HTTP and SOCKS5 on the same port.
|
||||
- The bot token is configured separately under `telegram.token` in config.yaml.
|
||||
- For full diagnostic checklist: `skill_view(name="clash-docker-workflow", file_path="references/proxy-diagnostics.md")`
|
||||
@@ -0,0 +1,229 @@
|
||||
# Mihomo Watchdog — Prevent Proxy Crashes from Killing Downstream Services
|
||||
|
||||
## Problem
|
||||
|
||||
mihomo periodically crashes or becomes unresponsive. When this happens, Hermes's Telegram adapter loses connectivity, hits 10 retries, sets a fatal error, and the `_platform_reconnect_watcher` may silently die — leaving Telegram permanently offline even after mihomo recovers.
|
||||
|
||||
## Solution: Watchdog Script
|
||||
|
||||
A lightweight watchdog that monitors mihomo's health and restarts it before downstream services notice.
|
||||
|
||||
### ⚠️ CRITICAL BUG: systemd cgroup kills mihomo on watchdog exit
|
||||
|
||||
**Root cause**: When the watchdog runs as a systemd user service (`Type=oneshot`) and calls `startup.sh restart mihomo`, the mihomo process becomes a child of the watchdog's service cgroup. When the watchdog service exits, systemd kills all processes in the cgroup — **including the just-started mihomo**.
|
||||
|
||||
**Death loop**: watchdog starts mihomo → watchdog exits → systemd kills mihomo → next watchdog cycle finds mihomo dead → repeat every 2 minutes.
|
||||
|
||||
**Symptoms**:
|
||||
- mihomo log shows `Mihomo shutting down` 3-6 seconds after each startup
|
||||
- watchdog.log shows repeated "mihomo 进程不存在，正在重启" entries
|
||||
- mihomo PID file is stale (process already dead)
|
||||
- Ports 7890/9090 never stay up
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check if watchdog is restarting in a loop
|
||||
tail -20 ~/startup/logs/watchdog.log
|
||||
# Check mihomo uptime pattern (should be minutes/hours, not seconds)
|
||||
grep "Mihomo shutting down" ~/startup/logs/mihomo.log | tail -10
|
||||
# Check if watchdog timer is active
|
||||
systemctl --user status mihomo-watchdog.timer
|
||||
```
|
||||
|
||||
**Fix — Option A (recommended): mihomo as dedicated systemd service**
|
||||
|
||||
mihomo MUST have its own systemd service so it runs in its own cgroup, independent of the watchdog. Full recipe: `skill_view(name="clash-docker-workflow", file_path="references/systemd-setup.md")`.
|
||||
|
||||
Then update the watchdog to call `systemctl --user restart mihomo` instead of `startup.sh restart mihomo`:
|
||||
**⚠️ Do NOT check proxy connectivity in the watchdog.** Proxy node issues (upstream provider downtime, node rotation) are NOT mihomo's fault — restarting mihomo won't fix them. Only check process existence + API responsiveness. Proxy connectivity checks cause unnecessary restarts that break stable connections (user lesson: 2026-06-23).
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# mihomo watchdog — uses systemctl, NO proxy connectivity check
|
||||
HEALTH_URL="http://127.0.0.1:9090/version"
|
||||
LOG="/home/openclaw/startup/logs/watchdog.log"
|
||||
|
||||
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG"; }
|
||||
|
||||
# 1. Process alive?
|
||||
if ! pgrep -f "mihomo -d" > /dev/null 2>&1; then
|
||||
log "mihomo 进程不存在，正在重启..."
|
||||
systemctl --user restart mihomo >> "$LOG" 2>&1; sleep 3
|
||||
fi
|
||||
|
||||
# 2. API responsive?
|
||||
if ! curl -s --max-time 3 "$HEALTH_URL" > /dev/null 2>&1; then
|
||||
log "mihomo API 无响应，正在重启..."
|
||||
systemctl --user restart mihomo >> "$LOG" 2>&1
|
||||
fi
|
||||
```
|
||||
|
||||
**Fix — Option B: Use `systemd-run` to escape cgroup** (if mihomo can't be a service)
|
||||
|
||||
```bash
|
||||
# In the watchdog script, replace:
|
||||
# $STARTUP restart mihomo
|
||||
# With:
|
||||
systemd-run --user --scope --slice=mihomo.slice bash -c "cd ~/clash && exec ./mihomo -d config"
|
||||
```
|
||||
|
||||
**NEVER do this**: Run `startup.sh restart mihomo` from inside a systemd oneshot service. The spawned process inherits the service's cgroup and gets killed on exit.
|
||||
|
||||
---
|
||||
|
||||
## Deployed Script: `~/startup/mihomo-watchdog.sh`
|
||||
|
||||
Uses `systemctl --user restart mihomo` (requires mihomo as a dedicated systemd service). **No proxy connectivity check** — proxy node issues ≠ mihomo issues.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# mihomo watchdog — 每 2 分钟检查一次，挂了自动拉起
|
||||
# ⚠️ 必须配合 mihomo 专用 systemd service 使用（不能用 startup.sh）
|
||||
HEALTH_URL="http://127.0.0.1:9090/version"
|
||||
LOG="/home/openclaw/startup/logs/watchdog.log"
|
||||
|
||||
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG"; }
|
||||
|
||||
# 1. Process alive?
|
||||
if ! pgrep -f "mihomo -d" > /dev/null 2>&1; then
|
||||
log "mihomo 进程不存在，正在重启..."
|
||||
systemctl --user restart mihomo >> "$LOG" 2>&1; sleep 3
|
||||
fi
|
||||
|
||||
# 2. API responsive?
|
||||
if ! curl -s --max-time 3 "$HEALTH_URL" > /dev/null 2>&1; then
|
||||
log "mihomo API 无响应，正在重启..."
|
||||
systemctl --user restart mihomo >> "$LOG" 2>&1
|
||||
fi
|
||||
```
|
||||
|
||||
**systemd timer** (runs every 2 minutes, 30s after boot):
|
||||
```ini
|
||||
# ~/.config/systemd/user/mihomo-watchdog.timer
|
||||
[Timer]
|
||||
OnBootSec=30s
|
||||
OnUnitActiveSec=2min
|
||||
AccuracySec=10s
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
```ini
|
||||
# ~/.config/systemd/user/mihomo-watchdog.service
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/home/openclaw/startup/mihomo-watchdog.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now mihomo-watchdog.timer
|
||||
```
|
||||
|
||||
Silent when healthy (no log output). Only writes to `watchdog.log` on failure/restart.
|
||||
|
||||
### Alternative Script: `~/clash/watchdog.sh` (standalone)
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Mihomo watchdog — checks proxy health every 60s, restarts if dead
|
||||
# Run via systemd timer or cron
|
||||
|
||||
MIHOMO_API="http://127.0.0.1:9090"
|
||||
MIHOMO_BIN="$HOME/clash/mihomo"
|
||||
MIHOMO_DIR="$HOME/clash/config"
|
||||
LOG="$HOME/clash/watchdog.log"
|
||||
MAX_LOG_SIZE=1048576 # 1MB
|
||||
|
||||
# Rotate log if too big
|
||||
if [ -f "$LOG" ] && [ "$(stat -c%s "$LOG" 2>/dev/null || echo 0)" -gt "$MAX_LOG_SIZE" ]; then
|
||||
mv "$LOG" "${LOG}.old"
|
||||
fi
|
||||
|
||||
check_health() {
|
||||
# Method 1: API health check
|
||||
if curl -s --max-time 5 "$MIHOMO_API/version" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
# Method 2: Check if process is alive
|
||||
if pgrep -f "mihomo -d" >/dev/null 2>&1; then
|
||||
# Process exists but API unresponsive — stuck
|
||||
return 1
|
||||
fi
|
||||
# Process dead
|
||||
return 1
|
||||
}
|
||||
|
||||
restart_mihomo() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo unhealthy, restarting..." >> "$LOG"
|
||||
# Kill stale processes
|
||||
pkill -f "mihomo -d" 2>/dev/null
|
||||
sleep 2
|
||||
# Start fresh
|
||||
cd "$MIHOMO_DIR" && exec "$MIHOMO_BIN" -d "$MIHOMO_DIR" &
|
||||
sleep 5
|
||||
if curl -s --max-time 5 "$MIHOMO_API/version" >/dev/null 2>&1; then
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo restarted successfully" >> "$LOG"
|
||||
else
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo restart FAILED" >> "$LOG"
|
||||
fi
|
||||
}
|
||||
|
||||
if ! check_health; then
|
||||
restart_mihomo
|
||||
fi
|
||||
```
|
||||
|
||||
### Deployment Options
|
||||
|
||||
**Option A: Cron (simplest)**
|
||||
```bash
|
||||
# Every 2 minutes
|
||||
*/2 * * * * bash ~/clash/watchdog.sh
|
||||
```
|
||||
|
||||
**Option B: Hermes cronjob (delivers alert on failure)**
|
||||
```python
|
||||
cronjob(action='create', schedule='2m', name='mihomo-watchdog',
|
||||
prompt='Run: bash ~/clash/watchdog.sh. Only report if restart was needed.',
|
||||
deliver='local', enabled_toolsets=['terminal'])
|
||||
```
|
||||
|
||||
**Option C: systemd timer (most robust)**
|
||||
```ini
|
||||
# /etc/systemd/system/mihomo-watchdog.timer
|
||||
[Unit]
|
||||
Description=Mihomo watchdog timer
|
||||
|
||||
[Timer]
|
||||
OnBootSec=30
|
||||
OnUnitActiveSec=120
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/mihomo-watchdog.service
|
||||
[Unit]
|
||||
Description=Mihomo watchdog
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/home/openclaw/clash/watchdog.sh
|
||||
User=openclaw
|
||||
```
|
||||
|
||||
## Why Not Just Restart the Gateway?
|
||||
|
||||
Restarting the gateway:
|
||||
- Kills all active agent sessions
|
||||
- Interrupts running cron jobs
|
||||
- Takes 10-20 seconds to fully initialize
|
||||
- Doesn't fix the underlying mihomo instability
|
||||
|
||||
A mihomo watchdog:
|
||||
- Restarts only the proxy (1-2 seconds)
|
||||
- No impact on gateway sessions or cron jobs
|
||||
- Prevents the cascade before it starts
|
||||
- Is the correct layer to fix (proxy stability, not gateway resilience)
|
||||
@@ -0,0 +1,188 @@
|
||||
# Proxy Diagnostics: "Is the proxy working?"
|
||||
|
||||
Systematic checklist when the user asks whether the proxy is healthy, or when Hermes services show connection timeouts despite the proxy running.
|
||||
|
||||
## Key Insight
|
||||
**Proxy works ≠ Hermes is using it.** These are independent checks. A curl test passing through the proxy proves the proxy is healthy, but Hermes may not be routing through it.
|
||||
|
||||
## Critical: Telegram Proxy Source
|
||||
|
||||
The Telegram adapter reads proxy from environment variables at the network layer.
|
||||
|
||||
Source code path:
|
||||
- `gateway/platforms/telegram_network.py` → `_resolve_proxy_url()` → `resolve_proxy_url("TELEGRAM_PROXY", ...)` from `gateway/platforms/base.py`
|
||||
- `gateway/platforms/telegram.py` line 2068 → same `resolve_proxy_url()` call
|
||||
|
||||
`resolve_proxy_url()` checks (in order):
|
||||
1. `TELEGRAM_PROXY` env var (Telegram-specific, highest priority)
|
||||
2. `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` env vars (and lowercase variants)
|
||||
3. macOS system proxy auto-detection
|
||||
|
||||
**Key: `telegram.proxy_url` in config.yaml IS bridged to `TELEGRAM_PROXY` env var** by `load_gateway_config()` at startup. So both config.yaml and direct env vars work. However, if `load_gateway_config()` fails to run (e.g. gateway started manually without the config bridge), only direct env vars work.
|
||||
|
||||
**Dual defense recommended**: set BOTH `telegram.proxy_url` in config.yaml AND `HTTPS_PROXY` in the systemd service file. This ensures proxy works whether the config bridge runs or not.
|
||||
|
||||
## Quick Diagnosis: Read Gateway Logs Directly
|
||||
|
||||
When `terminal()` is blocked (consent mechanism, security scanner, etc.), diagnose via log files:
|
||||
|
||||
```python
|
||||
# 1. Find log files
|
||||
search_files(pattern="*.log", path="~/.hermes/logs", target="files")
|
||||
|
||||
# 2. Read recent gateway log tail
|
||||
read_file(path="~/.hermes/logs/gateway.log", offset=-100) # last 100 lines
|
||||
|
||||
# 3. Search for Telegram errors
|
||||
search_files(pattern="telegram.*failed|Connect attempt|All connection",
|
||||
path="~/.hermes/logs/gateway.log", target="content")
|
||||
```
|
||||
|
||||
**Key log patterns to look for:**
|
||||
|
||||
| Log Pattern | Meaning |
|
||||
|---|---|
|
||||
| `Proxy detected; passing explicitly to HTTPXRequest: http://127.0.0.1:7890` | Proxy URL is configured in Hermes ✓ |
|
||||
| `Connect attempt N/8 failed: httpx.ConnectError: All connection attempts failed` | Proxy port unreachable (proxy down or wrong port) |
|
||||
| `connect timed out after 30s` | All retry attempts exhausted, will retry in N seconds |
|
||||
| `Reconnect telegram error: ... next retry in 300s` | Exponential backoff — retries increase: 60→120→240→300s |
|
||||
| `Primary api.telegram.org connection failed (); trying fallback IPs` | DNS resolution worked but TCP connection failed |
|
||||
| `DoH discovery yielded no usable IPs` | DNS-over-HTTPS failed, using hardcoded fallback IPs |
|
||||
|
||||
**Common failure: proxy itself is DOWN** (simplest case):
|
||||
- Log shows: `Proxy detected` + `All connection attempts failed` on every attempt
|
||||
- This means Hermes knows about the proxy (config bridge works), but the proxy port is not responding
|
||||
- Fix: restart mihomo (`systemctl --user restart mihomo` or `bash ~/clash/startup.sh restart mihomo`)
|
||||
|
||||
**This is different from**: proxy running but Hermes not routing through it (see "Common Failure Pattern" below).
|
||||
|
||||
## Diagnostic Steps (in order)
|
||||
|
||||
### 1. Proxy process alive?
|
||||
```bash
|
||||
ss -tlnp | grep 7890 # mihomo should be LISTEN
|
||||
# Expected: users:(("mihomo",pid=NNNN,fd=3))
|
||||
```
|
||||
|
||||
### 2. Proxy actually works? (curl test)
|
||||
```bash
|
||||
curl -x http://127.0.0.1:7890 -I https://www.google.com --connect-timeout 5
|
||||
curl -x http://127.0.0.1:7890 -I https://api.telegram.org --connect-timeout 5
|
||||
```
|
||||
Both should return HTTP 200/302, not timeout.
|
||||
|
||||
### 3. Gateway process has proxy env vars?
|
||||
```bash
|
||||
PID=$(pgrep -f "hermes_cli.main gateway" | head -1)
|
||||
tr '\0' '\n' < /proc/$PID/environ | grep -i proxy
|
||||
```
|
||||
If empty — the gateway process has NO proxy env vars and relies solely on the config bridge (`telegram.proxy_url` → `TELEGRAM_PROXY`). If the bridge fails, Telegram connects DIRECT.
|
||||
|
||||
### 4. Check systemd service file
|
||||
```bash
|
||||
grep -i proxy ~/.config/systemd/user/hermes-gateway.service
|
||||
```
|
||||
If no `HTTPS_PROXY` / `HTTP_PROXY` lines — the service file needs updating for dual defense.
|
||||
|
||||
### 5. Check config.yaml bridge
|
||||
```bash
|
||||
grep proxy_url ~/.hermes/config.yaml
|
||||
```
|
||||
Should show `proxy_url: http://127.0.0.1:7890` under `telegram:`.
|
||||
|
||||
### 6. Gateway status + logs
|
||||
```bash
|
||||
hermes gateway status
|
||||
journalctl --user -u hermes-gateway -n 50 | grep -i telegram
|
||||
```
|
||||
Look for: "Proxy detected" (good) or "connection failed" / "Timed out" (bad).
|
||||
|
||||
### 7. Run hermes doctor
|
||||
```bash
|
||||
hermes doctor
|
||||
```
|
||||
Shows API connectivity, missing configs, auth issues.
|
||||
|
||||
## Common Failure Pattern
|
||||
|
||||
**Symptom**: Proxy is running, curl tests pass, but gateway shows:
|
||||
```
|
||||
[Telegram] Primary api.telegram.org connection failed
|
||||
[Telegram] Fallback IP 149.154.167.220 failed
|
||||
[Telegram] Connect attempt N/8 failed: Timed out
|
||||
```
|
||||
|
||||
**Root cause**: Gateway systemd service file has no proxy environment variables AND the config bridge failed (or `telegram.proxy_url` not set). The Telegram adapter falls back to direct connection, which fails behind GFW.
|
||||
|
||||
**Fix** (dual defense — set both):
|
||||
```bash
|
||||
# 1. Set config bridge
|
||||
hermes config set telegram.proxy_url http://127.0.0.1:7890
|
||||
|
||||
# 2. Add proxy env vars to systemd service
|
||||
python3 -c "
|
||||
import pathlib
|
||||
p = pathlib.Path.home() / '.config/systemd/user/hermes-gateway.service'
|
||||
lines = p.read_text().splitlines()
|
||||
lines = [l for l in lines if 'HTTPS_PROXY' not in l and 'HTTP_PROXY' not in l]
|
||||
out = []
|
||||
for l in lines:
|
||||
out.append(l)
|
||||
if 'HERMES_HOME' in l and l.strip().startswith('Environment'):
|
||||
out.append('Environment=\"HTTPS_PROXY=http://127.0.0.1:7890\"')
|
||||
out.append('Environment=\"HTTP_PROXY=http://127.0.0.1:7890\"')
|
||||
p.write_text('\n'.join(out) + '\n')
|
||||
"
|
||||
systemctl --user daemon-reload
|
||||
hermes gateway restart
|
||||
```
|
||||
|
||||
**Quick fix**: `bash ~/clash/fix-proxy.sh --fix`
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
PID=$(pgrep -f "hermes_cli.main gateway" | head -1)
|
||||
tr '\0' '\n' < /proc/$PID/environ | grep -i proxy
|
||||
# Expected: TELEGRAM_PROXY=http://127.0.0.1:7890 (from config bridge)
|
||||
# HTTPS_PROXY=http://127.0.0.1:7890 (from systemd service)
|
||||
```
|
||||
|
||||
## Network/Proxy Audit Checklist
|
||||
|
||||
When the user asks "check my network/proxy config" or "any redundant configs?", cover these dimensions:
|
||||
|
||||
1. **代理进程**: mihomo running? Port 7890 listening?
|
||||
2. **代理连通性**: curl through proxy works? (Google, Telegram)
|
||||
3. **Gateway 代理模式**: Both config.yaml `proxy_url` AND systemd `HTTPS_PROXY` set? (dual defense)
|
||||
4. **旧服务残留**: Any old gateway services (e.g. `openclaw-gateway.service`) still in `~/.config/systemd/user/`? Disable with `systemctl --user disable --now <name>`
|
||||
5. **重复启动脚本**: Both `startup.sh` and systemd managing the same service? Pick one — systemd preferred. The other is dead code.
|
||||
6. **WireGuard 残留**: Check `/etc/wireguard/` for leftover `.conf` files. WireGuard can intercept ALL traffic if misconfigured.
|
||||
7. **环境变量代理**: Check `~/.bashrc` for `HTTP_PROXY`/`HTTPS_PROXY` — needed for terminal sessions but NOT sufficient for systemd services (they don't source .bashrc).
|
||||
8. **channel_prompts 重复**: Same prompt in slack/discord/telegram sections of config.yaml — only needed in platforms actually used.
|
||||
|
||||
## Quick Status Summary Template
|
||||
When reporting proxy status to the user, cover these four dimensions:
|
||||
1. **代理进程**: mihomo running? Port 7890 listening?
|
||||
2. **代理连通性**: curl through proxy works? (Google, Telegram)
|
||||
3. **Gateway 代理模式**: systemd service has proxy env vars? Config bridge set? Gateway process has them?
|
||||
4. **网关状态**: Gateway running? Telegram connected? No timeout errors in logs?
|
||||
|
||||
- **Gateway log path**: `~/.hermes/logs/gateway.log`, NOT `~/.hermes/gateway.log`. Also check `~/.hermes/logs/errors.log` for historical errors. Use `search_files(pattern="*.log", path="~/.hermes/logs", target="files")` to discover all logs.
|
||||
- **Telegram reconnect backoff**: When Telegram fails to connect, the gateway retries with exponential backoff: 60s → 120s → 240s → 300s (cap). After fixing the proxy, you may need to `hermes gateway restart` to reset the backoff instead of waiting for the next retry window.
|
||||
- **QQBot also fails when proxy is down**: The QQBot adapter routes through the same proxy. If `127.0.0.1:7890` is down, both Telegram AND QQ will show connection failures. Check QQBot logs for `Failed to get QQ Bot access token: All connection attempts failed`.
|
||||
|
||||
## Quick-Fix Script
|
||||
|
||||
`~/clash/fix-proxy.sh` — interactive menu with 8 functions:
|
||||
1. 📊 查看状态 (full status overview)
|
||||
2. 🔧 一键诊断修复 (auto-diagnose and fix)
|
||||
3. 🔄 重启 Mihomo
|
||||
4. 📡 更新/更换订阅
|
||||
5. 🔀 切换代理节点 (via Mihomo API)
|
||||
6. 🌐 Gateway 代理/直连切换
|
||||
7. 🔁 重启 Hermes Gateway
|
||||
8. 📋 查看日志
|
||||
|
||||
Usage:
|
||||
- Interactive: `bash ~/clash/fix-proxy.sh`
|
||||
- One-click fix: `bash ~/clash/fix-proxy.sh --fix`
|
||||
@@ -0,0 +1,153 @@
|
||||
# Systemd Service for Mihomo + YACD
|
||||
|
||||
Two approaches: **system-level** (recommended for always-on servers) and **user-level**.
|
||||
|
||||
---
|
||||
|
||||
## Option A: System-Level Services (recommended for servers)
|
||||
|
||||
System-level services run as root, survive logout without extra config, and start early in boot.
|
||||
|
||||
### Create service files
|
||||
|
||||
**Pitfall**: The Hermes security scanner blocks `sudo tee /etc/systemd/system/` and `sudo cp` into system paths. Workaround: write to `/tmp` first using `write_file`, then ask the user to manually run `sudo cp` + `sudo systemctl`.
|
||||
|
||||
```bash
|
||||
# Step 1: Agent writes to /tmp (allowed by write_file tool)
|
||||
# /tmp/mihomo.service:
|
||||
[Unit]
|
||||
Description=Mihomo (Clash Meta) Proxy
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=/home/openclaw/clash
|
||||
ExecStart=/home/openclaw/clash/mihomo -d /home/openclaw/clash/config
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
LimitNOFILE=65535
|
||||
StandardOutput=append:/tmp/mihomo.log
|
||||
StandardError=append:/tmp/mihomo.log
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
# /tmp/yacd.service:
|
||||
[Unit]
|
||||
Description=YACD - Clash Dashboard
|
||||
After=network-online.target mihomo.service
|
||||
Wants=network-online.target
|
||||
Requires=mihomo.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=/home/openclaw/clash/yacd
|
||||
ExecStart=/usr/bin/python3 -m http.server 9091 --bind 0.0.0.0
|
||||
Restart=on-failure
|
||||
RestartSec=3
|
||||
StandardOutput=append:/tmp/yacd.log
|
||||
StandardError=append:/tmp/yacd.log
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
### Step 2: User manually runs
|
||||
```bash
|
||||
sudo cp /tmp/mihomo.service /etc/systemd/system/
|
||||
sudo cp /tmp/yacd.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable mihomo yacd
|
||||
sudo systemctl start mihomo yacd
|
||||
sudo systemctl status mihomo yacd
|
||||
```
|
||||
|
||||
### Key design choices
|
||||
- `Requires=mihomo.service` on yacd: yacd won't start if mihomo fails
|
||||
- `After=network-online.target`: wait for network before starting
|
||||
- `Restart=on-failure`: auto-restart on crash
|
||||
- `LimitNOFILE=65535`: mihomo handles many connections
|
||||
- Logs to `/tmp/` via `append:` (matches startup.sh convention)
|
||||
|
||||
### Migration from startup.sh
|
||||
Once systemd manages the services, **disable** the startup.sh service definitions to prevent conflicts:
|
||||
```bash
|
||||
# Disable (don't delete — keep as backup)
|
||||
mv ~/startup/services/mihomo.sh ~/startup/services/mihomo.sh.disabled
|
||||
mv ~/startup/services/yacd.sh ~/startup/services/yacd.sh.disabled # if applicable
|
||||
|
||||
# Verify startup.sh no longer manages these
|
||||
bash ~/startup/startup.sh status # should not show mihomo/yacd
|
||||
|
||||
# Use systemd exclusively
|
||||
systemctl --user start|stop|restart|status mihomo # user-level
|
||||
# or: sudo systemctl start|stop|restart|status mihomo # system-level
|
||||
```
|
||||
|
||||
**Why disable instead of delete**: If systemd service has issues, you can quickly re-enable by renaming back. Also keeps the service definition as documentation.
|
||||
|
||||
**Pitfall**: If you forget to disable startup.sh definitions AND the watchdog still calls `startup.sh restart`, the two managers fight over PIDs. Always migrate atomically: create systemd service → enable → disable startup.sh definition → update watchdog.
|
||||
|
||||
---
|
||||
|
||||
## Option B: User-Level Services (for non-root setups)
|
||||
|
||||
User services run as the logged-in user, no root needed for service files.
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.config/systemd/user
|
||||
cat > ~/.config/systemd/user/mihomo.service << 'EOF'
|
||||
[Unit]
|
||||
Description=Mihomo (Clash Meta) Proxy
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=%h/clash
|
||||
ExecStart=%h/clash/mihomo -d %h/clash/config
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
EOF
|
||||
|
||||
cat > ~/.config/systemd/user/yacd-dashboard.service << 'EOF'
|
||||
[Unit]
|
||||
Description=YACD Dashboard (static HTTP)
|
||||
After=network.target mihomo.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=%h/clash/yacd
|
||||
ExecStart=/usr/bin/python3 -m http.server 9091 --bind 0.0.0.0 --directory %h/clash/yacd
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
EOF
|
||||
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable mihomo yacd-dashboard
|
||||
systemctl --user start mihomo yacd-dashboard
|
||||
```
|
||||
|
||||
**Critical**: Without linger, user services die on logout:
|
||||
```bash
|
||||
sudo loginctl enable-linger $USER
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verify (both options)
|
||||
```bash
|
||||
# System-level
|
||||
sudo systemctl status mihomo yacd
|
||||
# User-level
|
||||
systemctl --user status mihomo yacd-dashboard
|
||||
|
||||
curl -s http://127.0.0.1:9090/version
|
||||
curl -s http://127.0.0.1:9091/ | head -3
|
||||
```
|
||||
@@ -0,0 +1,108 @@
|
||||
# Telegram Adapter Diagnostics
|
||||
|
||||
## Quick Health Check
|
||||
|
||||
```bash
|
||||
# 1. Is the adapter connected?
|
||||
journalctl --user -u hermes-gateway --since '10 min ago' --no-pager | grep -i "telegram" | tail -5
|
||||
|
||||
# 2. Is the proxy working?
|
||||
curl -s -o /dev/null -w "%{http_code}" --proxy http://127.0.0.1:7890 https://api.telegram.org
|
||||
|
||||
# 3. Is the bot alive?
|
||||
# (replace TOKEN with actual bot token from ~/.hermes/.env)
|
||||
curl -s "https://api.telegram.org/bot<TOKEN>/getMe" --proxy http://127.0.0.1:7890
|
||||
|
||||
# 4. Is the bot in the target group?
|
||||
curl -s "https://api.telegram.org/bot<TOKEN>/getChatMember?chat_id=<GROUP_ID>&user_id=<BOT_ID>" --proxy http://127.0.0.1:7890
|
||||
```
|
||||
|
||||
## Diagnosing "Bot Not Processing Group Messages"
|
||||
|
||||
### Step 1: Check if adapter is alive
|
||||
```bash
|
||||
journalctl --user -u hermes-gateway --no-pager --since '1 hour ago' | grep "\[Telegram\]" | tail -10
|
||||
```
|
||||
- If only crash/reconnect-failed logs → adapter is dead, restart gateway
|
||||
- If "Reconnecting telegram" entries → watcher is alive, proxy may be down
|
||||
- If no Telegram logs at all → either adapter connected silently or watcher died
|
||||
|
||||
### Step 2: Check if proxy is the issue
|
||||
```bash
|
||||
curl -s -o /dev/null -w "%{http_code}" --proxy http://127.0.0.1:7890 https://api.telegram.org
|
||||
# 302 = proxy working, timeout/ConnectError = proxy down
|
||||
```
|
||||
|
||||
### Step 3: Check bot group membership
|
||||
```bash
|
||||
# Get bot info
|
||||
python3 -c "
|
||||
import json, urllib.request
|
||||
with open('/home/openclaw/.hermes/.env') as f:
|
||||
token = [l.split('=',1)[1].strip() for l in f if l.startswith('TELEGRAM_BOT_TOKEN')][0]
|
||||
proxy = urllib.request.ProxyHandler({'https': 'http://127.0.0.1:7890'})
|
||||
opener = urllib.request.build_opener(proxy)
|
||||
resp = opener.open(f'https://api.telegram.org/bot{token}/getChatMember?chat_id=<GROUP_ID>&user_id=<BOT_ID>')
|
||||
print(json.loads(resp.read()))
|
||||
"
|
||||
```
|
||||
|
||||
### Step 4: Check config
|
||||
In `~/.hermes/config.yaml`, under `telegram:`:
|
||||
- `free_response_chats` or `free_response_channels` must include the group ID
|
||||
- `channel_prompts` can be set per group for custom behavior
|
||||
- `allowed_chats` when set acts as a whitelist (empty = no restriction)
|
||||
- `require_mention: true` means bot only responds when @mentioned (unless in `free_response_chats`)
|
||||
|
||||
**Editing channel_prompts**: Values are unicode-escaped with YAML line continuations (`\` at line end). Don't use `patch()` — it breaks on multi-line unicode. Use Python regex instead:
|
||||
```python
|
||||
import re
|
||||
with open('/home/openclaw/.hermes/config.yaml') as f:
|
||||
content = f.read()
|
||||
# Match across line breaks with [\s\\]* for YAML continuations
|
||||
pattern = re.compile(r'\\uXXXX[\s\\]*\\uYYYY')
|
||||
content = pattern.sub(r'\\uNEW1\\uNEW2', content)
|
||||
with open('/home/openclaw/.hermes/config.yaml', 'w') as f:
|
||||
f.write(content)
|
||||
# Always verify YAML validity after
|
||||
import yaml; yaml.safe_load(open('/home/openclaw/.hermes/config.yaml'))
|
||||
```
|
||||
The same prompt string is duplicated across `slack`, `discord`, `mattermost`, and `telegram` sections — replace in all 4.
|
||||
|
||||
### Step 5: Check forwarder
|
||||
```bash
|
||||
# Is the forwarder running?
|
||||
docker ps | grep forwarder
|
||||
|
||||
# Check forwarder config
|
||||
sqlite3 ~/TelegramForwarder/db/forward.db "SELECT * FROM chats;"
|
||||
sqlite3 ~/TelegramForwarder/db/forward.db "SELECT * FROM forward_rules;"
|
||||
sqlite3 ~/TelegramForwarder/db/forward.db "SELECT * FROM keywords;"
|
||||
|
||||
# Check recent forwarding activity
|
||||
docker logs telegram-forwarder --tail 20 | grep "已发送"
|
||||
```
|
||||
|
||||
## Gateway Restart Constraints
|
||||
|
||||
- **Cannot restart from inside the gateway**: `hermes gateway restart` is blocked when run from a gateway session (SIGTERM propagation prevention)
|
||||
- **`kill <PID>` also blocked**: Security scanner intercepts `kill` commands targeting the gateway PID. Terminal returns `Blocked: cannot restart or stop the gateway from inside the gateway process.`
|
||||
- **Workaround 1**: User runs `hermes gateway restart` from a separate SSH shell (recommended)
|
||||
- **Workaround 2**: `systemctl --user restart hermes-gateway` from a separate shell
|
||||
- **Workaround 3**: Write a small helper script to `/tmp/` and ask user to run it manually
|
||||
|
||||
## Reconnect Watcher Architecture
|
||||
|
||||
When a platform adapter hits a fatal error:
|
||||
1. `_set_fatal_error()` → sets `adapter._running = False`
|
||||
2. `_handle_adapter_fatal_error()` → disconnects adapter, removes from `self.adapters`
|
||||
3. If `fatal_error_retryable=True` → adds to `self._failed_platforms` with backoff
|
||||
4. `_platform_reconnect_watcher()` → background asyncio task, retries with exponential backoff (30s→300s cap)
|
||||
|
||||
**Known bug**: The watcher task can silently die. No upstream fix yet. Workaround: external gateway restart.
|
||||
|
||||
**Root cause (observed 2026-06-23)**: When multiple platforms crash simultaneously (Telegram + WeChat + QQBot), the watcher may die. Evidence: `journalctl` shows the fatal error at 09:27 but ZERO "Reconnecting telegram" or "Reconnect telegram error" entries afterward — the watcher's `while self._running` loop exited without logging. On a previous crash (2026-06-22) with only Telegram affected, the watcher worked correctly and kept retrying with backoff.
|
||||
|
||||
**Edge case — already-queued platform**: `_handle_adapter_fatal_error()` checks `adapter.platform not in self._failed_platforms` before queueing. If Telegram was already in the queue from a prior failure, the new fatal error doesn't re-queue or reset the backoff. The existing entry's `next_retry` timer may be at the 300s cap, causing long silent gaps between reconnection attempts even when the watcher IS alive.
|
||||
|
||||
**Definitive diagnosis**: If after a crash you see zero reconnection attempt logs for >10 minutes, the watcher is dead. Restart the gateway from an external shell.
|
||||
Reference in New Issue
Block a user