Initial commit: Hermes Agent skills collection
- Trading skills (OKX, dividend, lottery, quantitative) - Creative skills (ASCII art, diagrams, video) - Development skills (GitHub, debugging, TDD) - Research skills (arXiv, blog monitoring) - Productivity skills (email, documents, notes) - MCP integration skills - Custom user skills
This commit is contained in:
Hermes Skills Manager
@@ -0,0 +1,262 @@
|
||||
---
|
||||
name: clash-docker-workflow
|
||||
description: Deploy and manage a high-performance proxy (Clash/Mihomo) — standalone binary (preferred) or Docker. Includes Web UI dashboards and Hermes Telegram proxy integration.
|
||||
tags: [proxy, clash, mihomo, networking]
|
||||
---
|
||||
# Clash / Mihomo Proxy Workflow
|
||||
|
||||
Deploy and manage a high-performance proxy (Clash/Mihomo) for routing traffic through China's GFW.
|
||||
|
||||
Two deployment modes: **standalone binary** (preferred by user) or **Docker**.
|
||||
|
||||
---
|
||||
|
||||
## Mode 1: Standalone Binary (preferred)
|
||||
|
||||
### Install
|
||||
```bash
|
||||
# Download mihomo — use GitHub mirror in China (direct github.com times out)
|
||||
cd ~/clash
|
||||
curl -L "https://ghfast.top/https://github.com/MetaCubeX/mihomo/releases/download/v1.19.8/mihomo-linux-amd64-v1.19.8.gz" -o mihomo.gz
|
||||
gunzip -f mihomo.gz && chmod +x mihomo
|
||||
./mihomo -v # verify
|
||||
```
|
||||
|
||||
### Configure
|
||||
```bash
|
||||
# 1. Create config dir
|
||||
mkdir -p ~/clash/config
|
||||
|
||||
# 2. Download subscription — MUST set User-Agent or WAF blocks (403)
|
||||
curl -L -H "User-Agent: clash-verge/v2.0.0" \
|
||||
"SUBSCRIPTION_URL" -o ~/clash/config/config.yaml
|
||||
|
||||
# 3. Inject required params if missing
|
||||
# Raw subscriptions often omit these:
|
||||
cat >> ~/clash/config/config.yaml << 'EOF'
|
||||
mixed-port: 7890
|
||||
allow-lan: true
|
||||
mode: rule
|
||||
log-level: info
|
||||
external-controller: 0.0.0.0:9090
|
||||
EOF
|
||||
```
|
||||
|
||||
### Pre-download GeoIP data (critical for China servers)
|
||||
mihomo downloads GeoIP/Geosite DBs from GitHub on first start. If GitHub is blocked, it **hangs forever** at "Can't find MMDB, start download". Pre-download via mirror:
|
||||
```bash
|
||||
cd ~/clash/config
|
||||
curl -L "https://ghfast.top/https://github.com/MetaCubeX/meta-rules-dat/releases/download/latest/geoip.db" -o geoip.db
|
||||
curl -L "https://ghfast.top/https://github.com/MetaCubeX/meta-rules-dat/releases/download/latest/geoip.metadb" -o geoip.metadb
|
||||
curl -L "https://ghfast.top/https://github.com/MetaCubeX/meta-rules-dat/releases/download/latest/geosite.db" -o geosite.db
|
||||
```
|
||||
|
||||
### Run
|
||||
```bash
|
||||
# Foreground (test, Ctrl+C to stop)
|
||||
~/clash/mihomo -d ~/clash/config
|
||||
|
||||
# Background — use Hermes terminal background mode:
|
||||
# terminal(background=true, command="cd ~/clash && exec ./mihomo -d config 2>&1")
|
||||
# Then check: terminal(command="ss -tlnp | grep -E '7890|9090'")
|
||||
```
|
||||
|
||||
**⚠️ `terminal(background=true)` is NOT persistent.** The mihomo process dies when:
|
||||
- The gateway session ends or restarts
|
||||
- The Hermes turn that started it finishes and gets garbage-collected
|
||||
- The system reboots
|
||||
|
||||
For production use, **always use systemd** (see references/systemd-setup.md). Only use `terminal(background=true)` for quick tests.
|
||||
|
||||
### Verify
|
||||
```bash
|
||||
curl -x http://127.0.0.1:7890 https://api.telegram.org # should return HTML/JSON (not timeout)
|
||||
curl http://127.0.0.1:9090/version # {"meta":true,"version":"v1.19.8"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Mode 2: Docker
|
||||
|
||||
### docker-compose.yml
|
||||
```yaml
|
||||
services:
|
||||
clash:
|
||||
image: orzly/mihomo:latest
|
||||
container_name: clash
|
||||
restart: always
|
||||
volumes:
|
||||
- ./config:/root/.config/mihomo
|
||||
ports:
|
||||
- "7890:7890"
|
||||
- "7891:7891"
|
||||
- "9090:9090"
|
||||
cap_add:
|
||||
- NET_ADMIN
|
||||
devices:
|
||||
- /dev/net/tun:/dev/net/tun
|
||||
clash-ui:
|
||||
image: yaknet/yacd:latest
|
||||
container_name: clash-ui
|
||||
restart: always
|
||||
ports:
|
||||
- "3412:80"
|
||||
environment:
|
||||
- API_URL=http://clash:9090
|
||||
depends_on:
|
||||
- clash
|
||||
```
|
||||
|
||||
### Workflow
|
||||
1. `mkdir -p ~/clash/config`
|
||||
2. Place `docker-compose.yml` in `~/clash/`
|
||||
3. Download subscription to `~/clash/config/config.yaml`
|
||||
4. `cd ~/clash && docker compose up -d`
|
||||
5. UI at `http://localhost:3412`
|
||||
|
||||
---
|
||||
|
||||
## Web UI (Dashboard)
|
||||
|
||||
### Option A: Online dashboard (zero setup, LAN only)
|
||||
Use a hosted dashboard pointing to your mihomo API:
|
||||
- **YACD**: https://yacd.haishan.me/ → API: `http://<server-ip>:9090`
|
||||
- **metacubexd**: https://metacubex.github.io/metacubexd/ → API: `http://<server-ip>:9090`
|
||||
|
||||
**⚠️ Mixed content blocker**: Online dashboards are served over HTTPS. If your mihomo API is HTTP (typical for LAN), browsers block the fetch due to mixed-content policy. The dashboard loads but shows an empty/disconnected state. Fix: use Option B (local deployment).
|
||||
|
||||
### Option B: Local static deployment (recommended for LAN)
|
||||
Download yacd static files and serve locally over HTTP:
|
||||
```bash
|
||||
cd ~/clash
|
||||
curl -L "https://ghfast.top/https://github.com/haishanh/yacd/releases/download/v0.3.8/yacd.tar.xz" -o yacd.tar.xz
|
||||
mkdir -p yacd && tar xf yacd.tar.xz -C yacd --strip-components=1
|
||||
# Serve on port 9091 (avoids conflict with mihomo API on 9090)
|
||||
python3 -m http.server 9091 --bind 0.0.0.0 --directory ~/clash/yacd &
|
||||
```
|
||||
Access with API pre-filled: `http://<server-ip>:9091/?host=http://<server-ip>:9090`
|
||||
|
||||
For persistence, add the python HTTP server to the same systemd service as mihomo (see references/systemd-setup.md).
|
||||
|
||||
### Option C: Docker container (see Mode 2 above)
|
||||
YACD at `http://localhost:3412`.
|
||||
|
||||
---
|
||||
|
||||
### Connecting Applications
|
||||
|
||||
Set proxy env vars in `~/.bashrc` (Hermes terminal auto-sources it via `auto_source_bashrc: true`):
|
||||
```bash
|
||||
# Proxy for all HTTP/HTTPS traffic
|
||||
export HTTP_PROXY=http://127.0.0.1:7890
|
||||
export HTTPS_PROXY=http://127.0.0.1:7890
|
||||
export http_proxy=http://127.0.0.1:7890
|
||||
export https_proxy=http://127.0.0.1:7890
|
||||
export ALL_PROXY=socks5://127.0.0.1:7890
|
||||
|
||||
# NO_PROXY: local/internal services that must NOT go through proxy
|
||||
# Include: localhost, local model servers, WireGuard subnet, .local mDNS
|
||||
export NO_PROXY=localhost,127.0.0.1,192.168.199.100,10.8.0.0/24,*.local
|
||||
```
|
||||
|
||||
**Important**: Set BOTH uppercase AND lowercase variants (`HTTP_PROXY` + `http_proxy`). Some libraries only read one form.
|
||||
|
||||
### Routing Specific Services via Mihomo Rules
|
||||
|
||||
When you need certain API services to **always** go through the proxy (even if MATCH rule changes), add explicit `DOMAIN-SUFFIX` rules to the top of the `rules:` section in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
rules:
|
||||
# LongPort / LongBridge API
|
||||
- 'DOMAIN-SUFFIX,longportapp.com,BiXin Network'
|
||||
- 'DOMAIN-SUFFIX,longport.com,BiXin Network'
|
||||
- 'DOMAIN-SUFFIX,longbridge.com,BiXin Network'
|
||||
- 'DOMAIN-SUFFIX,longbridge.hk,BiXin Network'
|
||||
# OKX 交易所
|
||||
- 'DOMAIN-SUFFIX,okx.com,BiXin Network'
|
||||
- 'DOMAIN-SUFFIX,okx.me,BiXin Network'
|
||||
- 'DOMAIN-SUFFIX,okx.co,BiXin Network'
|
||||
- 'DOMAIN-SUFFIX,oklink.com,BiXin Network'
|
||||
- 'DOMAIN-SUFFIX,oklink.org,BiXin Network'
|
||||
# ... existing rules below ...
|
||||
```
|
||||
|
||||
Place service-specific rules **before** the large DIRECT rule block for Chinese services — Mihomo evaluates rules top-down, first match wins.
|
||||
|
||||
### Mihomo API Hot-Reload (no restart needed)
|
||||
|
||||
After editing `config.yaml`, reload without killing the process:
|
||||
```bash
|
||||
# Method 1: REST API reload
|
||||
curl -s -X PUT http://127.0.0.1:9090/configs \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"path":"/home/openclaw/clash/config/config.yaml"}'
|
||||
|
||||
# Method 2: SIGHUP (simpler, same effect)
|
||||
pkill -HUP -f mihomo
|
||||
|
||||
# Method 3: Full restart (only if hot-reload fails)
|
||||
bash ~/clash/startup.sh restart mihomo
|
||||
```
|
||||
|
||||
### Hermes Proxy Configuration
|
||||
|
||||
Hermes supports Telegram proxy via config.yaml:
|
||||
|
||||
```bash
|
||||
hermes config set telegram.proxy_url http://127.0.0.1:7890
|
||||
hermes gateway restart
|
||||
```
|
||||
|
||||
`load_gateway_config()` bridges `telegram.proxy_url` to `TELEGRAM_PROXY` env var at startup. Verified working.
|
||||
|
||||
For full detail see: `skill_view(name="clash-docker-workflow", file_path="references/hermes-telegram-proxy.md")`/...`. Other mirrors: `ghproxy.com`, `mirror.ghproxy.com`.
|
||||
- **Subscription URL returns 403 (WAF)**: Many subscription providers block requests without a browser-like User-Agent. Add `-H "User-Agent: clash-verge/v2.0.0"` to curl. This was the fix for BiXin (sub-route.bxy.re).
|
||||
- **mihomo hangs at "Can't find MMDB, start download"**: GeoIP/Geosite DBs need to be pre-downloaded when GitHub is blocked. See "Pre-download GeoIP data" section above. Without these files, mihomo starts but never binds ports.
|
||||
- **mihomo.gz corrupted on failed download**: If `file mihomo.gz` shows `ASCII text` instead of gzip, the download failed (GitHub returned 404/HTML). Re-download using mirror.
|
||||
- **Missing Control Params**: A raw subscription file often lacks `external-controller` or `mode`. Always ensure these are present in `config.yaml` for the API/UI to work.
|
||||
- **Container Communication (Docker only)**: The UI container uses `API_URL=http://clash:9090`. This relies on the service name `clash` in the Docker network.
|
||||
- **Subscription config format**: mihomo (MetaCubeX) uses a slightly different config format than original Clash. Ensure your subscription provider supports mihomo/clash-meta format, or convert.
|
||||
- **external-controller binding**: Default `127.0.0.1:9090` only allows local access. Use `0.0.0.0:9090` if you need remote Web UI access from another machine (e.g., Mac → Linux server).
|
||||
- **Background process output lost**: When running mihomo with `terminal(background=true)`, stdout/stderr go to pipes. Use a wrapper script with `exec ./mihomo -d config 2>&1` or redirect to a log file to capture output.
|
||||
- **Online dashboard shows empty/disconnected**: Browser blocks HTTP API requests from HTTPS dashboard pages (mixed-content policy). Deploy yacd locally instead — see Web UI Option B.
|
||||
- **`{"hello":"mihomo"}` when opening API URL**: This is the mihomo API root response, NOT the Web UI. Users sometimes confuse this. The Web UI is a separate static app (yacd) that connects to the API.
|
||||
- **Hot-reload preferred over restart**: Editing `config.yaml` then calling `PUT /configs` or `pkill -HUP` reloads rules without dropping active connections. Only do a full restart if hot-reload fails or the binary itself needs updating.
|
||||
- **NO_PROXY must include internal IPs**: If you route all traffic through proxy, local services (Ollama on 192.168.x.x, WireGuard 10.8.0.x, localhost) will break. Always set NO_PROXY with internal subnets. Also include `*.local` for mDNS.
|
||||
- **Lowercase proxy vars**: Some Python libraries (requests, httpx, aiohttp) read `http_proxy`/`https_proxy` (lowercase) while others read `HTTP_PROXY`/`HTTPS_PROXY` (uppercase). Set both to be safe.
|
||||
- **Mihomo rule order matters**: Rules are evaluated top-down, first match wins. Place specific service rules (e.g., `DOMAIN-SUFFIX,okx.com`) BEFORE broad DIRECT rules (e.g., `DOMAIN-SUFFIX,qq.com`). If a broad DIRECT rule matches first, your specific proxy rule is never reached.
|
||||
- **Proxy down = ALL platforms fail**: When `127.0.0.1:7890` is unreachable, Telegram, QQBot, and WeChat all fail simultaneously. Gateway log pattern: `Connect attempt N/8 failed: httpx.ConnectError: All connection attempts failed` + `Proxy detected; passing explicitly to HTTPXRequest`. Quick diagnosis: read `~/.hermes/logs/gateway.log` via `read_file()` (last 100 lines). For full diagnostic checklist: `skill_view(name="clash-docker-workflow", file_path="references/proxy-diagnostics.md")`.
|
||||
- **mihomo systemd service**: The binary runs as a foreground process. For production, create a systemd service to survive reboots. Full recipe: `skill_view(name="clash-docker-workflow", file_path="references/systemd-setup.md")`.
|
||||
- **Security scanner blocks sudo writes to /etc/systemd/**: Hermes's security layer intercepts `sudo tee`, `sudo cp`, and `write_file` targeting `/etc/systemd/system/`. Workaround: write `.service` files to `/tmp/` via `write_file()`, then instruct the user to manually `sudo cp` and `sudo systemctl`. Do not retry blocked commands — they will keep failing.
|
||||
- **startup.sh vs systemd conflict**: If both `startup.sh` and systemd manage the same services, they can fight over PIDs. After enabling systemd, stop using `startup.sh` for start/stop — use `sudo systemctl` exclusively.
|
||||
- **startup-service-manager skill**: For generic service management (not just Clash), see the `startup-service-manager` skill — same `~/startup/` directory, manages mihomo + yacd + any other service.
|
||||
- **Subscription update MUST NOT use proxy (circular dependency)**: The proxy depends on the subscription for node configs. If subscription updates go through the proxy, you get a dead-lock: proxy needs subscription → subscription update needs proxy. **Always use direct connection (no `-x` flag) for subscription downloads.** This may be slower or fail on some networks, but it avoids the circular dependency. If direct download also fails (e.g., the subscription provider is blocked by ISP), the only escape is fetching from an alternate network (mobile hotspot, VPN) and manually placing the config file.
|
||||
- **Duplicate mihomo processes**: If mihomo is started multiple times (e.g., both `startup.sh start` and a manual `./mihomo -d config`), duplicate PIDs compete for the same ports. Check with `ps aux | grep mihomo | grep -v grep` and kill extras. The `scripts/update-sub.sh` uses `flock` to prevent concurrent updates, but the mihomo binary itself has no singleton guard.
|
||||
- **`hermes gateway restart` fails when run from inside the gateway**: Hermes blocks self-restart from within a gateway session (SIGTERM propagation). Instruct the user to run it from a separate shell. The agent cannot do this via terminal(). Workaround: `systemctl --user restart hermes-gateway` from a separate shell.
|
||||
- **mihomo crash can permanently kill Telegram adapter (reconnect watcher bug)**: When mihomo goes down, the Telegram adapter hits 10 network retries → sets fatal error → gateway's `_platform_reconnect_watcher` is supposed to auto-reconnect with exponential backoff (30s→300s cap). BUT the watcher task can silently die (asyncio task exit with no log), leaving the adapter permanently offline even after mihomo recovers. Symptoms: (1) zero `[Telegram]` logs in journalctl after the crash, (2) no "Reconnecting telegram" or "Reconnect telegram error" entries, (3) DMs may still work (existing session) but group messages are lost. Diagnosis: `journalctl --user -u hermes-gateway --since '1 hour ago' | grep -i 'telegram' | grep -v Traceback`. If only crash/reconnect-failed logs exist with no subsequent "Reconnecting" attempts, the watcher is dead. Fix: restart gateway from external shell. This is a Hermes upstream bug — the watcher's `while self._running` loop exits silently under certain conditions. Until fixed upstream, consider a mihomo watchdog (see `references/mihomo-watchdog.md`) to prevent the initial crash.
|
||||
- **Dual defense for proxy — set BOTH config bridge AND systemd env vars**: `telegram.proxy_url` in config.yaml IS bridged to `TELEGRAM_PROXY` by `load_gateway_config()`. But if the bridge fails (manual start, config parse error), the gateway falls back to direct connection. Always also set `HTTPS_PROXY` in the systemd service file for belt-and-suspenders reliability. Full diagnostic checklist: `skill_view(name="clash-docker-workflow", file_path="references/proxy-diagnostics.md")`.
|
||||
- **channel_prompts are platform-agnostic in config.yaml**: The same group ID prompt configured under `telegram.channel_prompts`, `discord.channel_prompts`, and `slack.channel_prompts` is redundant — only configure it under platforms actually in use. Check which platforms are connected in `gateway_state.json` before auditing.
|
||||
- **Old gateway services linger**: `~/.config/systemd/user/` may contain services from previous tools (e.g. `openclaw-gateway.service`). If the service is no longer needed, disable it: `systemctl --user disable --now <name>`. Otherwise it wastes resources on boot.
|
||||
- **Proxy works but Telegram still times out**: Two fixes available — (1) `hermes config set telegram.proxy_url http://127.0.0.1:7890` then `hermes gateway restart` (simpler — `load_gateway_config()` bridges this to `TELEGRAM_PROXY` env var), OR (2) add `Environment="HTTPS_PROXY=http://127.0.0.1:7890"` to `~/.config/systemd/user/hermes-gateway.service`, then `systemctl --user daemon-reload && hermes gateway restart`. Symptom: curl through proxy passes, but gateway logs show connection failures. Full diagnostic checklist: `skill_view(name="clash-docker-workflow", file_path="references/proxy-diagnostics.md")`.
|
||||
|
||||
---
|
||||
|
||||
- **Mihomo watchdog to prevent cascading failures**: mihomo crashes can silently kill the Telegram adapter's reconnect watcher. A lightweight watchdog script monitors mihomo health and restarts it before downstream services notice. Full recipe: `skill_view(name="clash-docker-workflow", file_path="references/mihomo-watchdog.md")`.
|
||||
- **⚠️ CRITICAL: systemd oneshot cgroup kills mihomo on watchdog exit**: If the watchdog runs as a systemd user service (`Type=oneshot`) and calls `startup.sh restart mihomo`, the mihomo process becomes a child of the watchdog's cgroup. When the watchdog service exits, **systemd kills all processes in the cgroup — including the just-started mihomo**. This creates a death loop: watchdog starts mihomo → watchdog exits → systemd kills mihomo → next watchdog cycle finds mihomo dead → repeat every 2 minutes. Symptoms: mihomo log shows `Mihomo shutting down` 3-6 seconds after each startup, watchdog.log shows repeated restarts. **Fix**: (1) mihomo MUST run as its own dedicated systemd service (not via startup.sh inside the watchdog), (2) watchdog calls `systemctl --user restart mihomo`, (3) watchdog must NOT check proxy connectivity — proxy node issues ≠ mihomo issues, unnecessary restarts break stable connections. See `references/mihomo-watchdog.md`.
|
||||
- **Telegram adapter diagnostics**: When the bot stops processing group messages, use the diagnostic checklist in `references/telegram-adapter-diagnostics.md` — covers adapter health, proxy checks, bot membership, config verification, forwarder DB inspection, and the reconnect watcher architecture.
|
||||
|
||||
## Subscription URL Management\n\nHardcoding the subscription URL in multiple scripts creates a maintenance headache — change one place and the other drifts. Use a single-source-of-truth file:\n\n**Create `~/clash/sub-url.conf`:**\n```bash\n# Clash 订阅地址 — 改这一行，所有脚本自动读取\nSUB_URL=\"https://sub-route.bxy.re/api/v1/client/subscribe?token=YOUR_TOKEN_HERE\"\n```\n\n**Have all scripts source it:**\n```bash\n# In startup.sh, update-sub.sh, or any other script:\nif [ -f \"$CLASH_DIR/sub-url.conf\" ]; then\n source \"$CLASH_DIR/sub-url.conf\"\nfi\n# Now $SUB_URL is available everywhere\n```\n\nThe `scripts/update-sub.sh` in this skill can also accept the URL as a CLI argument (`bash update-sub.sh \"URL\"`), which overrides the default for one-off tests.\n\n---\n\n## Subscription Auto-Update (Cron)
|
||||
|
||||
Proxy nodes rotate periodically. Set up a recurring cron job to keep the config fresh:
|
||||
|
||||
1. **Direct-connection update script**: `scripts/update-sub.sh` in this skill — downloads the subscription **directly (no proxy)**, injects required params (`external-controller`, `mixed-port`, etc.), and hot-reloads mihomo via `PUT /configs`. No restart needed unless hot-reload fails.
|
||||
|
||||
**Why no proxy?** The proxy itself depends on the subscription for node configs. If the update script routes through the proxy, you get a circular dependency: proxy needs subscription → update needs proxy → deadlock. Always download subscriptions via direct connection.
|
||||
|
||||
2. **Cron job**: Run every 30 minutes. Only notify on failure:
|
||||
```bash
|
||||
# Hermes cronjob approach:
|
||||
cronjob(action='create', schedule='30m', repeat=forever, name='Clash 订阅更新',
|
||||
prompt='bash ~/clash/update-sub.sh. Report only on failure.',
|
||||
enabled_toolsets=['terminal', 'file'])
|
||||
```
|
||||
@@ -0,0 +1,107 @@
|
||||
# Hermes Telegram Proxy Configuration
|
||||
|
||||
How to route Hermes Agent's Telegram gateway through a local proxy (Clash/Mihomo).
|
||||
|
||||
## Why needed
|
||||
Telegram API (`api.telegram.org`) is blocked in China. The Hermes gateway needs a proxy to connect.
|
||||
|
||||
## How the Telegram Adapter Reads Proxy
|
||||
|
||||
The Telegram adapter reads proxy from environment variables at the network layer.
|
||||
|
||||
Source code:
|
||||
- `gateway/platforms/telegram_network.py` → `_resolve_proxy_url()` → calls `resolve_proxy_url("TELEGRAM_PROXY", ...)` from `gateway/platforms/base.py`
|
||||
- `gateway/platforms/telegram.py` line 2068 → same `resolve_proxy_url()` call
|
||||
|
||||
`resolve_proxy_url()` checks (in order):
|
||||
1. `TELEGRAM_PROXY` env var (Telegram-specific, highest priority)
|
||||
2. `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` env vars (and lowercase variants)
|
||||
3. macOS system proxy auto-detection
|
||||
|
||||
**Key: `telegram.proxy_url` in config.yaml IS bridged to `TELEGRAM_PROXY` env var** by `load_gateway_config()` at startup. So both config.yaml and direct env vars work.
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### Option A — `telegram.proxy_url` in config.yaml (simpler, recommended)
|
||||
|
||||
```bash
|
||||
hermes config set telegram.proxy_url http://127.0.0.1:7890
|
||||
hermes gateway restart
|
||||
```
|
||||
|
||||
Verified working — `load_gateway_config()` reads `telegram.proxy_url` and sets `TELEGRAM_PROXY` env var.
|
||||
|
||||
### Option B — System env vars in systemd service file
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
import pathlib
|
||||
p = pathlib.Path.home() / '.config/systemd/user/hermes-gateway.service'
|
||||
lines = p.read_text().splitlines()
|
||||
# Remove any existing proxy lines first
|
||||
lines = [l for l in lines if 'HTTPS_PROXY' not in l and 'HTTP_PROXY' not in l]
|
||||
out = []
|
||||
for l in lines:
|
||||
out.append(l)
|
||||
# Insert after HERMES_HOME line
|
||||
if 'HERMES_HOME' in l and l.strip().startswith('Environment'):
|
||||
out.append('Environment=\"HTTPS_PROXY=http://127.0.0.1:7890\"')
|
||||
out.append('Environment=\"HTTP_PROXY=http://127.0.0.1:7890\"')
|
||||
p.write_text('\n'.join(out) + '\n')
|
||||
"
|
||||
|
||||
systemctl --user daemon-reload
|
||||
hermes gateway restart
|
||||
```
|
||||
|
||||
Or use the quick-fix script: `bash ~/clash/fix-proxy.sh` → option 6 → option 1.
|
||||
|
||||
### Option C — Dual defense (recommended)
|
||||
|
||||
Set BOTH config.yaml bridge AND systemd env vars. This ensures proxy works whether the config bridge runs or not:
|
||||
|
||||
```bash
|
||||
# 1. Config bridge
|
||||
hermes config set telegram.proxy_url http://127.0.0.1:7890
|
||||
|
||||
# 2. Systemd env vars (see Option B above)
|
||||
|
||||
# 3. Restart
|
||||
hermes gateway restart
|
||||
```
|
||||
|
||||
## Applying changes
|
||||
```bash
|
||||
hermes gateway restart
|
||||
# ⚠️ Cannot restart from inside the gateway — instruct user to run from separate shell
|
||||
```
|
||||
|
||||
## Verification
|
||||
```bash
|
||||
# Check config
|
||||
grep proxy_url ~/.hermes/config.yaml
|
||||
|
||||
# Check process env vars
|
||||
PID=$(pgrep -f "hermes_cli.main gateway" | head -1)
|
||||
tr '\0' '\n' < /proc/$PID/environ | grep -i proxy
|
||||
# Expected: TELEGRAM_PROXY=http://127.0.0.1:7890 (from config bridge)
|
||||
# HTTPS_PROXY=http://127.0.0.1:7890 (from systemd, if set)
|
||||
|
||||
# Check logs
|
||||
journalctl --user -u hermes-gateway -n 50 | grep -i telegram
|
||||
# Good: "connected" state
|
||||
# Bad: "connection failed" / "Timed out"
|
||||
|
||||
# Check gateway state
|
||||
cat ~/.hermes/gateway_state.json | python3 -m json.tool | grep -A3 telegram
|
||||
```
|
||||
|
||||
## Supported proxy types
|
||||
- HTTP proxy: `http://127.0.0.1:7890`
|
||||
- SOCKS5 proxy: `socks5://127.0.0.1:1080`
|
||||
|
||||
## Notes
|
||||
- The proxy applies to both the main Telegram connection and the fallback IP transport.
|
||||
- Mihomo/Clash mixed-port (7890) supports both HTTP and SOCKS5 on the same port.
|
||||
- The bot token is configured separately under `telegram.token` in config.yaml.
|
||||
- For full diagnostic checklist: `skill_view(name="clash-docker-workflow", file_path="references/proxy-diagnostics.md")`
|
||||
@@ -0,0 +1,229 @@
|
||||
# Mihomo Watchdog — Prevent Proxy Crashes from Killing Downstream Services
|
||||
|
||||
## Problem
|
||||
|
||||
mihomo periodically crashes or becomes unresponsive. When this happens, Hermes's Telegram adapter loses connectivity, hits 10 retries, sets a fatal error, and the `_platform_reconnect_watcher` may silently die — leaving Telegram permanently offline even after mihomo recovers.
|
||||
|
||||
## Solution: Watchdog Script
|
||||
|
||||
A lightweight watchdog that monitors mihomo's health and restarts it before downstream services notice.
|
||||
|
||||
### ⚠️ CRITICAL BUG: systemd cgroup kills mihomo on watchdog exit
|
||||
|
||||
**Root cause**: When the watchdog runs as a systemd user service (`Type=oneshot`) and calls `startup.sh restart mihomo`, the mihomo process becomes a child of the watchdog's service cgroup. When the watchdog service exits, systemd kills all processes in the cgroup — **including the just-started mihomo**.
|
||||
|
||||
**Death loop**: watchdog starts mihomo → watchdog exits → systemd kills mihomo → next watchdog cycle finds mihomo dead → repeat every 2 minutes.
|
||||
|
||||
**Symptoms**:
|
||||
- mihomo log shows `Mihomo shutting down` 3-6 seconds after each startup
|
||||
- watchdog.log shows repeated "mihomo 进程不存在，正在重启" entries
|
||||
- mihomo PID file is stale (process already dead)
|
||||
- Ports 7890/9090 never stay up
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check if watchdog is restarting in a loop
|
||||
tail -20 ~/startup/logs/watchdog.log
|
||||
# Check mihomo uptime pattern (should be minutes/hours, not seconds)
|
||||
grep "Mihomo shutting down" ~/startup/logs/mihomo.log | tail -10
|
||||
# Check if watchdog timer is active
|
||||
systemctl --user status mihomo-watchdog.timer
|
||||
```
|
||||
|
||||
**Fix — Option A (recommended): mihomo as dedicated systemd service**
|
||||
|
||||
mihomo MUST have its own systemd service so it runs in its own cgroup, independent of the watchdog. Full recipe: `skill_view(name="clash-docker-workflow", file_path="references/systemd-setup.md")`.
|
||||
|
||||
Then update the watchdog to call `systemctl --user restart mihomo` instead of `startup.sh restart mihomo`:
|
||||
**⚠️ Do NOT check proxy connectivity in the watchdog.** Proxy node issues (upstream provider downtime, node rotation) are NOT mihomo's fault — restarting mihomo won't fix them. Only check process existence + API responsiveness. Proxy connectivity checks cause unnecessary restarts that break stable connections (user lesson: 2026-06-23).
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# mihomo watchdog — uses systemctl, NO proxy connectivity check
|
||||
HEALTH_URL="http://127.0.0.1:9090/version"
|
||||
LOG="/home/openclaw/startup/logs/watchdog.log"
|
||||
|
||||
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG"; }
|
||||
|
||||
# 1. Process alive?
|
||||
if ! pgrep -f "mihomo -d" > /dev/null 2>&1; then
|
||||
log "mihomo 进程不存在，正在重启..."
|
||||
systemctl --user restart mihomo >> "$LOG" 2>&1; sleep 3
|
||||
fi
|
||||
|
||||
# 2. API responsive?
|
||||
if ! curl -s --max-time 3 "$HEALTH_URL" > /dev/null 2>&1; then
|
||||
log "mihomo API 无响应，正在重启..."
|
||||
systemctl --user restart mihomo >> "$LOG" 2>&1
|
||||
fi
|
||||
```
|
||||
|
||||
**Fix — Option B: Use `systemd-run` to escape cgroup** (if mihomo can't be a service)
|
||||
|
||||
```bash
|
||||
# In the watchdog script, replace:
|
||||
# $STARTUP restart mihomo
|
||||
# With:
|
||||
systemd-run --user --scope --slice=mihomo.slice bash -c "cd ~/clash && exec ./mihomo -d config"
|
||||
```
|
||||
|
||||
**NEVER do this**: Run `startup.sh restart mihomo` from inside a systemd oneshot service. The spawned process inherits the service's cgroup and gets killed on exit.
|
||||
|
||||
---
|
||||
|
||||
## Deployed Script: `~/startup/mihomo-watchdog.sh`
|
||||
|
||||
Uses `systemctl --user restart mihomo` (requires mihomo as a dedicated systemd service). **No proxy connectivity check** — proxy node issues ≠ mihomo issues.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# mihomo watchdog — 每 2 分钟检查一次，挂了自动拉起
|
||||
# ⚠️ 必须配合 mihomo 专用 systemd service 使用（不能用 startup.sh）
|
||||
HEALTH_URL="http://127.0.0.1:9090/version"
|
||||
LOG="/home/openclaw/startup/logs/watchdog.log"
|
||||
|
||||
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG"; }
|
||||
|
||||
# 1. Process alive?
|
||||
if ! pgrep -f "mihomo -d" > /dev/null 2>&1; then
|
||||
log "mihomo 进程不存在，正在重启..."
|
||||
systemctl --user restart mihomo >> "$LOG" 2>&1; sleep 3
|
||||
fi
|
||||
|
||||
# 2. API responsive?
|
||||
if ! curl -s --max-time 3 "$HEALTH_URL" > /dev/null 2>&1; then
|
||||
log "mihomo API 无响应，正在重启..."
|
||||
systemctl --user restart mihomo >> "$LOG" 2>&1
|
||||
fi
|
||||
```
|
||||
|
||||
**systemd timer** (runs every 2 minutes, 30s after boot):
|
||||
```ini
|
||||
# ~/.config/systemd/user/mihomo-watchdog.timer
|
||||
[Timer]
|
||||
OnBootSec=30s
|
||||
OnUnitActiveSec=2min
|
||||
AccuracySec=10s
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
```ini
|
||||
# ~/.config/systemd/user/mihomo-watchdog.service
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/home/openclaw/startup/mihomo-watchdog.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now mihomo-watchdog.timer
|
||||
```
|
||||
|
||||
Silent when healthy (no log output). Only writes to `watchdog.log` on failure/restart.
|
||||
|
||||
### Alternative Script: `~/clash/watchdog.sh` (standalone)
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Mihomo watchdog — checks proxy health every 60s, restarts if dead
|
||||
# Run via systemd timer or cron
|
||||
|
||||
MIHOMO_API="http://127.0.0.1:9090"
|
||||
MIHOMO_BIN="$HOME/clash/mihomo"
|
||||
MIHOMO_DIR="$HOME/clash/config"
|
||||
LOG="$HOME/clash/watchdog.log"
|
||||
MAX_LOG_SIZE=1048576 # 1MB
|
||||
|
||||
# Rotate log if too big
|
||||
if [ -f "$LOG" ] && [ "$(stat -c%s "$LOG" 2>/dev/null || echo 0)" -gt "$MAX_LOG_SIZE" ]; then
|
||||
mv "$LOG" "${LOG}.old"
|
||||
fi
|
||||
|
||||
check_health() {
|
||||
# Method 1: API health check
|
||||
if curl -s --max-time 5 "$MIHOMO_API/version" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
# Method 2: Check if process is alive
|
||||
if pgrep -f "mihomo -d" >/dev/null 2>&1; then
|
||||
# Process exists but API unresponsive — stuck
|
||||
return 1
|
||||
fi
|
||||
# Process dead
|
||||
return 1
|
||||
}
|
||||
|
||||
restart_mihomo() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo unhealthy, restarting..." >> "$LOG"
|
||||
# Kill stale processes
|
||||
pkill -f "mihomo -d" 2>/dev/null
|
||||
sleep 2
|
||||
# Start fresh
|
||||
cd "$MIHOMO_DIR" && exec "$MIHOMO_BIN" -d "$MIHOMO_DIR" &
|
||||
sleep 5
|
||||
if curl -s --max-time 5 "$MIHOMO_API/version" >/dev/null 2>&1; then
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo restarted successfully" >> "$LOG"
|
||||
else
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] WATCHDOG: mihomo restart FAILED" >> "$LOG"
|
||||
fi
|
||||
}
|
||||
|
||||
if ! check_health; then
|
||||
restart_mihomo
|
||||
fi
|
||||
```
|
||||
|
||||
### Deployment Options
|
||||
|
||||
**Option A: Cron (simplest)**
|
||||
```bash
|
||||
# Every 2 minutes
|
||||
*/2 * * * * bash ~/clash/watchdog.sh
|
||||
```
|
||||
|
||||
**Option B: Hermes cronjob (delivers alert on failure)**
|
||||
```python
|
||||
cronjob(action='create', schedule='2m', name='mihomo-watchdog',
|
||||
prompt='Run: bash ~/clash/watchdog.sh. Only report if restart was needed.',
|
||||
deliver='local', enabled_toolsets=['terminal'])
|
||||
```
|
||||
|
||||
**Option C: systemd timer (most robust)**
|
||||
```ini
|
||||
# /etc/systemd/system/mihomo-watchdog.timer
|
||||
[Unit]
|
||||
Description=Mihomo watchdog timer
|
||||
|
||||
[Timer]
|
||||
OnBootSec=30
|
||||
OnUnitActiveSec=120
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/mihomo-watchdog.service
|
||||
[Unit]
|
||||
Description=Mihomo watchdog
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/home/openclaw/clash/watchdog.sh
|
||||
User=openclaw
|
||||
```
|
||||
|
||||
## Why Not Just Restart the Gateway?
|
||||
|
||||
Restarting the gateway:
|
||||
- Kills all active agent sessions
|
||||
- Interrupts running cron jobs
|
||||
- Takes 10-20 seconds to fully initialize
|
||||
- Doesn't fix the underlying mihomo instability
|
||||
|
||||
A mihomo watchdog:
|
||||
- Restarts only the proxy (1-2 seconds)
|
||||
- No impact on gateway sessions or cron jobs
|
||||
- Prevents the cascade before it starts
|
||||
- Is the correct layer to fix (proxy stability, not gateway resilience)
|
||||
@@ -0,0 +1,188 @@
|
||||
# Proxy Diagnostics: "Is the proxy working?"
|
||||
|
||||
Systematic checklist when the user asks whether the proxy is healthy, or when Hermes services show connection timeouts despite the proxy running.
|
||||
|
||||
## Key Insight
|
||||
**Proxy works ≠ Hermes is using it.** These are independent checks. A curl test passing through the proxy proves the proxy is healthy, but Hermes may not be routing through it.
|
||||
|
||||
## Critical: Telegram Proxy Source
|
||||
|
||||
The Telegram adapter reads proxy from environment variables at the network layer.
|
||||
|
||||
Source code path:
|
||||
- `gateway/platforms/telegram_network.py` → `_resolve_proxy_url()` → `resolve_proxy_url("TELEGRAM_PROXY", ...)` from `gateway/platforms/base.py`
|
||||
- `gateway/platforms/telegram.py` line 2068 → same `resolve_proxy_url()` call
|
||||
|
||||
`resolve_proxy_url()` checks (in order):
|
||||
1. `TELEGRAM_PROXY` env var (Telegram-specific, highest priority)
|
||||
2. `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` env vars (and lowercase variants)
|
||||
3. macOS system proxy auto-detection
|
||||
|
||||
**Key: `telegram.proxy_url` in config.yaml IS bridged to `TELEGRAM_PROXY` env var** by `load_gateway_config()` at startup. So both config.yaml and direct env vars work. However, if `load_gateway_config()` fails to run (e.g. gateway started manually without the config bridge), only direct env vars work.
|
||||
|
||||
**Dual defense recommended**: set BOTH `telegram.proxy_url` in config.yaml AND `HTTPS_PROXY` in the systemd service file. This ensures proxy works whether the config bridge runs or not.
|
||||
|
||||
## Quick Diagnosis: Read Gateway Logs Directly
|
||||
|
||||
When `terminal()` is blocked (consent mechanism, security scanner, etc.), diagnose via log files:
|
||||
|
||||
```python
|
||||
# 1. Find log files
|
||||
search_files(pattern="*.log", path="~/.hermes/logs", target="files")
|
||||
|
||||
# 2. Read recent gateway log tail
|
||||
read_file(path="~/.hermes/logs/gateway.log", offset=-100) # last 100 lines
|
||||
|
||||
# 3. Search for Telegram errors
|
||||
search_files(pattern="telegram.*failed|Connect attempt|All connection",
|
||||
path="~/.hermes/logs/gateway.log", target="content")
|
||||
```
|
||||
|
||||
**Key log patterns to look for:**
|
||||
|
||||
| Log Pattern | Meaning |
|
||||
|---|---|
|
||||
| `Proxy detected; passing explicitly to HTTPXRequest: http://127.0.0.1:7890` | Proxy URL is configured in Hermes ✓ |
|
||||
| `Connect attempt N/8 failed: httpx.ConnectError: All connection attempts failed` | Proxy port unreachable (proxy down or wrong port) |
|
||||
| `connect timed out after 30s` | All retry attempts exhausted, will retry in N seconds |
|
||||
| `Reconnect telegram error: ... next retry in 300s` | Exponential backoff — retries increase: 60→120→240→300s |
|
||||
| `Primary api.telegram.org connection failed (); trying fallback IPs` | DNS resolution worked but TCP connection failed |
|
||||
| `DoH discovery yielded no usable IPs` | DNS-over-HTTPS failed, using hardcoded fallback IPs |
|
||||
|
||||
**Common failure: proxy itself is DOWN** (simplest case):
|
||||
- Log shows: `Proxy detected` + `All connection attempts failed` on every attempt
|
||||
- This means Hermes knows about the proxy (config bridge works), but the proxy port is not responding
|
||||
- Fix: restart mihomo (`systemctl --user restart mihomo` or `bash ~/clash/startup.sh restart mihomo`)
|
||||
|
||||
**This is different from**: proxy running but Hermes not routing through it (see "Common Failure Pattern" below).
|
||||
|
||||
## Diagnostic Steps (in order)
|
||||
|
||||
### 1. Proxy process alive?
|
||||
```bash
|
||||
ss -tlnp | grep 7890 # mihomo should be LISTEN
|
||||
# Expected: users:(("mihomo",pid=NNNN,fd=3))
|
||||
```
|
||||
|
||||
### 2. Proxy actually works? (curl test)
|
||||
```bash
|
||||
curl -x http://127.0.0.1:7890 -I https://www.google.com --connect-timeout 5
|
||||
curl -x http://127.0.0.1:7890 -I https://api.telegram.org --connect-timeout 5
|
||||
```
|
||||
Both should return HTTP 200/302, not timeout.
|
||||
|
||||
### 3. Gateway process has proxy env vars?
|
||||
```bash
|
||||
PID=$(pgrep -f "hermes_cli.main gateway" | head -1)
|
||||
tr '\0' '\n' < /proc/$PID/environ | grep -i proxy
|
||||
```
|
||||
If empty — the gateway process has NO proxy env vars and relies solely on the config bridge (`telegram.proxy_url` → `TELEGRAM_PROXY`). If the bridge fails, Telegram connects DIRECT.
|
||||
|
||||
### 4. Check systemd service file
|
||||
```bash
|
||||
grep -i proxy ~/.config/systemd/user/hermes-gateway.service
|
||||
```
|
||||
If no `HTTPS_PROXY` / `HTTP_PROXY` lines — the service file needs updating for dual defense.
|
||||
|
||||
### 5. Check config.yaml bridge
|
||||
```bash
|
||||
grep proxy_url ~/.hermes/config.yaml
|
||||
```
|
||||
Should show `proxy_url: http://127.0.0.1:7890` under `telegram:`.
|
||||
|
||||
### 6. Gateway status + logs
|
||||
```bash
|
||||
hermes gateway status
|
||||
journalctl --user -u hermes-gateway -n 50 | grep -i telegram
|
||||
```
|
||||
Look for: "Proxy detected" (good) or "connection failed" / "Timed out" (bad).
|
||||
|
||||
### 7. Run hermes doctor
|
||||
```bash
|
||||
hermes doctor
|
||||
```
|
||||
Shows API connectivity, missing configs, auth issues.
|
||||
|
||||
## Common Failure Pattern
|
||||
|
||||
**Symptom**: Proxy is running, curl tests pass, but gateway shows:
|
||||
```
|
||||
[Telegram] Primary api.telegram.org connection failed
|
||||
[Telegram] Fallback IP 149.154.167.220 failed
|
||||
[Telegram] Connect attempt N/8 failed: Timed out
|
||||
```
|
||||
|
||||
**Root cause**: Gateway systemd service file has no proxy environment variables AND the config bridge failed (or `telegram.proxy_url` not set). The Telegram adapter falls back to direct connection, which fails behind GFW.
|
||||
|
||||
**Fix** (dual defense — set both):
|
||||
```bash
|
||||
# 1. Set config bridge
|
||||
hermes config set telegram.proxy_url http://127.0.0.1:7890
|
||||
|
||||
# 2. Add proxy env vars to systemd service
|
||||
python3 -c "
|
||||
import pathlib
|
||||
p = pathlib.Path.home() / '.config/systemd/user/hermes-gateway.service'
|
||||
lines = p.read_text().splitlines()
|
||||
lines = [l for l in lines if 'HTTPS_PROXY' not in l and 'HTTP_PROXY' not in l]
|
||||
out = []
|
||||
for l in lines:
|
||||
out.append(l)
|
||||
if 'HERMES_HOME' in l and l.strip().startswith('Environment'):
|
||||
out.append('Environment=\"HTTPS_PROXY=http://127.0.0.1:7890\"')
|
||||
out.append('Environment=\"HTTP_PROXY=http://127.0.0.1:7890\"')
|
||||
p.write_text('\n'.join(out) + '\n')
|
||||
"
|
||||
systemctl --user daemon-reload
|
||||
hermes gateway restart
|
||||
```
|
||||
|
||||
**Quick fix**: `bash ~/clash/fix-proxy.sh --fix`
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
PID=$(pgrep -f "hermes_cli.main gateway" | head -1)
|
||||
tr '\0' '\n' < /proc/$PID/environ | grep -i proxy
|
||||
# Expected: TELEGRAM_PROXY=http://127.0.0.1:7890 (from config bridge)
|
||||
# HTTPS_PROXY=http://127.0.0.1:7890 (from systemd service)
|
||||
```
|
||||
|
||||
## Network/Proxy Audit Checklist
|
||||
|
||||
When the user asks "check my network/proxy config" or "any redundant configs?", cover these dimensions:
|
||||
|
||||
1. **代理进程**: mihomo running? Port 7890 listening?
|
||||
2. **代理连通性**: curl through proxy works? (Google, Telegram)
|
||||
3. **Gateway 代理模式**: Both config.yaml `proxy_url` AND systemd `HTTPS_PROXY` set? (dual defense)
|
||||
4. **旧服务残留**: Any old gateway services (e.g. `openclaw-gateway.service`) still in `~/.config/systemd/user/`? Disable with `systemctl --user disable --now <name>`
|
||||
5. **重复启动脚本**: Both `startup.sh` and systemd managing the same service? Pick one — systemd preferred. The other is dead code.
|
||||
6. **WireGuard 残留**: Check `/etc/wireguard/` for leftover `.conf` files. WireGuard can intercept ALL traffic if misconfigured.
|
||||
7. **环境变量代理**: Check `~/.bashrc` for `HTTP_PROXY`/`HTTPS_PROXY` — needed for terminal sessions but NOT sufficient for systemd services (they don't source .bashrc).
|
||||
8. **channel_prompts 重复**: Same prompt in slack/discord/telegram sections of config.yaml — only needed in platforms actually used.
|
||||
|
||||
## Quick Status Summary Template
|
||||
When reporting proxy status to the user, cover these four dimensions:
|
||||
1. **代理进程**: mihomo running? Port 7890 listening?
|
||||
2. **代理连通性**: curl through proxy works? (Google, Telegram)
|
||||
3. **Gateway 代理模式**: systemd service has proxy env vars? Config bridge set? Gateway process has them?
|
||||
4. **网关状态**: Gateway running? Telegram connected? No timeout errors in logs?
|
||||
|
||||
- **Gateway log path**: `~/.hermes/logs/gateway.log`, NOT `~/.hermes/gateway.log`. Also check `~/.hermes/logs/errors.log` for historical errors. Use `search_files(pattern="*.log", path="~/.hermes/logs", target="files")` to discover all logs.
|
||||
- **Telegram reconnect backoff**: When Telegram fails to connect, the gateway retries with exponential backoff: 60s → 120s → 240s → 300s (cap). After fixing the proxy, you may need to `hermes gateway restart` to reset the backoff instead of waiting for the next retry window.
|
||||
- **QQBot also fails when proxy is down**: The QQBot adapter routes through the same proxy. If `127.0.0.1:7890` is down, both Telegram AND QQ will show connection failures. Check QQBot logs for `Failed to get QQ Bot access token: All connection attempts failed`.
|
||||
|
||||
## Quick-Fix Script
|
||||
|
||||
`~/clash/fix-proxy.sh` — interactive menu with 8 functions:
|
||||
1. 📊 查看状态 (full status overview)
|
||||
2. 🔧 一键诊断修复 (auto-diagnose and fix)
|
||||
3. 🔄 重启 Mihomo
|
||||
4. 📡 更新/更换订阅
|
||||
5. 🔀 切换代理节点 (via Mihomo API)
|
||||
6. 🌐 Gateway 代理/直连切换
|
||||
7. 🔁 重启 Hermes Gateway
|
||||
8. 📋 查看日志
|
||||
|
||||
Usage:
|
||||
- Interactive: `bash ~/clash/fix-proxy.sh`
|
||||
- One-click fix: `bash ~/clash/fix-proxy.sh --fix`
|
||||
@@ -0,0 +1,153 @@
|
||||
# Systemd Service for Mihomo + YACD
|
||||
|
||||
Two approaches: **system-level** (recommended for always-on servers) and **user-level**.
|
||||
|
||||
---
|
||||
|
||||
## Option A: System-Level Services (recommended for servers)
|
||||
|
||||
System-level services run as root, survive logout without extra config, and start early in boot.
|
||||
|
||||
### Create service files
|
||||
|
||||
**Pitfall**: The Hermes security scanner blocks `sudo tee /etc/systemd/system/` and `sudo cp` into system paths. Workaround: write to `/tmp` first using `write_file`, then ask the user to manually run `sudo cp` + `sudo systemctl`.
|
||||
|
||||
```bash
|
||||
# Step 1: Agent writes to /tmp (allowed by write_file tool)
|
||||
# /tmp/mihomo.service:
|
||||
[Unit]
|
||||
Description=Mihomo (Clash Meta) Proxy
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=/home/openclaw/clash
|
||||
ExecStart=/home/openclaw/clash/mihomo -d /home/openclaw/clash/config
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
LimitNOFILE=65535
|
||||
StandardOutput=append:/tmp/mihomo.log
|
||||
StandardError=append:/tmp/mihomo.log
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
# /tmp/yacd.service:
|
||||
[Unit]
|
||||
Description=YACD - Clash Dashboard
|
||||
After=network-online.target mihomo.service
|
||||
Wants=network-online.target
|
||||
Requires=mihomo.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=/home/openclaw/clash/yacd
|
||||
ExecStart=/usr/bin/python3 -m http.server 9091 --bind 0.0.0.0
|
||||
Restart=on-failure
|
||||
RestartSec=3
|
||||
StandardOutput=append:/tmp/yacd.log
|
||||
StandardError=append:/tmp/yacd.log
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
### Step 2: User manually runs
|
||||
```bash
|
||||
sudo cp /tmp/mihomo.service /etc/systemd/system/
|
||||
sudo cp /tmp/yacd.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable mihomo yacd
|
||||
sudo systemctl start mihomo yacd
|
||||
sudo systemctl status mihomo yacd
|
||||
```
|
||||
|
||||
### Key design choices
|
||||
- `Requires=mihomo.service` on yacd: yacd won't start if mihomo fails
|
||||
- `After=network-online.target`: wait for network before starting
|
||||
- `Restart=on-failure`: auto-restart on crash
|
||||
- `LimitNOFILE=65535`: mihomo handles many connections
|
||||
- Logs to `/tmp/` via `append:` (matches startup.sh convention)
|
||||
|
||||
### Migration from startup.sh
|
||||
Once systemd manages the services, **disable** the startup.sh service definitions to prevent conflicts:
|
||||
```bash
|
||||
# Disable (don't delete — keep as backup)
|
||||
mv ~/startup/services/mihomo.sh ~/startup/services/mihomo.sh.disabled
|
||||
mv ~/startup/services/yacd.sh ~/startup/services/yacd.sh.disabled # if applicable
|
||||
|
||||
# Verify startup.sh no longer manages these
|
||||
bash ~/startup/startup.sh status # should not show mihomo/yacd
|
||||
|
||||
# Use systemd exclusively
|
||||
systemctl --user start|stop|restart|status mihomo # user-level
|
||||
# or: sudo systemctl start|stop|restart|status mihomo # system-level
|
||||
```
|
||||
|
||||
**Why disable instead of delete**: If systemd service has issues, you can quickly re-enable by renaming back. Also keeps the service definition as documentation.
|
||||
|
||||
**Pitfall**: If you forget to disable startup.sh definitions AND the watchdog still calls `startup.sh restart`, the two managers fight over PIDs. Always migrate atomically: create systemd service → enable → disable startup.sh definition → update watchdog.
|
||||
|
||||
---
|
||||
|
||||
## Option B: User-Level Services (for non-root setups)
|
||||
|
||||
User services run as the logged-in user, no root needed for service files.
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.config/systemd/user
|
||||
cat > ~/.config/systemd/user/mihomo.service << 'EOF'
|
||||
[Unit]
|
||||
Description=Mihomo (Clash Meta) Proxy
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=%h/clash
|
||||
ExecStart=%h/clash/mihomo -d %h/clash/config
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
EOF
|
||||
|
||||
cat > ~/.config/systemd/user/yacd-dashboard.service << 'EOF'
|
||||
[Unit]
|
||||
Description=YACD Dashboard (static HTTP)
|
||||
After=network.target mihomo.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=%h/clash/yacd
|
||||
ExecStart=/usr/bin/python3 -m http.server 9091 --bind 0.0.0.0 --directory %h/clash/yacd
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
EOF
|
||||
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable mihomo yacd-dashboard
|
||||
systemctl --user start mihomo yacd-dashboard
|
||||
```
|
||||
|
||||
**Critical**: Without linger, user services die on logout:
|
||||
```bash
|
||||
sudo loginctl enable-linger $USER
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verify (both options)
|
||||
```bash
|
||||
# System-level
|
||||
sudo systemctl status mihomo yacd
|
||||
# User-level
|
||||
systemctl --user status mihomo yacd-dashboard
|
||||
|
||||
curl -s http://127.0.0.1:9090/version
|
||||
curl -s http://127.0.0.1:9091/ | head -3
|
||||
```
|
||||
@@ -0,0 +1,108 @@
|
||||
# Telegram Adapter Diagnostics
|
||||
|
||||
## Quick Health Check
|
||||
|
||||
```bash
|
||||
# 1. Is the adapter connected?
|
||||
journalctl --user -u hermes-gateway --since '10 min ago' --no-pager | grep -i "telegram" | tail -5
|
||||
|
||||
# 2. Is the proxy working?
|
||||
curl -s -o /dev/null -w "%{http_code}" --proxy http://127.0.0.1:7890 https://api.telegram.org
|
||||
|
||||
# 3. Is the bot alive?
|
||||
# (replace TOKEN with actual bot token from ~/.hermes/.env)
|
||||
curl -s "https://api.telegram.org/bot<TOKEN>/getMe" --proxy http://127.0.0.1:7890
|
||||
|
||||
# 4. Is the bot in the target group?
|
||||
curl -s "https://api.telegram.org/bot<TOKEN>/getChatMember?chat_id=<GROUP_ID>&user_id=<BOT_ID>" --proxy http://127.0.0.1:7890
|
||||
```
|
||||
|
||||
## Diagnosing "Bot Not Processing Group Messages"
|
||||
|
||||
### Step 1: Check if adapter is alive
|
||||
```bash
|
||||
journalctl --user -u hermes-gateway --no-pager --since '1 hour ago' | grep "\[Telegram\]" | tail -10
|
||||
```
|
||||
- If only crash/reconnect-failed logs → adapter is dead, restart gateway
|
||||
- If "Reconnecting telegram" entries → watcher is alive, proxy may be down
|
||||
- If no Telegram logs at all → either adapter connected silently or watcher died
|
||||
|
||||
### Step 2: Check if proxy is the issue
|
||||
```bash
|
||||
curl -s -o /dev/null -w "%{http_code}" --proxy http://127.0.0.1:7890 https://api.telegram.org
|
||||
# 302 = proxy working, timeout/ConnectError = proxy down
|
||||
```
|
||||
|
||||
### Step 3: Check bot group membership
|
||||
```bash
|
||||
# Get bot info
|
||||
python3 -c "
|
||||
import json, urllib.request
|
||||
with open('/home/openclaw/.hermes/.env') as f:
|
||||
token = [l.split('=',1)[1].strip() for l in f if l.startswith('TELEGRAM_BOT_TOKEN')][0]
|
||||
proxy = urllib.request.ProxyHandler({'https': 'http://127.0.0.1:7890'})
|
||||
opener = urllib.request.build_opener(proxy)
|
||||
resp = opener.open(f'https://api.telegram.org/bot{token}/getChatMember?chat_id=<GROUP_ID>&user_id=<BOT_ID>')
|
||||
print(json.loads(resp.read()))
|
||||
"
|
||||
```
|
||||
|
||||
### Step 4: Check config
|
||||
In `~/.hermes/config.yaml`, under `telegram:`:
|
||||
- `free_response_chats` or `free_response_channels` must include the group ID
|
||||
- `channel_prompts` can be set per group for custom behavior
|
||||
- `allowed_chats` when set acts as a whitelist (empty = no restriction)
|
||||
- `require_mention: true` means bot only responds when @mentioned (unless in `free_response_chats`)
|
||||
|
||||
**Editing channel_prompts**: Values are unicode-escaped with YAML line continuations (`\` at line end). Don't use `patch()` — it breaks on multi-line unicode. Use Python regex instead:
|
||||
```python
|
||||
import re
|
||||
with open('/home/openclaw/.hermes/config.yaml') as f:
|
||||
content = f.read()
|
||||
# Match across line breaks with [\s\\]* for YAML continuations
|
||||
pattern = re.compile(r'\\uXXXX[\s\\]*\\uYYYY')
|
||||
content = pattern.sub(r'\\uNEW1\\uNEW2', content)
|
||||
with open('/home/openclaw/.hermes/config.yaml', 'w') as f:
|
||||
f.write(content)
|
||||
# Always verify YAML validity after
|
||||
import yaml; yaml.safe_load(open('/home/openclaw/.hermes/config.yaml'))
|
||||
```
|
||||
The same prompt string is duplicated across `slack`, `discord`, `mattermost`, and `telegram` sections — replace in all 4.
|
||||
|
||||
### Step 5: Check forwarder
|
||||
```bash
|
||||
# Is the forwarder running?
|
||||
docker ps | grep forwarder
|
||||
|
||||
# Check forwarder config
|
||||
sqlite3 ~/TelegramForwarder/db/forward.db "SELECT * FROM chats;"
|
||||
sqlite3 ~/TelegramForwarder/db/forward.db "SELECT * FROM forward_rules;"
|
||||
sqlite3 ~/TelegramForwarder/db/forward.db "SELECT * FROM keywords;"
|
||||
|
||||
# Check recent forwarding activity
|
||||
docker logs telegram-forwarder --tail 20 | grep "已发送"
|
||||
```
|
||||
|
||||
## Gateway Restart Constraints
|
||||
|
||||
- **Cannot restart from inside the gateway**: `hermes gateway restart` is blocked when run from a gateway session (SIGTERM propagation prevention)
|
||||
- **`kill <PID>` also blocked**: Security scanner intercepts `kill` commands targeting the gateway PID. Terminal returns `Blocked: cannot restart or stop the gateway from inside the gateway process.`
|
||||
- **Workaround 1**: User runs `hermes gateway restart` from a separate SSH shell (recommended)
|
||||
- **Workaround 2**: `systemctl --user restart hermes-gateway` from a separate shell
|
||||
- **Workaround 3**: Write a small helper script to `/tmp/` and ask user to run it manually
|
||||
|
||||
## Reconnect Watcher Architecture
|
||||
|
||||
When a platform adapter hits a fatal error:
|
||||
1. `_set_fatal_error()` → sets `adapter._running = False`
|
||||
2. `_handle_adapter_fatal_error()` → disconnects adapter, removes from `self.adapters`
|
||||
3. If `fatal_error_retryable=True` → adds to `self._failed_platforms` with backoff
|
||||
4. `_platform_reconnect_watcher()` → background asyncio task, retries with exponential backoff (30s→300s cap)
|
||||
|
||||
**Known bug**: The watcher task can silently die. No upstream fix yet. Workaround: external gateway restart.
|
||||
|
||||
**Root cause (observed 2026-06-23)**: When multiple platforms crash simultaneously (Telegram + WeChat + QQBot), the watcher may die. Evidence: `journalctl` shows the fatal error at 09:27 but ZERO "Reconnecting telegram" or "Reconnect telegram error" entries afterward — the watcher's `while self._running` loop exited without logging. On a previous crash (2026-06-22) with only Telegram affected, the watcher worked correctly and kept retrying with backoff.
|
||||
|
||||
**Edge case — already-queued platform**: `_handle_adapter_fatal_error()` checks `adapter.platform not in self._failed_platforms` before queueing. If Telegram was already in the queue from a prior failure, the new fatal error doesn't re-queue or reset the backoff. The existing entry's `next_retry` timer may be at the 300s cap, causing long silent gaps between reconnection attempts even when the watcher IS alive.
|
||||
|
||||
**Definitive diagnosis**: If after a crash you see zero reconnection attempt logs for >10 minutes, the watcher is dead. Restart the gateway from an external shell.
|
||||
@@ -0,0 +1,70 @@
|
||||
#!/bin/bash
|
||||
# Clash 订阅自动更新脚本
|
||||
# 直连下载订阅（不走代理，避免循环依赖）→ 注入 required params → 热重载
|
||||
# Usage: bash update-sub.sh [SUBSCRIPTION_URL]
|
||||
#
|
||||
# Customize these variables for your deployment:
|
||||
# SUB_URL — subscription URL (default: reads from ~/clash/sub-url.conf)
|
||||
# CONFIG — path to mihomo config.yaml
|
||||
|
||||
set -e
|
||||
|
||||
CLASH_DIR="${CLASH_DIR:-$HOME/clash}"
|
||||
|
||||
# Read subscription URL from conf file or CLI arg
|
||||
if [ -n "$1" ]; then
|
||||
SUB_URL="$1"
|
||||
elif [ -f "$CLASH_DIR/sub-url.conf" ]; then
|
||||
source "$CLASH_DIR/sub-url.conf"
|
||||
else
|
||||
echo "错误: 找不到 $CLASH_DIR/sub-url.conf，也未传入 URL 参数"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
CONFIG="${CONFIG:-$CLASH_DIR/config/config.yaml}"
|
||||
TMP_FILE=$(mktemp)
|
||||
LOCK_FILE="/tmp/clash-update.lock"
|
||||
|
||||
# 防止并发
|
||||
exec 200>"$LOCK_FILE"
|
||||
flock -n 200 || { echo "[$(date -Iseconds)] Already running, skipping"; exit 0; }
|
||||
|
||||
echo "[$(date -Iseconds)] Starting subscription update (direct, no proxy)..."
|
||||
|
||||
# 直连下载，不走代理
|
||||
# 原因：代理本身依赖订阅配置，走代理更新 = 循环依赖死锁
|
||||
HTTP_CODE=$(curl -L --connect-timeout 10 --max-time 30 \
|
||||
-H "User-Agent: clash-verge/v2.0.0" \
|
||||
-w "%{http_code}" -o "$TMP_FILE" \
|
||||
"$SUB_URL" 2>/dev/null)
|
||||
|
||||
if [ "$HTTP_CODE" != "200" ]; then
|
||||
echo " FAIL: Subscription download failed (HTTP $HTTP_CODE)"
|
||||
rm -f "$TMP_FILE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Inject required params that raw subscriptions often omit
|
||||
sed -i "/external-controller:/c\external-controller: '0.0.0.0:9090'" "$TMP_FILE"
|
||||
grep -q "^mixed-port:" "$TMP_FILE" || sed -i "1i mixed-port: 7890" "$TMP_FILE"
|
||||
grep -q "^allow-lan:" "$TMP_FILE" || sed -i "1a allow-lan: true" "$TMP_FILE"
|
||||
grep -q "^mode:" "$TMP_FILE" || sed -i "1a mode: rule" "$TMP_FILE"
|
||||
grep -q "^log-level:" "$TMP_FILE" || sed -i "1a log-level: info" "$TMP_FILE"
|
||||
|
||||
cp "$TMP_FILE" "$CONFIG"
|
||||
rm -f "$TMP_FILE"
|
||||
|
||||
NODE_COUNT=$(grep -c 'name:' "$CONFIG" 2>/dev/null || echo 0)
|
||||
echo " OK: Subscription updated, $NODE_COUNT nodes"
|
||||
|
||||
# Hot-reload via mihomo REST API (preferred over restart — preserves connections)
|
||||
if curl -s -o /dev/null http://127.0.0.1:9090/version 2>/dev/null; then
|
||||
curl -s -X PUT http://127.0.0.1:9090/configs \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"path\":\"$CONFIG\"}" > /dev/null 2>&1
|
||||
echo " OK: Hot-reload complete"
|
||||
else
|
||||
echo " WARN: mihomo not running, skipped hot-reload"
|
||||
fi
|
||||
|
||||
echo "[$(date -Iseconds)] Update complete"
|
||||
Reference in New Issue
Block a user