# Proxy Diagnostics: "Is the proxy working?" Systematic checklist when the user asks whether the proxy is healthy, or when Hermes services show connection timeouts despite the proxy running. ## Key Insight **Proxy works ≠ Hermes is using it.** These are independent checks. A curl test passing through the proxy proves the proxy is healthy, but Hermes may not be routing through it. ## Critical: Telegram Proxy Source The Telegram adapter reads proxy from environment variables at the network layer. Source code path: - `gateway/platforms/telegram_network.py` → `_resolve_proxy_url()` → `resolve_proxy_url("TELEGRAM_PROXY", ...)` from `gateway/platforms/base.py` - `gateway/platforms/telegram.py` line 2068 → same `resolve_proxy_url()` call `resolve_proxy_url()` checks (in order): 1. `TELEGRAM_PROXY` env var (Telegram-specific, highest priority) 2. `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` env vars (and lowercase variants) 3. macOS system proxy auto-detection **Key: `telegram.proxy_url` in config.yaml IS bridged to `TELEGRAM_PROXY` env var** by `load_gateway_config()` at startup. So both config.yaml and direct env vars work. However, if `load_gateway_config()` fails to run (e.g. gateway started manually without the config bridge), only direct env vars work. **Dual defense recommended**: set BOTH `telegram.proxy_url` in config.yaml AND `HTTPS_PROXY` in the systemd service file. This ensures proxy works whether the config bridge runs or not. ## Quick Diagnosis: Read Gateway Logs Directly When `terminal()` is blocked (consent mechanism, security scanner, etc.), diagnose via log files: ```python # 1. Find log files search_files(pattern="*.log", path="~/.hermes/logs", target="files") # 2. Read recent gateway log tail read_file(path="~/.hermes/logs/gateway.log", offset=-100) # last 100 lines # 3. Search for Telegram errors search_files(pattern="telegram.*failed|Connect attempt|All connection", path="~/.hermes/logs/gateway.log", target="content") ``` **Key log patterns to look for:** | Log Pattern | Meaning | |---|---| | `Proxy detected; passing explicitly to HTTPXRequest: http://127.0.0.1:7890` | Proxy URL is configured in Hermes ✓ | | `Connect attempt N/8 failed: httpx.ConnectError: All connection attempts failed` | Proxy port unreachable (proxy down or wrong port) | | `connect timed out after 30s` | All retry attempts exhausted, will retry in N seconds | | `Reconnect telegram error: ... next retry in 300s` | Exponential backoff — retries increase: 60→120→240→300s | | `Primary api.telegram.org connection failed (); trying fallback IPs` | DNS resolution worked but TCP connection failed | | `DoH discovery yielded no usable IPs` | DNS-over-HTTPS failed, using hardcoded fallback IPs | **Common failure: proxy itself is DOWN** (simplest case): - Log shows: `Proxy detected` + `All connection attempts failed` on every attempt - This means Hermes knows about the proxy (config bridge works), but the proxy port is not responding - Fix: restart mihomo (`systemctl --user restart mihomo` or `bash ~/clash/startup.sh restart mihomo`) **This is different from**: proxy running but Hermes not routing through it (see "Common Failure Pattern" below). ## Diagnostic Steps (in order) ### 1. Proxy process alive? ```bash ss -tlnp | grep 7890 # mihomo should be LISTEN # Expected: users:(("mihomo",pid=NNNN,fd=3)) ``` ### 2. Proxy actually works? (curl test) ```bash curl -x http://127.0.0.1:7890 -I https://www.google.com --connect-timeout 5 curl -x http://127.0.0.1:7890 -I https://api.telegram.org --connect-timeout 5 ``` Both should return HTTP 200/302, not timeout. ### 3. Gateway process has proxy env vars? ```bash PID=$(pgrep -f "hermes_cli.main gateway" | head -1) tr '\0' '\n' < /proc/$PID/environ | grep -i proxy ``` If empty — the gateway process has NO proxy env vars and relies solely on the config bridge (`telegram.proxy_url` → `TELEGRAM_PROXY`). If the bridge fails, Telegram connects DIRECT. ### 4. Check systemd service file ```bash grep -i proxy ~/.config/systemd/user/hermes-gateway.service ``` If no `HTTPS_PROXY` / `HTTP_PROXY` lines — the service file needs updating for dual defense. ### 5. Check config.yaml bridge ```bash grep proxy_url ~/.hermes/config.yaml ``` Should show `proxy_url: http://127.0.0.1:7890` under `telegram:`. ### 6. Gateway status + logs ```bash hermes gateway status journalctl --user -u hermes-gateway -n 50 | grep -i telegram ``` Look for: "Proxy detected" (good) or "connection failed" / "Timed out" (bad). ### 7. Run hermes doctor ```bash hermes doctor ``` Shows API connectivity, missing configs, auth issues. ## Common Failure Pattern **Symptom**: Proxy is running, curl tests pass, but gateway shows: ``` [Telegram] Primary api.telegram.org connection failed [Telegram] Fallback IP 149.154.167.220 failed [Telegram] Connect attempt N/8 failed: Timed out ``` **Root cause**: Gateway systemd service file has no proxy environment variables AND the config bridge failed (or `telegram.proxy_url` not set). The Telegram adapter falls back to direct connection, which fails behind GFW. **Fix** (dual defense — set both): ```bash # 1. Set config bridge hermes config set telegram.proxy_url http://127.0.0.1:7890 # 2. Add proxy env vars to systemd service python3 -c " import pathlib p = pathlib.Path.home() / '.config/systemd/user/hermes-gateway.service' lines = p.read_text().splitlines() lines = [l for l in lines if 'HTTPS_PROXY' not in l and 'HTTP_PROXY' not in l] out = [] for l in lines: out.append(l) if 'HERMES_HOME' in l and l.strip().startswith('Environment'): out.append('Environment=\"HTTPS_PROXY=http://127.0.0.1:7890\"') out.append('Environment=\"HTTP_PROXY=http://127.0.0.1:7890\"') p.write_text('\n'.join(out) + '\n') " systemctl --user daemon-reload hermes gateway restart ``` **Quick fix**: `bash ~/clash/fix-proxy.sh --fix` **Verification**: ```bash PID=$(pgrep -f "hermes_cli.main gateway" | head -1) tr '\0' '\n' < /proc/$PID/environ | grep -i proxy # Expected: TELEGRAM_PROXY=http://127.0.0.1:7890 (from config bridge) # HTTPS_PROXY=http://127.0.0.1:7890 (from systemd service) ``` ## Network/Proxy Audit Checklist When the user asks "check my network/proxy config" or "any redundant configs?", cover these dimensions: 1. **代理进程**: mihomo running? Port 7890 listening? 2. **代理连通性**: curl through proxy works? (Google, Telegram) 3. **Gateway 代理模式**: Both config.yaml `proxy_url` AND systemd `HTTPS_PROXY` set? (dual defense) 4. **旧服务残留**: Any old gateway services (e.g. `openclaw-gateway.service`) still in `~/.config/systemd/user/`? Disable with `systemctl --user disable --now ` 5. **重复启动脚本**: Both `startup.sh` and systemd managing the same service? Pick one — systemd preferred. The other is dead code. 6. **WireGuard 残留**: Check `/etc/wireguard/` for leftover `.conf` files. WireGuard can intercept ALL traffic if misconfigured. 7. **环境变量代理**: Check `~/.bashrc` for `HTTP_PROXY`/`HTTPS_PROXY` — needed for terminal sessions but NOT sufficient for systemd services (they don't source .bashrc). 8. **channel_prompts 重复**: Same prompt in slack/discord/telegram sections of config.yaml — only needed in platforms actually used. ## Quick Status Summary Template When reporting proxy status to the user, cover these four dimensions: 1. **代理进程**: mihomo running? Port 7890 listening? 2. **代理连通性**: curl through proxy works? (Google, Telegram) 3. **Gateway 代理模式**: systemd service has proxy env vars? Config bridge set? Gateway process has them? 4. **网关状态**: Gateway running? Telegram connected? No timeout errors in logs? - **Gateway log path**: `~/.hermes/logs/gateway.log`, NOT `~/.hermes/gateway.log`. Also check `~/.hermes/logs/errors.log` for historical errors. Use `search_files(pattern="*.log", path="~/.hermes/logs", target="files")` to discover all logs. - **Telegram reconnect backoff**: When Telegram fails to connect, the gateway retries with exponential backoff: 60s → 120s → 240s → 300s (cap). After fixing the proxy, you may need to `hermes gateway restart` to reset the backoff instead of waiting for the next retry window. - **QQBot also fails when proxy is down**: The QQBot adapter routes through the same proxy. If `127.0.0.1:7890` is down, both Telegram AND QQ will show connection failures. Check QQBot logs for `Failed to get QQ Bot access token: All connection attempts failed`. ## Quick-Fix Script `~/clash/fix-proxy.sh` — interactive menu with 8 functions: 1. 📊 查看状态 (full status overview) 2. 🔧 一键诊断修复 (auto-diagnose and fix) 3. 🔄 重启 Mihomo 4. 📡 更新/更换订阅 5. 🔀 切换代理节点 (via Mihomo API) 6. 🌐 Gateway 代理/直连切换 7. 🔁 重启 Hermes Gateway 8. 📋 查看日志 Usage: - Interactive: `bash ~/clash/fix-proxy.sh` - One-click fix: `bash ~/clash/fix-proxy.sh --fix`