Hermes-Skills/devops/clash-docker-workflow/references/proxy-diagnostics.md

Proxy Diagnostics: "Is the proxy working?"

Systematic checklist when the user asks whether the proxy is healthy, or when Hermes services show connection timeouts despite the proxy running.

Key Insight

Proxy works ≠ Hermes is using it. These are independent checks. A curl test passing through the proxy proves the proxy is healthy, but Hermes may not be routing through it.

Critical: Telegram Proxy Source

The Telegram adapter reads proxy from environment variables at the network layer.

Source code path:

• gateway/platforms/telegram_network.py → _resolve_proxy_url() → resolve_proxy_url("TELEGRAM_PROXY", ...) from gateway/platforms/base.py
• gateway/platforms/telegram.py line 2068 → same resolve_proxy_url() call

resolve_proxy_url() checks (in order):

1. TELEGRAM_PROXY env var (Telegram-specific, highest priority)
2. HTTPS_PROXY / HTTP_PROXY / ALL_PROXY env vars (and lowercase variants)
3. macOS system proxy auto-detection

Key: telegram.proxy_url in config.yaml IS bridged to TELEGRAM_PROXY env var by load_gateway_config() at startup. So both config.yaml and direct env vars work. However, if load_gateway_config() fails to run (e.g. gateway started manually without the config bridge), only direct env vars work.

Dual defense recommended: set BOTH telegram.proxy_url in config.yaml AND HTTPS_PROXY in the systemd service file. This ensures proxy works whether the config bridge runs or not.

Quick Diagnosis: Read Gateway Logs Directly

When terminal() is blocked (consent mechanism, security scanner, etc.), diagnose via log files:

# 1. Find log files
search_files(pattern="*.log", path="~/.hermes/logs", target="files")

# 2. Read recent gateway log tail
read_file(path="~/.hermes/logs/gateway.log", offset=-100)  # last 100 lines

# 3. Search for Telegram errors
search_files(pattern="telegram.*failed|Connect attempt|All connection", 
             path="~/.hermes/logs/gateway.log", target="content")

Key log patterns to look for:

Log Pattern	Meaning
Proxy detected; passing explicitly to HTTPXRequest: http://127.0.0.1:7890	Proxy URL is configured in Hermes ✓
Connect attempt N/8 failed: httpx.ConnectError: All connection attempts failed	Proxy port unreachable (proxy down or wrong port)
connect timed out after 30s	All retry attempts exhausted, will retry in N seconds
Reconnect telegram error: ... next retry in 300s	Exponential backoff — retries increase: 60→120→240→300s
Primary api.telegram.org connection failed (); trying fallback IPs	DNS resolution worked but TCP connection failed
DoH discovery yielded no usable IPs	DNS-over-HTTPS failed, using hardcoded fallback IPs

Common failure: proxy itself is DOWN (simplest case):

• Log shows: Proxy detected + All connection attempts failed on every attempt
• This means Hermes knows about the proxy (config bridge works), but the proxy port is not responding
• Fix: restart mihomo (systemctl --user restart mihomo or bash ~/clash/startup.sh restart mihomo)

This is different from: proxy running but Hermes not routing through it (see "Common Failure Pattern" below).

Diagnostic Steps (in order)

1. Proxy process alive?

ss -tlnp | grep 7890          # mihomo should be LISTEN
# Expected: users:(("mihomo",pid=NNNN,fd=3))

2. Proxy actually works? (curl test)

curl -x http://127.0.0.1:7890 -I https://www.google.com --connect-timeout 5
curl -x http://127.0.0.1:7890 -I https://api.telegram.org --connect-timeout 5

Both should return HTTP 200/302, not timeout.

3. Gateway process has proxy env vars?

PID=$(pgrep -f "hermes_cli.main gateway" | head -1)
tr '\0' '\n' < /proc/$PID/environ | grep -i proxy

If empty — the gateway process has NO proxy env vars and relies solely on the config bridge (telegram.proxy_url → TELEGRAM_PROXY). If the bridge fails, Telegram connects DIRECT.

4. Check systemd service file

grep -i proxy ~/.config/systemd/user/hermes-gateway.service

If no HTTPS_PROXY / HTTP_PROXY lines — the service file needs updating for dual defense.

5. Check config.yaml bridge

grep proxy_url ~/.hermes/config.yaml

Should show proxy_url: http://127.0.0.1:7890 under telegram:.

6. Gateway status + logs

hermes gateway status
journalctl --user -u hermes-gateway -n 50 | grep -i telegram

Look for: "Proxy detected" (good) or "connection failed" / "Timed out" (bad).

7. Run hermes doctor

hermes doctor

Shows API connectivity, missing configs, auth issues.

Common Failure Pattern

Symptom: Proxy is running, curl tests pass, but gateway shows:

[Telegram] Primary api.telegram.org connection failed
[Telegram] Fallback IP 149.154.167.220 failed
[Telegram] Connect attempt N/8 failed: Timed out

Root cause: Gateway systemd service file has no proxy environment variables AND the config bridge failed (or telegram.proxy_url not set). The Telegram adapter falls back to direct connection, which fails behind GFW.

Fix (dual defense — set both):

# 1. Set config bridge
hermes config set telegram.proxy_url http://127.0.0.1:7890

# 2. Add proxy env vars to systemd service
python3 -c "
import pathlib
p = pathlib.Path.home() / '.config/systemd/user/hermes-gateway.service'
lines = p.read_text().splitlines()
lines = [l for l in lines if 'HTTPS_PROXY' not in l and 'HTTP_PROXY' not in l]
out = []
for l in lines:
    out.append(l)
    if 'HERMES_HOME' in l and l.strip().startswith('Environment'):
        out.append('Environment=\"HTTPS_PROXY=http://127.0.0.1:7890\"')
        out.append('Environment=\"HTTP_PROXY=http://127.0.0.1:7890\"')
p.write_text('\n'.join(out) + '\n')
"
systemctl --user daemon-reload
hermes gateway restart

Quick fix: bash ~/clash/fix-proxy.sh --fix

Verification:

PID=$(pgrep -f "hermes_cli.main gateway" | head -1)
tr '\0' '\n' < /proc/$PID/environ | grep -i proxy
# Expected: TELEGRAM_PROXY=http://127.0.0.1:7890 (from config bridge)
#           HTTPS_PROXY=http://127.0.0.1:7890 (from systemd service)

Network/Proxy Audit Checklist

When the user asks "check my network/proxy config" or "any redundant configs?", cover these dimensions:

1. 代理进程: mihomo running? Port 7890 listening?
2. 代理连通性: curl through proxy works? (Google, Telegram)
3. Gateway 代理模式: Both config.yaml proxy_url AND systemd HTTPS_PROXY set? (dual defense)
4. 旧服务残留: Any old gateway services (e.g. openclaw-gateway.service) still in ~/.config/systemd/user/? Disable with systemctl --user disable --now <name>
5. 重复启动脚本: Both startup.sh and systemd managing the same service? Pick one — systemd preferred. The other is dead code.
6. WireGuard 残留: Check /etc/wireguard/ for leftover .conf files. WireGuard can intercept ALL traffic if misconfigured.
7. 环境变量代理: Check ~/.bashrc for HTTP_PROXY/HTTPS_PROXY — needed for terminal sessions but NOT sufficient for systemd services (they don't source .bashrc).
8. channel_prompts 重复: Same prompt in slack/discord/telegram sections of config.yaml — only needed in platforms actually used.

Quick Status Summary Template

When reporting proxy status to the user, cover these four dimensions:

1. 代理进程: mihomo running? Port 7890 listening?
2. 代理连通性: curl through proxy works? (Google, Telegram)
3. Gateway 代理模式: systemd service has proxy env vars? Config bridge set? Gateway process has them?
4. 网关状态: Gateway running? Telegram connected? No timeout errors in logs?

• Gateway log path: ~/.hermes/logs/gateway.log, NOT ~/.hermes/gateway.log. Also check ~/.hermes/logs/errors.log for historical errors. Use search_files(pattern="*.log", path="~/.hermes/logs", target="files") to discover all logs.
• Telegram reconnect backoff: When Telegram fails to connect, the gateway retries with exponential backoff: 60s → 120s → 240s → 300s (cap). After fixing the proxy, you may need to hermes gateway restart to reset the backoff instead of waiting for the next retry window.
• QQBot also fails when proxy is down: The QQBot adapter routes through the same proxy. If 127.0.0.1:7890 is down, both Telegram AND QQ will show connection failures. Check QQBot logs for Failed to get QQ Bot access token: All connection attempts failed.

Quick-Fix Script

~/clash/fix-proxy.sh — interactive menu with 8 functions:

1. 📊 查看状态 (full status overview)
2. 🔧 一键诊断修复 (auto-diagnose and fix)
3. 🔄 重启 Mihomo
4. 📡 更新/更换订阅
5. 🔀 切换代理节点 (via Mihomo API)
6. 🌐 Gateway 代理/直连切换
7. 🔁 重启 Hermes Gateway
8. 📋 查看日志

Usage:

• Interactive: bash ~/clash/fix-proxy.sh
• One-click fix: bash ~/clash/fix-proxy.sh --fix