Hermes-Skills/devops/clash-docker-workflow/references/proxy-diagnostics.md

# Proxy Diagnostics: "Is the proxy working?"

Systematic checklist when the user asks whether the proxy is healthy, or when Hermes services show connection timeouts despite the proxy running.

## Key Insight
**Proxy works ≠ Hermes is using it.** These are independent checks. A curl test passing through the proxy proves the proxy is healthy, but Hermes may not be routing through it.

## Critical: Telegram Proxy Source

The Telegram adapter reads proxy from environment variables at the network layer.

Source code path:
- `gateway/platforms/telegram_network.py` → `_resolve_proxy_url()` → `resolve_proxy_url("TELEGRAM_PROXY", ...)` from `gateway/platforms/base.py`
- `gateway/platforms/telegram.py` line 2068 → same `resolve_proxy_url()` call

`resolve_proxy_url()` checks (in order):
1. `TELEGRAM_PROXY` env var (Telegram-specific, highest priority)
2. `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` env vars (and lowercase variants)
3. macOS system proxy auto-detection

**Key: `telegram.proxy_url` in config.yaml IS bridged to `TELEGRAM_PROXY` env var** by `load_gateway_config()` at startup. So both config.yaml and direct env vars work. However, if `load_gateway_config()` fails to run (e.g. gateway started manually without the config bridge), only direct env vars work.

**Dual defense recommended**: set BOTH `telegram.proxy_url` in config.yaml AND `HTTPS_PROXY` in the systemd service file. This ensures proxy works whether the config bridge runs or not.

## Quick Diagnosis: Read Gateway Logs Directly

When `terminal()` is blocked (consent mechanism, security scanner, etc.), diagnose via log files:

```python
# 1. Find log files
search_files(pattern="*.log", path="~/.hermes/logs", target="files")

# 2. Read recent gateway log tail
read_file(path="~/.hermes/logs/gateway.log", offset=-100)  # last 100 lines

# 3. Search for Telegram errors
search_files(pattern="telegram.*failed|Connect attempt|All connection", 
             path="~/.hermes/logs/gateway.log", target="content")
```

**Key log patterns to look for:**

| Log Pattern | Meaning |
|---|---|
| `Proxy detected; passing explicitly to HTTPXRequest: http://127.0.0.1:7890` | Proxy URL is configured in Hermes ✓ |
| `Connect attempt N/8 failed: httpx.ConnectError: All connection attempts failed` | Proxy port unreachable (proxy down or wrong port) |
| `connect timed out after 30s` | All retry attempts exhausted, will retry in N seconds |
| `Reconnect telegram error: ... next retry in 300s` | Exponential backoff — retries increase: 60→120→240→300s |
| `Primary api.telegram.org connection failed (); trying fallback IPs` | DNS resolution worked but TCP connection failed |
| `DoH discovery yielded no usable IPs` | DNS-over-HTTPS failed, using hardcoded fallback IPs |

**Common failure: proxy itself is DOWN** (simplest case):
- Log shows: `Proxy detected` + `All connection attempts failed` on every attempt
- This means Hermes knows about the proxy (config bridge works), but the proxy port is not responding
- Fix: restart mihomo (`systemctl --user restart mihomo` or `bash ~/clash/startup.sh restart mihomo`)

**This is different from**: proxy running but Hermes not routing through it (see "Common Failure Pattern" below).

## Diagnostic Steps (in order)

### 1. Proxy process alive?
```bash
ss -tlnp | grep 7890          # mihomo should be LISTEN
# Expected: users:(("mihomo",pid=NNNN,fd=3))
```

### 2. Proxy actually works? (curl test)
```bash
curl -x http://127.0.0.1:7890 -I https://www.google.com --connect-timeout 5
curl -x http://127.0.0.1:7890 -I https://api.telegram.org --connect-timeout 5
```
Both should return HTTP 200/302, not timeout.

### 3. Gateway process has proxy env vars?
```bash
PID=$(pgrep -f "hermes_cli.main gateway" | head -1)
tr '\0' '\n' < /proc/$PID/environ | grep -i proxy
```
If empty — the gateway process has NO proxy env vars and relies solely on the config bridge (`telegram.proxy_url` → `TELEGRAM_PROXY`). If the bridge fails, Telegram connects DIRECT.

### 4. Check systemd service file
```bash
grep -i proxy ~/.config/systemd/user/hermes-gateway.service
```
If no `HTTPS_PROXY` / `HTTP_PROXY` lines — the service file needs updating for dual defense.

### 5. Check config.yaml bridge
```bash
grep proxy_url ~/.hermes/config.yaml
```
Should show `proxy_url: http://127.0.0.1:7890` under `telegram:`.

### 6. Gateway status + logs
```bash
hermes gateway status
journalctl --user -u hermes-gateway -n 50 | grep -i telegram
```
Look for: "Proxy detected" (good) or "connection failed" / "Timed out" (bad).

### 7. Run hermes doctor
```bash
hermes doctor
```
Shows API connectivity, missing configs, auth issues.

## Common Failure Pattern

**Symptom**: Proxy is running, curl tests pass, but gateway shows:
```
[Telegram] Primary api.telegram.org connection failed
[Telegram] Fallback IP 149.154.167.220 failed
[Telegram] Connect attempt N/8 failed: Timed out
```

**Root cause**: Gateway systemd service file has no proxy environment variables AND the config bridge failed (or `telegram.proxy_url` not set). The Telegram adapter falls back to direct connection, which fails behind GFW.

**Fix** (dual defense — set both):
```bash
# 1. Set config bridge
hermes config set telegram.proxy_url http://127.0.0.1:7890

# 2. Add proxy env vars to systemd service
python3 -c "
import pathlib
p = pathlib.Path.home() / '.config/systemd/user/hermes-gateway.service'
lines = p.read_text().splitlines()
lines = [l for l in lines if 'HTTPS_PROXY' not in l and 'HTTP_PROXY' not in l]
out = []
for l in lines:
    out.append(l)
    if 'HERMES_HOME' in l and l.strip().startswith('Environment'):
        out.append('Environment=\"HTTPS_PROXY=http://127.0.0.1:7890\"')
        out.append('Environment=\"HTTP_PROXY=http://127.0.0.1:7890\"')
p.write_text('\n'.join(out) + '\n')
"
systemctl --user daemon-reload
hermes gateway restart
```

**Quick fix**: `bash ~/clash/fix-proxy.sh --fix`

**Verification**:
```bash
PID=$(pgrep -f "hermes_cli.main gateway" | head -1)
tr '\0' '\n' < /proc/$PID/environ | grep -i proxy
# Expected: TELEGRAM_PROXY=http://127.0.0.1:7890 (from config bridge)
#           HTTPS_PROXY=http://127.0.0.1:7890 (from systemd service)
```

## Network/Proxy Audit Checklist

When the user asks "check my network/proxy config" or "any redundant configs?", cover these dimensions:

1. **代理进程**: mihomo running? Port 7890 listening?
2. **代理连通性**: curl through proxy works? (Google, Telegram)
3. **Gateway 代理模式**: Both config.yaml `proxy_url` AND systemd `HTTPS_PROXY` set? (dual defense)
4. **旧服务残留**: Any old gateway services (e.g. `openclaw-gateway.service`) still in `~/.config/systemd/user/`? Disable with `systemctl --user disable --now <name>`
5. **重复启动脚本**: Both `startup.sh` and systemd managing the same service? Pick one — systemd preferred. The other is dead code.
6. **WireGuard 残留**: Check `/etc/wireguard/` for leftover `.conf` files. WireGuard can intercept ALL traffic if misconfigured.
7. **环境变量代理**: Check `~/.bashrc` for `HTTP_PROXY`/`HTTPS_PROXY` — needed for terminal sessions but NOT sufficient for systemd services (they don't source .bashrc).
8. **channel_prompts 重复**: Same prompt in slack/discord/telegram sections of config.yaml — only needed in platforms actually used.

## Quick Status Summary Template
When reporting proxy status to the user, cover these four dimensions:
1. **代理进程**: mihomo running? Port 7890 listening?
2. **代理连通性**: curl through proxy works? (Google, Telegram)
3. **Gateway 代理模式**: systemd service has proxy env vars? Config bridge set? Gateway process has them?
4. **网关状态**: Gateway running? Telegram connected? No timeout errors in logs?

- **Gateway log path**: `~/.hermes/logs/gateway.log`, NOT `~/.hermes/gateway.log`. Also check `~/.hermes/logs/errors.log` for historical errors. Use `search_files(pattern="*.log", path="~/.hermes/logs", target="files")` to discover all logs.
- **Telegram reconnect backoff**: When Telegram fails to connect, the gateway retries with exponential backoff: 60s → 120s → 240s → 300s (cap). After fixing the proxy, you may need to `hermes gateway restart` to reset the backoff instead of waiting for the next retry window.
- **QQBot also fails when proxy is down**: The QQBot adapter routes through the same proxy. If `127.0.0.1:7890` is down, both Telegram AND QQ will show connection failures. Check QQBot logs for `Failed to get QQ Bot access token: All connection attempts failed`.

## Quick-Fix Script

`~/clash/fix-proxy.sh` — interactive menu with 8 functions:
1. 📊 查看状态 (full status overview)
2. 🔧 一键诊断修复 (auto-diagnose and fix)
3. 🔄 重启 Mihomo
4. 📡 更新/更换订阅
5. 🔀 切换代理节点 (via Mihomo API)
6. 🌐 Gateway 代理/直连切换
7. 🔁 重启 Hermes Gateway
8. 📋 查看日志

Usage:
- Interactive: `bash ~/clash/fix-proxy.sh`
- One-click fix: `bash ~/clash/fix-proxy.sh --fix`