--- name: hermes-gateway-troubleshooting description: > Diagnose and fix Hermes Gateway platform adapter connection issues. Covers checking gateway state, common adapter errors (is_reconnect, credential failures, dependency missing), safe restart procedures, and log inspection. trigger: - gateway not connecting - platform disconnected - adapter error - QQ收不到消息 - gateway retrying - connect() unexpected keyword argument - is_reconnect - gateway state check - 网关连不上 --- # Hermes Gateway Troubleshooting Diagnose why a Hermes Gateway platform adapter won't connect. ## 1. Check Gateway State The gateway state file is the single source of truth for platform connection status: ```bash cat ~/.hermes/gateway_state.json | python3 -m json.tool ``` Look at the `platforms` dict. Each platform has: - `state`: `"connected"` ✅, `"disconnected"`, `"retrying"` ❌ - `error_code`: machine-readable error type or `null` - `error_message`: human-readable error or `null` Also check if the gateway process is alive: ```bash ps aux | grep -i gateway | grep -v grep systemctl --user is-active hermes-gateway ``` ## 2. Categorize the Error ### Common Error Patterns | Error message pattern | Likely cause | Fix | |-----------------------|-------------|-----| | `X.connect() got an unexpected keyword argument 'is_reconnect'` | Adapter `connect()` signature doesn't match base class | Add `*, is_reconnect: bool = False` to adapter's `connect()` | | `failed to get access token` / `qq_missing_credentials` | Missing or wrong env vars/config | Check `.env` or `config.yaml` for the platform's API credentials | | `aiohttp not installed` / `httpx not installed` | Missing Python dependency | `pip install aiohttp httpx` (or `uv pip install ...`) | | `WebSocket closed (code=4914/4915)` | Bot offline or banned | Check bot status on platform's developer portal | | `database is locked` (Telethon/SQLite) | Container still running with session DB | Stop container, edit DB, restart | | `getUpdates returned 0` (Telegram) | No adapter polling / stale gateway | Kill old gateway process, restart fresh | | `Telegram polling conflict` | Two processes sharing same bot token | `systemctl --user stop hermes-gateway`, wait 30s, `start` | | `Connect timeout` / `Connection refused` | Outbound connectivity issue | Check proxy/VPN, firewal, or WSS_PROXY setting | ## 3. The `is_reconnect` Parameter Mismatch (Common Bug) ### Symptom Gateway state shows the platform as `"retrying"` with error: ``` XAdapter.connect() got an unexpected keyword argument 'is_reconnect' ``` ### Root Cause The base class `BasePlatformAdapter` defines: ```python @abstractmethod async def connect(self, *, is_reconnect: bool = False) -> bool: ``` The gateway's reconnect watcher calls `adapter.connect(is_reconnect=True)`. If the adapter's `connect()` doesn't accept the `is_reconnect` keyword-only arg, Python throws `TypeError`. ### Fix Add the missing parameter to the adapter's `connect()` method: ```python # Before (broken): async def connect(self) -> bool: ... # After (fixed): async def connect(self, *, is_reconnect: bool = False) -> bool: ... ``` The adapter doesn't need to *use* `is_reconnect` if it doesn't care about the distinction — just accepting the kwarg is enough to stop the TypeError. For adapters that do care (e.g., Telegram's cold boot drops the stale event queue vs reconnect preserves it), use `is_reconnect` to branch behavior. ### Verification After fixing, restart the gateway and check state again: ```bash systemctl --user stop hermes-gateway sleep 30 # critical: let Telegram session expire systemctl --user start hermes-gateway sleep 10 # wait for startup cat ~/.hermes/gateway_state.json | python3 -c "import sys,json; s=json.load(sys.stdin); [print(f' {k}: {v[\"state\"]}') for k,v in s['platforms'].items()]" ``` Also run the adapter's tests: ```bash cd ~/.hermes/hermes-agent source ./venv/bin/activate python -m pytest tests/gateway/test_.py -x -v --tb=short ``` ### Pattern: All Adapters Must Match Every Hermes gateway platform adapter must declare `connect()` with `*, is_reconnect: bool = False`. When adding a new adapter or updating an old one, verify this signature matches the base class. ## 4. Log Inspection ```bash # Gateway log (general platform issues) tail -100 ~/.hermes/logs/gateway.log | grep -iE "error|fail|disconnect|connect|retry" # Filter by platform tag (e.g., QQBot, Telegram, Weixin) tail -100 ~/.hermes/logs/gateway.log | grep -i "QQBot\|Telegram\|Weixin" # Check gateway update timestamps grep -E "state.*connected|state.*retrying" ~/.hermes/logs/gateway.log | tail -20 ``` ## 5. Safe Gateway Restart **Do NOT use `hermes gateway restart`** — it kills the Telegram polling session too quickly. The new adapter starts before the old one's Telegram session expires, causing a permanent polling conflict. ### Safe Sequence ```bash # Step 1: Stop systemctl --user stop hermes-gateway # Step 2: Wait for Telegram's session lock to expire (30 seconds minimum) sleep 30 # Step 3: Start fresh systemctl --user start hermes-gateway # Step 4: Wait for startup sleep 10 # Step 5: Verify cat ~/.hermes/gateway_state.json | python3 -m json.tool ``` ## 6. Platform-Specific Diagnostics ### QQ Bot - Check `QQ_APP_ID` and `QQ_CLIENT_SECRET` in `.env` - QQ WebSocket needs proxy env var `WSS_PROXY` or `HTTPS_PROXY` (set in `_open_ws`) - Error 4914 = bot offline/sandbox (fatal, stop reconnecting) - Error 4915 = bot banned (fatal, stop reconnecting) - Verify `adapter.py`'s `connect()` accepts `is_reconnect` ### Telegram - Only one bot token can poll at a time. Stale processes block new sessions. - Disable bot privacy mode in @BotFather for group message reading (kick + re-add bot after changing) - `free_response_chats` in config must be inside `telegram.extra` (NOT top-level) ### Weixin - Requires `aiohttp` and `cryptography` packages - Check `WX_APP_ID`, `WX_TOKEN`, `WX_AES_KEY` in `.env` ## Pitfalls - **🔴 Gateway state file can be stale.** If `gateway_state.json` shows an old error even after restart, the new gateway process may be still starting up. Check `ps aux` for PID changes and wait. - **🔴 sleep 30 is not optional** on Telegram restarts. Without it, the new adapter gets 409 Conflict and never connects. - **🔴 `is_reconnect` in adapter but not in base class** — the reverse check is also worth making when adding a new adapter: if the adapter uses `is_reconnect` internally but the base class doesn't pass it (hypothetical), it'll silently default to `False` during reconnect. The gateway always passes `is_reconnect=True` on watcher-initiated reconnects. - **Test mocks need the same signature.** If a test creates a fake adapter with `async def connect(self):` (no `is_reconnect`), it won't catch the real runtime error.