Hermes-Skills/devops/hermes-gateway-troubleshooting/SKILL.md

---
name: hermes-gateway-troubleshooting
description: >
  Diagnose and fix Hermes Gateway platform adapter connection issues.
  Covers checking gateway state, common adapter errors (is_reconnect,
  credential failures, dependency missing), safe restart procedures,
  and log inspection.
trigger:
  - gateway not connecting
  - platform disconnected
  - adapter error
  - QQ收不到消息
  - gateway retrying
  - connect() unexpected keyword argument
  - is_reconnect
  - gateway state check
  - 网关连不上
---

# Hermes Gateway Troubleshooting

Diagnose why a Hermes Gateway platform adapter won't connect.

## 1. Check Gateway State

The gateway state file is the single source of truth for platform connection status:

```bash
cat ~/.hermes/gateway_state.json | python3 -m json.tool
```

Look at the `platforms` dict. Each platform has:
- `state`: `"connected"` ✅, `"disconnected"`, `"retrying"` ❌
- `error_code`: machine-readable error type or `null`
- `error_message`: human-readable error or `null`

Also check if the gateway process is alive:
```bash
ps aux | grep -i gateway | grep -v grep
systemctl --user is-active hermes-gateway
```

## 2. Categorize the Error

### Common Error Patterns

| Error message pattern | Likely cause | Fix |
|-----------------------|-------------|-----|
| `X.connect() got an unexpected keyword argument 'is_reconnect'` | Adapter `connect()` signature doesn't match base class | Add `*, is_reconnect: bool = False` to adapter's `connect()` |
| `failed to get access token` / `qq_missing_credentials` | Missing or wrong env vars/config | Check `.env` or `config.yaml` for the platform's API credentials |
| `aiohttp not installed` / `httpx not installed` | Missing Python dependency | `pip install aiohttp httpx` (or `uv pip install ...`) |
| `WebSocket closed (code=4914/4915)` | Bot offline or banned | Check bot status on platform's developer portal |
| `database is locked` (Telethon/SQLite) | Container still running with session DB | Stop container, edit DB, restart |
| `getUpdates returned 0` (Telegram) | No adapter polling / stale gateway | Kill old gateway process, restart fresh |
| `Telegram polling conflict` | Two processes sharing same bot token | `systemctl --user stop hermes-gateway`, wait 30s, `start` |
| `Connect timeout` / `Connection refused` | Outbound connectivity issue | Check proxy/VPN, firewal, or WSS_PROXY setting |

## 3. The `is_reconnect` Parameter Mismatch (Common Bug)

### Symptom

Gateway state shows the platform as `"retrying"` with error:
```
XAdapter.connect() got an unexpected keyword argument 'is_reconnect'
```

### Root Cause

The base class `BasePlatformAdapter` defines:
```python
@abstractmethod
async def connect(self, *, is_reconnect: bool = False) -> bool:
```

The gateway's reconnect watcher calls `adapter.connect(is_reconnect=True)`. If the adapter's `connect()` doesn't accept the `is_reconnect` keyword-only arg, Python throws `TypeError`.

### Fix

Add the missing parameter to the adapter's `connect()` method:

```python
# Before (broken):
async def connect(self) -> bool:
    ...

# After (fixed):
async def connect(self, *, is_reconnect: bool = False) -> bool:
    ...
```

The adapter doesn't need to *use* `is_reconnect` if it doesn't care about the distinction — just accepting the kwarg is enough to stop the TypeError. For adapters that do care (e.g., Telegram's cold boot drops the stale event queue vs reconnect preserves it), use `is_reconnect` to branch behavior.

### Verification

After fixing, restart the gateway and check state again:

```bash
systemctl --user stop hermes-gateway
sleep 30   # critical: let Telegram session expire
systemctl --user start hermes-gateway
sleep 10   # wait for startup
cat ~/.hermes/gateway_state.json | python3 -c "import sys,json; s=json.load(sys.stdin); [print(f'  {k}: {v[\"state\"]}') for k,v in s['platforms'].items()]"
```

Also run the adapter's tests:
```bash
cd ~/.hermes/hermes-agent
source ./venv/bin/activate
python -m pytest tests/gateway/test_<adapter>.py -x -v --tb=short
```

### Pattern: All Adapters Must Match

Every Hermes gateway platform adapter must declare `connect()` with `*, is_reconnect: bool = False`. When adding a new adapter or updating an old one, verify this signature matches the base class.

## 4. Log Inspection

```bash
# Gateway log (general platform issues)
tail -100 ~/.hermes/logs/gateway.log | grep -iE "error|fail|disconnect|connect|retry"

# Filter by platform tag (e.g., QQBot, Telegram, Weixin)
tail -100 ~/.hermes/logs/gateway.log | grep -i "QQBot\|Telegram\|Weixin"

# Check gateway update timestamps
grep -E "state.*connected|state.*retrying" ~/.hermes/logs/gateway.log | tail -20
```

## 5. Safe Gateway Restart

**Do NOT use `hermes gateway restart`** — it kills the Telegram polling session too quickly. The new adapter starts before the old one's Telegram session expires, causing a permanent polling conflict.

### Safe Sequence

```bash
# Step 1: Stop
systemctl --user stop hermes-gateway

# Step 2: Wait for Telegram's session lock to expire (30 seconds minimum)
sleep 30

# Step 3: Start fresh
systemctl --user start hermes-gateway

# Step 4: Wait for startup
sleep 10

# Step 5: Verify
cat ~/.hermes/gateway_state.json | python3 -m json.tool
```

## 6. Platform-Specific Diagnostics

### QQ Bot
- Check `QQ_APP_ID` and `QQ_CLIENT_SECRET` in `.env`
- QQ WebSocket needs proxy env var `WSS_PROXY` or `HTTPS_PROXY` (set in `_open_ws`)
- Error 4914 = bot offline/sandbox (fatal, stop reconnecting)
- Error 4915 = bot banned (fatal, stop reconnecting)
- Verify `adapter.py`'s `connect()` accepts `is_reconnect`

### Telegram
- Only one bot token can poll at a time. Stale processes block new sessions.
- Disable bot privacy mode in @BotFather for group message reading (kick + re-add bot after changing)
- `free_response_chats` in config must be inside `telegram.extra` (NOT top-level)

### Weixin
- Requires `aiohttp` and `cryptography` packages
- Check `WX_APP_ID`, `WX_TOKEN`, `WX_AES_KEY` in `.env`

## Pitfalls

- **🔴 Gateway state file can be stale.** If `gateway_state.json` shows an old error even after restart, the new gateway process may be still starting up. Check `ps aux` for PID changes and wait.
- **🔴 sleep 30 is not optional** on Telegram restarts. Without it, the new adapter gets 409 Conflict and never connects.
- **🔴 `is_reconnect` in adapter but not in base class** — the reverse check is also worth making when adding a new adapter: if the adapter uses `is_reconnect` internally but the base class doesn't pass it (hypothetical), it'll silently default to `False` during reconnect. The gateway always passes `is_reconnect=True` on watcher-initiated reconnects.
- **Test mocks need the same signature.** If a test creates a fake adapter with `async def connect(self):` (no `is_reconnect`), it won't catch the real runtime error.