Hermes-Skills/devops/hermes-gateway-troubleshooting/SKILL.md

name, description, trigger

name	description	trigger
hermes-gateway-troubleshooting	Diagnose and fix Hermes Gateway platform adapter connection issues. Covers checking gateway state, common adapter errors (is_reconnect, credential failures, dependency missing), safe restart procedures, and log inspection.	gateway not connecting platform disconnected adapter error QQ收不到消息 gateway retrying connect() unexpected keyword argument is_reconnect gateway state check 网关连不上

Hermes Gateway Troubleshooting

Diagnose why a Hermes Gateway platform adapter won't connect.

1. Check Gateway State

The gateway state file is the single source of truth for platform connection status:

cat ~/.hermes/gateway_state.json | python3 -m json.tool

Look at the platforms dict. Each platform has:

• state: "connected" ✅, "disconnected", "retrying" ❌
• error_code: machine-readable error type or null
• error_message: human-readable error or null

Also check if the gateway process is alive:

ps aux | grep -i gateway | grep -v grep
systemctl --user is-active hermes-gateway

2. Categorize the Error

Common Error Patterns

Error message pattern	Likely cause	Fix
X.connect() got an unexpected keyword argument 'is_reconnect'	Adapter connect() signature doesn't match base class	Add *, is_reconnect: bool = False to adapter's connect()
failed to get access token / qq_missing_credentials	Missing or wrong env vars/config	Check .env or config.yaml for the platform's API credentials
aiohttp not installed / httpx not installed	Missing Python dependency	pip install aiohttp httpx (or uv pip install ...)
WebSocket closed (code=4914/4915)	Bot offline or banned	Check bot status on platform's developer portal
database is locked (Telethon/SQLite)	Container still running with session DB	Stop container, edit DB, restart
getUpdates returned 0 (Telegram)	No adapter polling / stale gateway	Kill old gateway process, restart fresh
Telegram polling conflict	Two processes sharing same bot token	systemctl --user stop hermes-gateway, wait 30s, start
Connect timeout / Connection refused	Outbound connectivity issue	Check proxy/VPN, firewal, or WSS_PROXY setting

3. The is_reconnect Parameter Mismatch (Common Bug)

Symptom

Gateway state shows the platform as "retrying" with error:

XAdapter.connect() got an unexpected keyword argument 'is_reconnect'

Root Cause

The base class BasePlatformAdapter defines:

@abstractmethod
async def connect(self, *, is_reconnect: bool = False) -> bool:

The gateway's reconnect watcher calls adapter.connect(is_reconnect=True). If the adapter's connect() doesn't accept the is_reconnect keyword-only arg, Python throws TypeError.

Fix

Add the missing parameter to the adapter's connect() method:

# Before (broken):
async def connect(self) -> bool:
    ...

# After (fixed):
async def connect(self, *, is_reconnect: bool = False) -> bool:
    ...

The adapter doesn't need to use is_reconnect if it doesn't care about the distinction — just accepting the kwarg is enough to stop the TypeError. For adapters that do care (e.g., Telegram's cold boot drops the stale event queue vs reconnect preserves it), use is_reconnect to branch behavior.

Verification

After fixing, restart the gateway and check state again:

systemctl --user stop hermes-gateway
sleep 30   # critical: let Telegram session expire
systemctl --user start hermes-gateway
sleep 10   # wait for startup
cat ~/.hermes/gateway_state.json | python3 -c "import sys,json; s=json.load(sys.stdin); [print(f'  {k}: {v[\"state\"]}') for k,v in s['platforms'].items()]"

Also run the adapter's tests:

cd ~/.hermes/hermes-agent
source ./venv/bin/activate
python -m pytest tests/gateway/test_<adapter>.py -x -v --tb=short

Pattern: All Adapters Must Match

Every Hermes gateway platform adapter must declare connect() with *, is_reconnect: bool = False. When adding a new adapter or updating an old one, verify this signature matches the base class.

4. Log Inspection

# Gateway log (general platform issues)
tail -100 ~/.hermes/logs/gateway.log | grep -iE "error|fail|disconnect|connect|retry"

# Filter by platform tag (e.g., QQBot, Telegram, Weixin)
tail -100 ~/.hermes/logs/gateway.log | grep -i "QQBot\|Telegram\|Weixin"

# Check gateway update timestamps
grep -E "state.*connected|state.*retrying" ~/.hermes/logs/gateway.log | tail -20

5. Safe Gateway Restart

Do NOT use hermes gateway restart — it kills the Telegram polling session too quickly. The new adapter starts before the old one's Telegram session expires, causing a permanent polling conflict.

Safe Sequence

# Step 1: Stop
systemctl --user stop hermes-gateway

# Step 2: Wait for Telegram's session lock to expire (30 seconds minimum)
sleep 30

# Step 3: Start fresh
systemctl --user start hermes-gateway

# Step 4: Wait for startup
sleep 10

# Step 5: Verify
cat ~/.hermes/gateway_state.json | python3 -m json.tool

6. Platform-Specific Diagnostics

QQ Bot

• Check QQ_APP_ID and QQ_CLIENT_SECRET in .env
• QQ WebSocket needs proxy env var WSS_PROXY or HTTPS_PROXY (set in _open_ws)
• Error 4914 = bot offline/sandbox (fatal, stop reconnecting)
• Error 4915 = bot banned (fatal, stop reconnecting)
• Verify adapter.py's connect() accepts is_reconnect

Telegram

• Only one bot token can poll at a time. Stale processes block new sessions.
• Disable bot privacy mode in @BotFather for group message reading (kick + re-add bot after changing)
• free_response_chats in config must be inside telegram.extra (NOT top-level)

Weixin

• Requires aiohttp and cryptography packages
• Check WX_APP_ID, WX_TOKEN, WX_AES_KEY in .env

Pitfalls

• 🔴 Gateway state file can be stale. If gateway_state.json shows an old error even after restart, the new gateway process may be still starting up. Check ps aux for PID changes and wait.
• 🔴 sleep 30 is not optional on Telegram restarts. Without it, the new adapter gets 409 Conflict and never connects.
• 🔴 is_reconnect in adapter but not in base class — the reverse check is also worth making when adding a new adapter: if the adapter uses is_reconnect internally but the base class doesn't pass it (hypothetical), it'll silently default to False during reconnect. The gateway always passes is_reconnect=True on watcher-initiated reconnects.
• Test mocks need the same signature. If a test creates a fake adapter with async def connect(self): (no is_reconnect), it won't catch the real runtime error.