Hermes-Skills/devops/telegram-message-forwarder/references/debugging-guide.md

# Signal Forwarding Debugging Guide

Systematic approach to diagnose "信号没收到" (signal not received) complaints.

## The Chain

```
Source Channel → TelegramForwarder → Target Group → Hermes → User DM
     (1)              (2)               (3)          (4)       (5)
```

Failure at ANY link breaks the chain. Debug from left to right.

## Step 1: Forwarder Receiving Messages?

```bash
docker logs telegram-forwarder 2>&1 | grep "收到新消息.*实盘监控" | tail -5
```

- **Has entries** → Forwarder is receiving from source. Go to Step 2.
- **No entries** → Source channel not bound, or Telethon session expired. Check DB:
  ```bash
  docker exec telegram-forwarder python3 -c "
  import sqlite3
  conn = sqlite3.connect('/app/db/forward.db')
  for r in conn.execute('SELECT id, telegram_chat_id, name FROM chats'): print(r)
  conn.close()"
  ```

## Step 2: Keyword Matching?

```bash
docker logs telegram-forwarder 2>&1 | grep -E "正则匹配成功|未匹配" | tail -5
```

- `正则匹配成功` → Match succeeded. Go to Step 3.
- `未匹配到普通白名单关键词` → Signal format changed, or regex doesn't match.
  - Check the actual signal text: `docker logs telegram-forwarder 2>&1 | grep "内容:" | tail -5`
  - Common issue: signal uses different brackets (e.g., `[币种]` vs `【币种】`)
  - Common issue: multi-line signals — `.*` doesn't match `\n`, use `[\s\S]*` instead

## Step 3: Forwarder Sending to Group?

```bash
docker logs telegram-forwarder 2>&1 | grep -E "已发送|成功获取目标" | tail -5
```

- `消息已发送到: XXX` → Forward succeeded. Go to Step 4.
- No send logs after match → Check for errors:
  ```bash
  docker logs telegram-forwarder 2>&1 | grep -iE "error|invalid Peer|banned|forbidden" | tail -5
  ```
  - `An invalid Peer was used` → Group migrated to supergroup, ID changed
  - `Bots can't send messages` → Trying to send to a bot, not a group

## Step 4: Hermes Receiving from Group?

**FIRST: Check for competing gateway processes** (most common hidden cause):
```bash
ps aux | grep "gateway\|openclaw" | grep -v grep
```
If you see an old `openclaw ... gateway` or duplicate `hermes gateway` process → **KILL IT FIRST**. Two processes sharing one bot token = permanent polling failure. See "Polling Conflict — Scenario A" below.

Then check polling status:
```bash
# Check for polling conflict
journalctl --user -u hermes-gateway --since "10 min ago" --no-pager | grep -i "conflict"
# If found → see "Polling Conflict" section below

# Check if "✓ telegram connected" appears in logs
journalctl --user -u hermes-gateway --since "5 min ago" --no-pager | grep "✓ telegram"
# If NOT found → adapter didn't connect (check for competing processes or conflict)

# Check gateway.log for inbound messages from the group
grep "inbound message" ~/.hermes/logs/gateway.log | grep "1003966251111" | tail -5
```

- **Polling conflict found** → Stop gateway, wait 30s, start fresh (see "Polling Conflict" section)
- **Has entries** → Hermes receives group messages. Go to Step 5.
- **No entries, no conflict** → Hermes can't see group messages. Check:
  1. Bot privacy mode: `curl -s --proxy http://127.0.0.1:7890 "https://api.telegram.org/bot<TOKEN>/getMe" | grep can_read_all_group_messages`
     - Must be `true`. If `false`, disable in @BotFather and **kick+re-add** bot to group.
  2. `free_response_chats` config (NOT `free_response_channels`):
     ```bash
     grep -A5 "extra:" ~/.hermes/config.yaml | grep free_response
     ```
     Must be under `telegram.extra.free_response_chats`, not top-level.
  3. Check if bot is actually in the group.

## Step 5: Hermes Processing and Responding?

```bash
grep "1003966251111" ~/.hermes/logs/gateway.log | grep -E "response ready|Suppressing" | tail -5
```

- `response ready` → Hermes processed and responded. Check delivery.
- `inbound` but no `response` → Agent failed or timed out. Check errors.log:
  ```bash
  tail -20 ~/.hermes/logs/errors.log
  ```

## Quick One-Liner Diagnostic

```bash
echo "=== Forwarder ===" && \
docker logs telegram-forwarder 2>&1 | grep -cE "正则匹配成功|已发送" && \
echo "=== Hermes ===" && \
grep -c "inbound.*1003966251111" ~/.hermes/logs/gateway.log 2>/dev/null && \
echo "=== Errors ===" && \
tail -3 ~/.hermes/logs/errors.log 2>/dev/null
```

## Log Pattern Reference

| Pattern | Meaning | Where |
|---------|---------|-------|
| `收到新消息 来自聊天: XXX` | Forwarder received message | forwarder logs |
| `正则匹配成功` | Keyword regex matched | forwarder logs |
| `未匹配到普通白名单关键词` | Keyword didn't match | forwarder logs |
| `所有白名单条件都满足，允许转发` | Forwarding allowed | forwarder logs |
| `消息已发送到: XXX` | Message sent to target | forwarder logs |
| `过滤器识别到机器人消息，忽略处理` | Bot message filtered (normal) | forwarder logs |
| `inbound message: platform=telegram chat=XXX` | Hermes received message | gateway.log |
| `response ready: platform=telegram chat=XXX` | Hermes responded | gateway.log |
| `Suppressing normal final send` | Response already delivered via stream | gateway.log |

## Common Scenarios

### "昨天能收到，今天收不到"
- Check if source channel stopped sending (forwarder logs)
- Check if Telethon session expired (container restart fixes this)
- Check if signal format changed

### "有信号进来但没收到你的消息"
- Could be timing: signal hasn't arrived yet, or arrived but user expected Hermes response
- Quick check: `docker logs telegram-forwarder 2>&1 | grep "收到新消息.*实盘监控" | tail -1` — compare timestamp to current time. If >30min stale, the signal may not have reached the forwarder.
- If forwarder received and forwarded but no Hermes response: check `tail -20 ~/.hermes/logs/gateway.log | grep inbound` — if no recent group messages, the bot may not be receiving (privacy mode).
- **Bot message filtering**: Hermes may ignore messages sent by other bots. Always test with a human account, not the forwarder bot.
- **⚠️ Check for competing gateway processes**: `ps aux | grep "gateway\|openclaw" | grep -v grep` — if an old OpenClaw Gateway or duplicate Hermes process is running, it steals the bot token polling session and Hermes NEVER receives any group messages. This was the actual root cause for a user whose group messages had NEVER worked for over a week.

### "信号在频道里能看到，但没转发到群"
- Step 2 failure: keyword mismatch
- Check actual signal text vs regex pattern

### "信号转发到群了，但Hermes没回复"
- Step 4 failure: privacy mode or config
- Most common: bot privacy mode is ON

### "重启了 gateway 还是不处理群消息"（Polling Conflict — 两种原因）

**This is the #1 hidden cause.** There are TWO distinct polling conflict scenarios:

#### Scenario A: Competing gateway processes (most dangerous)

Another process is using the same bot token for `getUpdates`. This causes PERMANENT failure — no amount of restarting Hermes will help until the other process is killed.

**Diagnosis**:
```bash
# 1. Check for competing processes
ps aux | grep "gateway\|openclaw" | grep -v grep
# Look for: node .../openclaw/dist/index.js gateway --port 18789
# Or: another hermes gateway run

# 2. Test getUpdates directly (bypass adapter)
python3 -c "
import json, urllib.request
with open('/home/openclaw/.hermes/.env') as f:
    for line in f:
        if line.startswith('TELEGRAM_BOT_TOKEN'):
            token = line.strip().split('=',1)[1]
            break
proxy = urllib.request.ProxyHandler({'http': 'http://127.0.0.1:7890', 'https': 'http://127.0.0.1:7890'})
opener = urllib.request.build_opener(proxy)
try:
    req = urllib.request.Request(f'https://api.telegram.org/bot{token}/getUpdates?timeout=1')
    resp = opener.open(req, timeout=5)
    data = json.loads(resp.read())
    print(f'getUpdates: {len(data.get(\"result\", []))} updates (no conflict = nobody polling)')
except Exception as e:
    if '409' in str(e):
        print('409 Conflict — adapter IS polling (normal)')
    else:
        print(f'Error: {e}')
"
```

**Key diagnostic signal**:
- `getUpdates` returns 0 without 409 conflict → **NOBODY is polling** → competing process consumed and dropped updates
- `getUpdates` returns 409 → adapter IS polling (conflict is expected, this is normal)
- `getUpdates` returns updates → adapter is NOT polling AND competing process dropped off → restart Hermes

**Fix**: Kill the stale process FIRST, then restart Hermes:
```bash
kill <stale_pid>
systemctl --user restart hermes-gateway
```

**Verify**: `journalctl --user -u hermes-gateway --since "1 min ago" | grep "✓ telegram connected"` — must appear after restart.

#### Scenario B: `hermes gateway restart` race condition

The restart command sends SIGTERM and immediately starts a new process. Telegram's servers keep the old polling session alive for ~30 seconds. The new process tries to poll → conflict → retry loop.

**Diagnosis**:
```bash
journalctl --user -u hermes-gateway --since "5 min ago" --no-pager | grep -i "conflict"
```

**Fix**:
```bash
systemctl --user stop hermes-gateway
sleep 30  # critical: wait for Telegram session to expire
systemctl --user start hermes-gateway
```

#### How to tell which scenario

```bash
# Check how long the gateway has been running
systemctl --user status hermes-gateway | grep "Active:"
# If running for days/weeks but group messages NEVER worked → Scenario A
# If just restarted and immediately getting conflicts → Scenario B

# Check for stale processes
ps aux | grep "gateway" | grep -v grep | grep -v $(pgrep -f "hermes_cli.main gateway")
# If you see old processes → Scenario A
```

## Gateway Logs (Binary Format — Use strings!)

Gateway logs at `~/.hermes/logs/gateway.log` are **binary format**. Use `strings` to extract:

```bash
strings ~/.hermes/logs/gateway.log | tail -30
strings ~/.hermes/logs/gateway.log | grep "inbound message" | tail -10
strings ~/.hermes/logs/gateway.log | grep "<group_chat_id>" | tail -10
strings ~/.hermes/logs/gateway.log | grep "Connected to"
```

## ⚠️ Testing: Use Human Account

**Never test with bot-sent messages.** Bots cannot see their own messages via getUpdates. Always use a human Telegram account to send test messages to the group or source channel.

## Agent-Side Filtering Architecture

When the user says "收所有的消息，在处理消息这边来处理规则过滤":
1. Forwarder: `.*` whitelist (pass everything)
2. Agent channel_prompts: classify messages as A(signal)/B(confirm)/C(close)/D(ignore)
3. Only process A/B/C, silently ignore D

Benefits: source format changes only require updating the agent prompt, not the forwarder database.