Hermes-Skills/devops/telegram-message-forwarder/references/debugging-guide.md

Signal Forwarding Debugging Guide

Systematic approach to diagnose "信号没收到" (signal not received) complaints.

The Chain

Source Channel → TelegramForwarder → Target Group → Hermes → User DM
     (1)              (2)               (3)          (4)       (5)

Failure at ANY link breaks the chain. Debug from left to right.

Step 1: Forwarder Receiving Messages?

docker logs telegram-forwarder 2>&1 | grep "收到新消息.*实盘监控" | tail -5

• Has entries → Forwarder is receiving from source. Go to Step 2.
• No entries → Source channel not bound, or Telethon session expired. Check DB:

  docker exec telegram-forwarder python3 -c "
  import sqlite3
  conn = sqlite3.connect('/app/db/forward.db')
  for r in conn.execute('SELECT id, telegram_chat_id, name FROM chats'): print(r)
  conn.close()"
  

Step 2: Keyword Matching?

docker logs telegram-forwarder 2>&1 | grep -E "正则匹配成功|未匹配" | tail -5

• 正则匹配成功 → Match succeeded. Go to Step 3.
• 未匹配到普通白名单关键词 → Signal format changed, or regex doesn't match.
  • Check the actual signal text: docker logs telegram-forwarder 2>&1 | grep "内容:" | tail -5
  • Common issue: signal uses different brackets (e.g., [币种] vs 【币种】)
  • Common issue: multi-line signals — .* doesn't match \n, use [\s\S]* instead

Step 3: Forwarder Sending to Group?

docker logs telegram-forwarder 2>&1 | grep -E "已发送|成功获取目标" | tail -5

• 消息已发送到: XXX → Forward succeeded. Go to Step 4.
• No send logs after match → Check for errors:

  docker logs telegram-forwarder 2>&1 | grep -iE "error|invalid Peer|banned|forbidden" | tail -5
  

  • An invalid Peer was used → Group migrated to supergroup, ID changed
  • Bots can't send messages → Trying to send to a bot, not a group

Step 4: Hermes Receiving from Group?

FIRST: Check for competing gateway processes (most common hidden cause):

ps aux | grep "gateway\|openclaw" | grep -v grep

If you see an old openclaw ... gateway or duplicate hermes gateway process → KILL IT FIRST. Two processes sharing one bot token = permanent polling failure. See "Polling Conflict — Scenario A" below.

Then check polling status:

# Check for polling conflict
journalctl --user -u hermes-gateway --since "10 min ago" --no-pager | grep -i "conflict"
# If found → see "Polling Conflict" section below

# Check if "✓ telegram connected" appears in logs
journalctl --user -u hermes-gateway --since "5 min ago" --no-pager | grep "✓ telegram"
# If NOT found → adapter didn't connect (check for competing processes or conflict)

# Check gateway.log for inbound messages from the group
grep "inbound message" ~/.hermes/logs/gateway.log | grep "1003966251111" | tail -5

• Polling conflict found → Stop gateway, wait 30s, start fresh (see "Polling Conflict" section)
• Has entries → Hermes receives group messages. Go to Step 5.
• No entries, no conflict → Hermes can't see group messages. Check:
  1. Bot privacy mode: curl -s --proxy http://127.0.0.1:7890 "https://api.telegram.org/bot<TOKEN>/getMe" | grep can_read_all_group_messages
     • Must be true. If false, disable in @BotFather and kick+re-add bot to group.
  2. free_response_chats config (NOT free_response_channels):

     grep -A5 "extra:" ~/.hermes/config.yaml | grep free_response
     

     Must be under telegram.extra.free_response_chats, not top-level.
  3. Check if bot is actually in the group.

Step 5: Hermes Processing and Responding?

grep "1003966251111" ~/.hermes/logs/gateway.log | grep -E "response ready|Suppressing" | tail -5

• response ready → Hermes processed and responded. Check delivery.
• inbound but no response → Agent failed or timed out. Check errors.log:

  tail -20 ~/.hermes/logs/errors.log
  

Quick One-Liner Diagnostic

echo "=== Forwarder ===" && \
docker logs telegram-forwarder 2>&1 | grep -cE "正则匹配成功|已发送" && \
echo "=== Hermes ===" && \
grep -c "inbound.*1003966251111" ~/.hermes/logs/gateway.log 2>/dev/null && \
echo "=== Errors ===" && \
tail -3 ~/.hermes/logs/errors.log 2>/dev/null

Log Pattern Reference

Pattern	Meaning	Where
收到新消息 来自聊天: XXX	Forwarder received message	forwarder logs
正则匹配成功	Keyword regex matched	forwarder logs
未匹配到普通白名单关键词	Keyword didn't match	forwarder logs
所有白名单条件都满足，允许转发	Forwarding allowed	forwarder logs
消息已发送到: XXX	Message sent to target	forwarder logs
过滤器识别到机器人消息，忽略处理	Bot message filtered (normal)	forwarder logs
inbound message: platform=telegram chat=XXX	Hermes received message	gateway.log
response ready: platform=telegram chat=XXX	Hermes responded	gateway.log
Suppressing normal final send	Response already delivered via stream	gateway.log

Common Scenarios

"昨天能收到，今天收不到"

• Check if source channel stopped sending (forwarder logs)
• Check if Telethon session expired (container restart fixes this)
• Check if signal format changed

"有信号进来但没收到你的消息"

• Could be timing: signal hasn't arrived yet, or arrived but user expected Hermes response
• Quick check: docker logs telegram-forwarder 2>&1 | grep "收到新消息.*实盘监控" | tail -1 — compare timestamp to current time. If >30min stale, the signal may not have reached the forwarder.
• If forwarder received and forwarded but no Hermes response: check tail -20 ~/.hermes/logs/gateway.log | grep inbound — if no recent group messages, the bot may not be receiving (privacy mode).
• Bot message filtering: Hermes may ignore messages sent by other bots. Always test with a human account, not the forwarder bot.
• ⚠️ Check for competing gateway processes: ps aux | grep "gateway\|openclaw" | grep -v grep — if an old OpenClaw Gateway or duplicate Hermes process is running, it steals the bot token polling session and Hermes NEVER receives any group messages. This was the actual root cause for a user whose group messages had NEVER worked for over a week.

"信号在频道里能看到，但没转发到群"

• Step 2 failure: keyword mismatch
• Check actual signal text vs regex pattern

"信号转发到群了，但Hermes没回复"

• Step 4 failure: privacy mode or config
• Most common: bot privacy mode is ON

"重启了 gateway 还是不处理群消息"（Polling Conflict — 两种原因）

This is the #1 hidden cause. There are TWO distinct polling conflict scenarios:

Scenario A: Competing gateway processes (most dangerous)

Another process is using the same bot token for getUpdates. This causes PERMANENT failure — no amount of restarting Hermes will help until the other process is killed.

Diagnosis:

# 1. Check for competing processes
ps aux | grep "gateway\|openclaw" | grep -v grep
# Look for: node .../openclaw/dist/index.js gateway --port 18789
# Or: another hermes gateway run

# 2. Test getUpdates directly (bypass adapter)
python3 -c "
import json, urllib.request
with open('/home/openclaw/.hermes/.env') as f:
    for line in f:
        if line.startswith('TELEGRAM_BOT_TOKEN'):
            token = line.strip().split('=',1)[1]
            break
proxy = urllib.request.ProxyHandler({'http': 'http://127.0.0.1:7890', 'https': 'http://127.0.0.1:7890'})
opener = urllib.request.build_opener(proxy)
try:
    req = urllib.request.Request(f'https://api.telegram.org/bot{token}/getUpdates?timeout=1')
    resp = opener.open(req, timeout=5)
    data = json.loads(resp.read())
    print(f'getUpdates: {len(data.get(\"result\", []))} updates (no conflict = nobody polling)')
except Exception as e:
    if '409' in str(e):
        print('409 Conflict — adapter IS polling (normal)')
    else:
        print(f'Error: {e}')
"

Key diagnostic signal:

• getUpdates returns 0 without 409 conflict → NOBODY is polling → competing process consumed and dropped updates
• getUpdates returns 409 → adapter IS polling (conflict is expected, this is normal)
• getUpdates returns updates → adapter is NOT polling AND competing process dropped off → restart Hermes

Fix: Kill the stale process FIRST, then restart Hermes:

kill <stale_pid>
systemctl --user restart hermes-gateway

Verify: journalctl --user -u hermes-gateway --since "1 min ago" | grep "✓ telegram connected" — must appear after restart.

Scenario B: hermes gateway restart race condition

The restart command sends SIGTERM and immediately starts a new process. Telegram's servers keep the old polling session alive for ~30 seconds. The new process tries to poll → conflict → retry loop.

Diagnosis:

journalctl --user -u hermes-gateway --since "5 min ago" --no-pager | grep -i "conflict"

Fix:

systemctl --user stop hermes-gateway
sleep 30  # critical: wait for Telegram session to expire
systemctl --user start hermes-gateway

How to tell which scenario

# Check how long the gateway has been running
systemctl --user status hermes-gateway | grep "Active:"
# If running for days/weeks but group messages NEVER worked → Scenario A
# If just restarted and immediately getting conflicts → Scenario B

# Check for stale processes
ps aux | grep "gateway" | grep -v grep | grep -v $(pgrep -f "hermes_cli.main gateway")
# If you see old processes → Scenario A

Gateway Logs (Binary Format — Use strings!)

Gateway logs at ~/.hermes/logs/gateway.log are binary format. Use strings to extract:

strings ~/.hermes/logs/gateway.log | tail -30
strings ~/.hermes/logs/gateway.log | grep "inbound message" | tail -10
strings ~/.hermes/logs/gateway.log | grep "<group_chat_id>" | tail -10
strings ~/.hermes/logs/gateway.log | grep "Connected to"

⚠️ Testing: Use Human Account

Never test with bot-sent messages. Bots cannot see their own messages via getUpdates. Always use a human Telegram account to send test messages to the group or source channel.

Agent-Side Filtering Architecture

When the user says "收所有的消息，在处理消息这边来处理规则过滤":

1. Forwarder: .* whitelist (pass everything)
2. Agent channel_prompts: classify messages as A(signal)/B(confirm)/C(close)/D(ignore)
3. Only process A/B/C, silently ignore D

Benefits: source format changes only require updating the agent prompt, not the forwarder database.