Hermes-Skills/devops/cron-job-management/SKILL.md

---
name: cron-job-management
triggers:
  - cron job transient error broken pipe
  - cron output logs debugging
  - cron agent fallback prompts
description: >
  Manage Hermes cron jobs: script deployment, path verification, environment
  variable setup, and debugging. Covers the key differences between no_agent
  scripts and agent-based cron tasks, plus the common .bashrc guard pitfall
  that silently breaks env var loading for cron-based scripts.
trigger:
  - 检查定时任务
  - cron job broken
  - script path wrong
  - environment variable not found in cron
  - 定时任务脚本路径
  - cron env vars
  - 检查cron脚本
  - cron script check
  - cron job verification
---

# Cron Job Management

Hermes cron jobs run in minimal shell environments — no interactive bash, no
`.bashrc`/`.bash_profile` sourcing, no inherited PATH or env vars. This skill
covers how to verify, deploy, and debug them.

---

## 1. Checking All Cron Jobs

```bash
hermes cron list                    # List all jobs (use cronjob action='list' in tool mode)
hermes cron logs <job_id>           # View last run logs
hermes cron run <job_id>            # Manual one-shot run for testing
```

### Systematic Script Path Verification

When asked to verify cron job scripts, follow this chain:

```
cronjob list              → get ALL jobs with their script/skill fields
→ check each script path  → stat ~/.hermes/scripts/<name>
→ check shell wrappers    → are they cd + python3? Do they use absolute paths?
→ check Python imports    → can the modules actually import?
→ check data files        → e.g., dca_positions.json, config files
→ factor mining skills    → verify skill path + Python import paths
```

**Batch verification command**:
```bash
cd ~/.hermes/scripts && for f in script1.py script2.sh ...; do
  [ -f "$f" ] && echo "✅ $f" || echo "❌ $f"
done
```

---

## 2. 🔴 Critical: `.bashrc` Non-Interactive Guard Blocks Cron

**The single most common cron env var failure.** Most Linux distros ship with:

```bash
# ~/.bashrc
# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac
```

**Impact**: When a cron job runs, bash starts as a non-interactive shell →
`.bashrc` returns immediately at the `return;` statement → **none of the
`export VAR=...` lines below the guard are ever executed**.

**Signs this is happening**:
- Manual test (`source ~/.bashrc && python script.py`) works fine
- Cron job fails with `missing environment variable: XXX`
- Cron job fails with auth/permission errors (LongPort 401003, etc.)
- The `.bashrc` file has the `case $- in` guard

**Diagnosis**:
```bash
# Test: does .bashrc actually export the vars?
bash -c "source ~/.bashrc && env | grep LONGPORT"
# If empty → guard is blocking, OR vars are below interactive-only sections
```

**Fixes (pick one)**:

| Approach | Method | Pros | Cons |
|----------|--------|------|------|
| **Extract to separate file** | Move exports to `~/.env` or `~/.config/hermes/env` | Clean, portable | One-time move |
| **Use /etc/environment** | Add `VAR=val` lines there | System-wide, read by all users | Needs sudo |
| **Inline in script** | Add `source ~/.bashrc` after the guard | Quick fix | Repetitive |
| **Patch .bashrc** | Move exports ABOVE the guard | Simplest | Only works for interactive sourcing |

**Recommended**: Create `~/.hermes/env` (referenced by cron scripts) or use
`/etc/environment` for system-wide availability.

---

## 3. Two Cron Job Modes

### `no_agent: True` — Pure Script Mode

The script IS the job. The scheduler runs it as a subprocess, captures stdout,
and delivers it verbatim.

```yaml
script: "hk_intraday_scanner.py"   # relative to ~/.hermes/scripts/
no_agent: true
```

**Environment rules**:
- Runs in a **clean shell** — no `.bashrc`, no `.profile`
- Only sees: `PATH=/usr/bin:/bin`, `HOME=/home/user`, `USER`, `HERMES_CONFIG_DIR`
- Must self-bootstrap: parse `.bashrc` manually (Python), or source env file
- Empty stdout = silent (no notification sent)
- Non-zero exit → error alert delivered

**Common pattern (shell wrapper)**:
```bash
#!/bin/bash
cd /home/user/.hermes/scripts
python3 my_script.py "$@"
```

### `no_agent: False` — Agent Mode

The cron prompt is fed to a Hermes agent session each tick. The agent can use
tools (terminal, web, file) to execute tasks.

- Inherits Hermes's own env vars (those from the gateway process)
- Can `source ~/.bashrc` in terminal tool calls
- Can use `enabled_toolsets` to limit tool availability (reduces token cost)

---

## 4. Shell Wrapper Pattern (for no_agent scripts)

All `no_agent: True` scripts under `~/.hermes/scripts/` follow this pattern:

```bash
#!/bin/bash
cd /home/openclaw/.hermes/scripts     # absolute path
python3 actual_script.py [args]       # run the real script
```

**Why this pattern**:
- `cd` ensures relative imports and file paths work
- Absolute `cd` path avoids PWD ambiguity
- Shell wrapper is stateless and re-runnable

---

## 5. Python Script: Manual Env Load Pattern

For `no_agent` Python scripts that need LongPort credentials:

```python
import os, re

# Parse ~/.bashrc manually (cron can't source it)
config = {}
with open(os.path.expanduser('~/.bashrc')) as f:
    for line in f:
        if line.startswith('export LONGBRIDGE_') or line.startswith('export LONGPORT_'):
            key, value = line.strip().split('=', 1)
            config[key.replace('export ', '')] = value

os.environ['LONGPORT_APP_KEY'] = config.get('LONGBRIDGE_APP_KEY', '')
os.environ['LONGPORT_APP_SECRET'] = config.get('LONGBRIDGE_APP_SECRET', '')
os.environ['LONGPORT_ACCESS_TOKEN'] = config.get('LONGBRIDGE_ACCESS_TOKEN', '')
```

**⚠️ Pitfall**: This only works if `.bashrc` values don't reference `${OTHER_VAR}`
syntax. If they do, use `re.sub(r'\$\{(\w+)\}', ...)` to resolve them.

---

## 6. Common Pitfalls

### 🔴 `.bashrc` Guard (most frequent — see §2)
Symptom: "works when I run it, fails in cron"

### 🔴 Script uses `os.environ` directly
Python scripts that do `os.environ['VAR']` will crash if the cron shell hasn't
exported the variable. Always use `.get('VAR', '')` with a fallback.

### 🔴 Relative paths in no_agent scripts
The cron scheduler's CWD may not be what you expect. Always use absolute paths
or `cd` to `~/.hermes/scripts/` first.

### 🔴 LongPort token expiry (401003 vs 401004)
LongPort access tokens expire every ~180 days. All scripts using LongPort API
simultaneously failing with the same error code points to token expiry.

**Two different error codes — different root causes:**

| Error | Code | Meaning | Action |
|-------|------|---------|--------|
| `token expired` | 401003 | Token was valid but expired | Go to [LongPort console](https://open.longportapp.com/) → Personal Access Token → regenerate |
| `token invalid` / missing env | 401004 | Bashrc has truncated placeholder (e.g. `m_eyJh...jb-k` with literal `...`) | Restore full token from `.env` |

**Diagnosis**: Run any script that uses LongPort manually:
```bash
cd ~/.hermes/scripts && bash scan_hk.sh
# If 401003 → needs refresh at LongPort console
# If 401004 or "missing env" → check .bashrc for truncated tokens
```

```python
from longport import openapi
try:
    ctx = openapi.QuoteContext(config=openapi.Config.from_env())
    ctx.quote(['700.HK'])
except openapi.OpenApiException as e:
    code = str(e)
    if '401003' in code:
        print("Token expired — needs refresh via LongPort developer console")
    elif '401004' in code:
        print("Token invalid/truncated — check .bashrc for placeholder values")
```

### 🔴 QQBot WebSocket session timeout (code=4009)
Symptom: Delivery error says `Not connected` / `All connection attempts failed`,
but the script itself ran fine. Gateway logs show:
```
[QQBot:102921794] WebSocket closed: code=4009 reason=Session timed out
```
Tencent's QQ Bot platform terminates idle WebSocket sessions after ~30 min.
The gateway auto-reconnects, but there's a window where delivery fails.

**Diagnosis**: Check gateway logs:
```bash
journalctl --user -u hermes-gateway --no-pager -n 50 | grep -iE "qqbot|websocket|closed"
```

**Mitigation**: The session timeout is server-side (Tencent), not configurable.
The main fix is ensuring `Not connected` delivery errors are NOT confused with
script failures — check `last_delivery_error` in `cronjob list` output first.

### 🔴 Delivery failures vs Script failures
Distinguish between:
- **Script failure** (`last_status=error`, no `last_delivery_error`): non-zero
  exit, stderr output occurred. Fix the script. **Diagnosis**: run the script
  manually via `bash ~/.hermes/scripts/<script>` or `python3 <script.py>`.
- **Delivery failure** (`last_status=error`, `last_delivery_error` present with
  DNS/adapter error): script succeeded, but output couldn't be sent. Fix the
  delivery pipeline, script is fine.
  - `Temporary failure in name resolution` → DNS issue (check `nslookup` / `dig`)
  - `Not connected` / `All connection attempts failed` → QQBot disconnected
    (see QQBot section above)
  - `Connection refused` → gateway/adapter not running

---

## 7. Transient Cron Errors: Broken Pipe

### What It Means

`[Errno 32] Broken pipe` in a cron delivery error does NOT mean the Python script crashed. It signals a pipe-level abort — typically the agent tried a blocked tool (Tirith-blocked terminal/execute_code) and the pipe was torn down before output could be written.

### Diagnosis

1. Re-run: `cronjob action=run job_id=<id>` — if it succeeds, it was transient
2. For agent-based cron jobs: check if the prompt tells the agent to run Python. Rewrite it to skip execution and use pre-computed data fallback instead.
3. For no_agent scripts: `bash ~/.hermes/scripts/<script>` manually

### Reading Cron Output Files

Output logs at `~/.hermes/cron/output/<job_id>/<timestamp>.md` contain the full prompt + skill injection + agent response. For no_agent scripts, they contain stdout.

**Common transient causes:**
- Tirith block on Python execution in cron (most common)
- CLS/Sina browser rate limits (policy news tasks)
- LongPort API rate limiting (429002) — add retry
- DNS/network blip

## 8. Quick Debugging Checklist

1. `cronjob action=list` → note all last_status values
2. For error jobs: check last_delivery_error field
3. For no_agent scripts: manually `bash ~/.hermes/scripts/<script>`
4. For agent scripts: `cronjob action=run job_id=<id>` to test
5. **If re-run fixes it** → transient error (blocked tool / network glitch)
6. **If re-run fails** → check output: `~/.hermes/cron/output/<job_id>/`
7. Check env loading: `bash -c "source ~/.bashrc && env | grep KEY"`
8. Check QQBot/Telegram DNS: `nslookup api.telegram.org`