---
name: task-reliability
description: Prevent tasks from silently stalling. Covers delegation patterns, failure recovery, watchdog monitoring, and session health.
trigger: "When starting any task that takes more than a few tool calls, or when a background process/delegate_task might fail"
---

# Task Reliability — Anti-Stall Protocol

## Core Problem
Tasks silently stop because:
1. Subagents via `hermes chat -q` fail with exit code 1 and nobody notices
2. `delegate_task` errors are returned but not acted upon
3. Background processes crash without notification
4. Session context compresses and loses track of in-progress items
5. No watchdog monitors task health between sessions

## Delegation Rules

### NEVER use `hermes chat -q` for subagents
This CLI command returns exit code 1 frequently with no useful error info.
**USE `delegate_task` instead:**
```
delegate_task(goal="任务描述", context="需要的所有背景信息", toolsets=["terminal", "file", "web"])
```

### Delegate Task Error Handling
When `delegate_task` returns:
- `status: "error"` → **MUST** retry once with simplified goal, or take over manually
- `status: "interrupted"` → Parent was cancelled, work is lost. Log and report to user
- `status: "success"` → Verify the output before telling user it's done

### Batch Delegation (parallel tasks)

When you have 3+ independent subtasks, launch them in ONE `delegate_task` call:

```python
delegate_task(tasks=[
    {"goal": "Task A description", "context": "full context for A", "toolsets": [...]},
    {"goal": "Task B description", "context": "full context for B", "toolsets": [...]},
    {"goal": "Task C description", "context": "full context for C", "toolsets": [...]},
])
```

**Results format:**
```json
{
  "results": [
    {"task_index": 0, "status": "completed", "summary": "..."},
    {"task_index": 1, "status": "completed", "summary": "..."},
    {"task_index": 2, "status": "completed", "summary": "..."}
  ],
  "total_duration_seconds": 461.08
}
```

**Rules:**
- Max 3 concurrent subagents per batch (hard limit)
- Each task object must be fully self-contained
- Check EACH result individually — some may succeed while others fail
- On mixed success/failure: log failed ones, manually complete what failed

## Streaming Failure Recovery

When `delegate_task` subagents return:
```
"status": "completed" but "summary": "API call failed after 3 retries: An error occurred during streaming"
```

This means the subagent's API call to the LLM provider failed — the subagent DID run but couldn't complete. **Recovery:**
1. Log the failure with details
2. Do NOT retry with `delegate_task` again (same provider, same failure likely)
3. **Take over manually** — do the work yourself in the current session
4. This is almost always a provider-side streaming issue, not a prompt issue

## Subagent Prompt Design
Each delegated task prompt MUST be:
1. Self-contained (no references to "earlier in the conversation")
2. Include exact file paths, error messages, and constraints
3. Specify expected output format (e.g., "Return the absolute path of the file you created")
4. Have a clear success criterion

## Cron Agent Failures (Critical)

### One-shot cron workers are unreliable
Cron agents running in isolated sessions frequently fail silently:
- **2/3 cron workers never executed** in a 2026-06-02 test (scheduled but no execution record)
- **1/3 executed but produced no output** (ran but didn't write results)
- Root cause unclear: possibly scheduler timing, session isolation, or gateway state

### Cron agents cannot call `delegate_task`
Cron agents run in isolated sessions that **cannot spawn sub-agents** via `delegate_task`. The isolation prevents the recursive delegation pattern.

**Workaround:** Use cron agents only for self-contained tasks (terminal commands, file operations). For multi-step work requiring delegation, use the main session's `delegate_task` or Kanban.

### `delegate_task` gets interrupted by user input
When `delegate_task` is running and the user sends a message, the subagent is **killed immediately** and all progress is lost.

**Mitigation:**
- Use Kanban for long-running tasks (isolated from user input)
- Or warn user: "Task X is running, please wait"
- Never rely on `delegate_task` for work > 2 minutes

### Cron `deliver=origin` fails on CLI
Cron jobs with `deliver=origin` fail with "no delivery target resolved for deliver=origin" when running in CLI-only sessions (no messaging gateway).

**Fix:** Use `deliver=local` for CLI sessions. Results are visible via `hermes cron list` and session search.

## Background Process Monitoring

### Pattern for long-running commands
```python
# Start with notify_on_complete=True
terminal(command="...", background=True, notify_on_complete=True)
# System auto-notifies on completion

# For servers that never exit:
terminal(command="...", background=True)
# Then verify: terminal("curl http://localhost:8080/health")
```

### Process health check
After starting any background process:
1. Wait 10-30 seconds (or appropriate startup time)
2. Check: `process(action="poll")` for output
3. Verify: health endpoint, log signal, or expected behavior
4. If no output after 2x expected startup time → process likely crashed

## User Input Logging (Non-Negotiable)

**Every user message must be logged** to `~/input-logs/YYYYMMDD_inputs.log`. This is not optional — it creates a permanent record of all user requests.

**CRITICAL: User input must NEVER interrupt a running task.** If a background process, delegate_task, or long-running command is active:
1. Log the input immediately
2. Note the task switch in the current task log
3. Decide whether to interleave the new request or queue it
4. The running task continues — do NOT kill or cancel it

```bash
python3 ~/.hermes/scripts/log_helper.py input "user message text"
```

Full logging system details are in the `task-and-input-logging` skill.

## Task State Tracking

### Before starting a task
1. Create task log: `log_helper.py start "task_name" "description"`
2. Add to todo list with status tracking
3. Note expected completion criteria

### During task execution
1. Append progress notes to task log: `log_helper.py update <path> running "progress note"`
2. If switching to a different task, log why

### On task completion
1. Update status: `log_helper.py update <path> completed "what was done"`
2. Verify the output exists and is correct
3. Remove from todo list or mark completed

### On task failure
1. **MUST** log: `log_helper.py update <path> failed "error details"`
2. **MUST** either retry (max 2 attempts) OR explain to user why it failed
3. **NEVER** silently move on to another topic

## Task Status Sync

When the user asks for current task status (or periodically syncs), distinguish between **historical logs** (what was done before) and **current state** (what happened today / what's happening now). The user wants real-time current state, not yesterday's summary.

### Multi-repo work reconstruction (today's commits)

Reusable script: `~/.hermes/skills/task-reliability/scripts/multi-repo-scan.sh`
- No args: scan commits from today
- Pass a number to scan N days back

Inline equivalent:
```bash
cd ~/repos && for d in $(find . -maxdepth 1 -type d); do
  if [ -d "$d/.git" ]; then
    commits=$(git -C "$d" log --since="$(date +%Y-%m-%dT00:00)" --oneline --all 2>/dev/null)
    if [ -n "$commits" ]; then
      echo "=== $d ==="; echo "$commits"; echo
    fi
  fi
done
```

### Status sync checklist (all four sources):
1. **Commits from today**: multi-repo scan above
2. **Dirty files**: `git status --short` per repo (uncommitted changes, untracked files)
3. **Unpushed commits**: `git log origin/main..HEAD --oneline` per repo
4. **Cron jobs**: `cronjob list` — check last_status, last_delivery_error. Note: `deliver=origin` fails on CLI-only sessions with "no delivery target resolved" — this is expected, not a real error. Only `deliver=local` jobs reliably work in CLI mode. **Scheduler-death signature**: `last_run_at=null` combined with `next_run_at` in the PAST means the job was NEVER executed (scheduler/gateway down), not that a run failed — flag it, the job's work has silently never happened.
5. **Active processes**: `process(action="list")`
6. **Todo list**: `todo list`

### What to present
- Table format: repo | commits | push status | dirty files
- Flag anything unpushed, dirty, or with cron job errors
- Keep it terse — bullet points and tables, not prose

## Session Health

### At session start
1. Check: `process(action="list")` — any zombie processes?
2. Check: recent task logs in `~/task-logs/` — any stuck in "running" for hours?
3. If session_search works, check recent sessions for unfinished tasks
4. Check for **dead agent sessions** (provider crash-loops): `ls -lt ~/.hermes/sessions/request_dump_* | head`. Dumps with `reason: max_retries_exhausted` / "Provider returned an empty stream" mean a session died repeatedly and the user's task in it was never executed. Confirm via `session_search(session_id=...)`: user messages but ZERO assistant messages = dead session. Report it and re-run the orphaned task; recommend a model fallback chain.

### On context compression
When `[Your active task list was preserved...]` appears:
1. Check each in_progress task — is it actually running or stalled?
2. If stalled for >30 minutes without progress → flag for user or retry

## Watchdog Cron Job

A cron job runs every 30 minutes to:
1. Scan `~/task-logs/` for tasks stuck in "running" state > 2 hours
2. Check for background processes that exited unexpectedly
3. Report findings to user

Set up:
```
cronjob(action="create",
  name="task-watchdog",
  schedule="every 30m",
  prompt="Scan ~/task-logs/ for tasks stuck in 'running' state for more than 2 hours. Check process list for crashed background processes. Report any issues. If nothing wrong, stay silent.",
  script="check_stuck_tasks.sh",
  no_agent=True,
  deliver="origin")
```

## Recovery Pattern

When discovering a stalled task:
1. Log the discovery
2. Assess: can I resume from where it left off?
3. If yes → resume and document the gap
4. If no → explain what was lost and propose restart
5. Never pretend the stall didn't happen

## Multi-Task Switching Rules

Users frequently switch between multiple concurrent tasks. This is normal and expected.

**When switching tasks:**
1. Note the switch in the current task log: `[time] User switched to task Y, paused here`
2. Resume the new task with its own log entry
3. On session start, list all `running` tasks so the user sees what's pending

**When a task fails (not just paused):**
1. Log the failure: `log_helper.py update <path> failed "error details"`
2. Report to user immediately — the failed task stays visible in INDEX.md
3. Do NOT silently drop it. It stays in "failed" state until user decides: retry, abandon, or fix
4. User may switch to other tasks while a failed task sits — that's fine, it's tracked in INDEX.md

**Key principle: visibility over serialization.**
Tasks can be interleaved, paused, resumed, or abandoned — but every task's state must be visible in `~/task-logs/INDEX.md`. The user should never have to ask "what happened to X?"

## Log Helper Script

**File**: `scripts/log_helper.py` (this skill directory)

```bash
# Log user input (call on every user message)
python3 .hermes/scripts/log_helper.py input "message text"

# Start a new task
python3 .hermes/scripts/log_helper.py start "task_name" "description"

# Update task status
python3 .hermes/scripts/log_helper.py update <log_path> running|completed|failed "note"

# List all tasks
python3 .hermes/scripts/log_helper.py list
```

- Input logs: `~/input-logs/YYYY-MM-DD.log`
- Task logs: `~/task-logs/YYYYMMDD_HHMMSS_taskname.log`
- Index: `~/task-logs/INDEX.md` (Markdown table of all tasks)